asdf2014 opened a new pull request, #65788:
URL: https://github.com/apache/doris/pull/65788
### What problem does this PR solve?
Issue Number: close #xxx
Related PR: #65482
Problem Summary:
Extends the stale query cache entry incremental merge (#65482) from the
shared-nothing (local storage) deployment to the compute-storage decoupled
(cloud) mode.
In cloud mode a tablet's rowsets, and for merge-on-write tables its delete
bitmap, are a lazily synced copy of the meta service. The base change refused
incremental merge on cloud outright. This brings the same optimization to cloud
by syncing each stale tablet's view to the queried version before capturing the
delta:
- Per-tablet meta-service syncs are fanned out in parallel and awaited with
a fast-fail budget (`query_cache_decision_sync_timeout_ms`, new cloud BE
config, default 2000 ms, changeable at runtime). The decision runs on the
bounded query-admission pool, so on a meta-service brownout it falls back to a
full recompute rather than holding that thread. A value <= 0 disables cloud
incremental merge as a fail-safe.
- The presync brings the delete bitmap to the queried version, so the
merge-on-write history-rewrite check sees exactly what the read path sees; the
later scan-node sync takes the same no-op shortcut, so the RPC count per query
does not grow.
- The cached-side rowsets are pinned during bitmap classification so a
concurrent unused-rowset GC cannot retire the evidence (both storage engines
drop a retired rowset's bitmap only once nothing else references it).
- A query reading a version-inexact view (`query_freshness_tolerance_ms` or
`enable_prefer_cached_rowset`) is excluded from incremental merge and, on
cloud, from the cache write-back, so no entry whose content mismatches its
version stamp is ever created.
Observability: new counter `query_cache_decision_sync_time_ms` (only
advances in cloud mode) records the wall time the decision spent syncing tablet
views before capturing the delta.
### Release note
Query Cache incremental merge now works in the compute-storage decoupled
(cloud) mode. New BE config `query_cache_decision_sync_timeout_ms` (default
2000, changeable at runtime) bounds the decision's tablet view sync; on timeout
the query falls back to one full recompute.
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [x] Regression test
- [x] Unit Test
- [x] Manual test (add detailed scripts or steps below)
- Ran the `query_cache_incremental` regression suite and a
four-phase benchmark (no cache, exact hit, incremental merge, incremental off)
on a real compute-storage decoupled cluster (FoundationDB + meta service +
MinIO): the stale-hit and fallback metrics match the shared-nothing baseline,
and the decision sync time stays in the tens of milliseconds.
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [ ] No.
- [x] Yes. <!-- Explain the behavior change --> On cloud, a stale cache
entry can now be reused through incremental merge instead of always recomputing
in full. The profile gains two cloud fallback reasons (`cloud rowset sync
failed`, `cloud rowset sync timed out`).
- Does this need documentation?
- [ ] No.
- [x] Yes. <!-- Add document PR link here. eg:
https://github.com/apache/doris-website/pull/1214 -->
https://github.com/apache/doris-website/pull/3999
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR should
merge into -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
