mrhhsg opened a new pull request, #70836:
URL: https://github.com/apache/doris/pull/70836
### Background
OLAP scanners report scan accounting after each scheduled batch. Scanners
belonging to one local state share a SegmentIterator profile, whose FileCache
subtree contains byte counters, detailed I/O counters, and timers.
### What problem does this PR solve?
Issue Number: None
Related PR: None
PR Type: Performance
Problem Summary:
Multi-batch OLAP scans repeatedly construct the full FileCache reporter and
look up existing counters under the shared profile-map mutex. Cache lazily
initialized counter handles per local state, publish only local-read,
remote-read, and cache-write byte deltas per batch, and merge detailed reader
statistics once at scanner termination. The fragment snapshot changes retain
late terminal updates after LIMIT or cancellation without waiting for scanner
I/O; most added lines are tests. Detailed counters become terminal rather than
realtime, with no change to SQL results.
### Details
- **Change map:** Production: +268/-115 lines across eleven files; tests:
+658/-0 across two files; no generated output or documentation changes. The
production changes have three parts:
1. Lazy reporter reuse and split byte/detail updates: 158 changed lines,
shared by the local state and reporter.
2. Batch deltas and idempotent terminal accounting: 160 changed lines in
the scanner.
3. Late final-snapshot publication and ownership: 65 changed lines in
fragment/query contexts.
- **Why it was slow:** Each full reporter construction makes 84 counter
registration/lookup calls. Each call takes the profile counter-map mutex even
when its counter already exists; scanners sharing a local state use the same
map.
- **Optimization:** Lazily share lightweight byte counters and a full
reporter per local state. Keep cumulative reader details until a terminal
flush, with per-scanner byte baselines preventing double counting. Refresh the
first saved fragment snapshot only when a later publication advances its epoch.
- **Results:** The source-level registration model changes from 84 calls per
eligible batch to at most 4 + 84 calls per local state. This is a structural
bound, not a measurement of contention, CPU savings, or query speedup. No
comparable Release benchmark has run.
- **Applicability and cost:** Targets multi-batch OLAP FileCache scans;
short scans may not benefit. Adds three byte baselines per scanner and shared
handles/once flags per local state, retains peer hosts until termination, and
may serialize the saved fragment profile once more after a late flush. Existing
full-reporter callers keep their update behavior; no configuration defaults or
protocols change.
- **Behavior change:** The three main byte counters remain realtime;
detailed FileCache counters and timers are finalized at scanner termination.
Empty builder statistics no longer create an empty subtree. Detail-only
statistics remain reportable, and terminal high-water marks preserve the
maximum current value across scanners.
### Testing
All runs below were local, on the final source content:
- ASAN BE UT: 20 focused tests passed, covering reporter/scanner accounting,
final snapshots, and existing LIMIT/RF cleanup paths.
- ASAN accounting/lifetime tests: nine tests passed in 20 shuffled
repetitions with seed 42, totaling 180 executions.
- Non-cloud SQL regression: `query_p0/limit/test_unified_limit_pushdown`
passed, one suite with no failures or skips.
- Negative controls failed as expected for per-batch detail publication and
epoch sampling after serialization; the final source was restored and
revalidated.
- Real cloud FileCache end-to-end validation, FE final-profile transport,
and comparable Release performance measurements have not run.
### Notes for Reviewers
The epoch is sampled before serialization so a flush during serialization
still forces a final refresh. Only the context that inserted the first fragment
snapshot may replace it; subsequent recursive stages sharing the ID cannot do
so. The local-byte fallback uses current-batch deltas, not cumulative cache
totals. Failed reader initialization permits valid I/O accounting but not full
reader-profile collection.
### Release note
OLAP FileCache byte counters remain live, while detailed cache statistics
are finalized when scanning stops rather than published after every batch.
Counter names, units, and hierarchy are preserved; scans without builder I/O no
longer create an empty FileCache subtree.
### Check List (For Author)
- Behavior changed: Yes (see Details)
- Does this need documentation: No
- Pre-submission status: Partially verified: real cloud FileCache end-to-end
behavior, FE final-profile transport, and comparable Release performance
measurements remain unverified.
### AI Disclosure
- Tool: Codex
- Generated: Code, tests, and this description
- Human review: not stated
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]