mrhhsg opened a new pull request, #70836:
URL: https://github.com/apache/doris/pull/70836

   ### Background
   
   OLAP scanners report scan accounting after each scheduled batch. Scanners 
belonging to one local state share a SegmentIterator profile, whose FileCache 
subtree contains byte counters, detailed I/O counters, and timers.
   
   ### What problem does this PR solve?
   
   Issue Number: None
   Related PR: None
   PR Type: Performance
   
   Problem Summary:
   
   Multi-batch OLAP scans repeatedly construct the full FileCache reporter and 
look up existing counters under the shared profile-map mutex. Cache lazily 
initialized counter handles per local state, publish only local-read, 
remote-read, and cache-write byte deltas per batch, and merge detailed reader 
statistics once at scanner termination. The fragment snapshot changes retain 
late terminal updates after LIMIT or cancellation without waiting for scanner 
I/O; most added lines are tests. Detailed counters become terminal rather than 
realtime, with no change to SQL results.
   
   ### Details
   
   - **Change map:** Production: +268/-115 lines across eleven files; tests: 
+658/-0 across two files; no generated output or documentation changes. The 
production changes have three parts:
     1. Lazy reporter reuse and split byte/detail updates: 158 changed lines, 
shared by the local state and reporter.
     2. Batch deltas and idempotent terminal accounting: 160 changed lines in 
the scanner.
     3. Late final-snapshot publication and ownership: 65 changed lines in 
fragment/query contexts.
   - **Why it was slow:** Each full reporter construction makes 84 counter 
registration/lookup calls. Each call takes the profile counter-map mutex even 
when its counter already exists; scanners sharing a local state use the same 
map.
   - **Optimization:** Lazily share lightweight byte counters and a full 
reporter per local state. Keep cumulative reader details until a terminal 
flush, with per-scanner byte baselines preventing double counting. Refresh the 
first saved fragment snapshot only when a later publication advances its epoch.
   - **Results:** The source-level registration model changes from 84 calls per 
eligible batch to at most 4 + 84 calls per local state. This is a structural 
bound, not a measurement of contention, CPU savings, or query speedup. No 
comparable Release benchmark has run.
   - **Applicability and cost:** Targets multi-batch OLAP FileCache scans; 
short scans may not benefit. Adds three byte baselines per scanner and shared 
handles/once flags per local state, retains peer hosts until termination, and 
may serialize the saved fragment profile once more after a late flush. Existing 
full-reporter callers keep their update behavior; no configuration defaults or 
protocols change.
   - **Behavior change:** The three main byte counters remain realtime; 
detailed FileCache counters and timers are finalized at scanner termination. 
Empty builder statistics no longer create an empty subtree. Detail-only 
statistics remain reportable, and terminal high-water marks preserve the 
maximum current value across scanners.
   
   ### Testing
   
   All runs below were local, on the final source content:
   
   - ASAN BE UT: 20 focused tests passed, covering reporter/scanner accounting, 
final snapshots, and existing LIMIT/RF cleanup paths.
   - ASAN accounting/lifetime tests: nine tests passed in 20 shuffled 
repetitions with seed 42, totaling 180 executions.
   - Non-cloud SQL regression: `query_p0/limit/test_unified_limit_pushdown` 
passed, one suite with no failures or skips.
   - Negative controls failed as expected for per-batch detail publication and 
epoch sampling after serialization; the final source was restored and 
revalidated.
   - Real cloud FileCache end-to-end validation, FE final-profile transport, 
and comparable Release performance measurements have not run.
   
   ### Notes for Reviewers
   
   The epoch is sampled before serialization so a flush during serialization 
still forces a final refresh. Only the context that inserted the first fragment 
snapshot may replace it; subsequent recursive stages sharing the ID cannot do 
so. The local-byte fallback uses current-batch deltas, not cumulative cache 
totals. Failed reader initialization permits valid I/O accounting but not full 
reader-profile collection.
   
   ### Release note
   
   OLAP FileCache byte counters remain live, while detailed cache statistics 
are finalized when scanning stops rather than published after every batch. 
Counter names, units, and hierarchy are preserved; scans without builder I/O no 
longer create an empty FileCache subtree.
   
   ### Check List (For Author)
   
   - Behavior changed: Yes (see Details)
   - Does this need documentation: No
   - Pre-submission status: Partially verified: real cloud FileCache end-to-end 
behavior, FE final-profile transport, and comparable Release performance 
measurements remain unverified.
   
   ### AI Disclosure
   
   - Tool: Codex
   - Generated: Code, tests, and this description
   - Human review: not stated


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to