Mryange opened a new pull request, #70621:
URL: https://github.com/apache/doris/pull/70621

   ### Background
   
   A scan may visit many segments before key-range or index filtering 
eliminates their candidate rows. SegmentIterator previously prepared 
column/index iterators and scan buffers before this filtering ran.
   
   ### What problem does this PR solve?
   
   Issue Number: None
   Related PR: None
   PR Type: Performance
   
   Problem Summary:
   
   Segments rejected by filtering still pay for resources that will never be 
used. Prepare key iterators on demand and reuse them, initialize 
column-condition resources inside their filtering stage, and defer output 
iterators, the batch predictor and rowid buffer growth until needed. Empty 
candidate sets finish between stages without requiring data-column buffers.
   
   ### Details
   
   - **Why it was slow:** Reader/cache lookups, iterator construction and 
buffer initialization occurred even for segments with no matching rows. A small 
column-reader cache also thrashed between key and output columns.
   - **Optimization:** Split initialization by dependency and stop after 
filtering proves the segment empty. Preserve condition-cache lookup before 
expression cloning and reuse the existing EOF contract without another 
completion flag.
   - **Results:** ClickBench, 99,997,497 rows, RANDOM 48 buckets, 1,680 
segments, CounterID as the leading sort key. Query: `SELECT SUM(JavaEnable) 
FROM hits WHERE CounterID = 1377`. Both versions return 1 and key-range 
filtering excludes 99,997,496 rows. The following are medians of 15 profiles 
per version and cache setting, after 20 warmups:
   
   | Column-reader cache capacity | Query total, ms | Scanner CPU, ms | Init + 
first-batch preparation, ms |
   |---|---:|---:|---:|
   | 1 | 62 → 43 (-31%) | 42.111 → 23.516 (-44%) | 25.328 → 8.375 (-67%) |
   | 100 | 47 → 41 (-13%) | 30.641 → 23.332 (-24%) | 13.928 → 8.307 (-40%) |
   
   Both BE binaries use RELEASE; the same FE process was retained during 
timing. Scanner peak concurrency was 1; SQL/query caches and automatic 
statistics collection were disabled. Results were rechecked by switching back 
to the baseline BE and then to the patched BE. The host was shared, so these 
percentages describe this workload rather than a general throughput guarantee.
   
   “Init + first-batch preparation” is the per-profile sum of 
`SegmentIteratorInitTimer` and `BlockInitTime`, followed by the median. It 
includes key seeking and accounts for initialization moved into lazy init; it 
is not a sum of nested sub-timers or the complete rowset initialization cost. 
In an additional warmed query at capacity 1, column-reader cache 
misses/evictions fell from 3,360 to 2 each; SegmentCache remained at 1,680 hits 
and 0 misses in both versions.
   
   - **Applicability and cost:** Benefits scans with many segments rejected by 
key/index filtering. Filtering algorithms and query results are unchanged; 
initialization timing and resource lifetime change. After key pruning, index 
binding still resolves metadata for all read-schema columns, including Variant 
type inference where required. No new settings or storage/protocol changes.
   
   ### Testing
   
   - Development-machine builds: `./build.sh --be -j48` and `./build.sh --fe 
-j48` passed.
   - Modified-line clang-format 16, build hygiene, diff whitespace checks and 
clang-tidy passed. Standalone header clang-tidy required the toolchain resource 
directory to resolve standard headers.
   - Nine before/after SQL result comparisons passed: point/range/IN key 
predicates, mixed scalar/expression predicates, descending LIMIT, output 
projection, full-table SUM and an empty result.
   - Full BE unit/regression suites and dedicated 
inverted-index/ANN/complex-type coverage have not been run. No test files are 
changed in this PR.
   
   ### Notes for Reviewers
   
   Start with `_lazy_init`, `_init_column_iterator` and `_finish_scan`. 
First-batch preparation is included above so moving work out of the init timer 
is not counted as eliminated work.
   
   ### Release note
   
   Reduce scan initialization CPU and latency when key-range or index filtering 
eliminates most segments.
   
   ### Check List (For Author)
   
   - Behavior changed: Yes (resource preparation and empty-scan completion 
timing; query semantics unchanged; see Details).
   - Does this need documentation: No
   - Pre-submission status: Partially verified: builds, static checks, SQL 
comparisons and performance measurements passed; full UT/regression and 
specialized index/type coverage remain unverified.
   
   ### AI Disclosure
   
   - Tool: Codex (GPT-6)
   - Generated: Code, benchmark collection/analysis scripts and this 
description.
   - Human review: The author reviewed and discussed the initialization order, 
iterator reuse, EOF handling and selected code changes; full test coverage is 
still pending.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to