Mryange opened a new pull request, #70621: URL: https://github.com/apache/doris/pull/70621
### Background A scan may visit many segments before key-range or index filtering eliminates their candidate rows. SegmentIterator previously prepared column/index iterators and scan buffers before this filtering ran. ### What problem does this PR solve? Issue Number: None Related PR: None PR Type: Performance Problem Summary: Segments rejected by filtering still pay for resources that will never be used. Prepare key iterators on demand and reuse them, initialize column-condition resources inside their filtering stage, and defer output iterators, the batch predictor and rowid buffer growth until needed. Empty candidate sets finish between stages without requiring data-column buffers. ### Details - **Why it was slow:** Reader/cache lookups, iterator construction and buffer initialization occurred even for segments with no matching rows. A small column-reader cache also thrashed between key and output columns. - **Optimization:** Split initialization by dependency and stop after filtering proves the segment empty. Preserve condition-cache lookup before expression cloning and reuse the existing EOF contract without another completion flag. - **Results:** ClickBench, 99,997,497 rows, RANDOM 48 buckets, 1,680 segments, CounterID as the leading sort key. Query: `SELECT SUM(JavaEnable) FROM hits WHERE CounterID = 1377`. Both versions return 1 and key-range filtering excludes 99,997,496 rows. The following are medians of 15 profiles per version and cache setting, after 20 warmups: | Column-reader cache capacity | Query total, ms | Scanner CPU, ms | Init + first-batch preparation, ms | |---|---:|---:|---:| | 1 | 62 → 43 (-31%) | 42.111 → 23.516 (-44%) | 25.328 → 8.375 (-67%) | | 100 | 47 → 41 (-13%) | 30.641 → 23.332 (-24%) | 13.928 → 8.307 (-40%) | Both BE binaries use RELEASE; the same FE process was retained during timing. Scanner peak concurrency was 1; SQL/query caches and automatic statistics collection were disabled. Results were rechecked by switching back to the baseline BE and then to the patched BE. The host was shared, so these percentages describe this workload rather than a general throughput guarantee. “Init + first-batch preparation” is the per-profile sum of `SegmentIteratorInitTimer` and `BlockInitTime`, followed by the median. It includes key seeking and accounts for initialization moved into lazy init; it is not a sum of nested sub-timers or the complete rowset initialization cost. In an additional warmed query at capacity 1, column-reader cache misses/evictions fell from 3,360 to 2 each; SegmentCache remained at 1,680 hits and 0 misses in both versions. - **Applicability and cost:** Benefits scans with many segments rejected by key/index filtering. Filtering algorithms and query results are unchanged; initialization timing and resource lifetime change. After key pruning, index binding still resolves metadata for all read-schema columns, including Variant type inference where required. No new settings or storage/protocol changes. ### Testing - Development-machine builds: `./build.sh --be -j48` and `./build.sh --fe -j48` passed. - Modified-line clang-format 16, build hygiene, diff whitespace checks and clang-tidy passed. Standalone header clang-tidy required the toolchain resource directory to resolve standard headers. - Nine before/after SQL result comparisons passed: point/range/IN key predicates, mixed scalar/expression predicates, descending LIMIT, output projection, full-table SUM and an empty result. - Full BE unit/regression suites and dedicated inverted-index/ANN/complex-type coverage have not been run. No test files are changed in this PR. ### Notes for Reviewers Start with `_lazy_init`, `_init_column_iterator` and `_finish_scan`. First-batch preparation is included above so moving work out of the init timer is not counted as eliminated work. ### Release note Reduce scan initialization CPU and latency when key-range or index filtering eliminates most segments. ### Check List (For Author) - Behavior changed: Yes (resource preparation and empty-scan completion timing; query semantics unchanged; see Details). - Does this need documentation: No - Pre-submission status: Partially verified: builds, static checks, SQL comparisons and performance measurements passed; full UT/regression and specialized index/type coverage remain unverified. ### AI Disclosure - Tool: Codex (GPT-6) - Generated: Code, benchmark collection/analysis scripts and this description. - Human review: The author reviewed and discussed the initialization order, iterator reuse, EOF handling and selected code changes; full test coverage is still pending. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
