HappenLee opened a new pull request, #70626:
URL: https://github.com/apache/doris/pull/70626

   ### Background
   
   Condition cache remembers which 2,048-row granules contain qualifying rows 
in each segment. TopN filters tighten a runtime boundary as candidates arrive. 
Previously, scans receiving TopN filters always disabled condition cache.
   
   ### What problem does this PR solve?
   
   Issue Number: None
   Related PR: None
   PR Type: Performance
   
   Problem Summary: Repeated TopN scans cannot reuse previously learned 
exclusions. FE now identifies the complete input snapshot and sort semantics; 
BE combines that identity with existing scan predicates. The change reuses the 
existing bitmap cache, with most added lines covering correctness tests and 
generated expectations.
   
   ### Details
   
   - **Change map:** 211 production lines changed (116 FE/protocol identity and 
eligibility; 95 BE digest and survivor tracking), 500 test lines, and 388 
generated expectation lines; no documentation files.
   - **Why it was slow:** The blanket TopN exclusion prevented condition-cache 
pruning on repeated scans.
   - **Optimization:** Hash finalized tablet versions/schema, sorting column, 
direction, NULL ordering, LIMIT and OFFSET. BE adds existing predicate digests 
before injecting the dynamic TopN expression. Record survivors independently of 
deferred column reads, and publish only completed scans.
   - **Results:** RELEASE, one FE and two BEs on a shared Xeon Platinum 8457C 
host; each BE configured for 8 cores/16 GiB, without dedicated CPU reservation. 
Five million rows, eight tablets, one replica. Table below reports medians of 
40 paired off/learn/hit samples, with warm page caches and SQL/query caches 
disabled. Client execute-plus-fetch time excludes SET/cache clearing/Profile 
collection. Control order alternates; every result matches the uncached 
reference.
   
   | Workload | Off (ms) | Learn (ms) | Hit (ms) | Change vs. off |
   |---|---:|---:|---:|---:|
   | Top10 | 11.093 | 11.481 | 10.688 | -3.65% |
   | WHERE + Top10 | 12.738 | 13.131 | 11.719 | -8.00% |
   | Top10000 | 43.465 | 43.253 | 43.850 | +0.89% |
   | All granules match | 9.685 | 10.185 | 9.994 | +3.20% |
   | New snapshot per query (20 pairs) | 11.811 | 12.312 | — | learning: +4.24% 
|
   
   A separate 15-sample run observed Top10 −6.23%, WHERE+Top10 −6.10%, Top10000 
−0.68%, and all-match −0.64%; six changing-snapshot pairs cost +7.43%. These 
short-query results show modest gains and shared-host variability. An untimed 
WHERE+Top10 Profile recorded 8/8 cache hits and TotalPagesNum 1,230 → 897; 
all-match filtered zero rows despite 8/8 hits.
   
   - **Applicability and cost:** Local Duplicate Key base-index scans, one 
direct-column sort key/source/target, and no intervening row-changing operator 
or join runtime filter. Existing `enable_condition_cache` controls usage. 
Cloud, sampling, TTL, debug reads and unsupported plans remain ineligible. Full 
snapshot hashing costs O(tablets log tablets); snapshot changes relearn 
entries. Existing bitmap allocation/LRU eviction applies, with no new cache. 
Missing protocol identity disables reuse during mixed-version operation. No 
benefit is guaranteed for large LIMITs or granules that all contain candidates.
   
   <details>
   <summary>Benchmark fixture and queries</summary>
   
   ```sql
   CREATE TABLE bench_topn (
       id BIGINT NOT NULL, score BIGINT NOT NULL,
       tie_score INT NOT NULL, payload VARCHAR(64)
   )
   DUPLICATE KEY(id)
   DISTRIBUTED BY HASH(id) BUCKETS 8
   PROPERTIES("replication_num"="1", "disable_auto_compaction"="true");
   INSERT INTO bench_topn
   SELECT number, (number * 48271) % 2147483647, number % 32,
          repeat(cast(number % 997 as string), 16)
   FROM numbers("number"="5000000");
   ANALYZE TABLE bench_topn WITH SYNC;
   SET enable_sql_cache=false;
   SET enable_query_cache=false;
   SET enable_profile=false;
   SELECT sum(id), sum(score), sum(tie_score) FROM bench_topn;
   
   SELECT id, score FROM bench_topn ORDER BY score LIMIT 10;
   SELECT id, score FROM bench_topn WHERE id % 7 = 0 ORDER BY score LIMIT 10;
   SELECT id, score FROM bench_topn ORDER BY score LIMIT 10000;
   SELECT tie_score FROM bench_topn ORDER BY tie_score LIMIT 10;
   ```
   
   For each pair, toggle `enable_condition_cache`; clear only ConditionCache on 
both BEs before learning, then repeat for the hit. Changing-snapshot pairs 
insert one nonwinning row before measurement.
   
   </details>
   
   ### Testing
   
   Local verification:
   
   - FE build/Checkstyle; 8 FE tests, including translated plans.
   - ASAN BE build; 54 tests across RuntimePredicate, TopN segment, scan-limit 
and candidate-pushdown suites.
   - `topn_condition_cache`, `condition_cache`, `condition_cache_eligibility` 
regressions passed with both one and two ASAN BEs. Cover actual hits, dropped 
best partition, inserts, compaction, parallel scans, NULLs/ties and OFFSET. 
Generated expectations independently checked: 36 sections/315 rows.
   - RELEASE build and benchmark above; formatting, header hygiene and 
changed-line clang-tidy passed. Two GTest complexity suppressions document 
assertion-macro expansion.
   
   ### Notes for Reviewers
   
   Removing a partition containing the best rows must invalidate exclusions in 
unchanged segments elsewhere. Therefore the identity covers the entire scan 
snapshot, not just the cached segment.
   
   ### Release note
   
   Eligible repeated TopN scans can reuse condition-cache exclusions for an 
unchanged input snapshot. Query results are unchanged.
   
   ### Check List (For Author)
   
   - Behavior changed: No
   - Does this need documentation: No
   - Pre-submission status: Ready
   
   ### AI Disclosure
   
   - Tool: Codex (GPT-6 family)
   - Generated: Implementation, tests, benchmark and PR description.
   - Human review: not stated
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to