HappenLee opened a new pull request, #70626:
URL: https://github.com/apache/doris/pull/70626
### Background
Condition cache remembers which 2,048-row granules contain qualifying rows
in each segment. TopN filters tighten a runtime boundary as candidates arrive.
Previously, scans receiving TopN filters always disabled condition cache.
### What problem does this PR solve?
Issue Number: None
Related PR: None
PR Type: Performance
Problem Summary: Repeated TopN scans cannot reuse previously learned
exclusions. FE now identifies the complete input snapshot and sort semantics;
BE combines that identity with existing scan predicates. The change reuses the
existing bitmap cache, with most added lines covering correctness tests and
generated expectations.
### Details
- **Change map:** 211 production lines changed (116 FE/protocol identity and
eligibility; 95 BE digest and survivor tracking), 500 test lines, and 388
generated expectation lines; no documentation files.
- **Why it was slow:** The blanket TopN exclusion prevented condition-cache
pruning on repeated scans.
- **Optimization:** Hash finalized tablet versions/schema, sorting column,
direction, NULL ordering, LIMIT and OFFSET. BE adds existing predicate digests
before injecting the dynamic TopN expression. Record survivors independently of
deferred column reads, and publish only completed scans.
- **Results:** RELEASE, one FE and two BEs on a shared Xeon Platinum 8457C
host; each BE configured for 8 cores/16 GiB, without dedicated CPU reservation.
Five million rows, eight tablets, one replica. Table below reports medians of
40 paired off/learn/hit samples, with warm page caches and SQL/query caches
disabled. Client execute-plus-fetch time excludes SET/cache clearing/Profile
collection. Control order alternates; every result matches the uncached
reference.
| Workload | Off (ms) | Learn (ms) | Hit (ms) | Change vs. off |
|---|---:|---:|---:|---:|
| Top10 | 11.093 | 11.481 | 10.688 | -3.65% |
| WHERE + Top10 | 12.738 | 13.131 | 11.719 | -8.00% |
| Top10000 | 43.465 | 43.253 | 43.850 | +0.89% |
| All granules match | 9.685 | 10.185 | 9.994 | +3.20% |
| New snapshot per query (20 pairs) | 11.811 | 12.312 | — | learning: +4.24%
|
A separate 15-sample run observed Top10 −6.23%, WHERE+Top10 −6.10%, Top10000
−0.68%, and all-match −0.64%; six changing-snapshot pairs cost +7.43%. These
short-query results show modest gains and shared-host variability. An untimed
WHERE+Top10 Profile recorded 8/8 cache hits and TotalPagesNum 1,230 → 897;
all-match filtered zero rows despite 8/8 hits.
- **Applicability and cost:** Local Duplicate Key base-index scans, one
direct-column sort key/source/target, and no intervening row-changing operator
or join runtime filter. Existing `enable_condition_cache` controls usage.
Cloud, sampling, TTL, debug reads and unsupported plans remain ineligible. Full
snapshot hashing costs O(tablets log tablets); snapshot changes relearn
entries. Existing bitmap allocation/LRU eviction applies, with no new cache.
Missing protocol identity disables reuse during mixed-version operation. No
benefit is guaranteed for large LIMITs or granules that all contain candidates.
<details>
<summary>Benchmark fixture and queries</summary>
```sql
CREATE TABLE bench_topn (
id BIGINT NOT NULL, score BIGINT NOT NULL,
tie_score INT NOT NULL, payload VARCHAR(64)
)
DUPLICATE KEY(id)
DISTRIBUTED BY HASH(id) BUCKETS 8
PROPERTIES("replication_num"="1", "disable_auto_compaction"="true");
INSERT INTO bench_topn
SELECT number, (number * 48271) % 2147483647, number % 32,
repeat(cast(number % 997 as string), 16)
FROM numbers("number"="5000000");
ANALYZE TABLE bench_topn WITH SYNC;
SET enable_sql_cache=false;
SET enable_query_cache=false;
SET enable_profile=false;
SELECT sum(id), sum(score), sum(tie_score) FROM bench_topn;
SELECT id, score FROM bench_topn ORDER BY score LIMIT 10;
SELECT id, score FROM bench_topn WHERE id % 7 = 0 ORDER BY score LIMIT 10;
SELECT id, score FROM bench_topn ORDER BY score LIMIT 10000;
SELECT tie_score FROM bench_topn ORDER BY tie_score LIMIT 10;
```
For each pair, toggle `enable_condition_cache`; clear only ConditionCache on
both BEs before learning, then repeat for the hit. Changing-snapshot pairs
insert one nonwinning row before measurement.
</details>
### Testing
Local verification:
- FE build/Checkstyle; 8 FE tests, including translated plans.
- ASAN BE build; 54 tests across RuntimePredicate, TopN segment, scan-limit
and candidate-pushdown suites.
- `topn_condition_cache`, `condition_cache`, `condition_cache_eligibility`
regressions passed with both one and two ASAN BEs. Cover actual hits, dropped
best partition, inserts, compaction, parallel scans, NULLs/ties and OFFSET.
Generated expectations independently checked: 36 sections/315 rows.
- RELEASE build and benchmark above; formatting, header hygiene and
changed-line clang-tidy passed. Two GTest complexity suppressions document
assertion-macro expansion.
### Notes for Reviewers
Removing a partition containing the best rows must invalidate exclusions in
unchanged segments elsewhere. Therefore the identity covers the entire scan
snapshot, not just the cached segment.
### Release note
Eligible repeated TopN scans can reuse condition-cache exclusions for an
unchanged input snapshot. Query results are unchanged.
### Check List (For Author)
- Behavior changed: No
- Does this need documentation: No
- Pre-submission status: Ready
### AI Disclosure
- Tool: Codex (GPT-6 family)
- Generated: Implementation, tests, benchmark and PR description.
- Human review: not stated
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]