airborne12 opened a new pull request, #70384: URL: https://github.com/apache/doris/pull/70384
### What problem does this PR solve? Issue Number: None Related PR: https://github.com/apache/doris/pull/70020 Backport #70020 to `branch-4.2` from commit `32accf9301cb2db4f3197287f54971e2a45787dd`. Segment readers keep page ranges computed before a TopN runtime bound arrives or tightens. Static scan-key normalization can also remove a shared runtime predicate. Publish versioned predicate snapshots, retain runtime predicates, and refresh unread page ranges between batches. Preserve consumed-row progress, bitmap holes, inclusive bounds, and read direction. ### Release note TopN scans can skip unread pages when runtime bounds arrive or tighten. Runtime predicates remain active after scan-key normalization. ### Check List (For Author) - Test: - [x] Regression test: both source TopN suites passed against the local ASAN FE/BE build. - [x] Unit Test: 57 targeted ASAN BE tests passed, including all 20 cases added by the source PR. - [x] Manual test: full C++ format preflight and English code-description check passed. - Behavior changed: - [ ] No. - [x] Yes. TopN scans can read fewer rows and pages. - Does this need documentation? - [x] No. - [ ] Yes. ### Validation - ASAN BE and FE build: `./build.sh --be --fe -j 96`; final BE incremental build after the iterator-construction adjustment also passed. - BE tests: `./run-be-ut.sh --run -j 96 --filter='SegmentIteratorTopNRuntimeFilterTest.*:Bounds/OlapScanRuntimePredicateTest.*:RuntimePredicateTest.*:ScanOperatorTest.*:SegmentIteratorLimitOptTest.*:CountEmitShortcut.*'` — 57 ran, 57 passed. `--gtest_list_tests` confirmed discovery first; the execution log contains all 20 required source cases. No sanitizer errors were reported with the repository's standard LSAN suppressions. - Regression: `./run-regression-test.sh --conf <private-cluster-config> --run -d topn_optimize -s topn_runtime_predicate_key_range,topn_runtime_range_update` — 2 suites passed, 0 failed, 0 skipped. The standalone ASAN BE used a 64 GB memory limit. Both Groovy suites and both expected-result files remain byte-for-byte identical to the source commit; results were compared without regenerating expected output. - C++ format: clang-format 16; preflight checked the complete merged C++ CI scope against the current `origin/branch-4.2`. - English code check: passed for all 8 changed source files; no data exceptions. - Static-analysis limitation: direct clang-tidy 20.1.8 checks with this branch's `.clang-tidy` configuration retain baseline diagnostics. The unchanged `be/src/core/types.h:578` has an unmatched `NOLINTEND`; existing function-complexity diagnostics remain above the threshold of 50 (base → backport: 108 → 113, 170 → 173, 187 → 190). The same target baseline was analyzed through a VFS overlay with identical dependencies. No other candidate diagnostic categories remain after using `std::make_unique`. Full clang-tidy is therefore not reported as passing. ### Adaptation and range-diff review ```text 1: 32accf9301c ! 1: 8454b59b005 [improvement](topn) Refresh TopN runtime filter page ranges (#70020) ``` The commit-message entry records the release-branch title, source provenance, conflict paths, and validation actually performed. The code differences are: - `segment_iterator.cpp/.h`: keep the branch's `LikeColumnPredicate` include and `RowLocation` declaration. Add the source-required cardinality cache and its three existing read sites; use `std::make_unique` for iterator construction. The runtime refresh and progress rules are preserved. - Scan-operator test: create the source runtime-predicate fixture as a new file because its original container file is absent here. Add `nullable_tuples` required by this branch; retain every source assertion. The production guard is unchanged apart from a two-sentence comment. - Segment test: replace master-only schema, writer, block-construction, and read-option initialization APIs with their branch equivalents. All eight source cases and their assertions remain. - `shared_predicate.h` and all four regression files have identical Git blob contents to the source commit. No source behavior hunk was dropped. Selected range-diff entries: ```diff ++ _range_iter = std::make_unique<BackwardBitmapRangeIterator>(_row_bitmap); ++ uint64_t _row_bitmap_cardinality = 0; ++ plan_node.__set_nullable_tuples({false}); -+ _schema = std::make_shared<ReadSchema>(_tablet_schema->columns()); ++ _schema = std::make_shared<Schema>(_tablet_schema->columns(), std::vector<ColumnId> {0}); -+ StorageReadOptions options(_stats); ++ StorageReadOptions options; ++ options.stats = &_stats; ++ options.tablet_schema = _tablet_schema; ``` <details> <summary>Source hunk audit</summary> | Source file | Source hunk | Result and target evidence | | --- | --- | --- | | `be/src/exec/operator/olap_scan_operator.cpp` | `@@ -1417,6 +1417,12 @@ Status OlapScanLocalState::_build_key_ranges_and_filters() {` | Adapted; preserve the runtime-filter guard and shorten its comment to two sentences under the repository's comment policy. | | `be/src/storage/predicate/shared_predicate.h` | Entire file | Ported(verbatim); source and candidate Git blob IDs match. | | `be/src/storage/segment/segment_iterator.cpp` | `@@ -98,6 +98,7 @@` | Ported; retain branch-4.2's LikeColumnPredicate include and add SharedPredicate. | | `be/src/storage/segment/segment_iterator.cpp` | `@@ -460,6 +461,11 @@ Status SegmentIterator::_init_impl(const StorageReadOptions& opts) {` | Ported; all source changes retained in the corresponding branch-4.2 method. | | `be/src/storage/segment/segment_iterator.cpp` | `@@ -617,15 +623,8 @@ Status SegmentIterator::_lazy_init(Block* block) {` | Adapted; rebuild the cursor in the existing lazy initialization; introduce the cardinality cache absent from branch-4.2 and use it at the three existing read sites. | | `be/src/storage/segment/segment_iterator.cpp` | `@@ -1282,6 +1281,75 @@ Status SegmentIterator::_apply_ann_topn_predicate() {` | Adapted; preserve the source refresh logic and use std::make_unique for the extracted iterator construction, as required by the branch's clang-tidy configuration. | | `be/src/storage/segment/segment_iterator.cpp` | `@@ -2665,6 +2733,11 @@ Status SegmentIterator::_read_columns_by_index(const std::vector<ColumnId>& read` | Ported; all source changes retained in the corresponding branch-4.2 method. | | `be/src/storage/segment/segment_iterator.cpp` | `@@ -3148,6 +3221,7 @@ Status SegmentIterator::_next_batch_internal(Block* block) {` | Ported; all source changes retained in the corresponding branch-4.2 method. | | `be/src/storage/segment/segment_iterator.cpp` | `@@ -3197,6 +3271,7 @@ Status SegmentIterator::_next_batch_internal(Block* block) {` | Ported; all source changes retained in the corresponding branch-4.2 method. | | `be/src/storage/segment/segment_iterator.h` | `@@ -61,6 +61,7 @@ namespace doris {` | Ported; retain branch-4.2's RowLocation declaration and add SharedPredicate. | | `be/src/storage/segment/segment_iterator.h` | `@@ -161,6 +162,9 @@ class SegmentIterator : public RowwiseIterator {` | Ported; all source changes retained in the corresponding branch-4.2 method. | | `be/src/storage/segment/segment_iterator.h` | `@@ -380,10 +384,19 @@ class SegmentIterator : public RowwiseIterator {` | Adapted; add the cache member absent from branch-4.2 together with the source TopN state members. | | `be/test/exec/operator/olap_scan_operator_test.cpp` | `@@ -19,17 +19,25 @@` | Adapted; create the new runtime-predicate fixture with its required includes; branch-4.2 has no preexisting file or unrelated changelog/TSO fixtures. Add the nullable_tuples metadata required by branch-4.2; preserve all source assertions. | | `be/test/exec/operator/olap_scan_operator_test.cpp` | `@@ -360,4 +368,126 @@ TEST_F(OlapScanOperatorTsoPruningTest, EmptyBootstrapDoesNotStartMinDeltaScanner` | Adapted; create the new runtime-predicate fixture with its required includes; branch-4.2 has no preexisting file or unrelated changelog/TSO fixtures. Add the nullable_tuples metadata required by branch-4.2; preserve all source assertions. | | `be/test/storage/segment/segment_iterator_topn_runtime_filter_test.cpp` | `@@ -0,0 +1,332 @@` | Adapted; use Schema(columns, {0}), TestSegmentWriter::finalize(), TabletSchema::create_block(), and explicit StorageReadOptions stats/tablet_schema initialization in place of master-only APIs; retain all eight source tests and assertions. | | `regression-test/data/topn_optimize/topn_runtime_predicate_key_range.out` | Entire file | Ported(verbatim); source and candidate Git blob IDs match. | | `regression-test/data/topn_optimize/topn_runtime_range_update.out` | Entire file | Ported(verbatim); source and candidate Git blob IDs match. | | `regression-test/suites/topn_optimize/topn_runtime_predicate_key_range.groovy` | Entire file | Ported(verbatim); source and candidate Git blob IDs match. | | `regression-test/suites/topn_optimize/topn_runtime_range_update.groovy` | Entire file | Ported(verbatim); source and candidate Git blob IDs match. | </details> ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
