matanper opened a new issue, #70135: URL: https://github.com/apache/doris/issues/70135
### Search before asking - [x] I searched the issues and found no similar issue. Related: #70117 (Top-N not read in key order when leading keys are fixed), #70020 (refresh Top-N runtime filter page ranges; keeps the shared predicate through scan-key normalization), #69646 (the same idea for Parquet row groups). ### Description For `SELECT ... WHERE <leading keys fixed> ORDER BY <next key> DESC LIMIT n` read in key order, the Top-N threshold usually becomes tight after the first rowset or two, yet the scan still opens every remaining rowset and segment. Opening a segment loads its footer and column metadata and builds its iterator. On a log table with about 1K segments per scan instance this open cost dominates the query once #70117's key-order read is in place. On master: - `VCollectIterator::_topn_next` (`be/src/storage/iterator/vcollect_iterator.cpp`, the loop over `_rs_splits`) calls `rs_reader->init(...)` for every rowset unconditionally. The current threshold only feeds the runtime predicate, which is evaluated after `init` has already opened the rowset's segments. - `BetaRowsetReader` has no step that compares the threshold with a segment's zone map (or key bounds) before the segment is opened, so a segment whose whole `[min, max]` lies beyond the threshold is still opened, only to have every row filtered out. A minor related point: in `SegmentIterator::_get_row_ranges_from_conditions` the zone-map row ranges are intersected twice, and `pre_size2` is taken after the first intersection, so `rows_stats_rp_filtered` (`RowsZoneMapRuntimePredicateFiltered`) is always 0. Filtering itself is correct; only the counter is wrong. ### Solution Check the current threshold before opening: 1. Before `rs_reader->init` in `_topn_next`, and when splitting rowsets into segment ranges, compare the Top-N predicate at its current threshold against each segment's zone map for the target column (inclusive bound; a segment holding nulls is kept when nulls sort first; `pass_all` zone maps are never pruned; only when the predicate's value type matches the column's). Skip segments it rules out, and skip the rowset when every segment is ruled out. 2. Re-check per rowset as the walk reaches it, because the threshold keeps tightening while earlier rowsets and the query's other scan instances read rows. 3. The zone maps need to be available without opening the segment, e.g. kept from an earlier open or read from the segment footer. 4. Report it: `SegmentsPrunedByTopNBeforeOpen`, `RowsetsPrunedByTopNBeforeInit`. Every published threshold comes from n rows the query has actually read, so a segment ruled out this way holds no row the result needs, including while the threshold is still being tightened. We implemented this in our 4.1 fork, on top of a cache of segment zone maps kept from earlier opens. On a 1-billion-row, 31-day table (16 buckets, `ORDER BY event_time DESC LIMIT 100` for one tenant), 107 of 110 segments in a scan instance were skipped before opening. Segment open time per instance went from 2 ms to 34 µs and warm latency from 68 ms to 53 ms, with identical results on 12 Top-N query shapes (ASC/DESC, OFFSET, extra filters, IN lists). On our production cluster, where each scan instance opens about 1–1.3K segments for the same query, segment opens are the largest remaining cost of the 30-day first page after #70117. We are happy to contribute a PR for master if this direction is welcome. ### Are you willing to submit PR? - [x] Yes I am willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
