matanper opened a new issue, #70442: URL: https://github.com/apache/doris/issues/70442
### Search before asking - [x] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues. ### Description `SegmentIterator::_get_row_ranges_from_conditions` (`be/src/storage/segment/segment_iterator.cpp:1353` on master `08231e3a635`, called at `:981`) evaluates the page indexes over the whole segment, even when the key-range seek has already narrowed the segment to a few candidate rows. Those indexes are the dictionary (`:1372`), bloom filter (`:1404`) and page zone map (`:1431`). Bloom filters are stored in index pages of up to 1 MiB, so an exact point lookup reads whole bloom-filter pages, plus the zone-map and dictionary indexes, to evaluate a row it could read directly. With the file cache cold, or on object storage, that is most of the lookup's IO. ### Solution 1. **Narrow the bloom-filter pass to the surviving rows.** Start the bloom-filter pass from the min/max envelope of the row bitmap that survives the key seek, not the whole segment. Carry the shrinking range from one bloom-filter column to the next, and stop once it is empty. The final intersection with the row bitmap still removes any gaps inside the envelope. 2. **One-candidate shortcut.** When a multi-row segment is down to exactly one candidate row and the predicates are ordinary scalar column predicates, skip the dictionary, bloom-filter and page zone-map indexes and let the normal row-predicate evaluation decide. - Delete predicates, common expressions and Top-N runtime filters keep the index path. - A segment that holds only one row also keeps the index path, so the existing pruning counters such as `RowsBloomFilterFiltered` keep their meaning. We have a patch and can open a PR. ### How to Reproduce? 1. Write one segment of about 1M rows with a bloom-filter column (`bloom_filter_columns`) and a short key. 2. Disable the page cache, and evict segment handles between runs. 3. Run an exact key lookup (`WHERE k = <value>`, with or without an additional predicate on another column). 4. Count the bytes read through the segment's file reader. ### Anything Else? We measured on a 1,048,576-row segment, page cache disabled, counting bytes read through the file reader: | Lookup | Current | With the change | | --- | ---: | ---: | | Matching exact key | 1,132,232 bytes | 45,764 bytes | | Exact key rejected by another predicate | 1,117,689 bytes | 31,221 bytes | | Broad scan (control) | 1,131,073 bytes | 1,131,073 bytes | The matching lookup reads 96% fewer bytes, almost all of it from the one-candidate shortcut (item 2). Because of page granularity, narrowing the bloom-filter range on its own saves little in this case. It helps when the envelope covers only some of a segment's bloom-filter pages. Results were identical to an unindexed reference for 71 point and range lookups, including IN lists, NULLs, LIKE and delete predicates. The existing `test_ngram_bloomfilter_index_change` and `test_ngram_bloomfilter_index_like_escape` suites pass unchanged. ### Are you willing to submit PR? - [x] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
