matanper commented on issue #70442: URL: https://github.com/apache/doris/issues/70442#issuecomment-6777388289
### Follow-up: the same cost when a few candidate rows remain in one data page The one-candidate shortcut above does not cover a key seek that leaves a small number of rows (2 to a few dozen) inside a single data page. For those, `ColumnReader::get_row_ranges_by_bloom_filter` (`be/src/storage/segment/column_reader.cpp:850` on master) still loads the bloom-filter page for that data page. Bloom pages are up to 1 MiB, so that page is often much larger than the data page it would let us skip. When the bloom filter is also not cached, the lookup reads the big page just to decide whether to read a small one. In our measurements the bloom pass then removed zero candidates. **Proposal.** The loaded ordinal indexes already give the page sizes without reading anything: - the data page that holds the narrowed range, plus its dictionary page; - the bloom-filter page for that data page. If all of these hold, keep the candidates and let the normal row-predicate evaluation decide: - it is a query reader; - the narrowed range has more than one row and fits in one data page; - the bloom page is larger than the data page plus dictionary; - the bloom page is not in the page cache. A bloom page that is already cached is still used for pruning, and is held until the bloom iterator takes it. Broad scans, ranges that span data pages, and ngram bloom filters keep the current path. One detail matters for "is it cached": use the cache setting the bloom-filter reader was loaded with, not the shared `ColumnReader::_use_index_page_cache`. `FileColumnIterator::init` clears that flag for good as soon as any reader opens with `use_page_cache=false`, and compaction readers and sessions with the page cache disabled both do. After that, a shared-flag check would treat every bloom page as cold and skip pruning even when the page is cached. **Measurements.** Segments of 1,048,576 rows, page cache disabled, counting bytes read through the segment's file reader. All cases return the expected rows. | Lookup | Current | With the change | Reduction | | --- | ---: | ---: | ---: | | Integer key, 16 candidates in one page, one match | 1,128,090 | 78,484 | 93.0% | | Integer key, 16 candidates in one page, no match | 1,113,547 | 63,941 | 94.3% | | UUID-like string key, 16 candidates in one page, one match | 2,423,226 | 1,374,606 | 43.3% | | UUID-like string key, 16 candidates in one page, missing id | 2,121,865 | 1,335,600 | 37.1% | | UUID-like string key, 64 candidates across data pages (control) | 2,423,227 | 2,423,227 | 0% | | UUID-like string key, broad scan (control) | 3,226,888 | 3,226,888 | 0% | With 2 and 8 candidates the reads are the same as with 16. With a warm cache, a bloom page that is already cached still prunes 16 rejected candidates. We also add four scan profile counters, to show how often a narrowed range fits one data page and how often the shortcut fires: - `BloomMultiCandidateRanges` - `BloomMultiCandidateSinglePageRanges` - `BloomColdPagesBypassed` - `BloomPayloadBytesBypassed` We have a patch for this too and can include it in the PR for this issue. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
