Mryange opened a new pull request, #70761: URL: https://github.com/apache/doris/pull/70761
### Background Doris can rule out a segment using a string column's dictionary when all of its data pages use dictionary encoding. The reader probes the last data page when it first needs to determine that encoding. During a scan, a dictionary-encoded data page also uses the dictionary to decode values. ### What problem does this PR solve? Issue Number: None Related PR: None PR Type: Performance Problem Summary: A predicate such as `s LIKE '%qzx%'` cannot prune a segment using its dictionary, but initializing the scan can still load and decode that dictionary. If a subsequent NGram filter eliminates the segment, the scan never uses it. Check predicate capability before dictionary pruning and defer dictionary preparation during the encoding probe. ### Details - **Why it was slow:** Both the last-page encoding probe and dictionary pruning could prepare the dictionary decoder and populate a per-iterator array of dictionary entries. - **Optimization:** Probe the last page using a temporary parsed page. For a conjunction, run dictionary pruning when any child supports it. Bind the dictionary on the first data-page read that needs it. - **Results:** A prior RELEASE baseline recorded a median of 2.820 ms in the dictionary phase across seven warm profiles of a LIKE query scanning 128 segments with 1,048,576 rows. The new unit test checks that the probe and unsupported-pruning path leave the dictionary unloaded. Matched before-and-after query timings are pending. - **Applicability and cost:** This avoids preparation when a scan can skip dictionary pruning and never needs dictionary-encoded rows. Scans that read those rows still prepare the dictionary, and a scan that revisits the probe page parses it again. ### Testing - ASAN BE build passed with `BUILD_TYPE=ASAN bash build.sh --be -j48`. - `BUILD_TYPE_UT=ASAN bash run-be-ut.sh --run -j48 --filter='ColumnReaderTest.*:BlockColumnPredicateTest.*'` passed all 108 tests, including both new dictionary tests and the existing read-ahead probe retry test. - The unfiltered run stalled in a file-cache teardown. That test passed in isolation. Continuing the suite encountered an ASAN crash in `HdfsFileSystemTest.Write` in `FileCacheFactory::get_by_path`, which also reproduced in isolation. The full suite did not finish. ### Release note None. ### Check List (For Author) - Behavior changed: No - Does this need documentation: No - Pre-submission status: Partially verified: ASAN build and affected suites passed. The full suite did not finish, and matched query timings are pending. ### AI Disclosure - Tool: OpenAI Codex - Generated: Implementation, unit tests, and description - Human review: The author reviewed and discussed the local implementation before remote validation. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
