Mryange opened a new pull request, #70761:
URL: https://github.com/apache/doris/pull/70761

   ### Background
   
   Doris can rule out a segment using a string column's dictionary when all of 
its data pages use dictionary encoding. The reader probes the last data page 
when it first needs to determine that encoding. During a scan, a 
dictionary-encoded data page also uses the dictionary to decode values.
   
   ### What problem does this PR solve?
   
   Issue Number: None
   Related PR: None
   PR Type: Performance
   
   Problem Summary:
   
   A predicate such as `s LIKE '%qzx%'` cannot prune a segment using its 
dictionary, but initializing the scan can still load and decode that 
dictionary. If a subsequent NGram filter eliminates the segment, the scan never 
uses it. Check predicate capability before dictionary pruning and defer 
dictionary preparation during the encoding probe.
   
   ### Details
   
   - **Why it was slow:** Both the last-page encoding probe and dictionary 
pruning could prepare the dictionary decoder and populate a per-iterator array 
of dictionary entries.
   - **Optimization:** Probe the last page using a temporary parsed page. For a 
conjunction, run dictionary pruning when any child supports it. Bind the 
dictionary on the first data-page read that needs it.
   - **Results:** A prior RELEASE baseline recorded a median of 2.820 ms in the 
dictionary phase across seven warm profiles of a LIKE query scanning 128 
segments with 1,048,576 rows. The new unit test checks that the probe and 
unsupported-pruning path leave the dictionary unloaded. Matched 
before-and-after query timings are pending.
   - **Applicability and cost:** This avoids preparation when a scan can skip 
dictionary pruning and never needs dictionary-encoded rows. Scans that read 
those rows still prepare the dictionary, and a scan that revisits the probe 
page parses it again.
   
   ### Testing
   
   - ASAN BE build passed with `BUILD_TYPE=ASAN bash build.sh --be -j48`.
   - `BUILD_TYPE_UT=ASAN bash run-be-ut.sh --run -j48 
--filter='ColumnReaderTest.*:BlockColumnPredicateTest.*'` passed all 108 tests, 
including both new dictionary tests and the existing read-ahead probe retry 
test.
   - The unfiltered run stalled in a file-cache teardown. That test passed in 
isolation. Continuing the suite encountered an ASAN crash in 
`HdfsFileSystemTest.Write` in `FileCacheFactory::get_by_path`, which also 
reproduced in isolation. The full suite did not finish.
   
   ### Release note
   
   None.
   
   ### Check List (For Author)
   
   - Behavior changed: No
   - Does this need documentation: No
   - Pre-submission status: Partially verified: ASAN build and affected suites 
passed. The full suite did not finish, and matched query timings are pending.
   
   ### AI Disclosure
   
   - Tool: OpenAI Codex
   - Generated: Implementation, unit tests, and description
   - Human review: The author reviewed and discussed the local implementation 
before remote validation.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to