Rich-T-kid opened a new pull request, #12260:
URL: https://github.com/apache/arrow-rs/pull/12260

   # Which issue does this PR close?
   
   - Closes #12195
   
   # Rationale for this change
   
   When a requested dictionary batch crossed from a dictionary-encoded page 
into a plain page, `DictionaryBuffer` expanded every existing key back into its 
byte value. Finalization then hashed those repeated bytes again to rebuild a 
dictionary. This made transition work proportional to the expanded row bytes 
instead of the unique dictionary bytes.
   
   # What changes are included in this PR?
   
   - Add a merged dictionary state that preserves existing keys, interns 
referenced dictionary values, remaps later dictionary keys, and interns 
plain-page values.
   - Keep the existing single-dictionary fast path unchanged.
   - Preserve null handling, key validation, requested value-type conversion, 
UTF-8 validation, fixed-size binary handling, and key-overflow behavior.
   - Add transition coverage and focused elapsed-time and cumulative-allocation 
benchmarks for low and medium cardinality with aligned and crossing batch 
boundaries.
   
   # Are these changes tested?
   
   The added tests cover dictionary-to-dictionary and dictionary-to-plain 
transitions, sparse referenced values, invalid keys, null padding, fixed-size 
binary values, and key compaction/overflow. The full Parquet library suite 
passes (1,589 passed, 1 ignored), as do the focused bench-target checks and 
Clippy with warnings denied.
   
   The focused 16,384-row, 32-byte string benchmark reads two 8,192-row 
batches. Offset 8,192 is the aligned control; offset 8,191 forces a 
dictionary-to-plain transition within the first batch.
   
   | Cardinality | Crossing elapsed time | Crossing cumulative allocation | 
Crossing value bytes hashed |
   | --- | ---: | ---: | ---: |
   | 10 | 148.6 us -> 88.4 us (1.68x) | 3,150,108 B -> 1,496,004 B (-52.5%) | 
524,288 B -> 262,496 B (-49.9%) |
   | 1,000 | 166.6 us -> 107.2 us (1.55x) | 3,189,708 B -> 1,630,284 B (-48.9%) 
| 524,288 B -> 294,176 B (-43.9%) |
   
   The aligned controls did not regress: 74.9 us -> 72.1 us and 85.7 us -> 83.9 
us. Their allocation and hashed-byte counts are exactly unchanged.
   
   I also built DataFusion at `adffd19e2c6c3ef2ce03cda30a20856a7d439d97` 
against the Arrow base and this implementation. A 10-million-row Parquet scan 
with ten dictionary-to-plain row-group transitions, 1,000,000-row batches, and 
dictionary preservation enabled produced these ABBA results over 40 iterations 
per revision:
   
   | Batch boundary | Base median | PR median | Result |
   | --- | ---: | ---: | ---: |
   | Crossing | 59.62 ms | 25.15 ms | 2.37x faster |
   | Aligned | 14.26 ms | 14.56 ms | within run noise |
   
   Every iteration returned 10,000,000 rows. ClickBench Q20/Q22/Q23 result 
files were byte-identical between revisions; the clean Q22 paired blocks ranged 
from -0.32% to +1.85%, consistent with noise. Page inspection showed those 
files expose only short mixed batches, so the targeted scan above provides the 
useful end-to-end amplification.
   
   # Are there any significant user-facing changes?
   
   No public API or output changes.
   
   # AI usage disclosure
   
   AI tools helped draft parts of the implementation, tests, and benchmark 
harness. I reviewed the changes and validated correctness and performance 
locally.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to