Rich-T-kid opened a new pull request, #12260: URL: https://github.com/apache/arrow-rs/pull/12260
# Which issue does this PR close? - Closes #12195 # Rationale for this change When a requested dictionary batch crossed from a dictionary-encoded page into a plain page, `DictionaryBuffer` expanded every existing key back into its byte value. Finalization then hashed those repeated bytes again to rebuild a dictionary. This made transition work proportional to the expanded row bytes instead of the unique dictionary bytes. # What changes are included in this PR? - Add a merged dictionary state that preserves existing keys, interns referenced dictionary values, remaps later dictionary keys, and interns plain-page values. - Keep the existing single-dictionary fast path unchanged. - Preserve null handling, key validation, requested value-type conversion, UTF-8 validation, fixed-size binary handling, and key-overflow behavior. - Add transition coverage and focused elapsed-time and cumulative-allocation benchmarks for low and medium cardinality with aligned and crossing batch boundaries. # Are these changes tested? The added tests cover dictionary-to-dictionary and dictionary-to-plain transitions, sparse referenced values, invalid keys, null padding, fixed-size binary values, and key compaction/overflow. The full Parquet library suite passes (1,589 passed, 1 ignored), as do the focused bench-target checks and Clippy with warnings denied. The focused 16,384-row, 32-byte string benchmark reads two 8,192-row batches. Offset 8,192 is the aligned control; offset 8,191 forces a dictionary-to-plain transition within the first batch. | Cardinality | Crossing elapsed time | Crossing cumulative allocation | Crossing value bytes hashed | | --- | ---: | ---: | ---: | | 10 | 148.6 us -> 88.4 us (1.68x) | 3,150,108 B -> 1,496,004 B (-52.5%) | 524,288 B -> 262,496 B (-49.9%) | | 1,000 | 166.6 us -> 107.2 us (1.55x) | 3,189,708 B -> 1,630,284 B (-48.9%) | 524,288 B -> 294,176 B (-43.9%) | The aligned controls did not regress: 74.9 us -> 72.1 us and 85.7 us -> 83.9 us. Their allocation and hashed-byte counts are exactly unchanged. I also built DataFusion at `adffd19e2c6c3ef2ce03cda30a20856a7d439d97` against the Arrow base and this implementation. A 10-million-row Parquet scan with ten dictionary-to-plain row-group transitions, 1,000,000-row batches, and dictionary preservation enabled produced these ABBA results over 40 iterations per revision: | Batch boundary | Base median | PR median | Result | | --- | ---: | ---: | ---: | | Crossing | 59.62 ms | 25.15 ms | 2.37x faster | | Aligned | 14.26 ms | 14.56 ms | within run noise | Every iteration returned 10,000,000 rows. ClickBench Q20/Q22/Q23 result files were byte-identical between revisions; the clean Q22 paired blocks ranged from -0.32% to +1.85%, consistent with noise. Page inspection showed those files expose only short mixed batches, so the targeted scan above provides the useful end-to-end amplification. # Are there any significant user-facing changes? No public API or output changes. # AI usage disclosure AI tools helped draft parts of the implementation, tests, and benchmark harness. I reviewed the changes and validated correctness and performance locally. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
