JonSnow1807 opened a new pull request, #12113: URL: https://github.com/apache/arrow-rs/pull/12113
# Which issue does this PR close? Closes #12053. # Rationale for this change The sparse-dictionary encoding path gathers one dictionary value per input row before encoding. The current gather performs a dynamic MutableArrayData extension for every key, which adds overhead for common string and binary dictionaries. # What changes are included in this PR? Add typed gather implementations for Utf8, LargeUtf8, Binary, and LargeBinary dictionary values. These build the offsets, value buffer, and validity bitmap directly. Other value types continue to use the generic MutableArrayData implementation. Offset growth is checked so that overflow continues to reach the existing fallback. The dictionary-gather thresholds are unchanged. # Are these changes tested? New tests cover all four specialized value types, sliced keys, null keys, null dictionary values, multiple signed and unsigned key widths, the generic fallback, and sparse binary dictionary encoding and decoding. All 110 arrow-row library tests pass. Workspace tests, workspace Clippy checks, formatting, and spelling checks also pass. Performance was measured using the existing prepared 8,192-row string-dictionary benchmark on an Intel Xeon E5-2673 v4, Linux x86_64, with Rust 1.99.1. This benchmark cycles through contiguous key chunks. Two paired Criterion runs used 50 samples, a one-second warmup, and a three-second measurement period. Execution order was reversed for the second pair. The table reports the average of the two runs’ mean estimates. | Dictionary values | Before | After | Change in mean time | | ---: | ---: | ---: | ---: | | 10,000 | 170.3 µs | 174.5 µs | +2.5% | | 32,768 | 310.8 µs | 242.0 µs | -22.1% | | 100,000 | 318.5 µs | 242.9 µs | -23.7% | | 1,000,000 | 309.4 µs | 244.9 µs | -20.8% | | 4,000,000 | 312.4 µs | 240.1 µs | -23.1% | The 10,000-value control does not enter the dictionary-gather path and averaged 2.5% slower in these measurements. Individual runs varied by several percent. The additional dictionary sizes were used through a separate local benchmark patch; they are not included in this implementation diff. # Are there any significant user-facing changes? No public API or intended result changes. AI assistance: AI tools assisted with implementation, tests, independent review, and validation. I reviewed the code, tests, and description before submission. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
