Rich-T-kid opened a new issue, #11414: URL: https://github.com/apache/arrow-rs/issues/11414
### Describe the problem When reading a dictionary-encoded parquet string column back as its native `Dictionary(Int32, Utf8)` type, the arrow reader is **slower than reading the same column as `Utf8View`** across the low-to-medium cardinality range, even though Dict should strictly do less work (no gather, no per-row copy). Measured on `main`, 1M rows, 16-byte strings, batch_size=8192, UNCOMPRESSED pages all `RLE_DICTIONARY`: | cardinality | Dict read | Utf8View read | View/Dict | |------------:|----------:|--------------:|----------:| | 10 | 629 µs | 629 µs | ~1.0× | | 100 | 582 µs | 582 µs | ~1.0× | | 1000 | 621 µs | 621 µs | ~1.0× | (Both paths read the same bytes; Dict should win by a wide margin.) Profiling (`sample`) shows the Dict path spends ~40% of CPU in bounds validation at `DictionaryBuffer::into_array` finalization, plus notable time in per-batch keys-vector alloc/resize/truncate. ### Describe the solution Three targeted changes to the dict reader: 1. Track a per-batch `key_upper_bound` so finalization can skip the full O(n) bounds check when every observed key is in range. 2. Replace `keys.resize(len, 0) -> decode -> truncate` with `reserve + spare_capacity_mut + set_len` to avoid zeroing + reshrinking the key buffer every batch. 3. Reuse the keys backing `Buffer` across batches (via `Buffer::into_vec` when refcount drops to 1) instead of allocating a fresh `Vec<K>` per batch. After all three, Dict at card=10 drops from 629 µs -> 218 µs (~2.9x), restoring the expected Dict < View ordering across the full matrix. ### Additional context - Benchmark added as a separate PR (`arrow_reader_string_encoding`). - Related: DataFusion #24227 (RLE->Dictionary) regressions on low-cardinality string groupings may share the same root cause. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
