viirya opened a new issue, #12045: URL: https://github.com/apache/arrow-rs/issues/12045
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.** Reading a dictionary encoded BYTE_ARRAY column into a `StringArray` / `BinaryArray` (the default for string columns, since most writers dictionary encode them) goes through `OffsetBuffer::extend_from_dictionary`. For every key it does: ```rust self.values.extend_from_slice(&dict_values[start_offset..end_offset]); let index_offset = I::from_usize(self.values.len())...; self.offsets.push(index_offset); ``` So every value costs a `memcpy` call with a data dependent length plus capacity checks on both `Vec`s. Most dictionary encoded strings are short (category names, codes, tags), so the call overhead dominates the copy. In the existing `arrow_reader` benchmark (dictionary values of 18-19 bytes) this loop takes ~300 µs for 80k values, about 3.8 ns per value. On an end-to-end benchmark that reads a file with a 4-16 byte low-cardinality string column through `ParquetRecordBatchReader`, this expansion is most of the decode time. Making it cheaper cuts that read by ~60%, and a mixed analytics table (ids, timestamps, dictionary strings, and so on) gets 17-27% faster. **Describe the solution you'd like** Write the values and offsets through raw pointers, and copy short values (up to 64 bytes) with fixed size 16 byte chunk copies into reserved slack instead of calling `memcpy`. Long values keep using `memcpy`. With this, the existing `arrow_reader` dictionary cases get 48-53% faster (no NULLs), and there's no regression for long values. **Describe alternatives you've considered** Validating all keys and sizing the output in a first pass, then copying in a second pass without any checks. It's fast for short values, but the extra pass makes values of 64 bytes or more 5-15% slower than today. A single pass that only checks capacity per value (a predictable branch) doesn't have this problem. **Additional context** The same per-value `extend_from_slice`/`try_push` pattern exists elsewhere, for example in `ByteArrayDecoderPlain::read` and the delta byte array decoders, and in the fixed length byte array dictionary path. Those can be looked at separately. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
