Rich-T-kid opened a new issue, #10590: URL: https://github.com/apache/arrow-rs/issues/10590
### Is your feature request related to a problem or challenge? When reading a PLAIN-encoded string/binary column into a Dictionary Arrow type, the current sad path in ByteArrayDictionaryReader materializes string data twice: 1. Page bytes -> OffsetBuffer (copy #/1) 2. OffsetBuffer -> StringArray -> GenericByteDictionaryBuilder which hashes every value and copies unique ones into the dict values buffer (copy #/2) each row also incurs a hash cost. Not sure if this is avoidable but worth flagging ### Describe the solution you'd like 1. add benchmarks and validate that the existing ones cover the changes this issues intends to make 2. Do a single-pass decode that maintains the dedup HashTable during page scanning. Since PLAIN byte array encoding is [4-byte length][bytes], the bytes are already in memory when we scan them. We can hash before copying, check the table, and only write new entries into the dictionary values buffer — emitting key indices directly. ### Describe alternatives you've considered n/a ### Additional context - https://github.com/apache/datafusion/issues/24111 - follow up flagged by @/alamb https://github.com/apache/datafusion/issues/24111#issuecomment-5206277128 - https://github.com/apache/datafusion/issues/24112 focuses on RLE encoded parquet columns, this PR would focus on speeding up reads for PLAIN encoded strings to dictionary arrays -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
