Dandandan commented on issue #18411: URL: https://github.com/apache/datafusion/issues/18411#issuecomment-3659234056
> > 🤔 figuring out how to avoid re-hashing the same short values over and over again (StringView can be thought of as a type of dictionary encoding) is a very interesting possibility > > [@alamb](https://github.com/alamb) in the GIS data format codecs I worked on long ago we solved this by interning strings. The decoder maintained a hash table of strings and single instanced everything and memoized hash values. I haven't read it in detail yet, but the DuckDB related thesis that [@camuel](https://github.com/camuel) linked to on Discord (https://homepages.cwi.nl/~boncz/msc/2025-OmidAfroozeh.pdf) seems to be using similar techniques. Unfortunately I don't see how you would be able to use this while sticking to the Arrow memory layouts. This could be done at the plan level perhaps? Keep the hash of the column around and share / use it in other operators -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
