pepijnve commented on issue #18411: URL: https://github.com/apache/datafusion/issues/18411#issuecomment-3497705454
> 🤔 figuring out how to avoid re-hashing the same short values over and over again (StringView can be thought of as a type of dictionary encoding) is a very interesting possibility @alamb in the GIS data format codecs I wrote long ago we solved this by interning strings. The decoder maintained a hash table of strings and single instanced everything. I haven't read it in detail yet, but the DuckDB related thesis that was linked to on Discord (https://homepages.cwi.nl/~boncz/msc/2025-OmidAfroozeh.pdf) seems to be using similar techniques. Unfortunately I don't see how you would be able to use this while sticking to the Arrow memory layouts. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
