brancz opened a new issue, #27014: URL: https://github.com/apache/datafusion/issues/27014
### Is your feature request related to a problem or challenge? IPC spill files can contain large dictionary value tables even when a batch references only a small subset of their entries. Slicing or merging dictionary-encoded batches can leave dictionary arrays retaining unused values. The existing spill garbage collection traverses nested arrays and compacts StringView/BinaryView buffers, but does not prune dictionary entries that are no longer referenced by the keys. This can cause a small batch to repeatedly write a much larger dictionary value table across spill and merge passes, increasing temporary disk usage and I/O. ### Describe the solution you'd like Compact oversized dictionary value tables before writing IPC spill batches, remap their keys, and then apply the existing child-array compaction. ### Describe alternatives you've considered n/a ### Additional context I already have a fix for this ready, it's quite simple since we already have a similar case for eg. StringView and we already have code for compacting dicts in arrow, so it's just a bit of plumbing and I'm just opening this for tracking purposes. ### Willingness to contribute None -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
