brancz opened a new pull request, #27043: URL: https://github.com/apache/datafusion/pull/27043
## Which issue does this PR close? Related to #26940. ## Rationale for this change Multi-key sorts can run out of memory while trying to spill. Initializing merge cursors for all buffered batches can expand dictionary keys beyond the available memory before any data is written. ## What changes are included in this PR? Spill buffered batches as independent sorted runs for large multi-key sorts with potentially expanding keys when the buffered reservation exceeds reserved merge workspace. Keep the existing path otherwise. The workspace comparison is a heuristic, not a bound on encoded key memory. Related to #26953 and #26987. #26953 reduces merge input batch sizes; this avoids the ingestion-time merge for selected spills, including small batches that #26953 leaves unchanged. #26987 bounds the batches written to spill files for subsequent merging. Also complementary to #27015, which reduces dictionary data written to disk. ## What is the testing strategy for this PR? Unit tests for wide dictionary keys and reservation lifetimes during async writes, including errors and cancellation. Extended workspace tests and Clippy pass. ## Are there any user-facing changes? No API changes, just an improvement to ingestion-time spilling. The tradeoff is more spill files and potentially more merge I/O for the selected path. @alamb @Jefffrey -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
