geoffreyclaude opened a new pull request, #27132: URL: https://github.com/apache/datafusion/pull/27132
## Which issue does this PR close? Follow-up to #24740 for the small-batch concatenation case of #24739. ## Rationale for this change A sort can finish spilling, then fail with `ResourcesExhausted` when it tries to reacquire workspace it already held. The small-batch concatenation path releases that workspace, allowing another consumer to take it or a reduced allocation limit to prevent reacquisition. ## What changes are included in this PR? Keep workspace reserved during small-batch spilling. If concatenation expands shared buffers and its reservation cannot grow, sort and merge the original batches instead. Original inputs remain reserved until their replacement is accepted. Final output continues to release unused workspace. ## What is the testing strategy for this PR? Two focused regressions cover: - Four integer rows spilling after the allocation limit decreases, so existing workspace can be reused but cannot be reacquired. - Aliased columns whose concatenation needs more memory, forcing fallback to the original batches in a full pool. Both check sorted output and cleanup. Existing coverage checks that final output releases unused workspace. ## Are there any user-facing changes? Affected sorts can continue spilling under memory pressure. Internal comments document the retention and fallback; existing configuration documentation remains accurate. This PR was prepared with AI assistance. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
