Weijun-H opened a new issue, #25405: URL: https://github.com/apache/datafusion/issues/25405
### Describe the bug The external sort batch builder can keep consumed input batches alive after producing a partial output batch. This delays releasing memory that is no longer needed and can increase memory pressure for sorts that produce output in multiple chunks. ### To Reproduce This can be reproduced with a `BatchBuilder` state that has: - one input batch whose rows have just been emitted, - another input batch still referenced by pending row indices, - another input batch still referenced by a live cursor, - and an empty stream whose default cursor should not keep any consumed batch alive. After emitting only the consumed rows, the consumed input batch is no longer needed but is retained until later output drains. ### Expected behavior After partial output, input batches that are no longer referenced by pending row indices or live cursors should be released immediately. Batches that are still referenced must remain available for later output. ### Additional context This issue is about earlier release of already-consumed sort input batches. It does not change sort output sizing, spill admission, or user-visible APIs. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
