geoffreyclaude opened a new issue, #26411: URL: https://github.com/apache/datafusion/issues/26411
### Describe the bug SortExec can retain a prefetched output batch after its sort-run reservation has been released. The run stream releases the batch's charge when polled, while merge read-ahead keeps the batch alive until it is consumed. This leaves live memory outside the corresponding reservations and can allow the memory pool to admit more work than the intended budget permits. The gap can occur once per multi-batch run, independently of the number of concurrent sorting jobs. With many runs, the aggregate retained memory can therefore be substantial. Runs producing only one output batch have no additional prefetched batch after handing it to the merger. This is pre-existing behavior: the buffered-producer implementation using `spawn_buffered(stream, 1)` can likewise retain a batch after releasing its reservation. The lookahead adapter in #26242 preserves that merge-time behavior while keeping all prepared output reserved before merging starts. ### To Reproduce A focused regression setup would use multiple output batches per run, more runs than sort permits, and a bounded memory pool. Wide payload columns with a narrow sort key help distinguish retained batch memory from cursor memory. Check reservation coverage while the merge holds a prefetched batch from each input. The relevant code is the [lookahead adapter in #26242](https://github.com/apache/datafusion/blob/9f5f04caa51a5963d07dece371bb802780d4d51d/datafusion/physical-plan/src/sorts/sort.rs#L853-L876), together with the reservation release in `sort_run` and the merger's batch/cursor reservations. ### Expected behavior - Retained read-ahead batches remain covered by memory reservations throughout the handoff from sort runs to the merger. - The merger has sufficient workspace for both its current batches and sort-key cursors. It already polls inputs before reserving them; removing lookahead alone can make tight-budget merges fail. - Regression coverage preserves correct sorted output, tight-workspace merging, and reservation cleanup when output is dropped or execution is cancelled. ### Additional context Raised in [Jayant's review comment on #26242](https://github.com/apache/datafusion/pull/26242#discussion_r4286579486), where he suggested addressing this accounting gap as follow-up work. Related broader tracking: #23760 (multi-level merge memory accounting). ### Willingness to contribute I cannot contribute a fix for this bug at this time. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
