sunchao commented on PR #24740: URL: https://github.com/apache/datafusion/pull/24740#issuecomment-5521182021
@jayzhan211 Thanks for the detailed repro. Fixed your [single-batch spill finding](https://github.com/apache/datafusion/pull/24740#issuecomment-5460002438) in [804fbca9c](https://github.com/apache/datafusion/pull/24740/commits/804fbca9c566795e5a7dfc76882bbc1a696c4c2d). Removed `release_unused()` from the empty and single-batch branches. Kept it in the concatenation branch and added the explanation you suggested: concatenation can grow the ordinary sort reservation, which cannot borrow merge workspace. That path can still require a fresh parent grant, as documented in the PR description. Added `test_single_batch_spill_preserves_workspace_after_limit_decreases`, covering both a spill triggered by inserting the second batch and the final leftover single-batch spill. The reduced limit makes the first spill fail if its workspace is released and must be reacquired. Also updated `test_single_batch_spill_returns_live_workspace_loan_on_drop` to exercise `in_mem_sort_stream(false, false)`, assert the remaining loan through the available workspace credit, and verify cleanup when the stream is dropped early. Both regressions fail with the old branch logic and pass with the fix. Local workspace and SQL logic tests, formatting, Clippy, and the full lint script passed; validation details are in the PR description. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
