jayzhan211 commented on PR #25853: URL: https://github.com/apache/datafusion/pull/25853#issuecomment-6702291445
I put together [this page](https://claude.ai/artifact/FREAr6m99WzAH8HPW7t5yw) to help me understand the overall sort spilling design. It has diagrams, code links and measurements, in case it helps others here. When memory runs out, `SortExec` and hash aggregates write sorted runs to disk, and `MultiLevelMergeBuilder` merges them in as many passes as memory allows. Before each pass it reserves 2 to 4 times the largest batch of each run, then bypasses the memory pool. Batches are cut by row count, so nothing limits that largest batch. Three workarounds exist only because of that gap: re-spilling a run with its batches halved (#22945), per-run row limits (#23286), and a scan and retry for runs that can't shrink (#25383). This PR adds a byte limit for aggregates, at the spill writer and the merge output. `SortExec` and those three workarounds don't change. Remaining work, as untested ideas that could go under #26711: - Apply one byte limit to every caller, including `SortExec`, and derive it from the memory budget instead of a constant. Drafts #25910 and #26987 go in this direction. - Once that's in place, delete the three workarounds. - Track the merge's real memory instead of bypassing the pool after admission. - Let replay release a finished group before it takes the next one, so it peaks at one large group instead of two. - Test that a query passing at one memory limit also passes at any larger limit. The page has a counterexample. Please correct anything I got wrong. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
