andygrove commented on PR #25853: URL: https://github.com/apache/datafusion/pull/25853#issuecomment-6222064846
@EmilyMatt I prototyped your suggestion on current main to check it: before inserting a batch, the final and single-stage hash aggregates spill their table if the table plus the batch would exceed a budget. I also tried inserting the input one row at a time, so that a large input batch can't force a large spill either. On the reproducer from #25851 (`-m 128M`): - Spilling before inserting never triggers. Each final partition receives a single 80-row batch that already holds the state of all 8 of its groups, so the table is empty when it arrives. - With one-row inserts and budgets of 8, 16 and 32 MB, each partition writes 80 runs of about 2.4 MB, well within what reading back 2x needs. It still fails. The merge's intermediate passes emit batches of `batch_size` rows, and with fewer rows than that, each pass writes everything it merged as one batch. The runs grow from 2.4 MB to 112 MB, and splitting the 112 MB batch reserves twice its size: `Failed to allocate additional 225.1 MB`. - The two-partition query in Case H (24 MB) fails at every budget I tried. This PR passes both. When the input arrives in small batches, as in the single-stage queries of Case H, spilling at 2–4 MB does let them pass on main, because the merge's split path eventually shrinks the batches of the runs it splits. But replay then runs one row per batch: the last Case H query spills 50 times instead of 6, emits 340K batches instead of 75, and takes about 1.1 s against 0.1 s with this PR. So the size of the batches replay has to hold is set by how spill files and merges cut rows into batches, not by when the aggregate spills. That is what this PR changes. Reserving before inserting would still be worth doing on its own, since today the aggregate can overshoot its budget by a whole input batch before it spills, which is what #25606 tracks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
