andygrove commented on issue #25606: URL: https://github.com/apache/datafusion/issues/25606#issuecomment-7013046957
I'm closing #26429 and have unassigned myself. @EmilyMatt, this is yours if you want it (comment `take`). Here is what I learned from the PR and from your and @comphead's reviews. **What #26429 did.** Before each batch, it reserved the larger hash table that the batch would need if every row, in every grouping set, were a new group. If the pool refused while the table held groups, Final and Single spilled first and Partial emitted first. That cut `ResourcesExhausted` failures on a high-cardinality spilling query (numbers in the PR description), but the trigger is wrong. **Reserving for the worst case spills when a batch adds no groups** ([comphead's review](https://github.com/apache/datafusion/pull/26429#discussion_r4470270128)). A table with 100 groups asks for 278,528 bytes before each 8192-row batch, even when every row hits an existing group. A spill or emit shrinks the table to capacity 0, so as soon as it holds its 100 groups again, it asks again. I ran `SUM(v) GROUP BY k` over 100 keys, in 20 batches of 8192 rows, under a 256 KB pool. With #26429, Single and Final spill 20 times, and Partial emits early 19 times, passing on 2,000 rows instead of 100. On main nothing spills or emits, and with a 1 MB pool the PR doesn't either. With 8 grouping sets, the request is 8x larger. **A better fix is to reserve when the table actually grows**, as comphead suggested. It overlaps with your slice-and-retry idea. `intern` stops before the insert that would grow the table and reports how many rows it consumed. The aggregate updates the accumulators for those rows, reserves the doubled table (or spills or emits if the pool refuses) and continues with the rest of the batch. Rows that hit existing groups cost nothing, and a batch that grows the table several times reserves each growth. Where the tables grow today: - `GroupValuesColumn` grows in `insert_accounted` (`datafusion/common/src/utils/proxy.rs`), which already branches on `len() == capacity()`. In the vectorized path, every map insert happens in `collect_vectorized_process_context`. `scalarized_intern_remaining` adds colliding groups to an existing entry through `group_index_lists` instead of inserting. So stopping in the first phase and running the later phases on the rows before the stop looks feasible. - `GroupValuesRows`, `ArrowBytesMap` and `ArrowBytesViewMap` look keys up first and grow only in `insert_unique`, when a new group goes into a full table. - `GroupValuesPrimitive` uses `HashTable::entry`, which in hashbrown 0.17 reserves room for one more entry before it looks the key up. A full table grows even when the key is already there, so the check has to come before `entry()`. Most of the work is in the streams: spilling or emitting in the middle of a batch (the partial stream's emit is async), then resuming at the right grouping set and row. **Also worth knowing:** - More early emits in Partial keep the skip-aggregation probe from firing, because each emit resets the group count it compares against (#27350, fix in #27351). - Your [first two points](https://github.com/apache/datafusion/pull/26429#issuecomment-6249539823), the emit peak and capacity versus data, are about the emit itself rather than table growth. But a fix that spills or emits more often makes them matter more. - Two things from #26429 can be a starting point. The three stream tests (`final_hash_spills_before_the_hash_table_grows` and its Single and Partial versions) fail on main. `estimate_hash_table_growth` matches hashbrown 0.17's allocation to within 16 bytes. If you use it for a whole batch, see [comphead's fix](https://github.com/apache/datafusion/pull/26429#discussion_r4470270091) for batches that grow the table more than once. **Why it's worth fixing.** With integer keys, the hash table is a large part of an aggregate's memory. A `SUM(v) GROUP BY k` table holding 7,340,032 groups accounts for 277 MB, 143 MB of which is the hash table. The next group allocates a 285 MB table before the old one is freed. So for a moment the aggregate holds 562 MB, about twice what it has reserved. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
