andygrove commented on issue #25606:
URL: https://github.com/apache/datafusion/issues/25606#issuecomment-7013046957

   I'm closing #26429 and have unassigned myself. @EmilyMatt, this is yours if 
you want it (comment `take`). Here is what I learned from the PR and from your 
and @comphead's reviews.
   
   **What #26429 did.** Before each batch, it reserved the larger hash table 
that the batch would need if every row, in every grouping set, were a new 
group. If the pool refused while the table held groups, Final and Single 
spilled first and Partial emitted first. That cut `ResourcesExhausted` failures 
on a high-cardinality spilling query (numbers in the PR description), but the 
trigger is wrong.
   
   **Reserving for the worst case spills when a batch adds no groups** 
([comphead's 
review](https://github.com/apache/datafusion/pull/26429#discussion_r4470270128)).
 A table with 100 groups asks for 278,528 bytes before each 8192-row batch, 
even when every row hits an existing group. A spill or emit shrinks the table 
to capacity 0, so as soon as it holds its 100 groups again, it asks again. I 
ran `SUM(v) GROUP BY k` over 100 keys, in 20 batches of 8192 rows, under a 256 
KB pool. With #26429, Single and Final spill 20 times, and Partial emits early 
19 times, passing on 2,000 rows instead of 100. On main nothing spills or 
emits, and with a 1 MB pool the PR doesn't either. With 8 grouping sets, the 
request is 8x larger.
   
   **A better fix is to reserve when the table actually grows**, as comphead 
suggested. It overlaps with your slice-and-retry idea. `intern` stops before 
the insert that would grow the table and reports how many rows it consumed. The 
aggregate updates the accumulators for those rows, reserves the doubled table 
(or spills or emits if the pool refuses) and continues with the rest of the 
batch. Rows that hit existing groups cost nothing, and a batch that grows the 
table several times reserves each growth. Where the tables grow today:
   
   - `GroupValuesColumn` grows in `insert_accounted` 
(`datafusion/common/src/utils/proxy.rs`), which already branches on `len() == 
capacity()`. In the vectorized path, every map insert happens in 
`collect_vectorized_process_context`. `scalarized_intern_remaining` adds 
colliding groups to an existing entry through `group_index_lists` instead of 
inserting. So stopping in the first phase and running the later phases on the 
rows before the stop looks feasible.
   - `GroupValuesRows`, `ArrowBytesMap` and `ArrowBytesViewMap` look keys up 
first and grow only in `insert_unique`, when a new group goes into a full table.
   - `GroupValuesPrimitive` uses `HashTable::entry`, which in hashbrown 0.17 
reserves room for one more entry before it looks the key up. A full table grows 
even when the key is already there, so the check has to come before `entry()`.
   
   Most of the work is in the streams: spilling or emitting in the middle of a 
batch (the partial stream's emit is async), then resuming at the right grouping 
set and row.
   
   **Also worth knowing:**
   
   - More early emits in Partial keep the skip-aggregation probe from firing, 
because each emit resets the group count it compares against (#27350, fix in 
#27351).
   - Your [first two 
points](https://github.com/apache/datafusion/pull/26429#issuecomment-6249539823),
 the emit peak and capacity versus data, are about the emit itself rather than 
table growth. But a fix that spills or emits more often makes them matter more.
   - Two things from #26429 can be a starting point. The three stream tests 
(`final_hash_spills_before_the_hash_table_grows` and its Single and Partial 
versions) fail on main. `estimate_hash_table_growth` matches hashbrown 0.17's 
allocation to within 16 bytes. If you use it for a whole batch, see [comphead's 
fix](https://github.com/apache/datafusion/pull/26429#discussion_r4470270091) 
for batches that grow the table more than once.
   
   **Why it's worth fixing.** With integer keys, the hash table is a large part 
of an aggregate's memory. A `SUM(v) GROUP BY k` table holding 7,340,032 groups 
accounts for 277 MB, 143 MB of which is the hash table. The next group 
allocates a 285 MB table before the old one is freed. So for a moment the 
aggregate holds 562 MB, about twice what it has reserved.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to