comphead commented on issue #7341:
URL: 
https://github.com/apache/datafusion-comet/issues/7341#issuecomment-6884877027

   Data point with #7345 in, on Comet main with DataFusion main and Spark 4.0: 
the final merge of 17 `hll_sketch_agg(CASE WHEN ... THEN id END, 18)` columns, 
398M partial rows into 257M groups over 216 tasks, 8 tasks per executor with a 
3 GiB task share.
   
   | | Spark (`ObjectHashAggregate`, sort fallback) | Comet 
(`CometHashAggregate`) |
   |---|---|---|
   | Final aggregate time, all tasks | 1,105 s | 10,554 s |
   | Partial aggregate time, all tasks | 997 s | 4,054 s |
   | Final stage task time | 15 to 21 s | 48 to 56 s for the 209 tasks that 
spilled, 30 to 31 s for the 7 that did not |
   
   A task's groups need about 5.1 GiB of reservation, about 4.6 KB per group 
for 17 sketches. `fair_unified` gives the aggregate 1.5 GiB, half of the task's 
share, because the native shuffle writer above it holds the other half. So 
almost every task spills, and spilling serializes, sorts and merges every 
sketch again.
   
   So with #7345 in, the remaining gap is item 2, the 
`GroupsAccumulatorAdapter` dispatch with one boxed accumulator per group per 
column, and the per-group footprint, which decides whether the merge spills. 
The split of the task's share between the aggregate and the shuffle writer is 
in #7628.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to