comphead commented on issue #7341: URL: https://github.com/apache/datafusion-comet/issues/7341#issuecomment-6884877027
Data point with #7345 in, on Comet main with DataFusion main and Spark 4.0: the final merge of 17 `hll_sketch_agg(CASE WHEN ... THEN id END, 18)` columns, 398M partial rows into 257M groups over 216 tasks, 8 tasks per executor with a 3 GiB task share. | | Spark (`ObjectHashAggregate`, sort fallback) | Comet (`CometHashAggregate`) | |---|---|---| | Final aggregate time, all tasks | 1,105 s | 10,554 s | | Partial aggregate time, all tasks | 997 s | 4,054 s | | Final stage task time | 15 to 21 s | 48 to 56 s for the 209 tasks that spilled, 30 to 31 s for the 7 that did not | A task's groups need about 5.1 GiB of reservation, about 4.6 KB per group for 17 sketches. `fair_unified` gives the aggregate 1.5 GiB, half of the task's share, because the native shuffle writer above it holds the other half. So almost every task spills, and spilling serializes, sorts and merges every sketch again. So with #7345 in, the remaining gap is item 2, the `GroupsAccumulatorAdapter` dispatch with one boxed accumulator per group per column, and the per-group footprint, which decides whether the merge spills. The split of the task's share between the aggregate and the shuffle writer is in #7628. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
