andygrove commented on PR #4798: URL: https://github.com/apache/datafusion-comet/pull/4798#issuecomment-6131551235
@rich7420 the benchmark you asked for is in cf2ff79e3f as `CometPivotBenchmark`. It pivots `sum(v)` over 4M rows on a string column with 4, 32 or 128 values, grouped into 1K, 100K or 1M groups. It compares Spark, Comet with `PivotFirst` disabled (the aggregates that use it fall back while the rest of the plan stays native), and Comet with `PivotFirst` native. Every pivot column goes to the noop sink. Before timing, each Comet run is checked against Spark's answer and for the plan its name claims. You were right to ask for it. The first version only had a row `Accumulator`, so grouped aggregation went through DataFusion's `GroupsAccumulatorAdapter`, which visits every group's state on each batch. From 100K groups up it was slower than falling back: at 128 values and 1M groups it took 22.7 s against 5.2 s for the fallback. 0c4279a65c adds a `GroupsAccumulator`. With it, best times in ms on an M3 Max (`local[1]`, optimized build without LTO): | Pivot values × groups | Spark | `PivotFirst` in Spark | `PivotFirst` native | | --- | ---: | ---: | ---: | | 4 × 1K | 293 | 164 | 128 | | 4 × 100K | 1,043 | 338 | 205 | | 4 × 1M | 3,333 | 1,882 | 833 | | 32 × 1K | 368 | 188 | 153 | | 32 × 100K | 2,872 | 1,307 | 750 | | 32 × 1M | 5,180 | 3,144 | 2,284 | | 128 × 1K | 754 | 293 | 296 | | 128 × 100K | 4,417 | 2,332 | 1,528 | | 128 × 1M | 8,256 | 5,616 | 5,490 | The two cases at parity are 128 values with 1K or 1M groups. At 1M groups, operator metrics put most of the time outside `PivotFirst`. The largest piece is the shuffle of 128 mostly null state columns, followed by the 128 `GetArrayItem` extractions in Spark's final projection. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
