tlfua commented on issue #7396:
URL:
https://github.com/apache/datafusion-comet/issues/7396#issuecomment-6703622409
Two caveats on the comparison above:
- **Limits of the harness.**
- The Java side is a mirror of Spark's code, not Spark itself. I checked
it line by line against Spark v4.1.1 `HllSketchAgg`, `HllUnionAgg`, `HllUnion`
and `HllSketchEstimate`.
- The Comet side calls Comet's own native HLL code, but the aggregate glue
copies the `HllSketchAccumulator` and `HllUnionAccumulator` logic rather than
running a real DataFusion plan.
- Partitioning and merge order are simulated (1 to 16 partitions, forward
and reverse), not taken from real Spark or Comet query plans.
- The SQL tests in the PR will cover the end-to-end path.
- **The 27.9% Spark self-variation is at lgConfigK 4.** On its own, that
number overstates how much Spark varies at typical lgConfigK values. Here is
the per-lgConfigK view for `hll_sketch_agg`. At every lgConfigK, the Comet
difference is still smaller than Spark's own variation.
| lgConfigK | Spark: max variation across partition counts and merge orders
| Comet vs Spark: cases with differing estimate bits | Comet vs Spark: max rel
diff |
|---|---|---|---|
| 4 | 27.92% | 0 | 0 |
| 5 | 20.94% | 0 | 0 |
| 6 | 14.22% | 0 | 0 |
| 7 | 12.95% | 0 | 0 |
| 8 | 7.84% | 2 | 0.007% |
| 9 | 3.61% | 4 | 0.033% |
| 10 | 2.25% | 2 | 0.001% |
| 11 | 1.96% | 4 | 0.813% |
| 12 | 1.80% | 7 | 0.231% |
| 13 | 0.86% | 6 | 0.316% |
| 14 | 0.60% | 2 | 0.091% |
| 15 | 0.46% | 2 | 0.013% |
| 16 | 0.50% | 4 | 0.022% |
| 17 | 0.02% | 0 | 0 |
| 18 | 0.24% | 2 | 0.011% |
| 19 | 0.22% | 4 | 0.004% |
| 20 | 0.02% | 4 | 0.003% |
| 21 | 0.02% | 0 | 0 |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
