Dandandan opened a new pull request, #2313: URL: https://github.com/apache/datafusion-ballista/pull/2313
## Which issue does this PR close? N/A — follow-up from benchmarking DataFusion's partial-aggregation skip heuristic under Ballista. ## Rationale for this change A `Partial` aggregate probes its first `skip_partial_aggregation_probe_rows_threshold` rows (100k) and, if distinct-groups/rows exceeds `skip_partial_aggregation_probe_ratio_threshold`, stops aggregating and passes its input straight through. The decision is latched for the rest of the stream, so a prefix that is unrepresentative of the whole input is never revisited. A mispredicted skip costs more here than it does in DataFusion. In-process, the unreduced rows are merely handed to the next operator. In Ballista the `Partial` sits directly beneath a shuffle boundary, so those rows are additionally hash-partitioned, serialized, written to shuffle files and read back by the `FinalPartitioned` stage. That asymmetry argues for skipping only when the probe is confident, which is what raising the ratio does. ## What changes are included in this PR? Ballista now defaults `datafusion.execution.skip_partial_aggregation_probe_ratio_threshold` to `0.95` (DataFusion's default is `0.8`), alongside the other documented Ballista deviations in `ballista_restricted_configuration()`. ### How the value was chosen The skip decision is a step function, so rather than chase wall-clock noise I compared the `skipped_aggregation_rows` metric across thresholds via `EXPLAIN ANALYZE` — TPC-H SF10 (all 22) and ClickBench `hits_partitioned` (43), `target_partitions=8`. The counts are deterministic and reproduced byte-identically across runs. | threshold | TPC-H q2/q11/q17 | TPC-H q20 | ClickBench q13/q30 | ClickBench q31/q32 | |---|---|---|---|---| | 0.8 (DataFusion) | skip | skip | **skip (mispredicted)** | skip | | 0.9 | skip | skip | no skip | skip | | **0.95** | skip | skip | no skip | skip | | 0.99 | **no skip** | skip | no skip | skip | Only six aggregations in the two suites ever skip at all. - **ClickBench q13 and q30 are mispredictions at 0.8.** Their `reduction_factor` over the full input is 81%/71% (q13) and 56% (q30) — aggregating them is clearly worthwhile — but their leading rows look near-unique, so 0.8 latches "skip" and 16.5M rows pass through unreduced. - **ClickBench q31 and q32 are genuine skips**, at `reduction_factor` 100%: truly unique keys, 12.3M and 99.2M rows. These keep skipping at every value tested, including 0.99. - **TPC-H q2/q11/q17/q20 are genuine skips** (1.50M, 217K, 59.1M, 8.25M rows) and are preserved at 0.95. 0.9 and 0.95 make **identical** decisions on both suites, so the useful range is `[0.9, 0.95]`; 0.95 was taken as the more conservative end. 0.99 also fixes the mispredictions but gives up the TPC-H q2/q11/q17 skips, q17's alone covering 59M rows. On TPC-H alone this change is a no-op — 0.8, 0.9 and 0.95 skip exactly the same rows there. The difference comes entirely from ClickBench. ## Are these changes tested? Yes — a unit test asserts the new default and that an explicit user override still survives `upgrade_for_ballista`. `cargo test -p ballista-core --lib` passes (280 tests). The measurement above is plan-level (skip decisions), which is deterministic. I was not able to collect trustworthy head-to-head wall-clock numbers: the benchmark host was heavily contended, and per-query variance reached ±20% even between configurations whose plans are provably identical. Independent timing confirmation on a quiet machine would be welcome. ## Are there any user-facing changes? Ballista sessions aggregate slightly more eagerly by default. Anyone wanting DataFusion's behaviour back can `SET datafusion.execution.skip_partial_aggregation_probe_ratio_threshold = 0.8`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
