ulysses-you commented on code in PR #57742: URL: https://github.com/apache/spark/pull/57742#discussion_r3735262711
########## docs/sql-performance-tuning.md: ########## @@ -181,6 +181,58 @@ Missing or inaccurate statistics will hinder Spark's ability to select an optima - **Query plan estimates**: You can inspect Spark's cost estimates in the optimized query plan via [`EXPLAIN COST`](sql-ref-syntax-qry-explain.html) or `DataFrame.explain(mode="cost")`. - **Runtime statistics**: You can inspect these statistics in the [SQL UI](web-ui.html#sql-tab) under the "Details" section as a query is running. Look for `Statistics(..., isRuntime=true)` in the plan. +## Optimizing the Aggregate + +### Adaptive Partial Aggregation + +A grouping aggregation normally runs in two phases: a partial aggregation before the shuffle and a +final aggregation after it. The partial aggregation is only worthwhile when it actually reduces the +number of rows; when the grouping keys are close to unique it maintains -- and possibly spills -- an +aggregation map roughly as large as its input while emitting almost as many rows as it consumed. + +When adaptive partial aggregation is enabled, hash aggregation measures the compaction ratio (the +number of processed rows divided by the number of keys held in its aggregation maps) at runtime +and, if the partial aggregation is not collapsing enough rows to be worthwhile, stops populating +the aggregation map and passes the remaining rows through as single-row partial aggregation buffers +for the final aggregation to merge. Query results are unchanged. The ratio is evaluated Review Comment: Confirmed. The counter only ever advances (`nextCheckRow += minRows`), so the ratio a periodic check sees is cumulative over the epoch, and your 1M-low-cardinality-then-1M-distinct input stays at ~2.0 throughout. The sentence promises something only the spill reset delivers. Two ways out, and they are not independent: 1. Make the docs match -- the periodic check catches inputs that are ineffective overall, and a late turn is caught only when a spill starts a new epoch. 2. Make each evaluation judge its own window, resetting the row count and the key baseline at every check point rather than only at a spill. Option 2 catches the late turn but makes a single window decisive, which is the same exposure as your item 2. I would rather settle the policy question there first and then make this consistent with it, instead of fixing the wording now and changing the behaviour after. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
