ulysses-you opened a new pull request, #57276:
URL: https://github.com/apache/spark/pull/57276
### What changes were proposed in this pull request?
Add a new physical rule `CombineAdjacentAggregation` in the query stage
preparation rules. When there is an adjacent aggregation pair with `Partial`
and `Final` mode that groups by the same expressions and refers to the same
logical aggregate, combine them into a single `Complete` mode aggregate so we
do not need to merge the aggregation buffer.
For example:
```
HashAggregate (Final) HashAggregate (Complete)
| |
HashAggregate (Partial) => Exchange
|
Exchange
```
It supports `HashAggregateExec`, `SortAggregateExec` and
`ObjectHashAggregateExec`.
The new rule runs before `ReplaceHashWithSortAgg`, so the "merge adjacent
partial and final aggregate" logic that used to live in
`ReplaceHashWithSortAgg` is removed and consolidated here.
`ReplaceHashWithSortAgg` can then replace the combined `Complete` hash
aggregate with a sort aggregate when the child ordering is satisfied.
The behavior is guarded by a new internal config
`spark.sql.execution.combineAdjacentAggregation`, which falls back to
`spark.sql.execution.replaceHashWithSortAgg` (disabled by default). So this
does not change plans out of the box, and enabling `replaceHashWithSortAgg`
also enables combining (preserving the previous merge behavior that was moved
out of that rule).
### Why are the changes needed?
Improve performance by skipping the aggregation buffer merge. It has more
benefits in the high cardinality case.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Add `CombineAdjacentAggregationSuite` and extend
`ReplaceHashWithSortAggSuite`.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
