ulysses-you opened a new pull request, #57464: URL: https://github.com/apache/spark/pull/57464
### What changes were proposed in this pull request? A Hive UDAF whose `GenericUDAFEvaluator` legally uses different aggregation-buffer classes per mode (one for consuming raw input in `PARTIAL1`, another for merging partial buffers in `FINAL` -- as `MockUDAF2` does in the test) throws a `ClassCastException` when the aggregate is planned in `Complete` mode. `Complete` mode is produced by `CombineAdjacentAggregation`, which merges an adjacent partial/final pair (no shuffle between them) into a single complete-mode aggregate. In `Complete` mode, `update` is called but `merge` is not, so the buffer reaching `eval` is a `PARTIAL1`-mode buffer. `eval` previously passed `buffer.buf` straight to the `FINAL` evaluator's `terminate`, which expects a `FINAL`-mode buffer, causing the cast failure. `merge` already performs an on-demand `PARTIAL1 -> FINAL` buffer conversion for the same reason. This PR extracts that logic into a `toFinalBuffer` helper and applies it in `eval` as well, so the `Complete`-mode path terminates on a `FINAL`-mode buffer. This is extracted from #57363 as a standalone correctness fix. The `Complete`-mode path is reachable today by setting `spark.sql.execution.combineAdjacentAggregation` to `true` (introduced in SPARK-43317), independent of the `spark.sql.execution.replaceHashWithSortAgg` default flip proposed there, so it can be reviewed, merged, and backported on its own. ### Why are the changes needed? Without the fix, a valid Hive UDAF that uses mode-specific buffer classes fails at runtime with a `ClassCastException` whenever the optimizer combines its partial/final aggregates into `Complete` mode. ### Does this PR introduce _any_ user-facing change? Yes. A Hive UDAF that previously threw a `ClassCastException` under `spark.sql.execution.combineAdjacentAggregation=true` now evaluates correctly. ### How was this patch tested? New test `SPARK-58294: Hive UDAF with two aggregation buffers in Complete mode` in `HiveUDAFSuite`, using the existing `MockUDAF2` (distinct buffer classes per mode). It enables `combineAdjacentAggregation`, asserts the plan is a single `Complete`-mode `ObjectHashAggregateExec`, and checks the result under both the sort-based fallback path (`OBJECT_AGG_SORT_BASED_FALLBACK_THRESHOLD = 1`, asserting `numTasksFallBacked > 0`) and the non-fallback path (`= 100`, asserting `== 0`). Mutation-tested: reverting the `eval` fix (passing `buffer.buf` instead of `toFinalBuffer(buffer).buf`) fails this test with the exact `ClassCastException: MockUDAFBuffer cannot be cast to MockUDAFBuffer2`; it passes with the fix. The existing `SPARK-24935` test (partial/final path) is left unchanged and still passes. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (Claude Opus 4.8) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
