ulysses-you opened a new pull request, #58335: URL: https://github.com/apache/spark/pull/58335
### What changes were proposed in this pull request? `KeyedShuffleSpec.reducers` now returns the reduced expression (a `TransformExpression`) alongside the reducer, and `GroupPartitionsExec.outputPartitioning` reports it instead of the original partition expressions when reducers are applied. ### Why are the changes needed? In a storage-partitioned join with compatible transforms whose result types differ (e.g. `identity(id)` on one side and `bucket(N, id)` on the other), the reducer maps the partition keys to the other side's value type. `GroupPartitionsExec.outputPartitioning` used to report the original expressions with the reduced keys, so the two had different data types and computing the key ordering threw: ``` java.lang.ClassCastException: class java.lang.Integer cannot be cast to class java.lang.Long ``` ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Added regression tests in `KeyGroupedPartitioningSuite` covering the identity-vs-bucket reducer, the multi-table reduce (three tables with different transforms), and the subset-join-key case. Each also asserts the storage-partitioned join introduces no shuffle. The tests fail with the `ClassCastException` before this patch and pass after. Ran `KeyGroupedPartitioningSuite`, `EnsureRequirementsSuite`, `GroupPartitionsExecSuite`, `ProjectedOrderingAndPartitioningSuite`, and `ShuffleSpecSuite`. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
