ulysses-you opened a new pull request, #58044: URL: https://github.com/apache/spark/pull/58044
### What changes were proposed in this pull request? Change `UnionExec.prepareOutputPartitioning` to map every child's partitioning (including the first child's) to this union's output attributes, instead of only remapping the non-first children to the first child's attributes. The now-redundant `toUnionOutput` step is removed, and `comparePartitioning` keeps using structural equality (`l == r`) for hash partitionings. ### Why are the changes needed? Previously `prepareOutputPartitioning` left the first child's `outputPartitioning` untouched while remapping the other children to the first child's output attributes. A child's `outputPartitioning` is not necessarily expressed in its own output attributes: for example, `FilterExec.outputPartitioning` passes through its child's partitioning (`child.outputPartitioning`) while `FilterExec.output` adjusts the nullability via `outputWithNullability` (e.g. `IsNotNull`). As a result, the intersection compared two hash partitionings over attributes with different nullability, the union reported `UnknownPartitioning`, and an extra shuffle was added on top of the union. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Added `SPARK-58819: union outputPartitioning ignores partition key nullability` in `DataFrameSetOperationsSuite`, which verifies that a union of two `HashPartitioning(id)` children (one with a `Filter` that adjusts `id`'s nullability) still propagates `HashPartitioning(id)` and that a downstream `groupBy(id).count()` reuses the partitioning instead of adding a shuffle. Also ran `DataFrameSetOperationsSuite`, `UnionCodegenSuite`, `CoalesceShufflePartitionsSuite`, `PlannerSuite`, `AdaptiveQueryExecSuite`, and the TPC-DS plan stability suites (`TPCDSV1_4_PlanStabilitySuite`, `TPCDSV2_7_PlanStabilitySuite`). ### Was this patch authored or co-authored using generative AI tooling? Yes. Generated-by: Claude Code -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
