ulysses-you opened a new pull request, #57077: URL: https://github.com/apache/spark/pull/57077
### What changes were proposed in this pull request? `CoalesceShufflePartitions.collectCoalesceGroups` treats every `UnionExec` as a node whose children do not need compatible partitioning, so each union child becomes an independent coalesce group and may be coalesced to a different number of partitions. Since SPARK-57881, `UnionExec.outputPartitioning` can report a real partitioning (`HashPartitioning` / `KeyedPartitioning` / `SinglePartition`) when its children are compatibly partitioned. When it does, downstream operators rely on that partitioning, so the children must stay co-partitioned. This PR makes `childrenNeedCompatiblePartitioning` return `true` for a `UnionExec` that is not a plain union (i.e. reports a real partitioning), so its shuffle stages are coalesced together as a single group. `UnionExec.isPlainUnion` is promoted to `private[spark]` and reused for this check. ### Why are the changes needed? When a partitioning-aware union's children are coalesced independently, they end up with different partition counts and are no longer co-partitioned. This does not produce a wrong result: `AdaptiveSparkPlanExec` runs `ValidateRequirements` after each `AQEShuffleReadRule` and reverts the entire `CoalesceShufflePartitions` application when the co-partitioning check fails. The net effect is that shuffle partition coalescing is silently lost whenever a union reports a real partitioning. ### Does this PR introduce _any_ user-facing change? No. Results are unchanged; the executed plan may now coalesce shuffle partitions that were previously left un-coalesced. ### How was this patch tested? New test in `AdaptiveQueryExecSuite`. Without the fix the coalescing is reverted (no `AQEShuffleReadExec`); with the fix both union children are coalesced to the same number of partitions. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Opus 4.8 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
