dongjoon-hyun opened a new pull request, #58315:
URL: https://github.com/apache/spark/pull/58315
### What changes were proposed in this pull request?
Fix the post-shuffle regrouping in `EnsureRequirements` for
storage-partitioned joins with
`spark.sql.sources.v2.bucketing.allowJoinKeysSubsetOfPartitionKeys=true`. The
unwrap that pushes join key positions down to the keyed side only matches a
bare `KeyedShuffleSpec`:
```scala
bestSpecOpt match {
case Some(KeyedShuffleSpec(_, _, Some(joinKeyPositions))) =>
withJoinKeyPositions(child, joinKeyPositions)
case _ => child
}
```
When the keyed child's `outputPartitioning` is a `PartitioningCollection`
(e.g. an inner SPJ join result, or a projection duplicating a partition column
under multiple aliases like `SELECT a AS a1, a AS a2, b`), its spec is a
`ShuffleSpecCollection` wrapping the `KeyedShuffleSpec`, so the match falls
through and no `GroupPartitionsExec` is inserted. This PR unwraps a (possibly
nested) `ShuffleSpecCollection` to its head spec — the same spec
`ShuffleSpecCollection.createPartitioning` delegates to — before the match.
### Why are the changes needed?
Without the fix, the keyed side keeps its N ungrouped partitions while the
other side is shuffled into the projected layout with M partitions, and
planning fails with:
```
java.lang.IllegalArgumentException: requirement failed:
All KeyedPartitionings in a PartitioningCollection must have equal
partitionKeys
```
### Does this PR introduce _any_ user-facing change?
No behavior change. The affected queries used to fail with the error above;
they now run correctly, shuffling only the non-keyed side.
### How was this patch tested?
Pass the CIs.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Fable 5
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
