peter-toth opened a new pull request, #59832:
URL: https://github.com/apache/spark/pull/59832
### Differences from #58273
This is the `branch-4.2` backport of #58273. The description below is the
original one.
- **The problem on 4.2 is wrong rows, not a planning failure.** 4.2 has no
`PartitioningCollection.fromPartitionings` check (SPARK-56877), so nothing
fails. `ShuffleExchangeExec` routes the re-shuffled side by the sorted keys,
while `createShuffleSpec` declares them in first-occurrence order.
- With AQE off, the keyed side's `GroupPartitionsExec` sorts its keys too,
so the two sides line up.
- With AQE on, the join is planned again over the materialized shuffle.
The `GroupPartitionsExec` planned above that shuffle reads it in the declared
order, so the two sides pair different keys. An inner join returns no rows, and
a LEFT JOIN returns nulls for the shuffled side.
- Measured on `v4.2.0` and on `branch-4.2`. `branch-4.0`, `branch-4.1` and
`master` return the right rows.
- **Code.** 4.2's `toGrouped` builds a new `KeyedPartitioning` and drops
`mayContainUnknownPartitionKeys`, so the backport takes only its sorted keys
and keeps the marker. On 4.3 and `master`, `toGrouped` keeps it through `copy`.
The code comment describes the 4.2 symptom instead of the
`PartitioningCollection` check.
- **Tests.**
- The original test runs on 4.2 and is kept. Its subset conf is
`V2_BUCKETING_ALLOW_JOIN_KEYS_SUBSET_OF_PARTITION_KEYS` here, and its comment
no longer mentions the collection check. It passes on 4.2 without the fix too,
since that failure needs the check.
- A new test, `SPARK-58988: one-side shuffle onto subset join keys whose
projected keys are unsorted`, covers the 4.2 symptom with an inner and a LEFT
JOIN, AQE on and off. It fails on `branch-4.2` without the fix and passes with
it.
- **Conflicts.**
- The projected branch of `createShuffleSpec` conflicted. On 4.3 and
`master`, this change landed before SPARK-59050, which added
`mayContainUnknownPartitionKeys`. On 4.2, the SPARK-59050 backport is already
in, so the lines this change replaces already pass the marker. The resolution
sorts the keys and keeps the marker, which is what 4.3 does today with both
changes in.
- The test conflicted only in its position, since its neighbour on 4.3
(SPARK-46367) is not on 4.2. Both tests are at the end of the suite.
- **What I ran on `branch-4.2`.**
- `ShuffleSpecSuite` and `DistributionSuite`: 30 tests.
- `KeyGroupedPartitioningSuite`, `EnsureRequirementsSuite`,
`GroupPartitionsExecSuite`, `ProjectedOrderingAndPartitioningSuite`,
`WriteDistributionAndOrderingSuite`, `PlannerSuite` and `SparkPlannerSuite`:
434 tests.
- `AdaptiveQueryExecSuite`, `DataSourceV2Suite`, `FileDataSourceV2Suite`,
`NestedDataSourceV2Suite` and `StreamingDataSourceV2Suite`: 211 tests.
- `dev/lint-scala`.
- This backport was prepared with Claude Code (Claude Opus 5.5).
### What changes were proposed in this pull request?
`KeyedPartitioning.createShuffleSpec` now sorts the distinct projected
partition keys (via
`toGrouped`) so they follow the same natural ascending ordering as
`GroupPartitionsExec`.
### Why are the changes needed?
SPARK-56877 added a check in `PartitioningCollection.fromPartitionings`
requiring all
`KeyedPartitioning`s to share equal `partitionKeys`. In a
storage-partitioned join whose join keys
are a subset of the partition keys (e.g. a v2 table partitioned by `[dt,
bucket(16, c1)]` joined
on `c1`), with `spark.sql.sources.v2.bucketing.shuffle.enabled` enabled so
only the non-keyed side
is re-shuffled, the keyed side's projected keys are sorted by
`GroupPartitionsExec` while
`createShuffleSpec` kept them in first-occurrence order. The two sides then
carry the same keys in
different orders and the query fails with:
```
java.lang.IllegalArgumentException: requirement failed: All
KeyedPartitionings in a
PartitioningCollection must have equal partitionKeys
```
### Does this PR introduce _any_ user-facing change?
No by default.
### How was this patch tested?
Added a regression test in `KeyGroupedPartitioningSuite`
(`SPARK-58988: v2 bucketed table with subset join keys joining v1 table`)
that reproduces the
failure and passes with the fix. Also ran `KeyGroupedPartitioningSuite`,
`EnsureRequirementsSuite`,
`GroupPartitionsExecSuite`, and `ProjectedOrderingAndPartitioningSuite`.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
