[
https://issues.apache.org/jira/browse/SPARK-58988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Toth updated SPARK-58988:
-------------------------------
Fix Version/s: 4.3.0
(was: 4.4.0)
> Fix PartitioningCollection invariant violation exception when
> allowKeysSubsetOfPartitionKeys
> --------------------------------------------------------------------------------------------
>
> Key: SPARK-58988
> URL: https://issues.apache.org/jira/browse/SPARK-58988
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 5.0.0
> Reporter: XiDuo You
> Assignee: XiDuo You
> Priority: Major
> Labels: pull-request-available
> Fix For: 4.3.0
>
>
> This is a follow-up to SPARK-56877, which added a check in
> `PartitioningCollection.fromPartitionings` requiring all `KeyedPartitioning`s
> to share equal `partitionKeys`.
> When a storage-partitioned join has join keys that are a subset of the
> partition keys (e.g. a v2 table partitioned by `[dt, bucket(16, c1)]` joined
> on `c1` only), and `spark.sql.sources.v2.bucketing.shuffle.enabled` is on so
> only the non-keyed side is re-shuffled, `KeyedPartitioning.createShuffleSpec`
> projects the keyed side to the join keys but keeps them in first-occurrence
> order (`distinct` without sorting). The keyed side, however, goes through
> `GroupPartitionsExec`, which sorts the projected keys. The two sides then
> carry the same keys in a different order, and
> `PartitioningCollection.fromPartitionings` throws:
> ```
> java.lang.IllegalArgumentException: requirement failed: All
> KeyedPartitionings in a PartitioningCollection must have equal partitionKeys
> ```
> The fix sorts the distinct projected keys in `createShuffleSpec` (using
> `toGrouped`, which sorts by the same natural ascending ordering as
> `GroupPartitionsExec`), aligning the two sides.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
