pan3793 commented on code in PR #57762:
URL: https://github.com/apache/spark/pull/57762#discussion_r3713600503
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/exchange/EnsureRequirements.scala:
##########
@@ -782,20 +782,7 @@ case class EnsureRequirements(
partitioning: Partitioning,
distribution: ClusteredDistribution): Option[KeyedShuffleSpec] = {
def tryCreate(partitioning: KeyedPartitioning): Option[KeyedShuffleSpec] =
{
- // The single-column invariant in KeyedPartitioning.supportsExpressions
guarantees one
- // attribute per partition expression.
- val attributes = partitioning.expressions.flatMap(_.references)
- val clustering = distribution.clustering
-
- val satisfies = if
(SQLConf.get.getConf(SQLConf.REQUIRE_ALL_CLUSTER_KEYS_FOR_CO_PARTITION)) {
- attributes.length == clustering.length &&
attributes.zip(clustering).forall {
- case (l, r) => l.semanticEquals(r)
- }
- } else {
- partitioning.satisfies(distribution)
- }
-
- if (satisfies) {
+ if (partitioning.satisfies(distribution)) {
Review Comment:
Adopted the suggested coverage check in bdafe4d46b4. The
partition-keys-cover-part-of-join-keys case stays gated by the config;
duplicated join keys and `allowKeysSubsetOfPartitionKeys=true` no longer need
`requireAllClusterKeysForCoPartition=false`.
##########
docs/sql-migration-guide.md:
##########
@@ -22,6 +22,10 @@ license: |
* Table of contents
{:toc}
+## Upgrading from Spark SQL 4.3 to 4.4
+
+- Since Spark 4.4, `spark.sql.requireAllClusterKeysForCoPartition` no longer
affects storage-partitioned joins (V2 data sources). A shuffle is now avoided
whenever all partition keys appear in the join keys, regardless of order;
joining on a subset of partition keys remains controlled by
`spark.sql.sources.v2.bucketing.allowKeysSubsetOfPartitionKeys.enabled`. Users
who previously set `spark.sql.requireAllClusterKeysForCoPartition` to `false`
solely to enable storage-partitioned joins no longer need to do so. The config
still applies to hash-partitioned children (e.g., V1 bucketing).
Review Comment:
Rewrote the entry in bdafe4d46b4: it states the coverage semantics and that
the partition-keys-cover-part-of-join-keys case still requires setting the
config to `false`.
##########
docs/sql-performance-tuning.md:
##########
@@ -551,7 +543,7 @@ The following SQL properties enable Storage Partition Join
in different join que
<td><code>spark.sql.sources.v2.bucketing.allowJoinKeysSubsetOfPartitionKeys.enabled</code></td>
Review Comment:
Fixed in bdafe4d46b4, switched to the current config name.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
