This is an automated email from the ASF dual-hosted git repository.

ulysses-you pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/branch-4.x by this push:
     new 677df8bbe295 [SPARK-60618][SQL] Skip the pass-through 
GroupPartitionsExec in a storage-partitioned join's alignment
677df8bbe295 is described below

commit 677df8bbe295ec181cca24599759e4e864e19a6d
Author: Xiduo You <[email protected]>
AuthorDate: Tue Nov 3 11:36:16 2026 +0800

    [SPARK-60618][SQL] Skip the pass-through GroupPartitionsExec in a 
storage-partitioned join's alignment
    
    ### What changes were proposed in this pull request?
    
    `EnsureRequirements.checkKeyGroupCompatible` skips an identity alignment 
`GroupPartitionsExec` when its output partitioning equals its child's, every 
leaf of the child is a shuffle query stage, and no replicate side was chosen.
    
    Other children retain the node and its check against the partitioning it 
was planned over. This protects the alignment if a scan is rebuilt with a 
different layout. When a replicate side is chosen, both alignment nodes are 
also retained so an `EnsureRequirements` re-run can reproduce the choice, 
including when the join type declines it.
    
    ### Why are the changes needed?
    
    A pass-through alignment node can sit above a keyed shuffle stage:
    
    ```text
    Sort
    +- GroupPartitions
       +- ShuffleQueryStage
          +- Exchange keyedpartitioning(...)
    ```
    
    The node retains grouping and partition-key information in the driver plan 
and can make the plan harder to interpret. It also prevents 
`ConvertSortMergeJoinToShuffledHashJoin` from reaching the shuffle stage. 
Removing it allows an eligible join to convert to a shuffled hash join. A 
production `MERGE INTO` had this shape, and the target side's sort spilled 10.1 
TiB.
    
    The node is constructed before the skip is decided, so the change reduces 
retention in the resulting plan, not the cost of constructing the node.
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes. `EXPLAIN` may no longer show an identity `GroupPartitions` node over a 
shuffle-stage subtree. Alignment nodes also exist in released versions such as 
Spark 4.2.0. On master, removing the node can allow an eligible sort merge join 
to convert to a shuffled hash join. No configuration or API changes are 
introduced.
    
    ### How was this patch tested?
    
    - Added an AQE storage-partitioned join test for one-side keyed shuffle and 
SHJ conversion, checking results and the retained source-side grouping.
    - Added plan tests for direct, wrapped, and multiple shuffle-stage leaves, 
mixed leaves, non-identity alignment, a chosen replicate side, and push logging.
    - Added a controlled scan-rebuild test for the retained child-partitioning 
guard and its no-guard control.
    - Added inner and left outer join coverage for replicate-side selection 
across an `EnsureRequirements` re-run.
    - Kept the existing scan-side alignment test shapes and scoped the 
SPARK-59050 identity grouping check to the second join's input.
    
    Ran:
    
    ```text
    build/sbt 'sql/scalastyle' 'sql/Test/scalastyle'
    build/sbt 'sql/testOnly 
org.apache.spark.sql.connector.KeyGroupedPartitioningSuite'             # 242
    build/sbt 'sql/testOnly 
org.apache.spark.sql.execution.exchange.EnsureRequirementsSuite'        # 83
    build/sbt 'sql/testOnly 
org.apache.spark.sql.execution.datasources.v2.GroupPartitionsExecSuite' # 38
    build/sbt 'sql/testOnly 
org.apache.spark.sql.execution.adaptive.AdaptiveQueryExecSuite'         # 146
    ```
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Code (deepseek-flash[1m], gpt-6.1-terra[1m])
    
    🤖 Generated with [Claude Code](https://claude.com/claude-code)
    
    Closes #59863 from ulysses-you/shj-through-grouppartitions.
    
    Authored-by: Xiduo You <[email protected]>
    Signed-off-by: Xiduo You <[email protected]>
    (cherry picked from commit 1a524e44b5a481085fc4e8902e2c89e4b3dfd862)
    Signed-off-by: Xiduo You <[email protected]>
---
 .../datasources/v2/GroupPartitionsExec.scala       |   8 +-
 .../execution/exchange/EnsureRequirements.scala    |  52 +++--
 .../connector/KeyGroupedPartitioningSuite.scala    | 236 ++++++++++++++++++++-
 .../exchange/EnsureRequirementsSuite.scala         | 123 +++++++++++
 4 files changed, 393 insertions(+), 26 deletions(-)

diff --git 
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/GroupPartitionsExec.scala
 
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/GroupPartitionsExec.scala
index 809896399a7e..5a59cfda9b3a 100644
--- 
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/GroupPartitionsExec.scala
+++ 
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/GroupPartitionsExec.scala
@@ -114,9 +114,11 @@ case class GroupPartitionsExec(
    * `plannedPartitioning` rebuilds its keyed members on top of the grouping. 
A rule that hands this
    * node a child reporting something else invalidates both, so the keyed 
claim is dropped and only
    * the physical output count is reported. `ValidateRequirements` then finds 
the parent's
-   * distribution unsatisfied and `AdaptiveSparkPlanExec` reverts the rewrite, 
which is what an
-   * `AQEShuffleReadExec` landing over a keyed shuffle stage relies on. Only 
an `AQEShuffleReadRule`
-   * is validated that way, so `checkChildStillMatches` covers the rest at 
execution.
+   * distribution unsatisfied and `AdaptiveSparkPlanExec` reverts the rewrite. 
A coalesced
+   * `AQEShuffleReadExec` over a keyed stage also reports 
`UnknownPartitioning` itself, so
+   * validation can reject it even without this node. Only an 
`AQEShuffleReadRule` is validated
+   * that way, so a retained node also checks its child at execution with 
`checkChildStillMatches`.
+   * A side whose identity node was skipped has no such execution-time check.
    *
    * Re-deriving instead would apply a recipe chosen for one pairing to a 
different child, yielding
    * a claim nothing validated against the other side. It would also buy 
nothing today: no rule is
diff --git 
a/sql/core/src/main/scala/org/apache/spark/sql/execution/exchange/EnsureRequirements.scala
 
b/sql/core/src/main/scala/org/apache/spark/sql/execution/exchange/EnsureRequirements.scala
index 45c4036bb8a0..b783cc62cae5 100644
--- 
a/sql/core/src/main/scala/org/apache/spark/sql/execution/exchange/EnsureRequirements.scala
+++ 
b/sql/core/src/main/scala/org/apache/spark/sql/execution/exchange/EnsureRequirements.scala
@@ -29,6 +29,7 @@ import org.apache.spark.sql.catalyst.rules.Rule
 import org.apache.spark.sql.catalyst.util.InternalRowComparableWrapper
 import org.apache.spark.sql.errors.QueryExecutionErrors
 import org.apache.spark.sql.execution._
+import org.apache.spark.sql.execution.adaptive.ShuffleQueryStageExec
 import org.apache.spark.sql.execution.datasources.v2.GroupPartitionsExec
 import org.apache.spark.sql.execution.joins.{ShuffledHashJoinExec, 
ShuffledJoin, SortMergeJoinExec}
 import org.apache.spark.sql.internal.SQLConf
@@ -807,6 +808,7 @@ case class EnsureRequirements(
       var replicateLeftSide = false
       var replicateRightSide = false
       var applyPartialClustering = false
+      var replicateSideChosen = false
 
       // This means we allow partitions that are not clustered on their values,
       // that is, multiple partitions with the same partition value. In the
@@ -829,6 +831,7 @@ case class EnsureRequirements(
           logInfo(log"Skipping partially clustered distribution as it cannot 
be applied for " +
             log"join type '${MDC(LogKeys.JOIN_TYPE, joinType)}'")
         } else {
+          replicateSideChosen = true
           // The statistics and the original partition keys come from the 
pre-alignment plans,
           // which is what both sides already are here.
           val leftLink = rawLeft.logicalLink
@@ -913,7 +916,8 @@ case class EnsureRequirements(
         }
       }
 
-      // Now we need to push-down the common partition information to the 
`GroupPartitionsExec`s.
+      // Push the common partition information to the sides that need it. An 
identity alignment
+      // whose leaves are shuffle stages can keep its child instead (see 
`regroupOrIdentity` below).
       //
       // Where `applyPartialClustering` holds, exactly one side repeats: 
`replicateRightSide` is
       // the negation of `replicateLeftSide`, and the branch above is taken 
only when the side it
@@ -924,15 +928,39 @@ case class EnsureRequirements(
       // them. Each side stamps why it is left ungrouped (`UngroupingOrigin`), 
so a reader of the
       // finished plan judges the pair on the layouts it holds; the stamp also 
decides the node's
       // routing (`GroupPartitionsExec.distributePartitions`).
-      (
-        GroupPartitionsExec(rawLeft, leftSpec.joinKeyPositions,
-          Some(mergedPartitionKeys), leftReducers,
-          ungroupingOrigin = Option.when(applyPartialClustering)(
-            if (replicateLeftSide) REPLICATED_FOR_JOIN else SPLIT_FOR_JOIN)),
-        GroupPartitionsExec(rawRight, rightSpec.joinKeyPositions,
-          Some(mergedPartitionKeys), rightReducers,
+
+      def canSkipIdentityGroup(gpe: GroupPartitionsExec): Boolean = {
+        val child = gpe.child
+        // Keep both nodes when a replicate side is chosen, including when the 
join type declines
+        // it, so a re-run can peel the local sorts and reproduce the choice. 
This also guards a
+        // plan with existing keyed shuffle stages: the planner does not 
create keyed shuffles when
+        // partially clustered distribution is enabled.
+        // Shuffle stages report their exchanges' fixed partitioning. Keep the 
node when any leaf
+        // is not a stage: if a scan is rebuilt with different partitioning 
after alignment, the
+        // node checks its child against the planned partitioning and fails 
instead of letting the
+        // join silently pair mismatched partitions and produce incorrect 
results.
+        !replicateSideChosen &&
+          child.collectLeaves().forall(_.isInstanceOf[ShuffleQueryStageExec]) 
&&
+          gpe.outputPartitioning == child.outputPartitioning
+      }
+
+      def regroupOrIdentity(
+          child: SparkPlan,
+          spec: KeyedShuffleSpec,
+          reducers: Option[Seq[Option[KeyReducer]]],
+          replicates: Boolean): SparkPlan = {
+        val gpe = GroupPartitionsExec(child, spec.joinKeyPositions,
+          Some(mergedPartitionKeys), reducers,
           ungroupingOrigin = Option.when(applyPartialClustering)(
-            if (replicateRightSide) REPLICATED_FOR_JOIN else SPLIT_FOR_JOIN)))
+            if (replicates) REPLICATED_FOR_JOIN else SPLIT_FOR_JOIN))
+        if (canSkipIdentityGroup(gpe)) {
+          child
+        } else {
+          gpe
+        }
+      }
+      (regroupOrIdentity(rawLeft, leftSpec, leftReducers, replicateLeftSide),
+        regroupOrIdentity(rawRight, rightSpec, rightReducers, 
replicateRightSide))
     }
 
     // The pairing is only worth committing to if both children still declare 
the same aligned key
@@ -965,11 +993,11 @@ case class EnsureRequirements(
     // Announced here rather than where the branch runs, since the gate can 
still discard what it
     // built, and a log that names a pushdown should name one that happens.
     if (pushCommonValues) {
-      if (committed) {
-        logInfo("Pushing common partition values for storage-partitioned join")
-      } else {
+      if (!committed) {
         logInfo("Not storage-partitioning this join after all: the two sides 
no longer declare " +
           "the same partition keys once regrouped onto the merged partition 
values")
+      } else if ((newLeft ne rawLeft) || (newRight ne rawRight)) {
+        logInfo("Pushing common partition values for storage-partitioned join")
       }
     }
     Option.when(committed)(Seq(newLeft, newRight))
diff --git 
a/sql/core/src/test/scala/org/apache/spark/sql/connector/KeyGroupedPartitioningSuite.scala
 
b/sql/core/src/test/scala/org/apache/spark/sql/connector/KeyGroupedPartitioningSuite.scala
index a65048a03357..d5caa8d13f43 100644
--- 
a/sql/core/src/test/scala/org/apache/spark/sql/connector/KeyGroupedPartitioningSuite.scala
+++ 
b/sql/core/src/test/scala/org/apache/spark/sql/connector/KeyGroupedPartitioningSuite.scala
@@ -30,7 +30,7 @@ import 
org.apache.spark.sql.catalyst.expressions.aggregate.Complete
 import org.apache.spark.sql.catalyst.plans.{Cross, ExistenceJoin, Inner, 
JoinType, LeftAnti, LeftSemi, LeftSingle, RightOuter}
 import org.apache.spark.sql.catalyst.plans.physical
 import org.apache.spark.sql.catalyst.plans.physical.{HASH_ROUTING, 
KeyedPartitioning, PartitioningCollection, REPLICATED_FOR_JOIN, ROUTING_LOST, 
SPLIT_FOR_JOIN}
-import org.apache.spark.sql.connector.catalog.{Column, Identifier, 
InMemoryBaseTable, InMemoryCatalystRuntimeFilterCatalog, InMemoryTable, 
InMemoryTableCatalog}
+import org.apache.spark.sql.connector.catalog.{BufferedRows, Column, 
Identifier, InMemoryBaseTable, InMemoryCatalystRuntimeFilterCatalog, 
InMemoryTable, InMemoryTableCatalog}
 import org.apache.spark.sql.connector.catalog.constraints.Constraint
 import org.apache.spark.sql.connector.catalog.functions._
 import org.apache.spark.sql.connector.distributions.{Distribution, 
Distributions}
@@ -50,7 +50,7 @@ import org.apache.spark.sql.execution.{
   SparkPlan,
   UnionExec,
   WholeStageCodegenExec}
-import org.apache.spark.sql.execution.adaptive.{AQEShuffleReadExec, 
ResultQueryStageExec}
+import org.apache.spark.sql.execution.adaptive.{AQEShuffleReadExec, 
ResultQueryStageExec, ShuffleQueryStageExec}
 import org.apache.spark.sql.execution.aggregate.{BaseAggregateExec, 
HashAggregateExec, SortAggregateExec}
 import org.apache.spark.sql.execution.datasources.v2.{BatchScanExec, 
DataSourceV2ScanRelation, GroupPartitionsExec}
 import org.apache.spark.sql.execution.exchange.{EnsureRequirements, 
ReusedExchangeExec, ShuffleExchangeExec, ShuffleExchangeLike, 
ValidateRequirements}
@@ -2091,6 +2091,88 @@ class KeyGroupedPartitioningSuite
     }
   }
 
+  test("SPARK-60618: scan rebuilt with different partition keys after 
alignment") {
+    val cols = Array(Column.create("id", IntegerType))
+    createTable("rebuild_left", cols, Array(identity("id")))
+    createTable("rebuild_right", cols, Array(identity("id")))
+    sql("INSERT INTO testcat.ns.rebuild_left VALUES (1), (2)")
+    sql("INSERT INTO testcat.ns.rebuild_right VALUES (1), (2), (3)")
+
+    withSQLConf(
+        SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "false",
+        SQLConf.WHOLESTAGE_CODEGEN_ENABLED.key -> "false",
+        SQLConf.V2_BUCKETING_PUSH_PART_VALUES_ENABLED.key -> "true",
+        SQLConf.V2_BUCKETING_PARTITION_FILTER_ENABLED.key -> "true") {
+      // The original left layout [1, 2] is already the inner join's 
intersection. The right
+      // grouping prunes key 3, so it keeps the original [1, 2] pairing after 
the left scan rebuild.
+      def plannedJoin(): SortMergeJoinExec = {
+        val df = sql("SELECT /*+ MERGE(l, r) */ l.id, r.id " +
+          "FROM testcat.ns.rebuild_left l JOIN testcat.ns.rebuild_right r ON 
l.id = r.id")
+        val joins = collect(df.queryExecution.executedPlan) { case j: 
SortMergeJoinExec => j }
+        assert(joins.size == 1)
+        val join = joins.head
+        assert(collectAllShuffles(join).isEmpty)
+        assert(collectAllGroupPartitions(join.left).size == 1)
+        assert(collectAllGroupPartitions(join.right).size == 1)
+        join
+      }
+
+      def run(
+          replacementKeys: Seq[Int],
+          keepIdentity: Boolean): Seq[Row] = {
+        val join = plannedJoin()
+        val leftScan = collect(join.left) { case s: BatchScanExec => s }.head
+        val inMemoryScan = 
leftScan.scan.asInstanceOf[InMemoryBaseTable#BatchScanBaseClass]
+        val oldData = inMemoryScan.data
+        val oldRows = oldData.map(_.asInstanceOf[BufferedRows])
+        val expected = collectAllGroupPartitions(join.left).head
+        assert(expected.grouping.isIdentity)
+        assert(expected.outputPartitioning == 
expected.child.outputPartitioning)
+        val newData = replacementKeys.map { key =>
+          oldRows.find(_.key == Seq(key)).getOrElse {
+            new BufferedRows(Seq(key), 
oldRows.head.schema).withRow(InternalRow(key))
+          }
+        }
+        try {
+          // A rewritten runtime filter makes the scan copy distinct for tree 
replacement, while
+          // the true literal leaves the connector's replacement partitions 
unchanged.
+          inMemoryScan.data = newData
+          val rebuilt = leftScan.copy(runtimeFilters = 
Seq(Literal.TrueLiteral))
+          assert(rebuilt.outputPartitioning.numPartitions == 
replacementKeys.size)
+          assert(rebuilt.outputPartitioning != leftScan.outputPartitioning ||
+            replacementKeys == Seq(1, 2))
+          val replaced = join.transformUp {
+            case scan: BatchScanExec if scan eq leftScan => rebuilt
+            case g: GroupPartitionsExec if !keepIdentity && g.child.exists(_ 
eq rebuilt) => g.child
+          }
+          val rebuiltScans = collect(replaced) {
+            case scan: BatchScanExec if scan eq rebuilt => scan
+          }
+          assert(rebuiltScans.size == 1)
+          assert(ValidateRequirements.validate(replaced) == (replacementKeys 
== Seq(1, 2)))
+          replaced.executeCollect().map(row => Row(row.getInt(0), 
row.getInt(1))).toSeq
+        } finally {
+          inMemoryScan.data = oldData
+        }
+      }
+
+      // The replacement scan is a controlled post-planning rewrite; it 
deliberately changes the
+      // connector's split set without re-running EnsureRequirements.
+      assert(run(Seq(1, 2), keepIdentity = false).toSet == Set(Row(1, 1), 
Row(2, 2)))
+      assert(run(Seq(1, 2), keepIdentity = true).toSet == Set(Row(1, 1), 
Row(2, 2)))
+      // Without the guard, key 2 is silently paired with key 3 instead of its 
match.
+      assert(run(Seq(2, 3), keepIdentity = false).isEmpty)
+      val error = intercept[SparkException] {
+        run(Seq(2, 3), keepIdentity = true)
+      }
+      assert(error.getMessage.contains("no longer reports the partitioning it 
was planned over"))
+      val countError = intercept[Exception] {
+        run(Seq(1, 2, 3), keepIdentity = false)
+      }
+      assert(countError.getMessage.contains("Can't zip RDDs with unequal 
numbers of partitions"))
+    }
+  }
+
   test("SPARK-59252: a planned scan keeps the partitioning it was planned 
with") {
     createBucketedIdTable("l4", 4)
     createBucketedIdTable("r4", 4)
@@ -2948,6 +3030,61 @@ class KeyGroupedPartitioningSuite
     }
   }
 
+  test("SPARK-60618: Skip the pass-through GroupPartitionsExec in a 
storage-partitioned " +
+    "join's alignment") {
+    // `items` reports more partition keys than the join uses, so the aligned 
layout has to be
+    // projected onto the join keys. The shuffled side is laid out on that 
same projection, so
+    // its `GroupPartitionsExec` would only re-state the keys its shuffle 
already has and must
+    // not be built.
+    withSQLConf(
+        SQLConf.AUTO_BROADCASTJOIN_THRESHOLD.key -> "-1",
+        SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "true",
+        SQLConf.ADAPTIVE_MAX_SHUFFLE_HASH_JOIN_LOCAL_MAP_THRESHOLD.key -> 
"100m",
+        SQLConf.V2_BUCKETING_SHUFFLE_ENABLED.key -> "true",
+        SQLConf.V2_BUCKETING_PUSH_PART_VALUES_ENABLED.key -> "true",
+        SQLConf.V2_BUCKETING_ALLOW_KEYS_SUBSET_OF_PARTITION_KEYS.key -> 
"true") {
+      withTable(s"testcat.ns.$items", s"testcat.ns.$purchases") {
+        createTable(items, itemsColumns, Array(identity("id"), 
identity("name")))
+        sql(s"INSERT INTO testcat.ns.$items VALUES " +
+            s"(1, 'aa', 40.0, cast('2020-01-01' as timestamp)), " +
+            s"(1, 'aa', 30.0, cast('2020-01-02' as timestamp)), " +
+            s"(3, 'bb', 10.0, cast('2020-01-01' as timestamp)), " +
+            s"(4, 'cc', 15.5, cast('2020-02-01' as timestamp))")
+        createTable(purchases, purchasesColumns, Array.empty)
+        sql(s"INSERT INTO testcat.ns.$purchases VALUES " +
+            s"(1, 42.0, cast('2020-01-01' as timestamp)), " +
+            s"(1, 89.0, cast('2020-01-03' as timestamp)), " +
+            s"(3, 19.5, cast('2020-02-01' as timestamp)), " +
+            s"(5, 26.0, cast('2023-01-01' as timestamp)), " +
+            s"(6, 50.0, cast('2023-02-01' as timestamp))")
+
+        val df = sql(s"SELECT i.id, i.name, i.price AS purchase_price, p.price 
AS sale_price " +
+            s"FROM testcat.ns.$items i JOIN testcat.ns.$purchases p ON i.id = 
p.item_id")
+        checkAnswer(df, Seq(
+          Row(1, "aa", 30.0, 42.0), Row(1, "aa", 30.0, 89.0),
+          Row(1, "aa", 40.0, 42.0), Row(1, "aa", 40.0, 89.0),
+          Row(3, "bb", 10.0, 19.5)))
+
+        val plan = df.queryExecution.executedPlan
+        // The one-side shuffle still happens; only the regrouping node over 
it goes away.
+        assert(collectAllShuffles(plan).size == 1, s"expected one 
shuffle:\n$plan")
+        assert(collect(plan) {
+          case g: GroupPartitionsExec
+              if g.child.isInstanceOf[ShuffleQueryStageExec] ||
+                g.child.isInstanceOf[AQEShuffleReadExec] => g
+        }.isEmpty, s"no GroupPartitionsExec should sit over a shuffle 
stage:\n$plan")
+        // Without the node over the stage the join converts to a shuffled 
hash join.
+        val shj = collect(plan) { case j: ShuffledHashJoinExec => j }
+        assert(shj.size == 1 && collect(plan) { case j: SortMergeJoinExec => j 
}.isEmpty,
+          s"the join should convert to a shuffled hash join:\n$plan")
+        // The source side keeps its node: `items` holds two splits for key 
(1, 'aa'), and merging
+        // them is work the join needs.
+        assert(collect(shj.head.left) { case _: GroupPartitionsExec => true 
}.nonEmpty,
+          s"the source side should keep its grouping node:\n$plan")
+      }
+    }
+  }
+
   test("SPARK-41471: shuffle one side: only one side reports partitioning") {
     val items_partitions = Array(identity("id"))
     createTable(items, itemsColumns, items_partitions)
@@ -5416,6 +5553,78 @@ class KeyGroupedPartitioningSuite
     }
   }
 
+  test("SPARK-60618: partially clustered join keeps its replicate-side choice 
when the " +
+      "alignment is the identity") {
+    // The smaller side by statistics is replicated: `pc_small` holds five 
splits in 200 bytes,
+    // `pc_large` three splits in 216. Both are bucketed, whose transform 
derives no ordering, so
+    // the join's sort has to be added. The larger side ends up with one split 
per bucket, so the
+    // alignment on it is the identity. Its leaf is a scan, so the node stays, 
and a re-run peels
+    // the local sort through it and reads the statistics from the scans 
again. Without the node
+    // the counts (five against three) would pick the other side.
+    val smallColumns = Array(Column.create("id", LongType), 
Column.create("data", StringType))
+    createTable("pc_small", smallColumns, Array(bucket(4, "id")))
+    // 1, 5 and 9 share bucket 1; 2 and 3 have buckets of their own.
+    sql("INSERT INTO testcat.ns.pc_small VALUES " +
+        "(1, 'a1'), (5, 'a2'), (9, 'a3'), (2, 'a4'), (3, 'a5')")
+    // Wider than `pc_small` per row, so three rows already outweigh its five.
+    val largeColumns = smallColumns ++ Array(
+      Column.create("p1", LongType), Column.create("p2", LongType),
+      Column.create("p3", LongType), Column.create("p4", LongType))
+    createTable("pc_large", largeColumns, Array(bucket(4, "id")))
+    // One bucket per row, all of them also held by `pc_small`, so nothing is 
pruned and this
+    // side's alignment is the identity.
+    sql("INSERT INTO testcat.ns.pc_large VALUES " +
+        "(1, 'b1', 0, 0, 0, 0), (2, 'b2', 0, 0, 0, 0), (3, 'b3', 0, 0, 0, 0)")
+
+    // Unpartitioned, so this branch materializes shuffle stages and is 
converted to a shuffled
+    // hash join, which hands the whole plan back to `EnsureRequirements`.
+    createTable("pc_np1", smallColumns, Array.empty)
+    sql("INSERT INTO testcat.ns.pc_np1 VALUES (7, 'x')")
+    createTable("pc_np2", smallColumns, Array.empty)
+    sql("INSERT INTO testcat.ns.pc_np2 VALUES (7, 'y')")
+
+    withSQLConf(
+        SQLConf.V2_BUCKETING_PUSH_PART_VALUES_ENABLED.key -> "true",
+        SQLConf.V2_BUCKETING_PARTIALLY_CLUSTERED_DISTRIBUTION_ENABLED.key -> 
"true",
+        SQLConf.ADAPTIVE_MAX_SHUFFLE_HASH_JOIN_LOCAL_MAP_THRESHOLD.key -> 
"100m") {
+      def check(
+          joinType: String,
+          expectedRows: Seq[Row],
+          expectedFlags: (Seq[Boolean], Seq[Boolean])): Unit = {
+        val df = sql(
+          s"""
+             |SELECT /*+ MERGE(a, b) */ a.id AS k
+             |FROM testcat.ns.pc_small a $joinType testcat.ns.pc_large b ON 
a.id = b.id
+             |UNION ALL
+             |SELECT c.id AS k
+             |FROM testcat.ns.pc_np1 c JOIN testcat.ns.pc_np2 d ON c.id = d.id
+             |""".stripMargin)
+        checkAnswer(df, expectedRows)
+
+        assert(collectShuffles(df.queryExecution.executedPlan).isEmpty,
+          s"$joinType: the storage-partitioned join must stay shuffle-free")
+        val join = collect(df.queryExecution.executedPlan) {
+          case j: SortMergeJoinExec => j
+        }.head
+        // The choice is read off `distributePartitions`: the replicated side 
answers false, the
+        // spread side true.
+        def flags(side: SparkPlan): Seq[Boolean] =
+          collectAllGroupPartitions(side).map(_.distributePartitions)
+        assert(flags(join.left) === expectedFlags._1,
+          s"$joinType: left side distributePartitions")
+        assert(flags(join.right) === expectedFlags._2,
+          s"$joinType: right side distributePartitions")
+      }
+
+      check("JOIN", Seq(Row(1L), Row(2L), Row(3L), Row(7L)),
+        (Seq(false), Seq(true)))
+      // Here the join type declines the side the statistics pick, so the 
pairing is never
+      // applied. A re-run has to decline it again instead of picking the side 
it may replicate.
+      check("LEFT JOIN", Seq(Row(1L), Row(2L), Row(3L), Row(5L), Row(9L), 
Row(7L)),
+        (Seq(false), Seq(false)))
+    }
+  }
+
   test("SPARK-58996: partially clustered subset-key join keeps its join key 
positions when " +
       "EnsureRequirements re-runs") {
     // The left table is partitioned by (extra, id) and the join uses only 
`id`, the second
@@ -10759,16 +10968,19 @@ class KeyGroupedPartitioningSuite
           val plan = df.queryExecution.executedPlan
           assert(ValidateRequirements.validate(plan),
             s"the executed plan must satisfy every operator's required 
distribution:\n$plan")
-          // Locate the marked side's node: no reducers, and output partition 
i holds exactly input
-          // partition i. That is weaker than `identityGrouping`, which also 
asks for one output per
-          // input, so the assertions below are what pin the claim and the 
three keys.
-          val identityGpe = collectAllGroupPartitions(plan).find { g =>
-            g.reducers.isEmpty && g.groupedPartitions.zipWithIndex.forall {
-              case ((_, inputIndices), outputIndex) =>
-                inputIndices.lengthCompare(1) == 0 && inputIndices.head == 
outputIndex
-            }
+          val secondJoin = collect(plan) { case j: SortMergeJoinExec => j 
}.head
+          assert(collectAllShuffles(plan).size == 1, s"only the first join 
shuffles:\n$plan")
+          // Look only through the second join's wrappers, not into the first 
join.
+          def alignment(plan: SparkPlan): Option[GroupPartitionsExec] = plan 
match {
+            case g: GroupPartitionsExec => Some(g)
+            case s: SortExec => alignment(s.child)
+            case w: WholeStageCodegenExec => alignment(w.child)
+            case i: InputAdapter => alignment(i.child)
+            case p: ProjectExec => alignment(p.child)
+            case _ => None
           }
-          assert(identityGpe.isDefined,
+          val identityGpe = alignment(secondJoin.left)
+          assert(identityGpe.exists(g => g.reducers.isEmpty && 
g.grouping.isIdentity),
             s"expected a reducer-free identity GroupPartitionsExec, 
got:\n$plan")
           identityGpe.get.outputPartitioning match {
             case k: KeyedPartitioning =>
@@ -10779,6 +10991,8 @@ class KeyGroupedPartitioningSuite
             case other =>
               fail(s"expected a KeyedPartitioning output, got $other")
           }
+          assert(secondJoin.right.outputPartitioning.numPartitions == 3,
+            s"the other side is padded to the marked side's three 
keys:\n$plan")
         }
       }
     }
diff --git 
a/sql/core/src/test/scala/org/apache/spark/sql/execution/exchange/EnsureRequirementsSuite.scala
 
b/sql/core/src/test/scala/org/apache/spark/sql/execution/exchange/EnsureRequirementsSuite.scala
index f79ea2efe15c..0f040e05540e 100644
--- 
a/sql/core/src/test/scala/org/apache/spark/sql/execution/exchange/EnsureRequirementsSuite.scala
+++ 
b/sql/core/src/test/scala/org/apache/spark/sql/execution/exchange/EnsureRequirementsSuite.scala
@@ -17,6 +17,8 @@
 
 package org.apache.spark.sql.execution.exchange
 
+import org.apache.logging.log4j.Level
+
 import org.apache.spark.api.python.PythonEvalType
 import org.apache.spark.rdd.RDD
 import org.apache.spark.sql.catalyst.InternalRow
@@ -32,6 +34,7 @@ import 
org.apache.spark.sql.catalyst.util.InternalRowComparableWrapper
 import org.apache.spark.sql.connector.catalog.functions._
 import org.apache.spark.sql.execution.{BinaryExecNode, DummySparkPlan, 
LeafExecNode, SafeForKWayMerge, SortExec, UnaryExecNode}
 import org.apache.spark.sql.execution.SparkPlan
+import org.apache.spark.sql.execution.adaptive.ShuffleQueryStageExec
 import org.apache.spark.sql.execution.datasources.v2.{BatchScanExec, 
GroupPartitionsExec}
 import org.apache.spark.sql.execution.joins.{ShuffledHashJoinExec, 
SortMergeAsOfJoinExec, SortMergeJoinExec}
 import org.apache.spark.sql.execution.python.FlatMapCoGroupsInPandasExec
@@ -1145,6 +1148,126 @@ class EnsureRequirementsSuite extends 
SharedSparkSession {
     }
   }
 
+  test("SPARK-60618: skip identity alignment only when all leaves are shuffle 
stages") {
+    val keys = Seq(InternalRow(1), InternalRow(2))
+    val extraKeys = keys :+ InternalRow(3)
+
+    def stage(partitionKeys: Seq[InternalRow]): ShuffleQueryStageExec = {
+      val leaf = DummySparkPlan(outputPartitioning = 
UnknownPartitioning(partitionKeys.size))
+      val exchange = ShuffleExchangeExec(KeyedPartitioning(Seq(exprA), 
partitionKeys), leaf)
+      ShuffleQueryStageExec(0, exchange, exchange)
+    }
+
+    def planned(left: SparkPlan, rightKeys: Seq[InternalRow]): 
SortMergeJoinExec = {
+      val right = DummySparkPlan(outputPartitioning = 
KeyedPartitioning(Seq(exprB), rightKeys))
+      val join = SortMergeJoinExec(Seq(exprA), Seq(exprB), Inner, None, left, 
right)
+      EnsureRequirements.apply(join).asInstanceOf[SortMergeJoinExec]
+    }
+
+    withSQLConf(
+        SQLConf.V2_BUCKETING_ENABLED.key -> "true",
+        SQLConf.V2_BUCKETING_PUSH_PART_VALUES_ENABLED.key -> "true",
+        SQLConf.V2_BUCKETING_PARTITION_FILTER_ENABLED.key -> "true",
+        SQLConf.V2_BUCKETING_PARTIALLY_CLUSTERED_DISTRIBUTION_ENABLED.key -> 
"false") {
+      val identityStage = stage(keys)
+      val identity = planned(identityStage, extraKeys)
+      assert(groupPartitionsNodes(identity.left).isEmpty)
+      assert(identity.left.exists(_ eq identityStage))
+      assert(groupPartitionsNodes(identity.right).size == 1)
+      assert(ValidateRequirements.validate(identity))
+
+      val wrapped = DummyPassthroughUnaryExec(identityStage)
+      val wrappedPlan = planned(wrapped, extraKeys)
+      assert(groupPartitionsNodes(wrappedPlan.left).isEmpty,
+        "an identity alignment over a wrapped shuffle stage is skipped")
+      assert(wrappedPlan.left.exists(_ eq identityStage))
+      assert(ValidateRequirements.validate(wrappedPlan))
+
+      val stagePair = DummyOrderFromLeftBinaryExec(identityStage, stage(keys))
+      val stagesPlan = planned(stagePair, extraKeys)
+      assert(groupPartitionsNodes(stagesPlan.left).isEmpty,
+        "an identity alignment over multiple shuffle-stage leaves is skipped")
+      assert(ValidateRequirements.validate(stagesPlan))
+
+      val mixed = DummyOrderFromLeftBinaryExec(identityStage,
+        DummySparkPlan(outputPartitioning = KeyedPartitioning(Seq(exprA), 
keys)))
+      val mixedPlan = planned(mixed, extraKeys)
+      assert(groupPartitionsNodes(mixedPlan.left).size == 1,
+        "a child with a non-stage leaf retains its identity alignment")
+      assert(groupPartitionsNodes(mixedPlan.left).head.grouping.isIdentity)
+      assert(ValidateRequirements.validate(mixedPlan))
+
+      val identityExchange = identityStage.plan
+      val exchangePlan = planned(identityExchange, extraKeys)
+      assert(groupPartitionsNodes(exchangePlan.left).size == 1,
+        "an exchange outside a query stage retains its alignment node")
+      assert(groupPartitionsNodes(exchangePlan.left).head.grouping.isIdentity)
+      assert(ValidateRequirements.validate(exchangePlan))
+
+      val pruningStage = stage(extraKeys)
+      val nonIdentity = planned(pruningStage, keys)
+      val grouping = groupPartitionsNodes(nonIdentity.left)
+      assert(grouping.size == 1 && !grouping.head.grouping.isIdentity,
+        "a shuffle stage still needs non-identity alignment")
+      assert(ValidateRequirements.validate(nonIdentity))
+    }
+
+    withSQLConf(
+        SQLConf.V2_BUCKETING_ENABLED.key -> "true",
+        SQLConf.V2_BUCKETING_PUSH_PART_VALUES_ENABLED.key -> "true",
+        SQLConf.V2_BUCKETING_PARTIALLY_CLUSTERED_DISTRIBUTION_ENABLED.key -> 
"true") {
+      def stageJoin(joinType: JoinType): SortMergeJoinExec = {
+        val rightExchange = ShuffleExchangeExec(KeyedPartitioning(Seq(exprB), 
extraKeys),
+          DummySparkPlan(outputPartitioning = 
UnknownPartitioning(extraKeys.size)))
+        SortMergeJoinExec(Seq(exprA), Seq(exprB), joinType, None,
+          stage(keys), ShuffleQueryStageExec(1, rightExchange, rightExchange))
+      }
+      Seq(Inner, LeftOuter).foreach { joinType =>
+        val planned = EnsureRequirements.apply(stageJoin(joinType))
+          .asInstanceOf[SortMergeJoinExec]
+        val left = groupPartitionsNodes(planned.left)
+        val right = groupPartitionsNodes(planned.right)
+        assert(left.size == 1 && right.size == 1,
+          s"$joinType: a chosen replicate side keeps both alignment nodes")
+        assert(left.head.grouping.isIdentity,
+          s"$joinType: the left alignment is the identity")
+        assert(ValidateRequirements.validate(planned))
+      }
+    }
+  }
+
+  test("SPARK-60618: log a push only when an alignment node is retained") {
+    val keys = Seq(InternalRow(1), InternalRow(2))
+    def stage(expr: Attribute, id: Int, values: Seq[InternalRow]): 
ShuffleQueryStageExec = {
+      val leaf = DummySparkPlan(outputPartitioning = 
UnknownPartitioning(values.size))
+      val exchange = ShuffleExchangeExec(KeyedPartitioning(Seq(expr), values), 
leaf)
+      ShuffleQueryStageExec(id, exchange, exchange)
+    }
+    def join(rightKeys: Seq[InternalRow]): SortMergeJoinExec =
+      SortMergeJoinExec(Seq(exprA), Seq(exprB), FullOuter, None,
+        stage(exprA, 0, keys), stage(exprB, 1, rightKeys))
+
+    withSQLConf(
+        SQLConf.V2_BUCKETING_ENABLED.key -> "true",
+        SQLConf.V2_BUCKETING_PUSH_PART_VALUES_ENABLED.key -> "true",
+        SQLConf.V2_BUCKETING_PARTIALLY_CLUSTERED_DISTRIBUTION_ENABLED.key -> 
"true") {
+      def check(rightKeys: Seq[InternalRow], expectPush: Boolean): Unit = {
+        val appender = new LogAppender("storage partition alignment")
+        withLogAppender(appender, loggerNames = 
Seq(classOf[EnsureRequirements].getName),
+            level = Some(Level.INFO)) {
+          val planned = EnsureRequirements.apply(join(rightKeys))
+          assert(ValidateRequirements.validate(planned))
+          assert(groupPartitionsNodes(planned).nonEmpty == expectPush)
+        }
+        val pushLogged = 
appender.loggingEvents.exists(_.getMessage.getFormattedMessage
+          .contains("Pushing common partition values for storage-partitioned 
join"))
+        assert(pushLogged == expectPush)
+      }
+      check(keys, expectPush = false)
+      check(keys :+ InternalRow(3), expectPush = true)
+    }
+  }
+
   test("SPARK-41471: shuffle right side when" +
     " spark.sql.sources.v2.bucketing.shuffle.enabled is true") {
     withSQLConf(SQLConf.V2_BUCKETING_SHUFFLE_ENABLED.key -> "true") {


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to