This is an automated email from the ASF dual-hosted git repository.

ulysses-you pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/master by this push:
     new 8717a3def607 [SPARK-60741][SQL] Keep the build side's keyed 
partitioning in the DPP subquery
8717a3def607 is described below

commit 8717a3def607bd99314c7f5a66c6af8cd23afa8f
Author: Xiduo You <[email protected]>
AuthorDate: Fri Nov 6 16:26:13 2026 +0800

    [SPARK-60741][SQL] Keep the build side's keyed partitioning in the DPP 
subquery
    
    ### What changes were proposed in this pull request?
    
    Dynamic partition pruning (DPP) builds a subquery that collects the 
distinct build-side join keys. Both `PlanDynamicPruningFilters` and 
`PlanAdaptiveDynamicPruningFilters` constructed it as `Aggregate(aliases, 
aliases, buildPlan)`, i.e. the grouping expressions were aliases of the join 
keys.
    
    This PR moves those aliases into a `Project` below the aggregate and groups 
by the attributes that projection produces:
    
    ```scala
    val groupingAttrs = aliases.map(_.toAttribute)
    val aggregate = Aggregate(groupingAttrs, groupingAttrs, Project(aliases, 
buildPlan))
    ```
    
    The subquery's output schema is unchanged; only where the alias is 
evaluated changes.
    
    Grouping by attributes produced by a lower `Project` is the shape that 
`PullOutGroupingExpressions` establishes for complex grouping expressions. DPP 
builds its subquery after logical optimization, so that rule never sees it; 
this change applies the same shape when the subquery is constructed.
    
    ### Why are the changes needed?
    
    When the build side is a storage-partitioned-join scan, it reports a 
`KeyedPartitioning`. Physical planning splits the aggregate into partial and 
final. The partial aggregate's `resultExpressions` use the grouping attributes 
(`groupingExpressions.map(_.toAttribute)`), so the `Alias` object is no longer 
part of `outputExpressions`. `BaseAggregateExec` extends 
`PartitioningPreservingUnaryExecNode`, whose alias map is built from the 
`Alias` nodes in `outputExpressions`, so it cannot proj [...]
    
    With a plain attribute as the grouping expression the partial aggregate 
keeps the keyed partitioning, and `EnsureRequirements` coalesces duplicate keys 
with `GroupPartitionsExec` instead of shuffling. With the default 
configurations, `CombineAdjacentAggregation` merges the adjacent aggregates and 
`ReplaceHashWithSortAgg` selects a sort aggregate, so the resulting subquery 
has a single `SortAggregate`:
    
    ```
    Before
    HashAggregate (Final)
    +- Exchange hashpartitioning(id, 5)
       +- HashAggregate (Partial)
          +- Project / Filter / BatchScan (KeyedPartitioning)
    
    After
    SortAggregate
    +- GroupPartitionsExec
       +- Project [id AS id]
          +- Filter
             +- BatchScan (KeyedPartitioning)
    ```
    
    The same alias-preservation issue affects a V1 build side bucketed on the 
join key. On master, the DPP subquery disables the bucketed scan and shuffles. 
With this PR, it keeps the bucketed scan's `HashPartitioning` and needs no 
shuffle.
    
    JIRA: https://issues.apache.org/jira/browse/SPARK-60741
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes. These plan changes only apply when the internal 
`spark.sql.optimizer.dynamicPartitionPruning.reuseBroadcastOnly` is false. It 
defaults to true, and with it DPP only reuses a broadcast and plans no 
subquery, so default plans do not change. Where it applies, query results are 
unchanged, but the physical plan and scan task counts of the DPP build-side 
subquery can change:
    
    * A build-side scan reporting `KeyedPartitioning` no longer needs a shuffle 
in the DPP subquery.
    * A bucketed V1 build side keeps its bucketed scan instead of falling back 
to a non-bucketed scan plus a shuffle.
    * Without the shuffle, the subquery coalesces splits by partition key, or 
reads a bucketed V1 build side with one task per bucket. Scan parallelism is 
therefore bounded by the number of keys or buckets rather than the number of 
splits. For example, 16 splits over 2 keys are read by 2 tasks instead of 16. A 
build side with many splits per key can take longer to scan and delay the 
pruned query. This matches ordinary distinct aggregates over keyed or bucketed 
scans.
    * With AQE, when the build side is itself a shuffled join on the key, the 
subquery keeps the join's hash partitioning and has no exchange of its own. AQE 
can no longer split a skewed partition of that join from inside the subquery, 
because the split partition no longer satisfies the aggregate's distribution 
requirement. A plain `SELECT DISTINCT` over the same join makes the same 
trade-off.
    
    The existing `spark.sql.sources.v2.bucketing.enabled` and 
`spark.sql.sources.bucketing.enabled` configurations can disable keyed or 
bucketed scans, but they also affect storage-partitioned and bucketed joins.
    
    ### How was this patch tested?
    
    The new test `SPARK-60741: dynamic partition filtering keeps the build 
side's keyed partitioning` in `KeyGroupedPartitioningRuntimeFilterTests` runs 
against both the `SupportsRuntimeFiltering` and 
`SupportsRuntimeCatalystFiltering` fixtures, with `ADAPTIVE_EXECUTION_ENABLED` 
both off and on. It covers `PlanDynamicPruningFilters` and 
`PlanAdaptiveDynamicPruningFilters`, and asserts that the DPP subquery exists, 
is adaptive exactly when AQE is enabled, has no `ShuffleExchangeLike`, cont 
[...]
    
    The new test `SPARK-60741: the DPP subquery keeps a bucketed build side's 
partitioning` in `DynamicPartitionPruningV1Suite` runs with AQE both off and 
on. It asserts that the DPP subquery has no shuffle, keeps the V1 bucketed 
scan, and returns the correct result.
    
    Regression:
    
    * All 8 concrete `DynamicPartitionPruning*` suites: 343 passed, 11 ignored, 
0 failed.
    * `KeyGroupedPartitioningRuntimeFilterSuite` and its Catalyst variant, full 
run: 10 passed, 0 failed.
    
    ```bash
    build/sbt 'sql/testOnly *DynamicPartitionPruningV1SuiteAEOff 
*DynamicPartitionPruningV1SuiteAEOn *DynamicPartitionPruningV2SuiteAEOff 
*DynamicPartitionPruningV2SuiteAEOn *DynamicPartitionPruningV2FilterSuiteAEOff 
*DynamicPartitionPruningV2FilterSuiteAEOn 
*DynamicPartitionPruningV2CatalystFilterSuiteAEOff 
*DynamicPartitionPruningV2CatalystFilterSuiteAEOn'
    build/sbt 'sql/testOnly 
org.apache.spark.sql.connector.KeyGroupedPartitioningRuntimeFilterSuite 
org.apache.spark.sql.connector.KeyGroupedPartitioningCatalystRuntimeFilterSuite'
    ```
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Code (deepseek-flash[1m], kimi-k3.1-pro[1m])
    
    🤖 Generated with [Claude Code](https://claude.com/claude-code)
    
    Closes #59950 from ulysses-you/dpp-agg-alias.
    
    Authored-by: Xiduo You <[email protected]>
    Signed-off-by: Xiduo You <[email protected]>
---
 .../sql/catalyst/expressions/DynamicPruning.scala  | 24 +++++++-
 .../PlanAdaptiveDynamicPruningFilters.scala        |  8 +--
 .../dynamicpruning/PlanDynamicPruningFilters.scala | 10 +--
 .../spark/sql/DynamicPartitionPruningSuite.scala   | 33 +++++++++-
 .../connector/KeyGroupedPartitioningSuite.scala    | 72 +++++++++++++++++++++-
 5 files changed, 133 insertions(+), 14 deletions(-)

diff --git 
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/DynamicPruning.scala
 
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/DynamicPruning.scala
index 959acbc762b4..e22a3f7588b2 100644
--- 
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/DynamicPruning.scala
+++ 
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/DynamicPruning.scala
@@ -20,7 +20,7 @@ package org.apache.spark.sql.catalyst.expressions
 import org.apache.spark.sql.catalyst.InternalRow
 import org.apache.spark.sql.catalyst.expressions.codegen.{CodegenContext, 
ExprCode}
 import org.apache.spark.sql.catalyst.plans.QueryPlan
-import org.apache.spark.sql.catalyst.plans.logical.{HintInfo, LogicalPlan}
+import org.apache.spark.sql.catalyst.plans.logical.{Aggregate, HintInfo, 
LogicalPlan, Project}
 import org.apache.spark.sql.catalyst.trees.TreePattern._
 import org.apache.spark.sql.catalyst.trees.UnaryLike
 
@@ -121,6 +121,28 @@ case class DynamicPruningSubquery(
     copy(pruningKey = newChild)(broadcastValueProjection)
 }
 
+object DynamicPruningSubquery {
+  /**
+   * Builds the plan that collects the distinct build-side keys a DPP filter 
prunes on.
+   *
+   * The keys are aliased in a `Project` below the aggregate, which then 
groups on plain
+   * attributes. A partial aggregate outputs only its grouping attributes, so 
an `Alias` among the
+   * grouping expressions never reaches the `outputExpressions` that
+   * `PartitioningPreservingUnaryExecNode` reads its alias map from. The build 
side's partitioning
+   * (e.g. `KeyedPartitioning`) would then be hidden, and `EnsureRequirements` 
would add a shuffle.
+   * A plain grouping attribute keeps it, and `GroupPartitionsExec` coalesces 
duplicate keys. This
+   * is the shape `PullOutGroupingExpressions` gives complex grouping 
expressions.
+   */
+  private[sql] def planDistinctBuildKeys(
+      buildPlan: LogicalPlan,
+      buildKeys: Seq[Expression],
+      keyIndices: Seq[Int]): LogicalPlan = {
+    val aliases = keyIndices.map(idx => Alias(buildKeys(idx), 
buildKeys(idx).toString)())
+    val groupingAttrs = aliases.map(_.toAttribute)
+    Aggregate(groupingAttrs, groupingAttrs, Project(aliases, buildPlan))
+  }
+}
+
 /**
  * Marker for a planned [[DynamicPruning]] expression.
  * The expression is created during planning, and it defers to its child for 
evaluation.
diff --git 
a/sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/PlanAdaptiveDynamicPruningFilters.scala
 
b/sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/PlanAdaptiveDynamicPruningFilters.scala
index 9374758d4731..6c18436771b2 100644
--- 
a/sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/PlanAdaptiveDynamicPruningFilters.scala
+++ 
b/sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/PlanAdaptiveDynamicPruningFilters.scala
@@ -17,9 +17,8 @@
 
 package org.apache.spark.sql.execution.adaptive
 
-import org.apache.spark.sql.catalyst.expressions.{Alias, BindReferences, 
BroadcastValueProjection, DynamicPruningExpression, Expression, Literal}
+import org.apache.spark.sql.catalyst.expressions.{BindReferences, 
BroadcastValueProjection, DynamicPruningExpression, DynamicPruningSubquery, 
Expression, Literal}
 import org.apache.spark.sql.catalyst.optimizer.{BuildLeft, BuildRight}
-import org.apache.spark.sql.catalyst.plans.logical.Aggregate
 import org.apache.spark.sql.catalyst.rules.Rule
 import org.apache.spark.sql.catalyst.trees.TreePattern._
 import org.apache.spark.sql.execution._
@@ -118,9 +117,8 @@ case class PlanAdaptiveDynamicPruningFilters(
           case None if onlyInBroadcast =>
             DynamicPruningExpression(Literal.TrueLiteral)
           case None =>
-            val aliases = indices.map(idx =>
-              Alias(buildKeys(idx), buildKeys(idx).toString)())
-            val aggregate = Aggregate(aliases, aliases, buildPlan)
+            val aggregate =
+              DynamicPruningSubquery.planDistinctBuildKeys(buildPlan, 
buildKeys, indices)
             val sparkPlan = QueryExecution.prepareExecutedPlan(aggregate, 
adaptivePlan.context)
             assert(sparkPlan.isInstanceOf[AdaptiveSparkPlanExec])
             val newAdaptivePlan = sparkPlan.asInstanceOf[AdaptiveSparkPlanExec]
diff --git 
a/sql/core/src/main/scala/org/apache/spark/sql/execution/dynamicpruning/PlanDynamicPruningFilters.scala
 
b/sql/core/src/main/scala/org/apache/spark/sql/execution/dynamicpruning/PlanDynamicPruningFilters.scala
index ca7407eb2a88..9fae03b9859f 100644
--- 
a/sql/core/src/main/scala/org/apache/spark/sql/execution/dynamicpruning/PlanDynamicPruningFilters.scala
+++ 
b/sql/core/src/main/scala/org/apache/spark/sql/execution/dynamicpruning/PlanDynamicPruningFilters.scala
@@ -17,9 +17,9 @@
 
 package org.apache.spark.sql.execution.dynamicpruning
 
-import org.apache.spark.sql.catalyst.expressions.{Alias, AttributeSeq, 
BindReferences, BroadcastValueProjection, DynamicPruningExpression, 
DynamicPruningSubquery, Expression, Literal}
+import org.apache.spark.sql.catalyst.expressions.{AttributeSeq, 
BindReferences, BroadcastValueProjection, DynamicPruningExpression, 
DynamicPruningSubquery, Expression, Literal}
 import org.apache.spark.sql.catalyst.optimizer.{BuildLeft, BuildRight}
-import org.apache.spark.sql.catalyst.plans.logical.{Aggregate, LogicalPlan}
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
 import org.apache.spark.sql.catalyst.plans.physical.BroadcastMode
 import org.apache.spark.sql.catalyst.rules.Rule
 import org.apache.spark.sql.catalyst.trees.TreePattern.DYNAMIC_PRUNING_SUBQUERY
@@ -123,9 +123,9 @@ case class PlanDynamicPruningFilters(sparkSession: 
SparkSession) extends Rule[Sp
           case None if onlyInBroadcast =>
             DynamicPruningExpression(Literal.TrueLiteral)
           case None =>
-            val aliases = broadcastKeyIndices.map(idx =>
-              Alias(buildKeys(idx), buildKeys(idx).toString)())
-            val aggregate = Aggregate(aliases, aliases, buildPlan)
+            val aggregate =
+              DynamicPruningSubquery.planDistinctBuildKeys(buildPlan, 
buildKeys,
+                broadcastKeyIndices)
             val sparkPlan = QueryExecution.prepareExecutedPlan(sparkSession, 
aggregate)
             val values = SubqueryExec(name, sparkPlan)
             DynamicPruningExpression(InSubqueryExec(value, values, exprId))
diff --git 
a/sql/core/src/test/scala/org/apache/spark/sql/DynamicPartitionPruningSuite.scala
 
b/sql/core/src/test/scala/org/apache/spark/sql/DynamicPartitionPruningSuite.scala
index f1b397209ade..17f87fedf839 100644
--- 
a/sql/core/src/test/scala/org/apache/spark/sql/DynamicPartitionPruningSuite.scala
+++ 
b/sql/core/src/test/scala/org/apache/spark/sql/DynamicPartitionPruningSuite.scala
@@ -38,7 +38,8 @@ import org.apache.spark.sql.execution._
 import org.apache.spark.sql.execution.adaptive._
 import org.apache.spark.sql.execution.datasources.v2.BatchScanExec
 import org.apache.spark.sql.execution.dynamicpruning.PlanDynamicPruningFilters
-import org.apache.spark.sql.execution.exchange.{BroadcastExchangeLike, 
ReusedExchangeExec}
+import org.apache.spark.sql.execution.exchange.{BroadcastExchangeLike, 
ReusedExchangeExec,
+  ShuffleExchangeLike}
 import org.apache.spark.sql.execution.joins.BroadcastHashJoinExec
 import org.apache.spark.sql.execution.metric.SQLLastAttemptMetrics
 import org.apache.spark.sql.execution.streaming.runtime.{MemoryStream, 
StreamingQueryWrapper}
@@ -2051,6 +2052,36 @@ abstract class DynamicPartitionPruningV1Suite extends 
DynamicPartitionPruningDat
 
   import testImplicits._
 
+  test("SPARK-60741: the DPP subquery keeps a bucketed build side's 
partitioning") {
+    withSQLConf(SQLConf.DYNAMIC_PARTITION_PRUNING_ENABLED.key -> "true",
+        SQLConf.DYNAMIC_PARTITION_PRUNING_REUSE_BROADCAST_ONLY.key -> "false",
+        SQLConf.DYNAMIC_PARTITION_PRUNING_FALLBACK_FILTER_RATIO.key -> "10",
+        SQLConf.AUTO_BROADCASTJOIN_THRESHOLD.key -> "-1") {
+      withTable("bucketed_dim", "partitioned_fact") {
+        spark.range(100).selectExpr("id", "id % 10 AS k")
+          .write.bucketBy(4, 
"k").format(tableFormat).saveAsTable("bucketed_dim")
+        spark.range(100).selectExpr("id AS v", "id % 10 AS p")
+          
.write.partitionBy("p").format(tableFormat).saveAsTable("partitioned_fact")
+        val df = sql("SELECT sum(f.v) FROM partitioned_fact f JOIN 
bucketed_dim d " +
+          "ON f.p = d.k WHERE d.id < 5")
+        checkAnswer(df, Row(2350))
+
+        // The subquery groups on the bucket column, so the bucketed scan is 
kept, and no shuffle
+        // is needed.
+        val subqueries = 
collectDynamicPruningExpressions(df.queryExecution.executedPlan)
+          .collect { case i: InSubqueryExec => i.plan }
+        assert(subqueries.nonEmpty)
+        subqueries.foreach {
+          case s: SubqueryExec =>
+            assert(collect(s.child) { case e: ShuffleExchangeLike => e 
}.isEmpty, s.child)
+            assert(collect(s.child) { case f: FileSourceScanExec => 
f.bucketedScan } === Seq(true),
+              s.child)
+          case other => fail(s"expected a SubqueryExec, got $other")
+        }
+      }
+    }
+  }
+
   test("DPP with a checkpointed filtering side and an expression partition 
key") {
     withSQLConf(SQLConf.DYNAMIC_PARTITION_PRUNING_ENABLED.key -> "true",
         SQLConf.DYNAMIC_PARTITION_PRUNING_REUSE_BROADCAST_ONLY.key -> "true") {
diff --git 
a/sql/core/src/test/scala/org/apache/spark/sql/connector/KeyGroupedPartitioningSuite.scala
 
b/sql/core/src/test/scala/org/apache/spark/sql/connector/KeyGroupedPartitioningSuite.scala
index 182b2f8d0df2..45491255334f 100644
--- 
a/sql/core/src/test/scala/org/apache/spark/sql/connector/KeyGroupedPartitioningSuite.scala
+++ 
b/sql/core/src/test/scala/org/apache/spark/sql/connector/KeyGroupedPartitioningSuite.scala
@@ -25,7 +25,7 @@ import org.apache.spark.{SparkConf, SparkException}
 import org.apache.spark.rdd.SortedMergeCoalescedRDD
 import org.apache.spark.sql.{AnalysisException, DataFrame, ExplainSuiteHelper, 
QueryTest, Row}
 import org.apache.spark.sql.catalyst.InternalRow
-import org.apache.spark.sql.catalyst.expressions.{Ascending, 
AttributeReference, Descending, ExprId, Literal, TransformExpression}
+import org.apache.spark.sql.catalyst.expressions.{Ascending, 
AttributeReference, Descending, DynamicPruningExpression, ExprId, Literal, 
TransformExpression}
 import org.apache.spark.sql.catalyst.expressions.aggregate.Complete
 import org.apache.spark.sql.catalyst.plans.{Cross, ExistenceJoin, Inner, 
JoinType, LeftAnti, LeftSemi, LeftSingle, RightOuter}
 import org.apache.spark.sql.catalyst.plans.physical
@@ -42,15 +42,17 @@ import org.apache.spark.sql.execution.{
   ExtendedMode,
   FormattedMode,
   InputAdapter,
+  InSubqueryExec,
   LocalTableScanExec,
   ProjectExec,
   RDDScanExec,
   SimpleMode,
   SortExec,
   SparkPlan,
+  SubqueryExec,
   UnionExec,
   WholeStageCodegenExec}
-import org.apache.spark.sql.execution.adaptive.{AQEShuffleReadExec, 
ResultQueryStageExec, ShuffleQueryStageExec}
+import org.apache.spark.sql.execution.adaptive.{AdaptiveSparkPlanExec, 
AQEShuffleReadExec, ResultQueryStageExec, ShuffleQueryStageExec}
 import org.apache.spark.sql.execution.aggregate.{BaseAggregateExec, 
HashAggregateExec, SortAggregateExec}
 import org.apache.spark.sql.execution.datasources.v2.{BatchScanExec, 
DataSourceV2ScanRelation, GroupPartitionsExec, PlannedScan}
 import org.apache.spark.sql.execution.exchange.{EnsureRequirements, 
ReusedExchangeExec, ShuffleExchangeExec, ShuffleExchangeLike, 
ValidateRequirements}
@@ -226,6 +228,72 @@ trait KeyGroupedPartitioningRuntimeFilterTests extends 
KeyGroupedPartitioningSui
     }
   }
 
+  test("SPARK-60741: dynamic partition filtering keeps the build side's keyed 
partitioning") {
+    withSQLConf(
+        SQLConf.AUTO_BROADCASTJOIN_THRESHOLD.key -> "-1",
+        SQLConf.DYNAMIC_PARTITION_PRUNING_ENABLED.key -> "true",
+        SQLConf.DYNAMIC_PARTITION_PRUNING_REUSE_BROADCAST_ONLY.key -> "false",
+        SQLConf.DYNAMIC_PARTITION_PRUNING_FALLBACK_FILTER_RATIO.key -> "10") {
+      val items_partitions = Array(identity("id"))
+      createTable(items, itemsColumns, items_partitions)
+      sql(s"INSERT INTO testcat.ns.$items VALUES " +
+          s"(1, 'aa', 40.0, cast('2020-01-01' as timestamp)), " +
+          s"(1, 'aa', 41.0, cast('2020-01-15' as timestamp)), " +
+          s"(2, 'bb', 10.0, cast('2020-01-01' as timestamp)), " +
+          s"(2, 'bb', 10.5, cast('2020-01-01' as timestamp)), " +
+          s"(3, 'cc', 15.5, cast('2020-02-01' as timestamp))")
+
+      val purchases_partitions = Array(identity("item_id"))
+      createTable(purchases, purchasesColumns, purchases_partitions)
+      sql(s"INSERT INTO testcat.ns.$purchases VALUES " +
+          s"(1, 42.0, cast('2020-01-01' as timestamp)), " +
+          s"(1, 44.0, cast('2020-01-15' as timestamp)), " +
+          s"(1, 45.0, cast('2020-01-15' as timestamp)), " +
+          s"(2, 11.0, cast('2020-01-01' as timestamp)), " +
+          s"(3, 19.5, cast('2020-02-01' as timestamp))")
+
+      // The false arm exercises PlanDynamicPruningFilters and the true arm
+      // PlanAdaptiveDynamicPruningFilters.
+      Seq(false, true).foreach { aqeEnabled =>
+        withSQLConf(SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> 
aqeEnabled.toString) {
+          val df = sql(s"SELECT sum(p.price) FROM testcat.ns.$items i, " +
+              s"testcat.ns.$purchases p WHERE i.id = p.item_id AND i.price > 
40.0")
+          checkAnswer(df, Seq(Row(131)))
+
+          // The subquery groups only on the build side's key column, which is 
a partition key of
+          // `items`, so its aggregate must keep the scan's keyed 
partitioning. These keys repeat
+          // across splits, so keeping the partitioning means a 
`GroupPartitionsExec` coalesces
+          // them, and no shuffle is needed.
+          val plan = df.queryExecution.executedPlan
+          val subqueries = collectScans(plan)
+            .flatMap(_.runtimeFilters)
+            .flatMap(_.collect { case d: DynamicPruningExpression => d })
+            .flatMap(_.collect { case i: InSubqueryExec => i.plan })
+          assert(subqueries.nonEmpty, s"expected a DPP subquery, got:\n$plan")
+          subqueries.foreach {
+            case subquery: SubqueryExec =>
+              val subPlan = subquery.child
+              assert(subPlan.isInstanceOf[AdaptiveSparkPlanExec] == aqeEnabled,
+                s"the DPP subquery should be adaptive only with 
AQE:\n$subPlan")
+              assert(collect(subPlan) { case s: ShuffleExchangeLike => s 
}.isEmpty,
+                s"the DPP subquery should not shuffle:\n$subPlan")
+              assert(collect(subPlan) { case g: GroupPartitionsExec => g 
}.nonEmpty,
+                s"the DPP subquery should group the keyed 
partitions:\n$subPlan")
+            case other =>
+              fail(s"expected a SubqueryExec, got $other")
+          }
+
+          // DPP keeps only the item_id = 1 splits of `purchases`.
+          assertFilteredPartitions(collectScans(plan), Seq(5, 5), Seq(0, 2))
+
+          // The outer query keeps its storage-partitioned join.
+          assert(collectShuffles(plan).isEmpty,
+            s"should not add shuffle for both sides of the join:\n$plan")
+        }
+      }
+    }
+  }
+
   test("SPARK-42038: partially clustered: with dynamic partition filtering") {
     val items_partitions = Array(identity("id"))
     createTable(items, itemsColumns, items_partitions)


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to