This is an automated email from the ASF dual-hosted git repository.
ulysses-you pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 8717a3def607 [SPARK-60741][SQL] Keep the build side's keyed
partitioning in the DPP subquery
8717a3def607 is described below
commit 8717a3def607bd99314c7f5a66c6af8cd23afa8f
Author: Xiduo You <[email protected]>
AuthorDate: Fri Nov 6 16:26:13 2026 +0800
[SPARK-60741][SQL] Keep the build side's keyed partitioning in the DPP
subquery
### What changes were proposed in this pull request?
Dynamic partition pruning (DPP) builds a subquery that collects the
distinct build-side join keys. Both `PlanDynamicPruningFilters` and
`PlanAdaptiveDynamicPruningFilters` constructed it as `Aggregate(aliases,
aliases, buildPlan)`, i.e. the grouping expressions were aliases of the join
keys.
This PR moves those aliases into a `Project` below the aggregate and groups
by the attributes that projection produces:
```scala
val groupingAttrs = aliases.map(_.toAttribute)
val aggregate = Aggregate(groupingAttrs, groupingAttrs, Project(aliases,
buildPlan))
```
The subquery's output schema is unchanged; only where the alias is
evaluated changes.
Grouping by attributes produced by a lower `Project` is the shape that
`PullOutGroupingExpressions` establishes for complex grouping expressions. DPP
builds its subquery after logical optimization, so that rule never sees it;
this change applies the same shape when the subquery is constructed.
### Why are the changes needed?
When the build side is a storage-partitioned-join scan, it reports a
`KeyedPartitioning`. Physical planning splits the aggregate into partial and
final. The partial aggregate's `resultExpressions` use the grouping attributes
(`groupingExpressions.map(_.toAttribute)`), so the `Alias` object is no longer
part of `outputExpressions`. `BaseAggregateExec` extends
`PartitioningPreservingUnaryExecNode`, whose alias map is built from the
`Alias` nodes in `outputExpressions`, so it cannot proj [...]
With a plain attribute as the grouping expression the partial aggregate
keeps the keyed partitioning, and `EnsureRequirements` coalesces duplicate keys
with `GroupPartitionsExec` instead of shuffling. With the default
configurations, `CombineAdjacentAggregation` merges the adjacent aggregates and
`ReplaceHashWithSortAgg` selects a sort aggregate, so the resulting subquery
has a single `SortAggregate`:
```
Before
HashAggregate (Final)
+- Exchange hashpartitioning(id, 5)
+- HashAggregate (Partial)
+- Project / Filter / BatchScan (KeyedPartitioning)
After
SortAggregate
+- GroupPartitionsExec
+- Project [id AS id]
+- Filter
+- BatchScan (KeyedPartitioning)
```
The same alias-preservation issue affects a V1 build side bucketed on the
join key. On master, the DPP subquery disables the bucketed scan and shuffles.
With this PR, it keeps the bucketed scan's `HashPartitioning` and needs no
shuffle.
JIRA: https://issues.apache.org/jira/browse/SPARK-60741
### Does this PR introduce _any_ user-facing change?
Yes. These plan changes only apply when the internal
`spark.sql.optimizer.dynamicPartitionPruning.reuseBroadcastOnly` is false. It
defaults to true, and with it DPP only reuses a broadcast and plans no
subquery, so default plans do not change. Where it applies, query results are
unchanged, but the physical plan and scan task counts of the DPP build-side
subquery can change:
* A build-side scan reporting `KeyedPartitioning` no longer needs a shuffle
in the DPP subquery.
* A bucketed V1 build side keeps its bucketed scan instead of falling back
to a non-bucketed scan plus a shuffle.
* Without the shuffle, the subquery coalesces splits by partition key, or
reads a bucketed V1 build side with one task per bucket. Scan parallelism is
therefore bounded by the number of keys or buckets rather than the number of
splits. For example, 16 splits over 2 keys are read by 2 tasks instead of 16. A
build side with many splits per key can take longer to scan and delay the
pruned query. This matches ordinary distinct aggregates over keyed or bucketed
scans.
* With AQE, when the build side is itself a shuffled join on the key, the
subquery keeps the join's hash partitioning and has no exchange of its own. AQE
can no longer split a skewed partition of that join from inside the subquery,
because the split partition no longer satisfies the aggregate's distribution
requirement. A plain `SELECT DISTINCT` over the same join makes the same
trade-off.
The existing `spark.sql.sources.v2.bucketing.enabled` and
`spark.sql.sources.bucketing.enabled` configurations can disable keyed or
bucketed scans, but they also affect storage-partitioned and bucketed joins.
### How was this patch tested?
The new test `SPARK-60741: dynamic partition filtering keeps the build
side's keyed partitioning` in `KeyGroupedPartitioningRuntimeFilterTests` runs
against both the `SupportsRuntimeFiltering` and
`SupportsRuntimeCatalystFiltering` fixtures, with `ADAPTIVE_EXECUTION_ENABLED`
both off and on. It covers `PlanDynamicPruningFilters` and
`PlanAdaptiveDynamicPruningFilters`, and asserts that the DPP subquery exists,
is adaptive exactly when AQE is enabled, has no `ShuffleExchangeLike`, cont
[...]
The new test `SPARK-60741: the DPP subquery keeps a bucketed build side's
partitioning` in `DynamicPartitionPruningV1Suite` runs with AQE both off and
on. It asserts that the DPP subquery has no shuffle, keeps the V1 bucketed
scan, and returns the correct result.
Regression:
* All 8 concrete `DynamicPartitionPruning*` suites: 343 passed, 11 ignored,
0 failed.
* `KeyGroupedPartitioningRuntimeFilterSuite` and its Catalyst variant, full
run: 10 passed, 0 failed.
```bash
build/sbt 'sql/testOnly *DynamicPartitionPruningV1SuiteAEOff
*DynamicPartitionPruningV1SuiteAEOn *DynamicPartitionPruningV2SuiteAEOff
*DynamicPartitionPruningV2SuiteAEOn *DynamicPartitionPruningV2FilterSuiteAEOff
*DynamicPartitionPruningV2FilterSuiteAEOn
*DynamicPartitionPruningV2CatalystFilterSuiteAEOff
*DynamicPartitionPruningV2CatalystFilterSuiteAEOn'
build/sbt 'sql/testOnly
org.apache.spark.sql.connector.KeyGroupedPartitioningRuntimeFilterSuite
org.apache.spark.sql.connector.KeyGroupedPartitioningCatalystRuntimeFilterSuite'
```
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (deepseek-flash[1m], kimi-k3.1-pro[1m])
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Closes #59950 from ulysses-you/dpp-agg-alias.
Authored-by: Xiduo You <[email protected]>
Signed-off-by: Xiduo You <[email protected]>
---
.../sql/catalyst/expressions/DynamicPruning.scala | 24 +++++++-
.../PlanAdaptiveDynamicPruningFilters.scala | 8 +--
.../dynamicpruning/PlanDynamicPruningFilters.scala | 10 +--
.../spark/sql/DynamicPartitionPruningSuite.scala | 33 +++++++++-
.../connector/KeyGroupedPartitioningSuite.scala | 72 +++++++++++++++++++++-
5 files changed, 133 insertions(+), 14 deletions(-)
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/DynamicPruning.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/DynamicPruning.scala
index 959acbc762b4..e22a3f7588b2 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/DynamicPruning.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/DynamicPruning.scala
@@ -20,7 +20,7 @@ package org.apache.spark.sql.catalyst.expressions
import org.apache.spark.sql.catalyst.InternalRow
import org.apache.spark.sql.catalyst.expressions.codegen.{CodegenContext,
ExprCode}
import org.apache.spark.sql.catalyst.plans.QueryPlan
-import org.apache.spark.sql.catalyst.plans.logical.{HintInfo, LogicalPlan}
+import org.apache.spark.sql.catalyst.plans.logical.{Aggregate, HintInfo,
LogicalPlan, Project}
import org.apache.spark.sql.catalyst.trees.TreePattern._
import org.apache.spark.sql.catalyst.trees.UnaryLike
@@ -121,6 +121,28 @@ case class DynamicPruningSubquery(
copy(pruningKey = newChild)(broadcastValueProjection)
}
+object DynamicPruningSubquery {
+ /**
+ * Builds the plan that collects the distinct build-side keys a DPP filter
prunes on.
+ *
+ * The keys are aliased in a `Project` below the aggregate, which then
groups on plain
+ * attributes. A partial aggregate outputs only its grouping attributes, so
an `Alias` among the
+ * grouping expressions never reaches the `outputExpressions` that
+ * `PartitioningPreservingUnaryExecNode` reads its alias map from. The build
side's partitioning
+ * (e.g. `KeyedPartitioning`) would then be hidden, and `EnsureRequirements`
would add a shuffle.
+ * A plain grouping attribute keeps it, and `GroupPartitionsExec` coalesces
duplicate keys. This
+ * is the shape `PullOutGroupingExpressions` gives complex grouping
expressions.
+ */
+ private[sql] def planDistinctBuildKeys(
+ buildPlan: LogicalPlan,
+ buildKeys: Seq[Expression],
+ keyIndices: Seq[Int]): LogicalPlan = {
+ val aliases = keyIndices.map(idx => Alias(buildKeys(idx),
buildKeys(idx).toString)())
+ val groupingAttrs = aliases.map(_.toAttribute)
+ Aggregate(groupingAttrs, groupingAttrs, Project(aliases, buildPlan))
+ }
+}
+
/**
* Marker for a planned [[DynamicPruning]] expression.
* The expression is created during planning, and it defers to its child for
evaluation.
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/PlanAdaptiveDynamicPruningFilters.scala
b/sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/PlanAdaptiveDynamicPruningFilters.scala
index 9374758d4731..6c18436771b2 100644
---
a/sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/PlanAdaptiveDynamicPruningFilters.scala
+++
b/sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/PlanAdaptiveDynamicPruningFilters.scala
@@ -17,9 +17,8 @@
package org.apache.spark.sql.execution.adaptive
-import org.apache.spark.sql.catalyst.expressions.{Alias, BindReferences,
BroadcastValueProjection, DynamicPruningExpression, Expression, Literal}
+import org.apache.spark.sql.catalyst.expressions.{BindReferences,
BroadcastValueProjection, DynamicPruningExpression, DynamicPruningSubquery,
Expression, Literal}
import org.apache.spark.sql.catalyst.optimizer.{BuildLeft, BuildRight}
-import org.apache.spark.sql.catalyst.plans.logical.Aggregate
import org.apache.spark.sql.catalyst.rules.Rule
import org.apache.spark.sql.catalyst.trees.TreePattern._
import org.apache.spark.sql.execution._
@@ -118,9 +117,8 @@ case class PlanAdaptiveDynamicPruningFilters(
case None if onlyInBroadcast =>
DynamicPruningExpression(Literal.TrueLiteral)
case None =>
- val aliases = indices.map(idx =>
- Alias(buildKeys(idx), buildKeys(idx).toString)())
- val aggregate = Aggregate(aliases, aliases, buildPlan)
+ val aggregate =
+ DynamicPruningSubquery.planDistinctBuildKeys(buildPlan,
buildKeys, indices)
val sparkPlan = QueryExecution.prepareExecutedPlan(aggregate,
adaptivePlan.context)
assert(sparkPlan.isInstanceOf[AdaptiveSparkPlanExec])
val newAdaptivePlan = sparkPlan.asInstanceOf[AdaptiveSparkPlanExec]
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/execution/dynamicpruning/PlanDynamicPruningFilters.scala
b/sql/core/src/main/scala/org/apache/spark/sql/execution/dynamicpruning/PlanDynamicPruningFilters.scala
index ca7407eb2a88..9fae03b9859f 100644
---
a/sql/core/src/main/scala/org/apache/spark/sql/execution/dynamicpruning/PlanDynamicPruningFilters.scala
+++
b/sql/core/src/main/scala/org/apache/spark/sql/execution/dynamicpruning/PlanDynamicPruningFilters.scala
@@ -17,9 +17,9 @@
package org.apache.spark.sql.execution.dynamicpruning
-import org.apache.spark.sql.catalyst.expressions.{Alias, AttributeSeq,
BindReferences, BroadcastValueProjection, DynamicPruningExpression,
DynamicPruningSubquery, Expression, Literal}
+import org.apache.spark.sql.catalyst.expressions.{AttributeSeq,
BindReferences, BroadcastValueProjection, DynamicPruningExpression,
DynamicPruningSubquery, Expression, Literal}
import org.apache.spark.sql.catalyst.optimizer.{BuildLeft, BuildRight}
-import org.apache.spark.sql.catalyst.plans.logical.{Aggregate, LogicalPlan}
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
import org.apache.spark.sql.catalyst.plans.physical.BroadcastMode
import org.apache.spark.sql.catalyst.rules.Rule
import org.apache.spark.sql.catalyst.trees.TreePattern.DYNAMIC_PRUNING_SUBQUERY
@@ -123,9 +123,9 @@ case class PlanDynamicPruningFilters(sparkSession:
SparkSession) extends Rule[Sp
case None if onlyInBroadcast =>
DynamicPruningExpression(Literal.TrueLiteral)
case None =>
- val aliases = broadcastKeyIndices.map(idx =>
- Alias(buildKeys(idx), buildKeys(idx).toString)())
- val aggregate = Aggregate(aliases, aliases, buildPlan)
+ val aggregate =
+ DynamicPruningSubquery.planDistinctBuildKeys(buildPlan,
buildKeys,
+ broadcastKeyIndices)
val sparkPlan = QueryExecution.prepareExecutedPlan(sparkSession,
aggregate)
val values = SubqueryExec(name, sparkPlan)
DynamicPruningExpression(InSubqueryExec(value, values, exprId))
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/DynamicPartitionPruningSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/DynamicPartitionPruningSuite.scala
index f1b397209ade..17f87fedf839 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/DynamicPartitionPruningSuite.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/DynamicPartitionPruningSuite.scala
@@ -38,7 +38,8 @@ import org.apache.spark.sql.execution._
import org.apache.spark.sql.execution.adaptive._
import org.apache.spark.sql.execution.datasources.v2.BatchScanExec
import org.apache.spark.sql.execution.dynamicpruning.PlanDynamicPruningFilters
-import org.apache.spark.sql.execution.exchange.{BroadcastExchangeLike,
ReusedExchangeExec}
+import org.apache.spark.sql.execution.exchange.{BroadcastExchangeLike,
ReusedExchangeExec,
+ ShuffleExchangeLike}
import org.apache.spark.sql.execution.joins.BroadcastHashJoinExec
import org.apache.spark.sql.execution.metric.SQLLastAttemptMetrics
import org.apache.spark.sql.execution.streaming.runtime.{MemoryStream,
StreamingQueryWrapper}
@@ -2051,6 +2052,36 @@ abstract class DynamicPartitionPruningV1Suite extends
DynamicPartitionPruningDat
import testImplicits._
+ test("SPARK-60741: the DPP subquery keeps a bucketed build side's
partitioning") {
+ withSQLConf(SQLConf.DYNAMIC_PARTITION_PRUNING_ENABLED.key -> "true",
+ SQLConf.DYNAMIC_PARTITION_PRUNING_REUSE_BROADCAST_ONLY.key -> "false",
+ SQLConf.DYNAMIC_PARTITION_PRUNING_FALLBACK_FILTER_RATIO.key -> "10",
+ SQLConf.AUTO_BROADCASTJOIN_THRESHOLD.key -> "-1") {
+ withTable("bucketed_dim", "partitioned_fact") {
+ spark.range(100).selectExpr("id", "id % 10 AS k")
+ .write.bucketBy(4,
"k").format(tableFormat).saveAsTable("bucketed_dim")
+ spark.range(100).selectExpr("id AS v", "id % 10 AS p")
+
.write.partitionBy("p").format(tableFormat).saveAsTable("partitioned_fact")
+ val df = sql("SELECT sum(f.v) FROM partitioned_fact f JOIN
bucketed_dim d " +
+ "ON f.p = d.k WHERE d.id < 5")
+ checkAnswer(df, Row(2350))
+
+ // The subquery groups on the bucket column, so the bucketed scan is
kept, and no shuffle
+ // is needed.
+ val subqueries =
collectDynamicPruningExpressions(df.queryExecution.executedPlan)
+ .collect { case i: InSubqueryExec => i.plan }
+ assert(subqueries.nonEmpty)
+ subqueries.foreach {
+ case s: SubqueryExec =>
+ assert(collect(s.child) { case e: ShuffleExchangeLike => e
}.isEmpty, s.child)
+ assert(collect(s.child) { case f: FileSourceScanExec =>
f.bucketedScan } === Seq(true),
+ s.child)
+ case other => fail(s"expected a SubqueryExec, got $other")
+ }
+ }
+ }
+ }
+
test("DPP with a checkpointed filtering side and an expression partition
key") {
withSQLConf(SQLConf.DYNAMIC_PARTITION_PRUNING_ENABLED.key -> "true",
SQLConf.DYNAMIC_PARTITION_PRUNING_REUSE_BROADCAST_ONLY.key -> "true") {
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/connector/KeyGroupedPartitioningSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/connector/KeyGroupedPartitioningSuite.scala
index 182b2f8d0df2..45491255334f 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/connector/KeyGroupedPartitioningSuite.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/connector/KeyGroupedPartitioningSuite.scala
@@ -25,7 +25,7 @@ import org.apache.spark.{SparkConf, SparkException}
import org.apache.spark.rdd.SortedMergeCoalescedRDD
import org.apache.spark.sql.{AnalysisException, DataFrame, ExplainSuiteHelper,
QueryTest, Row}
import org.apache.spark.sql.catalyst.InternalRow
-import org.apache.spark.sql.catalyst.expressions.{Ascending,
AttributeReference, Descending, ExprId, Literal, TransformExpression}
+import org.apache.spark.sql.catalyst.expressions.{Ascending,
AttributeReference, Descending, DynamicPruningExpression, ExprId, Literal,
TransformExpression}
import org.apache.spark.sql.catalyst.expressions.aggregate.Complete
import org.apache.spark.sql.catalyst.plans.{Cross, ExistenceJoin, Inner,
JoinType, LeftAnti, LeftSemi, LeftSingle, RightOuter}
import org.apache.spark.sql.catalyst.plans.physical
@@ -42,15 +42,17 @@ import org.apache.spark.sql.execution.{
ExtendedMode,
FormattedMode,
InputAdapter,
+ InSubqueryExec,
LocalTableScanExec,
ProjectExec,
RDDScanExec,
SimpleMode,
SortExec,
SparkPlan,
+ SubqueryExec,
UnionExec,
WholeStageCodegenExec}
-import org.apache.spark.sql.execution.adaptive.{AQEShuffleReadExec,
ResultQueryStageExec, ShuffleQueryStageExec}
+import org.apache.spark.sql.execution.adaptive.{AdaptiveSparkPlanExec,
AQEShuffleReadExec, ResultQueryStageExec, ShuffleQueryStageExec}
import org.apache.spark.sql.execution.aggregate.{BaseAggregateExec,
HashAggregateExec, SortAggregateExec}
import org.apache.spark.sql.execution.datasources.v2.{BatchScanExec,
DataSourceV2ScanRelation, GroupPartitionsExec, PlannedScan}
import org.apache.spark.sql.execution.exchange.{EnsureRequirements,
ReusedExchangeExec, ShuffleExchangeExec, ShuffleExchangeLike,
ValidateRequirements}
@@ -226,6 +228,72 @@ trait KeyGroupedPartitioningRuntimeFilterTests extends
KeyGroupedPartitioningSui
}
}
+ test("SPARK-60741: dynamic partition filtering keeps the build side's keyed
partitioning") {
+ withSQLConf(
+ SQLConf.AUTO_BROADCASTJOIN_THRESHOLD.key -> "-1",
+ SQLConf.DYNAMIC_PARTITION_PRUNING_ENABLED.key -> "true",
+ SQLConf.DYNAMIC_PARTITION_PRUNING_REUSE_BROADCAST_ONLY.key -> "false",
+ SQLConf.DYNAMIC_PARTITION_PRUNING_FALLBACK_FILTER_RATIO.key -> "10") {
+ val items_partitions = Array(identity("id"))
+ createTable(items, itemsColumns, items_partitions)
+ sql(s"INSERT INTO testcat.ns.$items VALUES " +
+ s"(1, 'aa', 40.0, cast('2020-01-01' as timestamp)), " +
+ s"(1, 'aa', 41.0, cast('2020-01-15' as timestamp)), " +
+ s"(2, 'bb', 10.0, cast('2020-01-01' as timestamp)), " +
+ s"(2, 'bb', 10.5, cast('2020-01-01' as timestamp)), " +
+ s"(3, 'cc', 15.5, cast('2020-02-01' as timestamp))")
+
+ val purchases_partitions = Array(identity("item_id"))
+ createTable(purchases, purchasesColumns, purchases_partitions)
+ sql(s"INSERT INTO testcat.ns.$purchases VALUES " +
+ s"(1, 42.0, cast('2020-01-01' as timestamp)), " +
+ s"(1, 44.0, cast('2020-01-15' as timestamp)), " +
+ s"(1, 45.0, cast('2020-01-15' as timestamp)), " +
+ s"(2, 11.0, cast('2020-01-01' as timestamp)), " +
+ s"(3, 19.5, cast('2020-02-01' as timestamp))")
+
+ // The false arm exercises PlanDynamicPruningFilters and the true arm
+ // PlanAdaptiveDynamicPruningFilters.
+ Seq(false, true).foreach { aqeEnabled =>
+ withSQLConf(SQLConf.ADAPTIVE_EXECUTION_ENABLED.key ->
aqeEnabled.toString) {
+ val df = sql(s"SELECT sum(p.price) FROM testcat.ns.$items i, " +
+ s"testcat.ns.$purchases p WHERE i.id = p.item_id AND i.price >
40.0")
+ checkAnswer(df, Seq(Row(131)))
+
+ // The subquery groups only on the build side's key column, which is
a partition key of
+ // `items`, so its aggregate must keep the scan's keyed
partitioning. These keys repeat
+ // across splits, so keeping the partitioning means a
`GroupPartitionsExec` coalesces
+ // them, and no shuffle is needed.
+ val plan = df.queryExecution.executedPlan
+ val subqueries = collectScans(plan)
+ .flatMap(_.runtimeFilters)
+ .flatMap(_.collect { case d: DynamicPruningExpression => d })
+ .flatMap(_.collect { case i: InSubqueryExec => i.plan })
+ assert(subqueries.nonEmpty, s"expected a DPP subquery, got:\n$plan")
+ subqueries.foreach {
+ case subquery: SubqueryExec =>
+ val subPlan = subquery.child
+ assert(subPlan.isInstanceOf[AdaptiveSparkPlanExec] == aqeEnabled,
+ s"the DPP subquery should be adaptive only with
AQE:\n$subPlan")
+ assert(collect(subPlan) { case s: ShuffleExchangeLike => s
}.isEmpty,
+ s"the DPP subquery should not shuffle:\n$subPlan")
+ assert(collect(subPlan) { case g: GroupPartitionsExec => g
}.nonEmpty,
+ s"the DPP subquery should group the keyed
partitions:\n$subPlan")
+ case other =>
+ fail(s"expected a SubqueryExec, got $other")
+ }
+
+ // DPP keeps only the item_id = 1 splits of `purchases`.
+ assertFilteredPartitions(collectScans(plan), Seq(5, 5), Seq(0, 2))
+
+ // The outer query keeps its storage-partitioned join.
+ assert(collectShuffles(plan).isEmpty,
+ s"should not add shuffle for both sides of the join:\n$plan")
+ }
+ }
+ }
+ }
+
test("SPARK-42038: partially clustered: with dynamic partition filtering") {
val items_partitions = Array(identity("id"))
createTable(items, itemsColumns, items_partitions)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]