This is an automated email from the ASF dual-hosted git repository.
ulysses-you pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new 3fa64e15c55b [SPARK-60458][SQL] Optimize skewed joins per join instead
of all-or-nothing
3fa64e15c55b is described below
commit 3fa64e15c55baa494f0dfea512457a7da9b6ab3d
Author: Xiduo You <[email protected]>
AuthorDate: Tue Nov 3 12:03:25 2026 +0800
[SPARK-60458][SQL] Optimize skewed joins per join instead of all-or-nothing
### What changes were proposed in this pull request?
`OptimizeSkewedJoin` applied every split, validated the whole plan once,
and returned the plan unchanged when that validation failed. One join whose
split broke a requirement therefore cost every skewed join in the plan the
optimization.
This PR makes the strategy per join. The all-at-once attempt stays as the
fast path (one candidate pass, one `ValidateRequirements`). When it fails and
`spark.sql.adaptive.forceOptimizeSkewedJoin` is off,
`applySkewJoinsSatisfyingRequirements` re-applies the candidates one at a time
in the bottom-up order the plan is visited: a join keeps its split when the
plan holding the kept splits and this one still satisfies every requirement.
The internal `spark.sql.adaptive.skewJoin.perJoinFal [...]
Implementation notes:
- `collectSplitSkewJoins` gathers the candidates once, so the fallback
reuses the splits the fast path built.
- Candidates are matched by node identity: structurally equal joins in
different places are separate candidates.
- `applySkewJoins` walks top-down. A broadcast-hash join only constrains
its streamed side, so another candidate can sit in its build side; replacing
the outer join first keeps the inner candidate matchable, and the inner split
is then applied inside the replacement.
- Each trial is rebuilt from the original plan, so acceptance does not
depend on how an earlier rebuild kept node instances.
- The public `OptimizeSkewedJoin.optimizeSkewJoin` is removed. Its "split
every join" behavior now lives in the fast path, and it had no in-tree caller.
- The force path and the disabled path are unchanged.
`CoalesceShufflePartitions` changes too: it grouped a union's children into
one coalesce group whenever the stamped decision said the union passes a layout
through. That decision outlives the layout, so with one branch split the whole
group gives up coalescing. The rule now asks the union what it currently
reports as well, so its children coalesce independently once the union can only
concatenate.
### Why are the changes needed?
The rule was all-or-nothing. Two failure classes make the whole-plan
validation fail:
1. a split introduces an extra shuffle, because the consumer above the join
needs the rows on its key, and
2. an ancestor loses a partitioning claim, e.g. a `UnionExec` that was
passing a branch's layout through reports `UnknownPartitioning` once that
branch's join is split.
In both cases only the affected join is unsound; the other joins can keep
their split. Plans where several joins share a stage used to get no
optimization at all, and now they get the sound ones. The coalescing change
keeps the stages under such a union coalesced, which the mixed split would
otherwise have given up for every stage in the group.
### Does this PR introduce _any_ user-facing change?
Yes, in plan shape: an AQE query whose skewed joins previously all went
unoptimized can now split the ones whose plan still validates, and the stages
under a union that loses its layout stay coalesced. Results are unchanged,
since a split preserves them and every accepted subset is checked with
`ValidateRequirements`. `spark.sql.adaptive.forceOptimizeSkewedJoin` keeps its
meaning.
### How was this patch tested?
Three new tests in `AdaptiveQueryExecSuite`:
- `SPARK-60458: skip only the skewed join whose split the plan cannot
take`: a skewed join under a `GROUP BY` is rejected while the join feeding the
union directly is split; the rejected join's two reads are asserted coalesced,
and the result is checked to be exactly one aggregated row plus 334000
un-aggregated rows. An arm with the fallback config off asserts neither join is
split.
- `SPARK-60458: keep the layout a union passes through`: both union
branches report the join's hash layout and a window above depends on it, so
neither join is split; removing the window keeps both splits. Both arms check
the 668000 result rows and their value/rank distributions. Both arms also pass
on the previous rule, and the first is a guard that the per-join fallback keeps
no split the claim cannot take.
- `SPARK-60458: split a join nested in a broadcast join's build side`: a
plan-level test that nests a sort-merge join in a broadcast join's build side
and checks both splits survive.
The third case of `SPARK-37652` now expects 1 of 3 joins optimized, up from
0.
Commands:
```
build/sbt 'sql/testOnly *AdaptiveQueryExecSuite'
build/sbt sql/scalastyle
```
Full suite 149/149, scalastyle clean.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (deepseek-flash[1m])
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Closes #59703 from ulysses-you/SPARK-60458.
Authored-by: Xiduo You <[email protected]>
Signed-off-by: Xiduo You <[email protected]>
(cherry picked from commit 22428725853f497c0d8dbe6b5d678e09be88c1f0)
Signed-off-by: Xiduo You <[email protected]>
---
.../org/apache/spark/sql/internal/SQLConf.scala | 12 ++
.../adaptive/CoalesceShufflePartitions.scala | 15 +-
.../execution/adaptive/OptimizeSkewedJoin.scala | 143 +++++++++++++++----
.../adaptive/AdaptiveQueryExecSuite.scala | 157 ++++++++++++++++++++-
4 files changed, 286 insertions(+), 41 deletions(-)
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
index 35c5c57411a2..069417b8edf1 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
@@ -1575,6 +1575,18 @@ object SQLConf {
.booleanConf
.createWithDefault(true)
+ val SKEW_JOIN_PER_JOIN_FALLBACK_ENABLED =
+ buildConf("spark.sql.adaptive.skewJoin.perJoinFallback.enabled")
+ .internal()
+ .doc("When true, a plan whose all-at-once skew join optimization breaks
a requirement " +
+ "keeps the splits the plan can still satisfy, instead of giving up the
optimization for " +
+ "every join. When false, such a plan is left unchanged. Only takes
effect when " +
+ "'spark.sql.adaptive.forceOptimizeSkewedJoin' is false.")
+ .version("4.4.0")
+ .withBindingPolicy(ConfigBindingPolicy.NOT_APPLICABLE)
+ .booleanConf
+ .createWithDefault(true)
+
val SKEW_JOIN_SKEWED_PARTITION_FACTOR =
buildConf("spark.sql.adaptive.skewJoin.skewedPartitionFactor")
.doc("A partition is considered as skewed if its size is larger than
this factor " +
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/CoalesceShufflePartitions.scala
b/sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/CoalesceShufflePartitions.scala
index 32021da21171..642ef5b215ae 100644
---
a/sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/CoalesceShufflePartitions.scala
+++
b/sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/CoalesceShufflePartitions.scala
@@ -20,7 +20,7 @@ package org.apache.spark.sql.execution.adaptive
import scala.collection.mutable
import org.apache.spark.sql.SparkSession
-import org.apache.spark.sql.catalyst.plans.physical.SinglePartition
+import org.apache.spark.sql.catalyst.plans.physical.{SinglePartition,
UnknownPartitioning}
import org.apache.spark.sql.execution.{ShufflePartitionSpec, SparkPlan,
UnaryExecNode, UnionExec}
import org.apache.spark.sql.execution.exchange.{ENSURE_REQUIREMENTS,
REBALANCE_PARTITIONS_BY_COL, REBALANCE_PARTITIONS_BY_NONE, REPARTITION_BY_COL,
ShuffleExchangeLike, ShuffleOrigin}
import org.apache.spark.sql.execution.joins.{BroadcastHashJoinExec,
BroadcastNestedLoopJoinExec, CartesianProductExec}
@@ -182,11 +182,14 @@ case class CoalesceShufflePartitions(session:
SparkSession) extends AQEShuffleRe
private def childrenNeedCompatiblePartitioning(p: SparkPlan): Boolean = p
match {
// TODO: match more plan nodes here.
- // A UnionExec that reports a real outputPartitioning (i.e. it is not a
plain union) has that
- // partitioning relied on by downstream operators, so its children must
remain co-partitioned
- // -- coalesce them as a single group. When it reports UnknownPartitioning
(config off, or
- // children not compatibly partitioned), the children may still be
coalesced independently.
- case u: UnionExec => !u.isPlainUnion
+ // A UnionExec that reports a concrete outputPartitioning has that
partitioning relied on by
+ // downstream operators, so its children must remain co-partitioned --
coalesce them as a
+ // single group. When it reports UnknownPartitioning (a plain union, or
children that are not
+ // compatibly partitioned), the children may still be coalesced
independently. Ask what the
+ // union reports now rather than only the stamped decision: a rule that
runs after the stamp,
+ // such as `OptimizeSkewedJoin` splitting one branch, can take the layout
away, and the union
+ // then concatenates whatever its children produce.
+ case u: UnionExec =>
!u.outputPartitioning.isInstanceOf[UnknownPartitioning]
case _: CartesianProductExec => false
case _: BroadcastHashJoinExec => false
case _: BroadcastNestedLoopJoinExec => false
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/OptimizeSkewedJoin.scala
b/sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/OptimizeSkewedJoin.scala
index cea464fa6b84..6ee158681f2e 100644
---
a/sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/OptimizeSkewedJoin.scala
+++
b/sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/OptimizeSkewedJoin.scala
@@ -29,7 +29,7 @@ import
org.apache.spark.sql.catalyst.plans.physical.Partitioning
import org.apache.spark.sql.catalyst.rules.Rule
import org.apache.spark.sql.execution._
import org.apache.spark.sql.execution.exchange.{ENSURE_REQUIREMENTS,
EnsureRequirements, ValidateRequirements}
-import org.apache.spark.sql.execution.joins.{BroadcastHashJoinExec,
ShuffledHashJoinExec, ShuffledJoin, SortMergeJoinExec}
+import org.apache.spark.sql.execution.joins.{BaseJoinExec,
BroadcastHashJoinExec, ShuffledHashJoinExec, ShuffledJoin, SortMergeJoinExec}
import org.apache.spark.sql.internal.SQLConf
import org.apache.spark.util.Utils
@@ -211,65 +211,142 @@ case class OptimizeSkewedJoin(ensureRequirements:
EnsureRequirements)
}
}
- def optimizeSkewJoin(plan: SparkPlan): SparkPlan = {
- val localShuffleReaderEnabled =
conf.getConf(SQLConf.LOCAL_SHUFFLE_READER_ENABLED)
- plan.transformUp {
+ /**
+ * Returns the candidate for `plan` when it is a sort-merge, shuffled-hash
or broadcast-hash join
+ * with a skewed shuffled side this rule can split, and `None` otherwise.
+ */
+ private def splitSkewJoin(
+ plan: SparkPlan,
+ localShuffleReaderEnabled: Boolean): Option[SkewJoinCandidate] =
+ plan match {
case smj @ SortMergeJoinExec(_, _, joinType, _,
s1 @ SortExec(_, _, ShuffleStage(left: ShuffleQueryStageExec), _),
s2 @ SortExec(_, _, ShuffleStage(right: ShuffleQueryStageExec), _),
false) =>
tryOptimizeJoinChildren(left, right, joinType).map {
case (newLeft, newRight) =>
- smj.copy(
- left = s1.copy(child = newLeft), right = s2.copy(child =
newRight), isSkewJoin = true)
- }.getOrElse(smj)
+ SkewJoinCandidate(smj, smj.copy(
+ left = s1.copy(child = newLeft), right = s2.copy(child =
newRight),
+ isSkewJoin = true))
+ }
case shj @ ShuffledHashJoinExec(_, _, joinType, _, _,
ShuffleStage(left: ShuffleQueryStageExec),
ShuffleStage(right: ShuffleQueryStageExec), false) =>
tryOptimizeJoinChildren(left, right, joinType).map {
case (newLeft, newRight) =>
- shj.copy(left = newLeft, right = newRight, isSkewJoin = true)
- }.getOrElse(shj)
+ SkewJoinCandidate(shj, shj.copy(left = newLeft, right = newRight,
isSkewJoin = true))
+ }
// When localShuffleReader is disabled, the streamedPlan of
BroadcastHashJoin may be skewed,
// so we try to optimize it.
case bhj @ BroadcastHashJoinExec(_, _, _, BuildRight, _,
ShuffleStage(left: ShuffleQueryStageExec), _, _, false) if
!localShuffleReaderEnabled =>
tryOptimizeBroadcastHashJoinStreamedPlan(left).map {
- case newLeft => bhj.copy(left = newLeft, isSkewJoin = true)
- }.getOrElse(bhj)
+ case newLeft => SkewJoinCandidate(bhj, bhj.copy(left = newLeft,
isSkewJoin = true))
+ }
case bhj @ BroadcastHashJoinExec(_, _, _, BuildLeft, _, _,
ShuffleStage(right: ShuffleQueryStageExec), _, false) if
!localShuffleReaderEnabled =>
tryOptimizeBroadcastHashJoinStreamedPlan(right).map {
- case newRight => bhj.copy(right = newRight, isSkewJoin = true)
- }.getOrElse(bhj)
+ case newRight => SkewJoinCandidate(bhj, bhj.copy(right = newRight,
isSkewJoin = true))
+ }
+
+ case _ => None
}
+
+ /**
+ * Every join in `plan` this rule can split, in the bottom-up order
`foreachUp` visits them.
+ */
+ private def collectSplitSkewJoins(plan: SparkPlan): Seq[SkewJoinCandidate] =
{
+ val localShuffleReaderEnabled =
conf.getConf(SQLConf.LOCAL_SHUFFLE_READER_ENABLED)
+ val candidates = mutable.ArrayBuffer.empty[SkewJoinCandidate]
+ plan.foreachUp { operator =>
+ splitSkewJoin(operator, localShuffleReaderEnabled).foreach(candidates +=
_)
+ }
+ candidates.toSeq
}
- override def apply(plan: SparkPlan): SparkPlan = {
- if (!conf.getConf(SQLConf.SKEW_JOIN_ENABLED)) {
+ /**
+ * `plan` with `candidates` replaced by their splits. Candidates are matched
by node identity,
+ * so structurally equal joins in different places are distinct candidates.
A subset may be
+ * passed, keeping the others out of the result.
+ *
+ * The walk is top-down: a shuffled-hash candidate has shuffle stages as
children and a
+ * sort-merge one has sorts over them, so neither contains another
candidate; a broadcast-hash
+ * join only constrains its streamed side, so another candidate can sit in
its build side.
+ * Replacing the outer join first keeps that inner candidate an instance
`plan` holds, and the
+ * walk then splits it inside the replacement.
+ */
+ private[adaptive] def applySkewJoins(
+ plan: SparkPlan,
+ candidates: Seq[SkewJoinCandidate]): SparkPlan = {
+ if (candidates.isEmpty) {
return plan
}
+ plan.transformDown { operator =>
+ candidates.find(_.join eq
operator).map(_.replacement).getOrElse(operator)
+ }
+ }
- // We try to optimize every skewed sort-merge/shuffle-hash joins in the
query plan. If this
- // introduces extra shuffles, we give up the optimization and return the
original query plan, or
- // accept the extra shuffles if the force-apply config is true.
- // TODO: It's possible that only one skewed join in the query plan leads
to extra shuffles and
- // we only need to skip optimizing that join. We should make the
strategy smarter here.
- val optimized = optimizeSkewJoin(plan)
- val requirementSatisfied = if
(ensureRequirements.requiredDistribution.isDefined) {
- ValidateRequirements.validate(optimized,
ensureRequirements.requiredDistribution.get)
- } else {
- ValidateRequirements.validate(optimized)
+ private def requirementsSatisfied(plan: SparkPlan): Boolean = {
+ ensureRequirements.requiredDistribution match {
+ case Some(distribution) => ValidateRequirements.validate(plan,
distribution)
+ case None => ValidateRequirements.validate(plan)
}
- if (requirementSatisfied) {
- optimized.transform {
- case SkewJoinChildWrapper(child) => child
+ }
+
+ private def stripSkewJoinChildWrappers(plan: SparkPlan): SparkPlan = {
+ plan.transform { case SkewJoinChildWrapper(child) => child }
+ }
+
+ /**
+ * Applies the candidates one at a time, in the order a bottom-up pass
visits them, keeping a
+ * candidate when the plan holding the kept ones and this one still
satisfies every requirement.
+ * A split the plan cannot take therefore costs only its own join, where the
whole-plan fast path
+ * would have cost every join.
+ *
+ * Each trial is rebuilt from `plan` rather than from the previous trial, so
acceptance never
+ * depends on how a rebuild kept node instances; the handful of skewed joins
keeps that cheap.
+ */
+ private def applySkewJoinsSatisfyingRequirements(
+ plan: SparkPlan,
+ candidates: Seq[SkewJoinCandidate]): SparkPlan = {
+ var accepted = Seq.empty[SkewJoinCandidate]
+ candidates.zipWithIndex.foreach { case (candidate, index) =>
+ // With every earlier candidate kept, this trial is the plan the fast
path rejected.
+ val rejectedAlready = index == candidates.length - 1 && accepted.length
== index
+ if (!rejectedAlready && requirementsSatisfied(applySkewJoins(plan,
accepted :+ candidate))) {
+ accepted = accepted :+ candidate
+ } else {
+ logDebug(s"Skip splitting the skewed join " +
+ s"${candidate.join.simpleString(conf.maxToStringFields)}: with it
the plan fails " +
+ "ValidateRequirements")
}
+ }
+ logDebug(s"Optimized ${accepted.length} of ${candidates.length} skewed
joins")
+ if (accepted.isEmpty) {
+ plan
+ } else {
+ stripSkewJoinChildWrappers(applySkewJoins(plan, accepted))
+ }
+ }
+
+ override def apply(plan: SparkPlan): SparkPlan = {
+ if (!conf.getConf(SQLConf.SKEW_JOIN_ENABLED)) {
+ return plan
+ }
+
+ // Try every split at once first: one pass to find the candidates and one
validation cover the
+ // common case. When that breaks a requirement of the plan, either accept
the extra shuffles if
+ // the force-apply config is true, or keep only the joins whose splits the
plan can take. The
+ // candidates are collected once, so the fallback reuses the splits the
fast path built.
+ val candidates = collectSplitSkewJoins(plan)
+ val optimized = applySkewJoins(plan, candidates)
+ if (requirementsSatisfied(optimized)) {
+ stripSkewJoinChildWrappers(optimized)
} else if (conf.getConf(SQLConf.ADAPTIVE_FORCE_OPTIMIZE_SKEWED_JOIN)) {
- ensureRequirements.apply(optimized).transform {
- case SkewJoinChildWrapper(child) => child
- }
+ stripSkewJoinChildWrappers(ensureRequirements.apply(optimized))
+ } else if (conf.getConf(SQLConf.SKEW_JOIN_PER_JOIN_FALLBACK_ENABLED)) {
+ applySkewJoinsSatisfyingRequirements(plan, candidates)
} else {
plan
}
@@ -285,6 +362,10 @@ case class OptimizeSkewedJoin(ensureRequirements:
EnsureRequirements)
}
}
+// A skewed join `OptimizeSkewedJoin` can split: the join node in the plan,
and the join it is
+// replaced by, with the skewed side split.
+private case class SkewJoinCandidate(join: BaseJoinExec, replacement:
SparkPlan)
+
// After optimizing skew joins, we need to run EnsureRequirements again to add
necessary shuffles
// caused by skew join optimization. However, this shouldn't apply to the
sub-plan under skew join,
// as it's guaranteed to satisfy distribution requirement.
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/execution/adaptive/AdaptiveQueryExecSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/execution/adaptive/AdaptiveQueryExecSuite.scala
index 138f27a4fd9e..32d9df8d839b 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/execution/adaptive/AdaptiveQueryExecSuite.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/execution/adaptive/AdaptiveQueryExecSuite.scala
@@ -46,7 +46,7 @@ import
org.apache.spark.sql.execution.columnar.{InMemoryTableScanExec, InMemoryT
import org.apache.spark.sql.execution.command.DataWritingCommandExec
import org.apache.spark.sql.execution.datasources.noop.NoopDataSource
import org.apache.spark.sql.execution.datasources.v2.V2TableWriteExec
-import org.apache.spark.sql.execution.exchange.{BroadcastExchangeExec,
ENSURE_REQUIREMENTS, EnsureRequirements, Exchange, REPARTITION_BY_COL,
REPARTITION_BY_NUM, ReusedExchangeExec, ShuffleExchangeExec,
ShuffleExchangeLike, ShuffleOrigin}
+import org.apache.spark.sql.execution.exchange.{BroadcastExchangeExec,
ENSURE_REQUIREMENTS, EnsureRequirements, Exchange, REPARTITION_BY_COL,
REPARTITION_BY_NUM, ReusedExchangeExec, ShuffleExchangeExec,
ShuffleExchangeLike, ShuffleOrigin, ValidateRequirements}
import org.apache.spark.sql.execution.joins.{BaseJoinExec,
BroadcastHashJoinExec, BroadcastNestedLoopJoinExec,
HashedRelationBroadcastMode, ShuffledHashJoinExec, ShuffledJoin,
SortMergeJoinExec}
import org.apache.spark.sql.execution.metric.SQLShuffleReadMetricsReporter
import org.apache.spark.sql.execution.streaming.runtime.{MemoryStream,
StreamingQueryWrapper}
@@ -1901,6 +1901,155 @@ class AdaptiveQueryExecSuite
}
}
+ test("SPARK-60458: skip only the skewed join whose split the plan cannot
take") {
+ withSQLConf(
+ SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "true",
+ SQLConf.AUTO_BROADCASTJOIN_THRESHOLD.key -> "-1",
+ SQLConf.SKEW_JOIN_SKEWED_PARTITION_THRESHOLD.key -> "100",
+ SQLConf.ADVISORY_PARTITION_SIZE_IN_BYTES.key -> "100") {
+ withTempView("skewData1", "skewData2") {
+ spark
+ .range(0, 1000, 1, 10)
+ .selectExpr("id % 3 as key1", "id as value1")
+ .createOrReplaceTempView("skewData1")
+ spark
+ .range(0, 1000, 1, 10)
+ .selectExpr("id % 1 as key2", "id as value2")
+ .createOrReplaceTempView("skewData2")
+
+ val query =
+ """
+ |SELECT key1, count(*) FROM skewData1 JOIN skewData2 ON key1 =
key2 GROUP BY key1
+ |UNION ALL
+ |SELECT key1, 1 FROM skewData1 JOIN skewData2 ON key1 = key2
+ |""".stripMargin
+
+ // The aggregate above the first join needs its input on `key1`, which
the split breaks.
+ // The second join feeds the union directly, so its split breaks
nothing: it is kept.
+ val (_, adaptivePlan) = runAdaptiveAndVerifyResult(query)
+
+ // key2 is always 0, so only the 334 rows of key1 = 0 join, each
against all 1000 rows of
+ // the other side: the aggregated branch collapses them into one row,
the other keeps one
+ // row per joined row.
+ val rows = sql(query).collect()
+ assert(rows.length == 334001)
+ assert(rows.count(_ == Row(0L, 334000L)) == 1)
+ assert(rows.count(_ == Row(0L, 1L)) == 334000)
+
+ val rejectedJoins = collect(adaptivePlan) {
+ case agg: BaseAggregateExec => findTopLevelSortMergeJoin(agg)
+ }.flatten.distinct
+ assert(rejectedJoins.size == 1, adaptivePlan.toString)
+ val rejected = rejectedJoins.head
+ assert(!rejected.isSkewJoin)
+ // The rejected join does not lose partition coalescing to the kept
split.
+ val rejectedReads = collect(rejected) { case r: AQEShuffleReadExec =>
r }
+ assert(rejectedReads.size == 2 &&
rejectedReads.forall(_.hasCoalescedPartition),
+ adaptivePlan.toString)
+
+ val keptJoins = findTopLevelSortMergeJoin(adaptivePlan).filterNot(_ eq
rejected)
+ assert(keptJoins.size == 1, adaptivePlan.toString)
+ assert(keptJoins.head.isSkewJoin)
+ assert(collect(keptJoins.head) {
+ case r: AQEShuffleReadExec => r
+ }.exists(_.hasSkewedPartition))
+ assert(ValidateRequirements.validate(adaptivePlan))
+
+ // Turning the fallback off gives the plan back to the all-or-nothing
path.
+ withSQLConf(SQLConf.SKEW_JOIN_PER_JOIN_FALLBACK_ENABLED.key ->
"false") {
+ val (_, withoutFallback) = runAdaptiveAndVerifyResult(query)
+ val joins = findTopLevelSortMergeJoin(withoutFallback)
+ assert(joins.size == 2 && joins.forall(!_.isSkewJoin),
withoutFallback.toString)
+ }
+ }
+ }
+ }
+
+ test("SPARK-60458: keep the layout a union passes through") {
+ withSQLConf(
+ SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "true",
+ SQLConf.AUTO_BROADCASTJOIN_THRESHOLD.key -> "-1",
+ SQLConf.SKEW_JOIN_SKEWED_PARTITION_THRESHOLD.key -> "100",
+ SQLConf.ADVISORY_PARTITION_SIZE_IN_BYTES.key -> "100") {
+ withTempView("skewData1", "skewData2") {
+ spark
+ .range(0, 1000, 1, 10)
+ .selectExpr("id % 3 as key1", "id as value1")
+ .createOrReplaceTempView("skewData1")
+ spark
+ .range(0, 1000, 1, 10)
+ .selectExpr("id % 1 as key2", "id as value2")
+ .createOrReplaceTempView("skewData2")
+
+ def splitJoins(query: String): Seq[SortMergeJoinExec] = {
+ val (_, adaptivePlan) = runAdaptiveAndVerifyResult(query)
+ assert(ValidateRequirements.validate(adaptivePlan))
+ val joins = findTopLevelSortMergeJoin(adaptivePlan)
+ assert(joins.size == 2, adaptivePlan.toString)
+ joins
+ }
+
+ val union =
+ """
+ |SELECT key1 AS k, value1 AS v FROM skewData1 JOIN skewData2 ON
key1 = key2
+ |UNION ALL
+ |SELECT key1 AS k, value1 AS v FROM skewData1 JOIN skewData2 ON
key1 = key2
+ |""".stripMargin
+ val windowedQuery =
+ s"SELECT k, rank() OVER (PARTITION BY k ORDER BY v) FROM ($union) u"
+
+ // Both branches report the join's hash layout, so the union claims it
and the window above
+ // depends on it: a split takes the claim away from both branches. The
all-at-once attempt
+ // rejects this plan too; this arm pins that the per-join fallback
keeps no split the claim
+ // cannot take, and the arm below that the same splits are kept when
nothing reads the
+ // claim.
+ val windowed = splitJoins(windowedQuery)
+ assert(windowed.forall(j => !j.isSkewJoin))
+ // Each branch holds the 334000 joined rows of key1 = 0, so across the
union every v of
+ // 0, 3, ..., 999 appears 2000 times and shares its rank with its
duplicates.
+ val windowedRows = sql(windowedQuery).collect()
+ assert(windowedRows.length == 668000)
+ assert(windowedRows.forall(_.getLong(0) == 0L))
+
assert(windowedRows.groupBy(_.getInt(1)).view.mapValues(_.length).toMap ==
+ (1 to 666001 by 2000).map(_ -> 2000).toMap)
+
+ // Nothing above reads the union's layout, so the same splits break
nothing and are kept.
+ val plain = splitJoins(union)
+ assert(plain.forall(_.isSkewJoin))
+ val plainRows = sql(union).collect()
+ assert(plainRows.length == 668000)
+ assert(plainRows.groupBy(_.getLong(1)).view.mapValues(_.length).toMap
==
+ (0L until 1000L by 3).map(_ -> 2000).toMap)
+ }
+ }
+ }
+
+ test("SPARK-60458: split a join nested in a broadcast join's build side") {
+ // A broadcast-hash join constrains only its streamed side, so another
candidate can sit in
+ // its build side. Replacing the outer join must keep the inner split
instead of dropping the
+ // outer one and carrying the original build subtree.
+ val k1 = AttributeReference("k1", IntegerType)()
+ val k2 = AttributeReference("k2", IntegerType)()
+ val innerJoin = SortMergeJoinExec(
+ Seq(k1), Seq(k2), Inner, None,
+ LocalTableScanExec(Seq(k1), Nil, None),
+ LocalTableScanExec(Seq(k2), Nil, None))
+ val bhj = BroadcastHashJoinExec(
+ Seq(k2), Seq(k2), Inner, BuildRight, None,
+ LocalTableScanExec(Seq(k2), Nil, None),
+ BroadcastExchangeExec(HashedRelationBroadcastMode(Seq(k2)), innerJoin))
+ val rule = OptimizeSkewedJoin(EnsureRequirements())
+
+ val result = rule.applySkewJoins(bhj, Seq(
+ SkewJoinCandidate(innerJoin, innerJoin.copy(isSkewJoin = true)),
+ SkewJoinCandidate(bhj, bhj.copy(isSkewJoin = true))))
+
+ assert(result.isInstanceOf[BroadcastHashJoinExec])
+ assert(result.asInstanceOf[BroadcastHashJoinExec].isSkewJoin)
+ val innerJoins = findTopLevelSortMergeJoin(result)
+ assert(innerJoins.size == 1 && innerJoins.head.isSkewJoin)
+ }
+
test("SPARK-58099: Remove local sort dangling below the shuffle added by
skew join") {
// A ShuffledHashJoin feeding a SortMergeJoin in the same stage. When the
SHJ is skewed and
// skew-join optimization is force-applied, an extra shuffle is inserted
between the two joins.
@@ -4741,11 +4890,11 @@ class AdaptiveQueryExecSuite
"UNION ALL SELECT key2 FROM skewData2 GROUP BY key2", 1, 1)
// skewJoin1 union (skewJoin2 join aggregate)
- // skewJoin2 will lead to extra shuffles, but skew1 cannot be optimized
- checkSkewJoin(
+ // skewJoin2's split would break the join above it, so only skewJoin2
is skipped
+ checkSkewJoin(
"SELECT key1 FROM skewData1 JOIN skewData2 ON key1 = key2 UNION ALL
" +
"SELECT key1 from (SELECT key1 FROM skewData1 JOIN skewData2 ON
key1 = key2) tmp1 " +
- "JOIN (SELECT key2 FROM skewData2 GROUP BY key2) tmp2 ON key1 =
key2", 3, 0)
+ "JOIN (SELECT key2 FROM skewData2 GROUP BY key2) tmp2 ON key1 =
key2", 3, 1)
}
}
}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]