This is an automated email from the ASF dual-hosted git repository.
wenchen pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new fe4f8eac3efe [SPARK-46590][SQL][FOLLOWUP] Update
CoalesceShufflePartitions comments
fe4f8eac3efe is described below
commit fe4f8eac3efee42d53f7f24763a59c82ef03d343
Author: zml1206 <[email protected]>
AuthorDate: Wed Jan 24 15:06:55 2024 +0800
[SPARK-46590][SQL][FOLLOWUP] Update CoalesceShufflePartitions comments
### What changes were proposed in this pull request?
After #44661 ,In addition to Union, children of CartesianProduct,
BroadcastHashJoin and BroadcastNestedLoopJoin can also be coalesced
independently, update comments.
### Why are the changes needed?
Improve the readability and maintainability.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
N/A
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes #44854 from zml1206/SPARK-46590-FOLLOWUP.
Authored-by: zml1206 <[email protected]>
Signed-off-by: Wenchen Fan <[email protected]>
---
.../sql/execution/adaptive/CoalesceShufflePartitions.scala | 11 ++++++-----
1 file changed, 6 insertions(+), 5 deletions(-)
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/CoalesceShufflePartitions.scala
b/sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/CoalesceShufflePartitions.scala
index 26e5ac649dbb..db4a6b7dcf2e 100644
---
a/sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/CoalesceShufflePartitions.scala
+++
b/sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/CoalesceShufflePartitions.scala
@@ -66,9 +66,9 @@ case class CoalesceShufflePartitions(session: SparkSession)
extends AQEShuffleRe
}
}
- // Sub-plans under the Union operator can be coalesced independently, so
we can divide them
- // into independent "coalesce groups", and all shuffle stages within each
group have to be
- // coalesced together.
+ // Sub-plans under the
Union/CartesianProduct/BroadcastHashJoin/BroadcastNestedLoopJoin
+ // operator can be coalesced independently, so we can divide them into
independent
+ // "coalesce groups", and all shuffle stages within each group have to be
coalesced together.
val coalesceGroups = collectCoalesceGroups(plan)
// Divide minimum task parallelism among coalesce groups according to
their data sizes.
@@ -137,8 +137,9 @@ case class CoalesceShufflePartitions(session: SparkSession)
extends AQEShuffleRe
}
/**
- * Gather all coalesce-able groups such that the shuffle stages in each
child of a Union operator
- * are in their independent groups if:
+ * Gather all coalesce-able groups such that the shuffle stages in each
child of a
+ * Union/CartesianProduct/BroadcastHashJoin/BroadcastNestedLoopJoin operator
are in their
+ * independent groups if:
* 1) all leaf nodes of this child are exchange stages; and
* 2) all these shuffle stages support coalescing.
*/
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
