maryannxue commented on a change in pull request #27226: [SPARK-30524] [SQL]
Disable OptimizeSkewedJoin rule when introducing additional shuffle
URL: https://github.com/apache/spark/pull/27226#discussion_r367521851
##########
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/OptimizeSkewedJoin.scala
##########
@@ -204,11 +245,27 @@ case class OptimizeSkewedJoin(conf: SQLConf) extends
Rule[SparkPlan] {
val shuffleStages = collectShuffleStages(plan)
if (shuffleStages.length == 2) {
- // Currently we only support handling skewed join for 2 table join.
- handleSkewJoin(plan)
+ // When multi table join, there will be too many complex combination to
consider.
+ // Currently we only handle 2 table join like following two use cases.
+ // SMJ SMJ
Review comment:
Sorry that my previous comment was wrong. Once we have shuffle, there should
always be a sort. So we don't need to match this.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
