maryannxue commented on a change in pull request #27669: [SPARK-30918][SQL]
improve the splitting of skewed partitions
URL: https://github.com/apache/spark/pull/27669#discussion_r382679191
##########
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/OptimizeSkewedJoin.scala
##########
@@ -34,6 +34,29 @@ import
org.apache.spark.sql.execution.exchange.{EnsureRequirements, ShuffleExcha
import org.apache.spark.sql.execution.joins.SortMergeJoinExec
import org.apache.spark.sql.internal.SQLConf
+/**
+ * A rule to optimize skewed joins to avoid one or a little tasks processing
most of the data.
+ *
+ * The general idea is to treat a Sort-Merge join as many sub-joins that each
sub-join processes
Review comment:
It's not accurate to say "sub-join" here. How about:
The general idea is to divide each skew partition into smaller partitions
and replicate its matching partition on the other side of the join so that they
can run in parallel tasks. Note that when matching partitions from the left
side and the right side both have skew, it will become a cartesian product of
splits from left and right joining together.
And let's replace the term "sub-join" accordingly in the example below as
well.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
