cloud-fan commented on a change in pull request #27833: [SPARK-31070][SQL] make
skew join split skewed partitions more evenly
URL: https://github.com/apache/spark/pull/27833#discussion_r389820495
##########
File path:
sql/core/src/test/scala/org/apache/spark/sql/execution/adaptive/AdaptiveQueryExecSuite.scala
##########
@@ -641,11 +641,11 @@ class AdaptiveQueryExecSuite
// into 2 splits and right side is divided into 4 splits,
so
// 2 x 4 sub-partitions.
// Partition 1, 2, 3: not skewed, and coalesced into 1 partition.
- // Partition 4: only left side is skewed, and divide into 3 splits, so
- // 3 sub-partitions.
+ // Partition 4: only left side is skewed, and divide into 2 splits, so
+ // 2 sub-partitions.
Review comment:
This is definitely better with 2 splits, as the target size is 2000 and the
total size is 4014.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
