maropu commented on a change in pull request #29079:
URL: https://github.com/apache/spark/pull/29079#discussion_r454827484
##########
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
##########
@@ -2645,21 +2645,22 @@ object SQLConf {
.booleanConf
.createWithDefault(true)
- val COALESCE_BUCKETS_IN_SORT_MERGE_JOIN_ENABLED =
- buildConf("spark.sql.bucketing.coalesceBucketsInSortMergeJoin.enabled")
+ val COALESCE_BUCKETS_IN_JOIN_ENABLED =
+ buildConf("spark.sql.bucketing.coalesceBucketsInJoin.enabled")
.doc("When true, if two bucketed tables with the different number of
buckets are joined, " +
"the side with a bigger number of buckets will be coalesced to have
the same number " +
- "of buckets as the other side. Bucket coalescing is applied only to
sort-merge joins " +
- "and only when the bigger number of buckets is divisible by the
smaller number of buckets.")
+ "of buckets as the other side. Bigger number of buckets is divisible
by the smaller " +
+ "number of buckets. Bucket coalescing is applied to sort-merge joins
and " +
+ "shuffled hash join.")
.version("3.1.0")
.booleanConf
.createWithDefault(false)
- val COALESCE_BUCKETS_IN_SORT_MERGE_JOIN_MAX_BUCKET_RATIO =
-
buildConf("spark.sql.bucketing.coalesceBucketsInSortMergeJoin.maxBucketRatio")
+ val COALESCE_BUCKETS_IN_JOIN_MAX_BUCKET_RATIO =
+ buildConf("spark.sql.bucketing.coalesceBucketsInJoin.maxBucketRatio")
Review comment:
Is it okay to share this parameter between sort-merge/hash joins? As
@viirya suggested, we have some risk of OOM. So, I think we need a different
threshold policy for the hash-join case.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
