imback82 commented on a change in pull request #29473:
URL: https://github.com/apache/spark/pull/29473#discussion_r479590003
##########
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/bucketing/CoalesceOrRepartitionBucketsInJoin.scala
##########
@@ -83,23 +82,39 @@ case class CoalesceBucketsInJoin(conf: SQLConf) extends
Rule[SparkPlan] {
}
def apply(plan: SparkPlan): SparkPlan = {
- if (!conf.coalesceBucketsInJoinEnabled) {
+ if (!conf.coalesceBucketsInJoinEnabled &&
!conf.repartitionBucketsInJoinEnabled) {
return plan
}
+ if (conf.coalesceBucketsInJoinEnabled &&
conf.repartitionBucketsInJoinEnabled) {
+ throw new AnalysisException("Both
'spark.sql.bucketing.coalesceBucketsInJoin.enabled' and " +
+ "'spark.sql.bucketing.repartitionBucketsInJoin.enabled' cannot be set
to true at the" +
+ "same time")
Review comment:
Thanks for the suggestion! I think the new config makes more sense. I
renamed few, and let me know if it doesn't make sense.
Btw, do you think I can introduce `AUTOMATIC` as a follow up since this PR
is sizable? Let me know if you want to see it in this PR. Thanks.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
