maropu commented on a change in pull request #28123: [SPARK-31350][SQL]
Coalesce bucketed tables for join if applicable
URL: https://github.com/apache/spark/pull/28123#discussion_r406569745
##########
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/DataSourceScanExec.scala
##########
@@ -267,6 +268,23 @@ case class FileSourceScanExec(
}
}
+ /**
+ * A bucket can be coalesced if the number of buckets for this relation is
divisible
+ * by the number of buckets on the other side of table for join.
+ */
+ private lazy val coalescedNumBuckets: Option[Int] = {
Review comment:
Could we compute this value in the optimizer rule side and then set this
value in a constructor of `DataSourceScanExec` at that time? Probably, I think
we don't need to inject the value below in relation options.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
