maropu commented on a change in pull request #29804:
URL: https://github.com/apache/spark/pull/29804#discussion_r495698286
##########
File path:
sql/core/src/test/scala/org/apache/spark/sql/sources/BucketedReadSuite.scala
##########
@@ -1012,4 +1014,182 @@ abstract class BucketedReadSuite extends QueryTest with
SQLTestUtils {
}
}
}
+
+ private def checkDisableBucketedScan(
+ query: String,
+ expectedNumBucketedScanEnabled: Int,
Review comment:
nit: This param name looks a bit confusing and this means the number of
bucketed scans when `AUTO_BUCKETED_SCAN_ENABLED` enabled?
##########
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
##########
@@ -951,6 +951,14 @@ object SQLConf {
.checkValue(_ > 0, "the value of spark.sql.sources.bucketing.maxBuckets
must be greater than 0")
.createWithDefault(100000)
+ val AUTO_BUCKETED_SCAN_ENABLED =
+ buildConf("spark.sql.sources.bucketing.autoBucketedScan.enabled")
+ .doc("When true, decide whether to do bucketed scan on input tables
based on query plan " +
+ "automatically.")
+ .version("3.1.0")
Review comment:
btw, we need to make this config external? If we just add this config
for keeping the current behaviour, is it okay to add it as internal one?
##########
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
##########
@@ -951,6 +951,14 @@ object SQLConf {
.checkValue(_ > 0, "the value of spark.sql.sources.bucketing.maxBuckets
must be greater than 0")
.createWithDefault(100000)
+ val AUTO_BUCKETED_SCAN_ENABLED =
+ buildConf("spark.sql.sources.bucketing.autoBucketedScan.enabled")
+ .doc("When true, decide whether to do bucketed scan on input tables
based on query plan " +
Review comment:
Since our user documents are generated based on this statement, could
you describe a bit more about how to decide whether to do bucketed scans or not?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
