c21 commented on a change in pull request #32482:
URL: https://github.com/apache/spark/pull/32482#discussion_r628934263
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/CachedTableSuite.scala
##########
@@ -1554,4 +1554,39 @@ class CachedTableSuite extends QueryTest with
SQLTestUtils
assert(!spark.catalog.isCached(viewName))
}
}
+
+ test("SPARK-35332: Make cache plan disable configs configurable") {
+ val msg = intercept[IllegalArgumentException] {
+ spark.sessionState.conf.setConfString(SQLConf.CACHE_DISABLE_CONFIGS.key,
+ SQLConf.COALESCE_PARTITIONS_MIN_PARTITION_NUM.key)
+ }.getMessage
+ assert(msg.contains("config should be boolean type"))
+
+ Seq(SQLConf.ADAPTIVE_EXECUTION_ENABLED.key, "").foreach { disableConfig =>
Review comment:
could we also test for auto bucketed scan? thanks.
##########
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
##########
@@ -1090,6 +1090,18 @@ object SQLConf {
.booleanConf
.createWithDefault(true)
+ val CACHE_DISABLE_CONFIGS =
+ buildConf("spark.sql.cache.disableConfigs")
+ .doc("Configurations needs to be turned off, to avoid regression for
cached query, so that " +
+ "the outputPartitioning of the underlying cached query plan can be
leveraged later.")
+ .version("3.2.0")
+ .stringConf
+ .toSequence
+ .checkValue(_.forall(v => sqlConfEntries.containsKey(v) &&
+ sqlConfEntries.get(v).defaultValue.exists(_.isInstanceOf[Boolean])),
+ "config should be boolean type")
+ .createWithDefault(Seq(ADAPTIVE_EXECUTION_ENABLED.key,
AUTO_BUCKETED_SCAN_ENABLED.key))
Review comment:
+1, I agree with starting with simpler config format if possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
