maropu commented on a change in pull request #28852:
URL: https://github.com/apache/spark/pull/28852#discussion_r445894796
##########
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
##########
@@ -2656,6 +2656,16 @@ object SQLConf {
.checkValue(_ > 0, "The difference must be positive.")
.createWithDefault(4)
+ val METADATA_CACHE_TTL = buildConf("spark.sql.metadataCacheTTL")
+ .doc("Time-to-live (TTL) value for the metadata caches: partition file
metadata cache and " +
+ "session catalog cache. This configuration only has an effect when
this value having " +
+ "a positive value. It also requires setting `hive` to " +
+ s"${StaticSQLConf.CATALOG_IMPLEMENTATION} to be applied to the
partition file " +
Review comment:
More conditions for this option to be enabled?
https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileStatusCache.scala#L43-L44
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
