maropu commented on a change in pull request #28852:
URL: https://github.com/apache/spark/pull/28852#discussion_r443083542
##########
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
##########
@@ -835,6 +835,15 @@ object SQLConf {
.longConf
.createWithDefault(250 * 1024 * 1024)
+ val HIVE_FILESOURCE_PARTITION_FILE_CACHE_TTL =
+ buildConf("spark.sql.hive.filesourcePartitionFileCacheTTL")
Review comment:
> @gatorsmile could you elaborate a bit? Are you just referring to
dropping the hive prefix or is there something else I'm missing? Thanks
We have the cache mechanism for metadata-related items, e.g.,
https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalog.scala#L138
IIUC he suggested its better for them to have the same TTL.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
