dutyu commented on code in PR #18102:
URL: https://github.com/apache/doris/pull/18102#discussion_r1148316840
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/hive/HiveMetaStoreCache.java:
##########
@@ -120,14 +121,22 @@ public HivePartition load(PartitionCacheKey key) throws
Exception {
}
}, executor));
- fileCache =
CacheBuilder.newBuilder().maximumSize(Config.max_external_file_cache_num)
-
.expireAfterAccess(Config.external_cache_expire_time_minutes_after_access,
TimeUnit.MINUTES)
- .build(CacheLoader.asyncReloading(new
CacheLoader<FileCacheKey, ImmutableList<InputSplit>>() {
- @Override
- public ImmutableList<InputSplit> load(FileCacheKey key)
throws Exception {
- return loadFiles(key);
- }
- }, executor));
+ // if the file.meta.cache.ttl-second is equal or greater than 0, the
cache expired will be set to that value
+ int fileMetaCacheTtlSecond = NumberUtils.toInt(
+
(catalog.getProperties().getOrDefault(HMSExternalCatalog.FILE_META_CACHE_TTL_SECOND,
"")), HMSExternalCatalog.FILE_META_CACHE_NO_TTL);
+
+ CacheBuilder<Object, Object> fileCacheBuilder =
CacheBuilder.newBuilder().maximumSize(Config.max_external_file_cache_num);
Review Comment:
If `file.meta.cache.ttl-second` = '0', maybe we should not initialize the
loading cache, just set `fileCache` = null and let the current thread to load
the file splits. Because loading file splits of a hive table with many
partitions is a time-consuming task, it can easily drain the cache loader
thread pool(pool size = 10 now), it will block all querys with external tables
if any cache (partitionValuesCache/partitionCache/fileCache/schemaCache) is
expired.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
