924060929 commented on code in PR #66473:
URL: https://github.com/apache/doris/pull/66473#discussion_r3780924364
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/hive/source/HiveScanNode.java:
##########
@@ -320,8 +324,29 @@ private void
getFileSplitByPartitions(HiveExternalMetaCache cache, List<HivePart
}
} else {
boolean withCache = Config.max_external_file_cache_num > 0;
- fileCaches = cache.getFilesByPartitions(partitions, withCache,
partitions.size() > 1,
- directoryLister, hmsTable);
+ if (isBatchMode || !withCache) {
+ // Batch mode bounds FE memory by retaining only the
partitions currently in flight.
+ // Keeping every completed partition in the statement cache
would materialize the
+ // full scan again and defeat that bound. When the global file
cache is disabled,
+ // statement retention must not become an uncapped replacement
for that memory fence.
+ fileCaches = cache.getFilesByPartitions(partitions, withCache,
partitions.size() > 1,
+ directoryLister, hmsTable);
+ } else {
+ HiveFileScanTaskCacheKey cacheKey = new
HiveFileScanTaskCacheKey(
Review Comment:
Fixed in 7b739e0e344a60a719fc8f5ce361c88fa0229562. The existing Hive
file-cache invalidation generation is now advanced immediately before each
corresponding catalog, database, table, and partition file-cache invalidation.
Added a focused HiveMetaStoreCacheTest that observes the generation from inside
the invalidation predicate and verifies it has already advanced before the
cached entry is removed. Also ran HiveScanNodeTest (12/12),
HiveMetaStoreCacheTest (3/3), git diff --check, and the full FE build with JDK
17 successfully.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
