github-actions[bot] commented on code in PR #66473:
URL: https://github.com/apache/doris/pull/66473#discussion_r3774982788
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/hive/source/HiveScanNode.java:
##########
@@ -320,8 +324,29 @@ private void
getFileSplitByPartitions(HiveExternalMetaCache cache, List<HivePart
}
} else {
boolean withCache = Config.max_external_file_cache_num > 0;
- fileCaches = cache.getFilesByPartitions(partitions, withCache,
partitions.size() > 1,
- directoryLister, hmsTable);
+ if (isBatchMode || !withCache) {
+ // Batch mode bounds FE memory by retaining only the
partitions currently in flight.
+ // Keeping every completed partition in the statement cache
would materialize the
+ // full scan again and defeat that bound. When the global file
cache is disabled,
+ // statement retention must not become an uncapped replacement
for that memory fence.
+ fileCaches = cache.getFilesByPartitions(partitions, withCache,
partitions.size() > 1,
+ directoryLister, hmsTable);
+ } else {
+ HiveFileScanTaskCacheKey cacheKey = new
HiveFileScanTaskCacheKey(
Review Comment:
[P1] Fence Hive statement tasks against file-cache invalidation
This key captures only table and partition identity. If alias A hits a
global file entry populated by an earlier query, this query's directory cache
remains empty; after `HiveExternalMetaCache.invalidateTable()` or partition
invalidation, alias B used to miss globally and re-list, but the equal
statement key now returns A's retained `FileCacheValue`. HMS notifications,
manual refresh, and post-commit writes can run between sequential scan-node
finalizations, and ordinary Hive has no file-list MVCC snapshot, so this can
miss added files or schedule removed/replaced files. Please include a
file-entry invalidation generation ordered with global invalidation (or
synchronously invalidate the relevant statement entries), and cover equal
aliases separated by table and partition invalidation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
