github-actions[bot] commented on code in PR #66473:
URL: https://github.com/apache/doris/pull/66473#discussion_r3754922356
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/hive/source/HiveScanNode.java:
##########
@@ -320,8 +322,17 @@ private void
getFileSplitByPartitions(HiveExternalMetaCache cache, List<HivePart
}
} else {
boolean withCache = Config.max_external_file_cache_num > 0;
- fileCaches = cache.getFilesByPartitions(partitions, withCache,
partitions.size() > 1,
- directoryLister, hmsTable);
+ HiveFileScanTaskCacheKey cacheKey = new HiveFileScanTaskCacheKey(
+ hmsTable.getCatalog().getId(), hmsTable.getId(),
partitions);
+ try {
+ fileCaches = getOrLoadExternalScanTasks(cacheKey,
Review Comment:
[P1] Preserve bounded memory in the batch split paths
When Hive switches to batch mode, `startSplit` intentionally limits
in-flight partition planning, but this call stores every singleton partition's
`FileCacheValue` list in the statement cache until the whole statement closes.
Completed partitions therefore never become collectible; on a wide scan the map
grows with the entire table and can even pin entries after the capped global
Hive file cache evicts them. Hudi's per-partition `getPartitionSplits` has the
same retention pattern for `HudiSplit` graphs. The Iceberg manifest-cache
production branch also uses this cache in batch/explicit-size modes (the new
direct `splitFiles()` bypass test does not cover that dispatch). Please bypass
statement caching for streaming/batch planning, or evict entries when their
batch is queued/consumed, so the existing bounded-memory contract remains
intact.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
