github-actions[bot] commented on code in PR #66473:
URL: https://github.com/apache/doris/pull/66473#discussion_r3783052247
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/FileQueryScanNode.java:
##########
@@ -136,6 +142,46 @@ public FileQueryScanNode(PlanNodeId id, TupleDescriptor
desc, String planNodeNam
StatisticalType statisticalType, ScanContext scanContext, boolean
needCheckColumnPriv, SessionVariable sv) {
super(id, desc, planNodeName, statisticalType, scanContext,
needCheckColumnPriv);
this.sessionVariable = sv;
+ ConnectContext context = ConnectContext.get();
+ StatementContext statementContext = context == null ? null :
context.getStatementContext();
+ this.externalScanTaskCache = statementContext == null
+ ? null : statementContext.getExternalScanTaskCache();
+ }
+
+ protected <T> List<T> getOrLoadExternalScanTasks(
+ ExternalScanTaskCacheKey<T> key, Callable<List<T>> loader) throws
Exception {
+ if (!sessionVariable.enableExternalScanTaskReuse ||
externalScanTaskCache == null) {
+ return loader.call();
+ }
+ return externalScanTaskCache.getOrLoad(key, loader);
+ }
+
+ protected <T> List<T> getOrLoadExternalScanTasks(
+ ExternalScanTaskCacheKey<T> key, Callable<List<T>> loader,
+ ToLongFunction<List<T>> weigher, long maxRetainedWeight) throws
Exception {
+ if (!sessionVariable.enableExternalScanTaskReuse ||
externalScanTaskCache == null) {
+ return loader.call();
+ }
+ return externalScanTaskCache.getOrLoad(key, loader, weigher,
maxRetainedWeight);
+ }
+
+ protected <T> List<T> getOrLoadExternalScanTasks(
+ ExternalScanTaskCacheKey<T> key,
+ StatementContext.ExternalScanTaskCache.WeightedLoader<T> loader,
+ ToLongFunction<List<T>> weigher,
+ StatementContext.ExternalScanTaskCache.WeightBudget weightBudget,
+ long maxEntryWeight, long maxRetainedWeight,
+ boolean reserveBeforeLoad) throws Exception {
+ if (!sessionVariable.enableExternalScanTaskReuse ||
externalScanTaskCache == null) {
Review Comment:
[P2] Skip serialization when scan-task reuse is inactive
When reuse is disabled (or this scan node has no statement cache), this
still calls the cache-value loader with the full entry limit. Iceberg and
Paimon use that loader to serialize every fitting task list and then
immediately deserialize it, even though this path can never produce a cache
hit; only oversized plans reach the raw-task fallback. That leaves the session
opt-out paying the feature's full encode/decode and transient-buffer cost on
every ordinary scan. Please give these connectors a direct raw fallback that
invokes the planner exactly once, so serialization only runs when an entry can
actually be retained.
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/hive/HiveExternalMetaCache.java:
##########
@@ -119,6 +121,8 @@ public class HiveExternalMetaCache extends
AbstractExternalMetaCache {
public static final String ERR_CACHE_INCONSISTENCY =
"ERR_CACHE_INCONSISTENCY: ";
private final ExecutorService fileListingExecutor;
+ private final Map<Long, AtomicLong> fileCacheInvalidationGenerations = new
ConcurrentHashMap<>();
Review Comment:
[P2] Prune Hive generation state on terminal catalog removal
These process-wide generation maps are not pruned when a catalog is actually
dropped. `CatalogMgr` removes the catalog and routes
`ExternalMetaCacheMgr.removeCatalog()` to Hive; `invalidateCatalog()` then
removes the bounded catalog entry group but advances (and therefore retains) an
invalidation-generation tombstone, while any value-generation entry also
remains. Since replacement catalogs receive fresh IDs, repeated
create/load/drop churn grows this singleton state for the FE lifetime. Please
add terminal-removal-specific cleanup backed by never-reset generation fencing,
rather than resetting counters in the refresh-shaped invalidation path, so
dropped IDs can be discarded without allowing an in-flight old load to collide.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
