github-actions[bot] commented on code in PR #66913:
URL: https://github.com/apache/doris/pull/66913#discussion_r3829547282
##########
fe/fe-core/src/main/java/org/apache/doris/qe/StmtExecutor.java:
##########
@@ -969,16 +971,64 @@ public boolean isDeferredForArrowFlight() {
return deferredForArrowFlight;
}
+ void deferArrowFlightQuery() {
+ Closeable resources = statementContext.detachStatementResources();
+ deferredArrowFlightStatementResources = resources;
+ deferredForArrowFlight = true;
+ try {
+ context.addFlightSqlDeferredExecutor(this);
Review Comment:
[P1] Seal deferred registration against session teardown
Session teardown can run concurrently here: token eviction or `CloseSession`
may drain an empty deferred-executor list and remove this `ConnectContext`
after `deferredForArrowFlight` is set but before this append. The registry has
no sealed state, so the late add succeeds into an unreachable context; both
eager cleanup sites then skip this executor, leaving the coordinator/query
registration and detached Hudi/Iceberg resources without a terminal owner.
Please make registration atomic with teardown (seal the registry under the same
lock and reject post-seal adds into immediate exact-once finalization), with a
barrier test that drains between detachment and registration.
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/hive/HMSExternalCatalog.java:
##########
@@ -233,12 +249,59 @@ public void setDefaultPropsIfMissing(boolean isReplay) {
}
}
- public IcebergMetadataOps getIcebergMetadataOps() {
+ public synchronized IcebergMetadataOps getIcebergMetadataOps() {
makeSureInitialized();
if (icebergMetadataOps == null) {
HiveCatalog icebergHiveCatalog =
IcebergUtils.createIcebergHiveCatalog(this, getName());
icebergMetadataOps =
ExternalMetadataOperations.newIcebergMetadataOps(this, icebergHiveCatalog);
}
return icebergMetadataOps;
}
+
+ /** Retains the exact HMS Iceberg runtime while a table cache generation
is being loaded or borrowed. */
+ public synchronized IcebergTableLoadContext beginIcebergTableLoad() {
+ makeSureInitialized();
+ IcebergMetadataOps ops = getIcebergMetadataOps();
+ return new IcebergTableLoadContext(ops, threadPoolWithPreAuth,
icebergResourceTracker.beginLoad());
+ }
+
+ @Override
+ public synchronized void resetToUninitialized(boolean invalidCache) {
+
Env.getCurrentEnv().getExtMetaCacheMgr().removeCatalogByEngine(getId(),
IcebergExternalMetaCache.ENGINE);
Review Comment:
[P1] Fence the complete Hudi/Hive runtime generation on reset
This reset removes only Iceberg, so ordinary property updates can leave the
Hudi G1 meta client/view and the Hive G1 file enumeration reachable by
post-reset scans. Rotating both groups is necessary, but it is not sufficient
for in-flight scans: Hudi captures its client/instant during initialization,
acquires the fs view later, and rereads mutable authenticator/storage
properties; Hive retains file tasks while a same-ID group rebuild resets its
generation counters, allowing a G2 key to collide with retained G1 work. This
can mix endpoint A names/runtime state with endpoint B current properties.
Please rotate both groups and carry one exact HMS runtime generation through
initialization, split planning, and range serialization (or detect the fence
and restart all phases), while keeping Hive statement generations monotonic.
Add endpoint-rotation tests for both new and already-started Hudi/Hive scans.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
