924060929 commented on code in PR #66913:
URL: https://github.com/apache/doris/pull/66913#discussion_r3894206495
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergExternalCatalog.java:
##########
@@ -173,17 +221,108 @@ protected List<String>
listTableNamesFromRemote(SessionContext ctx, String dbNam
}
@Override
- public void onClose() {
+ public synchronized void onClose() {
+ ThreadPoolExecutor retiredExecutor = threadPoolWithPreAuth;
+ threadPoolWithPreAuth = null;
super.onClose();
- if (null != catalog) {
- try {
- if (catalog instanceof AutoCloseable) {
- ((AutoCloseable) catalog).close();
- }
- catalog = null;
- } catch (Exception e) {
- LOG.warn("Failed to close iceberg catalog: {}", getName(), e);
+ Catalog retiredCatalog = catalog;
+ catalog = null;
+ resourceTracker.retireCurrent(() -> {
Review Comment:
Already covered on the current head: every Iceberg catalog operation enters
executeCatalogOperation, which holds either
IcebergExternalCatalog.beginCatalogOperation(this) or
HMSExternalCatalog.beginIcebergCatalogOperation(this) through the authenticated
remote catalog call. The generation therefore cannot retire during
create/drop/rename table, namespace, or view I/O. No additional change was
needed for this thread.
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/hudi/source/HudiScanNode.java:
##########
@@ -552,59 +634,311 @@ private void initPrunedPartitions() throws UserException
{
throw new UserException(ExceptionUtils.getRootCauseMessage(e), e);
}
partitionInit = true;
+ ensureHmsRuntimeGeneration();
}
@Override
public void startSplit(int numBackends) {
+ ensureHmsRuntimeGeneration();
if (prunedPartitions.isEmpty()) {
splitAssignment.finishSchedule();
+ releaseFsViewOnce();
return;
}
- AtomicInteger numFinishedPartitions = new AtomicInteger(0);
+ acquireFsView();
Review Comment:
Already fixed in 69cf0c1b77c: the fs-view generation captures its HMS
authenticator, cold load and sync run under that captured identity, and the
exact fs-view generation remains retained until the scan node releases it. The
deterministic cold-load/reset test covers the handoff and stale-view close path.
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergExternalMetaCache.java:
##########
@@ -133,7 +144,32 @@ IcebergSnapshotEntryKey.class,
IcebergSnapshotCacheValue.class, defaultEntryCach
public Table getIcebergTable(ExternalTable dorisTable) {
NameMapping nameMapping = dorisTable.getOrBuildNameMapping();
- return
tableEntry.get(nameMapping.getCtlId()).get(nameMapping).getIcebergTable();
+ IcebergTableCacheValue.Lease lease = statementLease(nameMapping);
+ if (lease != null) {
+ return lease.getIcebergTable();
+ }
+ // Background callers have no deterministic statement boundary. Use a
live catalog load
+ // instead of returning a cache generation that can be evicted
immediately after lookup.
+ return getWritableIcebergTable(dorisTable);
+ }
+
+ ThreadPoolExecutor getIcebergTableExecutor(ExternalTable dorisTable) {
+ IcebergTableCacheValue.Lease lease =
statementLease(dorisTable.getOrBuildNameMapping());
+ if (lease == null || lease.getPlanningExecutor() == null) {
+ return dorisTable.getCatalog().getThreadPoolWithPreAuth();
+ }
+ return lease.getPlanningExecutor();
Review Comment:
No additional change is needed here: the HMS writable-table path already
calls ensureCatalogGenerationStable(...) immediately after context.loadTable()
and before context.promote(). A reset during the remote load therefore rejects
the stale table instead of publishing it into the replacement generation.
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/hudi/source/HudiScanNode.java:
##########
@@ -27,6 +27,7 @@
import org.apache.doris.catalog.Type;
import org.apache.doris.common.AnalysisException;
import org.apache.doris.common.UserException;
+import org.apache.doris.common.security.authentication.ExecutionAuthenticator;
Review Comment:
Fixed in 32043d3f448: Hudi schema resolution now completes into a local
InternalSchema, then rechecks the exact HMS runtime generation before
publishing query history, schemaId, table-format parameters, or the range
descriptor. A deterministic test resets during getCommitInstantInternalSchema()
and verifies that descriptor publication is rejected.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
