github-actions[bot] commented on code in PR #66913:
URL: https://github.com/apache/doris/pull/66913#discussion_r3891853940
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergExternalMetaCache.java:
##########
@@ -133,7 +144,32 @@ IcebergSnapshotEntryKey.class,
IcebergSnapshotCacheValue.class, defaultEntryCach
public Table getIcebergTable(ExternalTable dorisTable) {
NameMapping nameMapping = dorisTable.getOrBuildNameMapping();
- return
tableEntry.get(nameMapping.getCtlId()).get(nameMapping).getIcebergTable();
+ IcebergTableCacheValue.Lease lease = statementLease(nameMapping);
+ if (lease != null) {
+ return lease.getIcebergTable();
+ }
+ // Background callers have no deterministic statement boundary. Use a
live catalog load
+ // instead of returning a cache generation that can be evicted
immediately after lookup.
+ return getWritableIcebergTable(dorisTable);
+ }
+
+ ThreadPoolExecutor getIcebergTableExecutor(ExternalTable dorisTable) {
+ IcebergTableCacheValue.Lease lease =
statementLease(dorisTable.getOrBuildNameMapping());
+ if (lease == null || lease.getPlanningExecutor() == null) {
+ return dorisTable.getCatalog().getThreadPoolWithPreAuth();
+ }
+ return lease.getPlanningExecutor();
Review Comment:
[P1] Recheck the HMS generation before promoting this load. The
IcebergExternalCatalog branch fences after `context.loadTable()`, but this HMS
branch promotes immediately; an ALTER/reset during the remote load can replace
ops/authenticator/mapping settings and retire the tracker while this old table
is still published. Call `ensureCatalogGenerationStable` with the captured HMS
ops/authenticator/options before `context.promote()`, and add a
reset-during-HMS-load regression.
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergMetadataOps.java:
##########
@@ -87,12 +88,15 @@
import java.util.ArrayList;
import java.util.Arrays;
import java.util.Collections;
+import java.util.HashMap;
Review Comment:
[P1] Hold the generation lease through every Iceberg table commit. This new
scoped lease protects only `updateTableProperties`; branch/tag, schema, and
partition-spec methods still call `getWritableIcebergTable(dorisTable, this)`,
which releases its temporary guard after load before
`ManageSnapshots`/`UpdateSchema`/`UpdatePartitionSpec.commit()`. A concurrent
reset can close the retired catalog/FileIO during the remote commit. Route all
table-handle mutations through `acquireWritableIcebergTable` (or an equivalent
guard) and add reset-barrier tests.
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/hudi/source/HudiScanNode.java:
##########
@@ -27,6 +27,7 @@
import org.apache.doris.catalog.Type;
import org.apache.doris.common.AnalysisException;
import org.apache.doris.common.UserException;
+import org.apache.doris.common.security.authentication.ExecutionAuthenticator;
Review Comment:
[P1] Recheck the HMS generation after resolving the schema cache. This guard
runs before `getSchemaCacheValue()`, which can lazily load a replacement cache
after a reset; `setHudiParams()` then writes schemaId/history into the range
descriptor without another fence. Batch scheduling can therefore combine G1
split fields with G2 schema metadata. Carry the captured schema generation or
call `ensureHmsRuntimeGeneration()` after schema conversion and before
publishing the descriptor, with a reset-barrier regression.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
