924060929 commented on code in PR #66913:
URL: https://github.com/apache/doris/pull/66913#discussion_r3891717968
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/hudi/source/HudiScanNode.java:
##########
@@ -552,59 +634,311 @@ private void initPrunedPartitions() throws UserException
{
throw new UserException(ExceptionUtils.getRootCauseMessage(e), e);
}
partitionInit = true;
+ ensureHmsRuntimeGeneration();
}
@Override
public void startSplit(int numBackends) {
+ ensureHmsRuntimeGeneration();
if (prunedPartitions.isEmpty()) {
splitAssignment.finishSchedule();
+ releaseFsViewOnce();
return;
}
- AtomicInteger numFinishedPartitions = new AtomicInteger(0);
+ acquireFsView();
Review Comment:
Fixed in 69cf0c1b77c. The scan now captures the exact Hudi fs-view cache
generation together with the HMS runtime authenticator. Cold loading happens
outside lifecycle monitors; a short identity-and-lease handoff prevents a stale
scan from crossing into a replacement generation; and fsView.sync() runs under
the captured authenticator after the exact view is pinned. A deterministic
cold-load/reset concurrency test verifies retirement completes without deadlock
and the stale view is closed without sync.
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergMetadataOps.java:
##########
@@ -117,11 +127,18 @@ public IcebergMetadataOps(ExternalCatalog dorisCatalog,
Catalog catalog) {
this.catalog = catalog;
nsCatalog = (SupportsNamespaces) catalog;
this.executionAuthenticator = dorisCatalog.getExecutionAuthenticator();
+ this.threadPoolWithPreAuth = dorisCatalog.getThreadPoolWithPreAuth();
+ this.catalogProperties = Collections.unmodifiableMap(new
HashMap<>(dorisCatalog.getProperties()));
+ this.icebergCatalogType =
catalogProperties.get(IcebergExternalCatalog.ICEBERG_CATALOG_TYPE);
Review Comment:
Fixed in 69cf0c1b77c. IcebergMetadataOps now normalizes the frozen catalog
subtype with Locale.ROOT, and the database property/location tests cover
uppercase HMS so a valid mixed-case catalog configuration is not rejected later.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
