This is an automated email from the ASF dual-hosted git repository.
Gabriel39 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 0526f419746 [fix](iceberg) Tolerate concurrent namespace drops during
listing (#68248)
0526f419746 is described below
commit 0526f419746f80c2823b79a6a1c1966da0868ab2
Author: Gabriel <[email protected]>
AuthorDate: Sun Sep 20 18:18:11 2026 +0800
[fix](iceberg) Tolerate concurrent namespace drops during listing (#68248)
### What problem does this PR solve?
When another session drops an Iceberg namespace between listing its
parent and recursively listing its children, nested namespace discovery
fails for the entire catalog. Unrelated `SHOW DATABASES` queries then
fail with `NoSuchNamespaceException`, and database initialization can
surface this as `Unknown database` for an existing database.
Skip only child branches that disappear during recursive traversal,
including their stale namespace names. Preserve failures when listing
the root and propagate authorization and service errors. Also make the
existing nested-namespace regression suite fail when its refresh retries
are exhausted, instead of logging and passing.
### Release note
Fix Iceberg REST nested namespace listings failing when another session
concurrently drops an unrelated namespace.
### Check List (For Author)
- Test
- [x] Unit Test
- [x] Regression test (strengthened existing retry assertion; cluster
execution pending CI)
- [x] Manual test (isolated Groovy retry helper)
- Behavior changed:
- [x] Yes. A concurrently deleted child namespace no longer aborts the
catalog listing. Other listing errors remain visible.
- Does this need documentation?
- [x] No.
Validation:
- New deterministic namespace-deletion test fails on the original
implementation; all 27 `CatalogBackedIcebergCatalogOpsTest` tests pass
with the fix, including missing-root, permission, and service-error
checks. Maven Checkstyle and the connector import gate pass.
- Full Iceberg module test run: 1,450 tests, 1 failure, 5 skipped. The
sole failure,
`IcebergWritePlanProviderTest#planMergePreservesExplicitlyEmptyReadAcrossConcurrentFirstAppend`
(expected `null`, actual `-1`), also fails on unmodified master
`4e3b9673`; it is outside this change.
- Executed the regression suite's actual retry closure in an isolated
Groovy harness: exhaustion previously returned successfully and now
throws the original exception; a transient failure still recovers after
refresh.
- No live Doris/Iceberg cluster regression was run locally.
Commands:
```bash
mvn -f fe/pom.xml -pl :fe-connector-iceberg -am test \
-Dtest=CatalogBackedIcebergCatalogOpsTest -DfailIfNoTests=false \
-Dmaven.build.cache.enabled=false
mvn -f fe/pom.xml -pl :fe-connector-iceberg -am test \
-Dmaven.build.cache.enabled=false
```
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
---
.../doris/connector/iceberg/IcebergCatalogOps.java | 15 ++++-
.../CatalogBackedIcebergCatalogOpsTest.java | 64 ++++++++++++++++++++++
.../iceberg_and_internal_nested_namespace.groovy | 4 +-
3 files changed, 78 insertions(+), 5 deletions(-)
diff --git
a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergCatalogOps.java
b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergCatalogOps.java
index 3b55a98be7c..cb2d420ed6d 100644
---
a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergCatalogOps.java
+++
b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergCatalogOps.java
@@ -40,6 +40,7 @@ import org.apache.iceberg.catalog.Namespace;
import org.apache.iceberg.catalog.SupportsNamespaces;
import org.apache.iceberg.catalog.TableIdentifier;
import org.apache.iceberg.catalog.ViewCatalog;
+import org.apache.iceberg.exceptions.NoSuchNamespaceException;
import org.apache.iceberg.expressions.Expressions;
import org.apache.iceberg.expressions.Term;
import org.apache.iceberg.types.Type;
@@ -323,9 +324,17 @@ public interface IcebergCatalogOps {
SupportsNamespaces nsCatalog = (SupportsNamespaces) catalog;
if (restFlavor && nestedNamespaceEnabled) {
return nsCatalog.listNamespaces(parentNs).stream()
- .flatMap(childNs -> Stream.concat(
- Stream.of(childNs.toString()),
- listNestedNamespaces(childNs).stream()))
+ .flatMap(childNs -> {
+ try {
+ List<String> descendants =
listNestedNamespaces(childNs);
+ return
Stream.concat(Stream.of(childNs.toString()), descendants.stream());
+ } catch (NoSuchNamespaceException e) {
+ // A child can be dropped after its parent was
listed. Skip that branch,
+ // including the stale child name, without
hiding failures to list the root.
+ LOG.debug("Namespace {} was dropped during
listing", childNs, e);
+ return Stream.empty();
+ }
+ })
.collect(Collectors.toList());
}
return nsCatalog.listNamespaces(parentNs).stream()
diff --git
a/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/CatalogBackedIcebergCatalogOpsTest.java
b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/CatalogBackedIcebergCatalogOpsTest.java
index 119b0ce953f..5b8f1bfd5e7 100644
---
a/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/CatalogBackedIcebergCatalogOpsTest.java
+++
b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/CatalogBackedIcebergCatalogOpsTest.java
@@ -23,6 +23,9 @@ import org.apache.doris.connector.spi.DorisConnectorException;
import org.apache.iceberg.catalog.Catalog;
import org.apache.iceberg.catalog.Namespace;
import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.exceptions.ForbiddenException;
+import org.apache.iceberg.exceptions.NoSuchNamespaceException;
+import org.apache.iceberg.exceptions.ServiceFailureException;
import org.apache.iceberg.view.View;
import org.junit.jupiter.api.Assertions;
import org.junit.jupiter.api.Test;
@@ -30,7 +33,9 @@ import org.junit.jupiter.api.Test;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.Collections;
+import java.util.HashMap;
import java.util.List;
+import java.util.Map;
import java.util.Optional;
/**
@@ -84,6 +89,52 @@ public class CatalogBackedIcebergCatalogOpsTest {
"REST + nested-enabled must recurse and emit dotted namespace
names depth-first");
}
+ @Test
+ public void listDatabaseNamesSkipsNamespacesDeletedDuringTraversal() {
+ FailingNamespaceCatalog catalog = new FailingNamespaceCatalog();
+ Namespace deleted = Namespace.of("deleted");
+ Namespace parent = Namespace.of("parent");
+ Namespace deletedChild = Namespace.of("parent", "deleted");
+ Namespace emptyChild = Namespace.of("parent", "empty");
+ Namespace sibling = Namespace.of("sibling");
+ catalog.childNamespaces.put(Namespace.empty(), Arrays.asList(deleted,
parent, sibling));
+ catalog.childNamespaces.put(parent, Arrays.asList(deletedChild,
emptyChild));
+ // The parent listing still contains namespaces that a concurrent DROP
has already removed.
+ catalog.failures.put(deleted, new NoSuchNamespaceException("Namespace
does not exist: %s", deleted));
+ catalog.failures.put(deletedChild, new
NoSuchNamespaceException("Namespace does not exist: %s", deletedChild));
+
+ List<String> result = ops(catalog, true, true, true,
Optional.empty()).listDatabaseNames();
+
+ Assertions.assertEquals(Arrays.asList("parent", "parent.empty",
"sibling"), result);
+ }
+
+ @Test
+ public void listDatabaseNamesPropagatesMissingRootNamespace() {
+ for (Optional<String> externalCatalogName :
Arrays.asList(Optional.<String>empty(), Optional.of("cat"))) {
+ FailingNamespaceCatalog catalog = new FailingNamespaceCatalog();
+ Namespace root =
externalCatalogName.map(Namespace::of).orElse(Namespace.empty());
+ NoSuchNamespaceException failure = new
NoSuchNamespaceException("Missing root: %s", root);
+ catalog.failures.put(root, failure);
+
+ Assertions.assertSame(failure,
Assertions.assertThrows(NoSuchNamespaceException.class,
+ () -> ops(catalog, true, true, true,
externalCatalogName).listDatabaseNames()));
+ }
+ }
+
+ @Test
+ public void listDatabaseNamesPropagatesOtherTraversalFailures() {
+ for (RuntimeException failure : Arrays.asList(
+ new ForbiddenException("Access denied"), new
ServiceFailureException("Service unavailable"))) {
+ FailingNamespaceCatalog catalog = new FailingNamespaceCatalog();
+ Namespace child = Namespace.of("child");
+ catalog.childNamespaces.put(Namespace.empty(),
Collections.singletonList(child));
+ catalog.failures.put(child, failure);
+
+ Assertions.assertSame(failure,
Assertions.assertThrows(failure.getClass(),
+ () -> ops(catalog, true, true, true,
Optional.empty()).listDatabaseNames()));
+ }
+ }
+
@Test
public void listDatabaseNamesDoesNotRecurseWhenNestedFlagButNotRest() {
// WHY: legacy gates recursion on `dorisCatalog instanceof
IcebergRestExternalCatalog` AS WELL AS
@@ -403,4 +454,17 @@ public class CatalogBackedIcebergCatalogOpsTest {
Assertions.assertEquals(TableIdentifier.of(Namespace.of("a", "b"),
"t1"), catalog.lastTableExistsId,
"tableExists must build the identifier from the split
namespace");
}
+
+ private static class FailingNamespaceCatalog extends FakeIcebergCatalog {
+ private final Map<Namespace, RuntimeException> failures = new
HashMap<>();
+
+ @Override
+ public List<Namespace> listNamespaces(Namespace ns) {
+ RuntimeException failure = failures.get(ns);
+ if (failure != null) {
+ throw failure;
+ }
+ return super.listNamespaces(ns);
+ }
+ }
}
diff --git
a/regression-test/suites/external_table_p0/iceberg/iceberg_and_internal_nested_namespace.groovy
b/regression-test/suites/external_table_p0/iceberg/iceberg_and_internal_nested_namespace.groovy
index effb1b2fc8c..abf060b85ce 100644
---
a/regression-test/suites/external_table_p0/iceberg/iceberg_and_internal_nested_namespace.groovy
+++
b/regression-test/suites/external_table_p0/iceberg/iceberg_and_internal_nested_namespace.groovy
@@ -57,8 +57,8 @@ suite("iceberg_and_internal_nested_namespace", "p0,external")
{
owner.sql("""refresh catalog ${catalog_name}""")
sleep(500) // Sleep 500ms before retry
} else {
- // log but not throw exception
- logger.error("Query failed after ${maxRetries}
attempts: ${errorMsg}")
+ // Exhausted retries must fail the suite instead of
hiding a namespace listing regression.
+ throw e
}
} else {
throw e // Rethrow if it's a different exception
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
