This is an automated email from the ASF dual-hosted git repository.
roryqi pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/gravitino.git
The following commit(s) were added to refs/heads/main by this push:
new da99abcc36 [#11943] improvement(iceberg,core): validate warehouse at
catalog creation and fail fast with an actionable error (#11959)
da99abcc36 is described below
commit da99abcc36207f2c15c563dde657ab05d67ace82
Author: Nevin Zheng <[email protected]>
AuthorDate: Sun Jul 19 01:50:53 2026 -0700
[#11943] improvement(iceberg,core): validate warehouse at catalog creation
and fail fast with an actionable error (#11959)
### What changes were proposed in this pull request?
This PR fixes the visible configuration error reported in #11943 for a
`lakehouse-iceberg` catalog using `catalog-backend=rest`.
For the Iceberg REST backend, `warehouse` is forwarded to `GET
/v1/config?warehouse=...` as a server-side catalog selector. It is not a
storage location. Previously, a catalog could be created successfully
with an unresolvable value such as `s3://warehouse/`, then fail
cryptically on first use.
The changes are:
- **Validate the REST `warehouse` selector at creation time.**
`IcebergCatalog` opts into create-time initialization only when the
backend is REST and `warehouse` is nonblank. Hive, JDBC, and REST
catalogs without `warehouse` keep their existing creation behavior.
- **Return an actionable configuration error.** If the REST server
rejects the selector with `NoSuchWarehouseException`, creation fails
with `IllegalArgumentException` / HTTP 400 and explains that `warehouse`
should be removed to use the default catalog or set to a catalog name
the REST server recognizes.
- **Keep dependency failures distinct from invalid configuration.**
Because the new validation contacts the remote REST service, a transport
failure or HTTP 503 during REST initialization is returned as
`ConnectionFailedException` / HTTP 502 instead of being presented as an
invalid warehouse or a generic internal error. With a nonblank
`warehouse`, the failed creation is rolled back and no catalog is
persisted.
- **Preserve the exception across Gravitino boundaries.** The catalog
classloader, server handler, and Java client preserve
`IllegalArgumentException` and `ConnectionFailedException` so callers
receive the intended 400 or 502 result.
- **Clarify the property in OpenAPI examples.** The examples contrast
storage backends, where `warehouse` is a location, with the REST
backend, where it is a catalog selector and may be omitted.
#### Scope and future work
The primary scope of this PR is the `NoSuchWarehouseException` case from
#11943. The adjacent transport/503 mapping is included only so the new
create-time validation does not confuse a temporarily unavailable REST
dependency with an invalid warehouse value.
This PR does **not** attempt to normalize every response or exception
produced by the Iceberg REST client. Iceberg also has other response
types and endpoint-specific errors, including bad-request,
authorization, forbidden, service-failure, and operation-time failures
after initialization. Mapping those consistently into Gravitino's error
taxonomy should be handled as future work with dedicated scenarios and
tests.
### Why are the changes needed?
On a lakehouse-iceberg catalog with `catalog-backend=rest`, the
`warehouse` property is a catalog selector sent to the Iceberg REST
server, not a storage path. A user copying a Hive or JDBC example may
set it to a storage URI such as `s3://warehouse/`. Before this change,
catalog creation returned success and every later attempt to use the
catalog failed with a cryptic `NoSuchWarehouseException`.
The catalog should reject that specific misconfiguration at creation
time with a useful message. At the same time, a temporarily unavailable
remote server must remain distinguishable from an invalid selector.
Fix: #11943
### Does this PR introduce _any_ user-facing change?
- Creating a REST-backed lakehouse-iceberg catalog with a nonblank
`warehouse` that the remote server cannot resolve now fails at creation
with HTTP 400 and an actionable message.
- If the remote service is unreachable or returns HTTP 503 while REST
initialization is running, the failure is returned as HTTP 502
(`ConnectionFailedException`, error code `1007`). This can occur during
creation when `warehouse` is configured, or during lazy initialization
on first use when it is omitted.
- Creation behavior for REST catalogs without `warehouse` is unchanged:
initialization remains deferred until first use.
- Hive and JDBC behavior is unchanged.
- No property keys are added or removed.
### How was this patch tested?
**Unit tests** (`./gradlew :core:test
:catalogs:catalog-lakehouse-iceberg:test -PskipITs`):
- `TestCatalogManager#testCreateCatalogValidatesBackendConnection`
verifies opt-in create-time validation, rollback after validation
failure, and that catalogs which do not opt in are not eagerly
initialized.
- `TestIcebergCatalog#testShouldValidateWarehouseProperty` covers REST
with a value, missing/blank warehouse, JDBC, and a null backend.
-
`TestIcebergCatalogOperations#testHandleRestExceptionTranslatesUnavailableToConnectionFailed`
verifies the scoped initialization mapping for a transport failure and
HTTP 503.
-
`TestIcebergCatalogOperations#testTestConnectionNonRestWarehouseHasNoHint`
verifies that the REST-specific warehouse guidance is not shown for
other backends.
**Integration tests** (`CatalogIcebergRestIT`, against the embedded
Iceberg REST service):
- `testCreateRestCatalogWithUnresolvableWarehouseFailsAtCreate` verifies
that an invalid selector returns the actionable error and leaves no
catalog behind.
- `testCreateRestCatalogWithValidWarehouseSucceeds` verifies that a
named selector recognized by the REST server creates successfully and
can list schemas.
- `testCreateRestCatalogWithUnreachableServerAtCreate` verifies that a
dependency outage during create-time validation returns
`ConnectionFailedException` and persists nothing.
- `testOperationOnUnreachableRestCatalogFailsWithConnectionError`
verifies the lazy-initialization path: creation is allowed without
`warehouse`, and the first operation reports the unavailable REST
dependency.
Full `CatalogIcebergRestIT` run: 43 tests, 1 skipped, 0 failures.
Manual verification covered the three warehouse-selector cases:
- Recognized named selector → HTTP 200 and usable catalog.
- Unrecognized storage-URI-shaped selector → HTTP 400 with the
selector-versus-location explanation.
- Omitted selector → HTTP 200 using the server's default catalog.
---------
Co-authored-by: Claude Opus 4.8 <[email protected]>
---
.../catalog/lakehouse/iceberg/IcebergCatalog.java | 16 +++
.../iceberg/IcebergCatalogOperations.java | 47 ++++++--
.../lakehouse/iceberg/TestIcebergCatalog.java | 11 ++
.../iceberg/TestIcebergCatalogOperations.java | 53 +++++++++
.../integration/test/CatalogIcebergRestIT.java | 130 +++++++++++++++++++++
.../org/apache/gravitino/client/ErrorHandlers.java | 3 +
.../apache/gravitino/catalog/CatalogManager.java | 11 +-
.../apache/gravitino/connector/BaseCatalog.java | 9 ++
.../gravitino/utils/IsolatedClassLoader.java | 26 +++++
.../java/org/apache/gravitino/TestCatalog.java | 7 ++
.../gravitino/catalog/TestCatalogManager.java | 74 ++++++++++++
.../gravitino/connector/TestCatalogOperations.java | 10 +-
docs/open-api/catalogs.yaml | 49 ++++++++
.../server/web/rest/ExceptionHandlers.java | 10 ++
14 files changed, 446 insertions(+), 10 deletions(-)
diff --git
a/catalogs/catalog-lakehouse-iceberg/src/main/java/org/apache/gravitino/catalog/lakehouse/iceberg/IcebergCatalog.java
b/catalogs/catalog-lakehouse-iceberg/src/main/java/org/apache/gravitino/catalog/lakehouse/iceberg/IcebergCatalog.java
index f7dc4acc06..935112c66d 100644
---
a/catalogs/catalog-lakehouse-iceberg/src/main/java/org/apache/gravitino/catalog/lakehouse/iceberg/IcebergCatalog.java
+++
b/catalogs/catalog-lakehouse-iceberg/src/main/java/org/apache/gravitino/catalog/lakehouse/iceberg/IcebergCatalog.java
@@ -68,6 +68,22 @@ public class IcebergCatalog extends
BaseCatalog<IcebergCatalog> {
return (ViewCatalog) ops();
}
+ @Override
+ public boolean shouldValidateOnCreate() {
+ Map<String, String> properties = entity().getProperties();
+ return properties != null
+ && shouldValidateWarehouseProperty(
+ properties.get(IcebergConstants.CATALOG_BACKEND),
+ properties.get(IcebergConstants.WAREHOUSE));
+ }
+
+ // REST resolves `warehouse` as a server-side catalog selector, so validate
it at create; hive and
+ // jdbc use it as a storage location and need no create-time connection.
+ static boolean shouldValidateWarehouseProperty(String backend, String
warehouse) {
+ return IcebergCatalogBackend.REST.name().equalsIgnoreCase(backend)
+ && StringUtils.isNotBlank(warehouse);
+ }
+
@Override
public Capability newCapability() {
return new
IcebergCatalogCapability(HierarchicalSchemaUtil.schemaSeparator());
diff --git
a/catalogs/catalog-lakehouse-iceberg/src/main/java/org/apache/gravitino/catalog/lakehouse/iceberg/IcebergCatalogOperations.java
b/catalogs/catalog-lakehouse-iceberg/src/main/java/org/apache/gravitino/catalog/lakehouse/iceberg/IcebergCatalogOperations.java
index 0e7f8d48bd..3a75bad5ee 100644
---
a/catalogs/catalog-lakehouse-iceberg/src/main/java/org/apache/gravitino/catalog/lakehouse/iceberg/IcebergCatalogOperations.java
+++
b/catalogs/catalog-lakehouse-iceberg/src/main/java/org/apache/gravitino/catalog/lakehouse/iceberg/IcebergCatalogOperations.java
@@ -80,6 +80,9 @@ import org.apache.iceberg.catalog.TableIdentifier;
import org.apache.iceberg.exceptions.AlreadyExistsException;
import org.apache.iceberg.exceptions.NamespaceNotEmptyException;
import org.apache.iceberg.exceptions.NoSuchNamespaceException;
+import org.apache.iceberg.exceptions.NoSuchWarehouseException;
+import org.apache.iceberg.exceptions.RESTException;
+import org.apache.iceberg.exceptions.ServiceUnavailableException;
import org.apache.iceberg.rest.requests.RenameTableRequest;
import org.apache.iceberg.rest.requests.UpdateNamespacePropertiesRequest;
import org.apache.iceberg.rest.responses.GetNamespaceResponse;
@@ -128,14 +131,42 @@ public class IcebergCatalogOperations
IcebergCatalogWrapper rawWrapper = new
IcebergCatalogWrapper(icebergConfig);
- AuthenticationConfig authenticationConfig = new
AuthenticationConfig(resultConf);
- this.icebergCatalogWrapper =
- authenticationConfig.isKerberosAuth() && rawWrapper.getCatalog()
instanceof SupportsKerberos
- ? new
KerberosAwareIcebergCatalogProxy(rawWrapper).getProxy(icebergConfig)
- : rawWrapper;
- this.icebergCatalogWrapperHelper =
- new IcebergCatalogWrapperHelper(icebergCatalogWrapper.getCatalog());
- this.icebergViewCatalogOperations = new
IcebergViewCatalogOperations(icebergCatalogWrapper);
+ try {
+ AuthenticationConfig authenticationConfig = new
AuthenticationConfig(resultConf);
+ this.icebergCatalogWrapper =
+ authenticationConfig.isKerberosAuth()
+ && rawWrapper.getCatalog() instanceof SupportsKerberos
+ ? new
KerberosAwareIcebergCatalogProxy(rawWrapper).getProxy(icebergConfig)
+ : rawWrapper;
+ this.icebergCatalogWrapperHelper =
+ new IcebergCatalogWrapperHelper(icebergCatalogWrapper.getCatalog());
+ this.icebergViewCatalogOperations = new
IcebergViewCatalogOperations(icebergCatalogWrapper);
+ } catch (NoSuchWarehouseException e) {
+ // A reachable server rejecting the `warehouse` selector is a user
error. See issue #11943.
+ throw new IllegalArgumentException(
+ String.format(
+ "The 'warehouse' value '%s' could not be resolved by the Iceberg
REST server. On "
+ + "the REST backend 'warehouse' selects a catalog by name on
the remote server "
+ + "and is not a storage location; remove 'warehouse' to use
the server's default "
+ + "catalog, or set it to a catalog name/identifier that the
server recognizes.",
+ icebergConfig.get(IcebergConfig.CATALOG_WAREHOUSE)),
+ e);
+ } catch (RESTException e) {
+ throw handleRestException(e);
+ }
+ }
+
+ // Maps an Iceberg REST-client exception into Gravitino's taxonomy
+ @VisibleForTesting
+ static RuntimeException handleRestException(RESTException e) {
+ // Base RESTException (server unreachable) or ServiceUnavailableException
(503): the downstream
+ // dependency is not available. getClass() matches the base type only, so
the 4xx/5xx subtypes
+ // are excluded and pass through unchanged.
+ if (e.getClass() == RESTException.class || e instanceof
ServiceUnavailableException) {
+ return new ConnectionFailedException(
+ e, "The Iceberg REST backend is unavailable: %s", e.getMessage());
+ }
+ return e;
}
/** Closes the Iceberg catalog and releases the associated client pool. */
diff --git
a/catalogs/catalog-lakehouse-iceberg/src/test/java/org/apache/gravitino/catalog/lakehouse/iceberg/TestIcebergCatalog.java
b/catalogs/catalog-lakehouse-iceberg/src/test/java/org/apache/gravitino/catalog/lakehouse/iceberg/TestIcebergCatalog.java
index 9249088593..622140e200 100644
---
a/catalogs/catalog-lakehouse-iceberg/src/test/java/org/apache/gravitino/catalog/lakehouse/iceberg/TestIcebergCatalog.java
+++
b/catalogs/catalog-lakehouse-iceberg/src/test/java/org/apache/gravitino/catalog/lakehouse/iceberg/TestIcebergCatalog.java
@@ -114,6 +114,17 @@ public class TestIcebergCatalog {
Assertions.assertTrue(listNamespacesResponse.namespaces().isEmpty());
}
+ @Test
+ public void testShouldValidateWarehouseProperty() {
+ Assertions.assertTrue(
+ IcebergCatalog.shouldValidateWarehouseProperty("rest",
"s3://warehouse/"));
+
Assertions.assertFalse(IcebergCatalog.shouldValidateWarehouseProperty("rest",
null));
+ Assertions.assertFalse(
+ IcebergCatalog.shouldValidateWarehouseProperty("jdbc",
"file:///tmp/iceberg-jdbc"));
+
Assertions.assertFalse(IcebergCatalog.shouldValidateWarehouseProperty("rest", "
"));
+
Assertions.assertFalse(IcebergCatalog.shouldValidateWarehouseProperty(null,
"s3://warehouse/"));
+ }
+
@Test
void testCatalogProperty() {
AuditInfo auditInfo =
diff --git
a/catalogs/catalog-lakehouse-iceberg/src/test/java/org/apache/gravitino/catalog/lakehouse/iceberg/TestIcebergCatalogOperations.java
b/catalogs/catalog-lakehouse-iceberg/src/test/java/org/apache/gravitino/catalog/lakehouse/iceberg/TestIcebergCatalogOperations.java
index dfbd129a23..841d3ecfaa 100644
---
a/catalogs/catalog-lakehouse-iceberg/src/test/java/org/apache/gravitino/catalog/lakehouse/iceberg/TestIcebergCatalogOperations.java
+++
b/catalogs/catalog-lakehouse-iceberg/src/test/java/org/apache/gravitino/catalog/lakehouse/iceberg/TestIcebergCatalogOperations.java
@@ -27,8 +27,13 @@ import java.util.Arrays;
import org.apache.gravitino.Catalog;
import org.apache.gravitino.NameIdentifier;
import org.apache.gravitino.Namespace;
+import org.apache.gravitino.exceptions.ConnectionFailedException;
import org.apache.gravitino.exceptions.GravitinoRuntimeException;
+import org.apache.gravitino.iceberg.common.IcebergConfig;
import org.apache.gravitino.iceberg.common.ops.IcebergCatalogWrapper;
+import org.apache.iceberg.exceptions.BadRequestException;
+import org.apache.iceberg.exceptions.RESTException;
+import org.apache.iceberg.exceptions.ServiceUnavailableException;
import org.apache.iceberg.rest.responses.ListNamespacesResponse;
import org.junit.jupiter.api.Assertions;
import org.junit.jupiter.api.Test;
@@ -55,6 +60,54 @@ public class TestIcebergCatalogOperations {
exception.getMessage().contains("Failed to run listNamespace on
Iceberg catalog"));
}
+ @Test
+ public void testHandleRestExceptionTranslatesUnavailableToConnectionFailed()
{
+ // Transport failure (server unreachable) is the base RESTException.
+ RESTException unreachable = new RESTException("Error occurred while
processing GET request");
+ Assertions.assertInstanceOf(
+ ConnectionFailedException.class,
IcebergCatalogOperations.handleRestException(unreachable));
+
+ // The server answered 503.
+ RESTException unavailable = new ServiceUnavailableException("Service
Unavailable: %s", "down");
+ Assertions.assertInstanceOf(
+ ConnectionFailedException.class,
IcebergCatalogOperations.handleRestException(unavailable));
+
+ // A subtype we don't translate yet is returned unchanged (surfaces as a
runtime error).
+ RESTException badRequest = new BadRequestException("Malformed request:
%s", "bad");
+ Assertions.assertSame(badRequest,
IcebergCatalogOperations.handleRestException(badRequest));
+ }
+
+ @Test
+ public void testTestConnectionNonRestWarehouseHasNoHint() {
+ IcebergCatalogWrapper mockWrapper = mock(IcebergCatalogWrapper.class);
+ IcebergConfig config =
+ new IcebergConfig(
+ ImmutableMap.of(
+ IcebergConstants.CATALOG_BACKEND, "hive",
+ IcebergConstants.WAREHOUSE, "s3://warehouse/"));
+ when(mockWrapper.getIcebergConfig()).thenReturn(config);
+ when(mockWrapper.listNamespace(any())).thenThrow(new
RuntimeException("boom"));
+
+ IcebergCatalogOperations catalogOperations = new
IcebergCatalogOperations();
+ catalogOperations.icebergCatalogWrapper = mockWrapper;
+
+ Exception exception =
+ Assertions.assertThrows(
+ ConnectionFailedException.class,
+ () ->
+ catalogOperations.testConnection(
+ NameIdentifier.of(METALAKE, CATALOG),
+ Catalog.Type.RELATIONAL,
+ "iceberg",
+ "comment",
+ ImmutableMap.of()));
+ Assertions.assertTrue(
+ exception.getMessage().contains("Failed to run listNamespace on
Iceberg catalog"),
+ exception.getMessage());
+ Assertions.assertFalse(
+ exception.getMessage().contains("selects a catalog by name"),
exception.getMessage());
+ }
+
@Test
public void testListSchemasConvertsMultiLevelNamespacesToLogicalNames() {
IcebergCatalogWrapper mockWrapper = mock(IcebergCatalogWrapper.class);
diff --git
a/catalogs/catalog-lakehouse-iceberg/src/test/java/org/apache/gravitino/catalog/lakehouse/iceberg/integration/test/CatalogIcebergRestIT.java
b/catalogs/catalog-lakehouse-iceberg/src/test/java/org/apache/gravitino/catalog/lakehouse/iceberg/integration/test/CatalogIcebergRestIT.java
index f992821a8d..f6cb49554b 100644
---
a/catalogs/catalog-lakehouse-iceberg/src/test/java/org/apache/gravitino/catalog/lakehouse/iceberg/integration/test/CatalogIcebergRestIT.java
+++
b/catalogs/catalog-lakehouse-iceberg/src/test/java/org/apache/gravitino/catalog/lakehouse/iceberg/integration/test/CatalogIcebergRestIT.java
@@ -18,16 +18,46 @@
*/
package org.apache.gravitino.catalog.lakehouse.iceberg.integration.test;
+import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.Maps;
import java.util.Map;
+import org.apache.gravitino.Catalog;
import org.apache.gravitino.catalog.lakehouse.iceberg.IcebergConstants;
+import org.apache.gravitino.exceptions.ConnectionFailedException;
import org.apache.gravitino.iceberg.common.IcebergConfig;
import org.apache.gravitino.integration.test.container.HiveContainer;
+import org.apache.gravitino.integration.test.util.GravitinoITUtils;
import org.apache.gravitino.server.web.JettyServerConfig;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.BeforeAll;
import org.junit.jupiter.api.Tag;
+import org.junit.jupiter.api.Test;
@Tag("gravitino-docker-test")
public class CatalogIcebergRestIT extends CatalogIcebergBaseIT {
+ // A named (non-default) catalog registered on the embedded Iceberg REST
service so a `warehouse`
+ // selector that resolves to a real catalog can be exercised at create time.
+ private static final String NAMED_CATALOG = "it_named_catalog";
+
+ @Override
+ @BeforeAll
+ public void startup() throws Exception {
+ // Register the named catalog before the server (and its Iceberg REST
service) starts, so the
+ // REST service serves `NAMED_CATALOG` in addition to its default catalog.
Must run before
+ // super.startup() launches the embedded server.
+ String namedWarehouse = System.getProperty("java.io.tmpdir") + "/" +
NAMED_CATALOG + "-wh";
+ String prefix =
+ "gravitino." + IcebergConstants.GRAVITINO_ICEBERG_REST_SERVICE_NAME +
".catalog.";
+ registerCustomConfigs(
+ ImmutableMap.of(
+ prefix + NAMED_CATALOG + "." + IcebergConstants.CATALOG_BACKEND,
+ "memory",
+ prefix + NAMED_CATALOG + "." + IcebergConstants.WAREHOUSE,
+ namedWarehouse));
+ super.startup();
+ }
+
@Override
protected void initIcebergCatalogProperties() {
Map<String, String> map =
@@ -44,4 +74,104 @@ public class CatalogIcebergRestIT extends
CatalogIcebergBaseIT {
containerSuite.getHiveContainer().getContainerIpAddress(),
HiveContainer.HDFS_DEFAULTFS_PORT);
}
+
+ // Checks an Iceberg REST warehouse misconfiguration is caught on create.
See issue #11943.
+ @Test
+ void testCreateRestCatalogWithUnresolvableWarehouseFailsAtCreate() {
+ String badCatalogName =
GravitinoITUtils.genRandomName("iceberg_rest_bad_warehouse");
+ Map<String, String> properties = Maps.newHashMap();
+ properties.put(IcebergConfig.CATALOG_BACKEND.getKey(), "rest");
+ properties.put(IcebergConfig.CATALOG_URI.getKey(), URIS);
+ // A URI-shaped value copied from a hive/jdbc example; the REST server
does not know a catalog
+ // by this name, so it cannot be resolved.
+ properties.put(IcebergConfig.CATALOG_WAREHOUSE.getKey(),
"s3://not-a-real-catalog/");
+
+ // A reachable REST server rejecting the warehouse selector is a
user/configuration error, so
+ // it surfaces as an IllegalArgumentException (HTTP 400) at create time.
+ IllegalArgumentException exception =
+ Assertions.assertThrows(
+ IllegalArgumentException.class,
+ () ->
+ metalake.createCatalog(
+ badCatalogName,
+ Catalog.Type.RELATIONAL,
+ "lakehouse-iceberg",
+ "comment",
+ properties));
+
+ Assertions.assertTrue(
+ exception.getMessage() != null
+ && exception.getMessage().contains("selects a catalog by name")
+ && exception.getMessage().contains("s3://not-a-real-catalog/"),
+ "expected the REST warehouse name-vs-location hint, but got: " +
exception.getMessage());
+
+ Assertions.assertFalse(
+ metalake.catalogExists(badCatalogName),
+ "catalog must not be created when its warehouse cannot be resolved at
create time");
+ }
+
+ // A warehouse the REST server serves must still create and be usable. See
issue #11943.
+ @Test
+ void testCreateRestCatalogWithValidWarehouseSucceeds() {
+ String catalogName =
GravitinoITUtils.genRandomName("iceberg_rest_named_warehouse");
+ Map<String, String> properties = Maps.newHashMap();
+ properties.put(IcebergConfig.CATALOG_BACKEND.getKey(), "rest");
+ properties.put(IcebergConfig.CATALOG_URI.getKey(), URIS);
+ // A valid selector: NAMED_CATALOG is a catalog the embedded REST server
serves.
+ properties.put(IcebergConfig.CATALOG_WAREHOUSE.getKey(), NAMED_CATALOG);
+
+ Catalog created =
+ metalake.createCatalog(
+ catalogName, Catalog.Type.RELATIONAL, "lakehouse-iceberg",
"comment", properties);
+ try {
+ Assertions.assertTrue(metalake.catalogExists(catalogName));
+ // Resolvable end-to-end: listing schemas against the selected catalog
succeeds.
+ Assertions.assertDoesNotThrow(() -> created.asSchemas().listSchemas());
+ } finally {
+ metalake.disableCatalog(catalogName);
+ metalake.dropCatalog(catalogName);
+ }
+ }
+
+ // An unreachable remote at create fails as a ConnectionFailedException and
persists nothing.
+ @Test
+ void testCreateRestCatalogWithUnreachableServerAtCreate() {
+ String name =
GravitinoITUtils.genRandomName("iceberg_rest_unreachable_server");
+ Map<String, String> properties = Maps.newHashMap();
+ properties.put(IcebergConfig.CATALOG_BACKEND.getKey(), "rest");
+ // Port 1 refuses connections fast, so create-time validation cannot reach
the REST server.
+ properties.put(IcebergConfig.CATALOG_URI.getKey(),
"http://127.0.0.1:1/iceberg/";);
+ properties.put(IcebergConfig.CATALOG_WAREHOUSE.getKey(), "some_warehouse");
+
+ Assertions.assertThrows(
+ ConnectionFailedException.class,
+ () ->
+ metalake.createCatalog(
+ name, Catalog.Type.RELATIONAL, "lakehouse-iceberg", "comment",
properties));
+ Assertions.assertFalse(metalake.catalogExists(name));
+ }
+
+ // A catalog with valid config whose backend is unreachable: it is created
(no warehouse, so no
+ // create-time validation), but querying the dead backend fails as a
ConnectionFailedException.
+ @Test
+ void testOperationOnUnreachableRestCatalogFailsWithConnectionError() {
+ String name =
GravitinoITUtils.genRandomName("iceberg_rest_unreachable_op");
+ Map<String, String> properties = Maps.newHashMap();
+ properties.put(IcebergConfig.CATALOG_BACKEND.getKey(), "rest");
+ // Port 1 refuses connections fast; no warehouse means create-time
validation is skipped.
+ properties.put(IcebergConfig.CATALOG_URI.getKey(),
"http://127.0.0.1:1/iceberg/";);
+
+ Catalog created =
+ metalake.createCatalog(
+ name, Catalog.Type.RELATIONAL, "lakehouse-iceberg", "comment",
properties);
+ try {
+ Assertions.assertTrue(metalake.catalogExists(name));
+ // Config is valid but the backend is down, so querying it is a
dependency failure.
+ Assertions.assertThrows(
+ ConnectionFailedException.class, () ->
created.asSchemas().listSchemas());
+ } finally {
+ metalake.disableCatalog(name);
+ metalake.dropCatalog(name);
+ }
+ }
}
diff --git
a/clients/client-java/src/main/java/org/apache/gravitino/client/ErrorHandlers.java
b/clients/client-java/src/main/java/org/apache/gravitino/client/ErrorHandlers.java
index dc14f4c6d8..79d4ca6903 100644
---
a/clients/client-java/src/main/java/org/apache/gravitino/client/ErrorHandlers.java
+++
b/clients/client-java/src/main/java/org/apache/gravitino/client/ErrorHandlers.java
@@ -1374,6 +1374,9 @@ public class ErrorHandlers {
@Override
public void accept(ErrorResponse errorResponse) {
+ if (errorResponse.getCode() == ErrorConstants.CONNECTION_FAILED_CODE) {
+ throw new ConnectionFailedException("%s",
formatErrorMessage(errorResponse));
+ }
if (errorResponse.getCode() == ErrorConstants.FORBIDDEN_CODE) {
throw new ForbiddenException("Forbidden error :%s",
errorResponse.getMessage());
}
diff --git
a/core/src/main/java/org/apache/gravitino/catalog/CatalogManager.java
b/core/src/main/java/org/apache/gravitino/catalog/CatalogManager.java
index 4df78ad21b..e26e28d3a9 100644
--- a/core/src/main/java/org/apache/gravitino/catalog/CatalogManager.java
+++ b/core/src/main/java/org/apache/gravitino/catalog/CatalogManager.java
@@ -86,6 +86,7 @@ import org.apache.gravitino.connector.capability.Capability;
import org.apache.gravitino.exceptions.CatalogAlreadyExistsException;
import org.apache.gravitino.exceptions.CatalogInUseException;
import org.apache.gravitino.exceptions.CatalogNotInUseException;
+import org.apache.gravitino.exceptions.ConnectionFailedException;
import org.apache.gravitino.exceptions.GravitinoRuntimeException;
import org.apache.gravitino.exceptions.NoSuchCatalogException;
import org.apache.gravitino.exceptions.NoSuchEntityException;
@@ -1138,9 +1139,17 @@ public class CatalogManager implements
CatalogDispatcher, Closeable {
// so that AppClassLoader can get the value of properties.
wrapper.catalog.properties();
wrapper.catalog.capability();
+
+ // Eagerly initialize opted-in catalogs so a misconfiguration fails
at create rather than
+ // on first use. The backend translates failures into the
caller-error vs
+ // dependency-unavailable taxonomy; both types are forwarded by the
passthrough below.
+ if (propsToValidate != null &&
wrapper.catalog.shouldValidateOnCreate()) {
+ wrapper.catalog.ops();
+ }
return null;
},
- IllegalArgumentException.class);
+ IllegalArgumentException.class,
+ ConnectionFailedException.class);
return wrapper;
}
diff --git a/core/src/main/java/org/apache/gravitino/connector/BaseCatalog.java
b/core/src/main/java/org/apache/gravitino/connector/BaseCatalog.java
index 4ee0b9c794..0c66ea2242 100644
--- a/core/src/main/java/org/apache/gravitino/connector/BaseCatalog.java
+++ b/core/src/main/java/org/apache/gravitino/connector/BaseCatalog.java
@@ -211,6 +211,15 @@ public abstract class BaseCatalog<T extends BaseCatalog>
return ops;
}
+ /**
+ * Whether to validate this catalog against its backend at create time
rather than on first use.
+ *
+ * @return {@code true} to validate at create; {@code false} (default) to
defer
+ */
+ public boolean shouldValidateOnCreate() {
+ return false;
+ }
+
public void checkMetalakeAndCatalogInUse() {
Map<String, String> catalogProperties = entity.getProperties();
if (catalogProperties == null) {
diff --git
a/core/src/main/java/org/apache/gravitino/utils/IsolatedClassLoader.java
b/core/src/main/java/org/apache/gravitino/utils/IsolatedClassLoader.java
index 6c80a6caba..82eb7213e7 100644
--- a/core/src/main/java/org/apache/gravitino/utils/IsolatedClassLoader.java
+++ b/core/src/main/java/org/apache/gravitino/utils/IsolatedClassLoader.java
@@ -111,6 +111,32 @@ public class IsolatedClassLoader implements Closeable {
}
}
+ /**
+ * Executes {@code fn} within the isolated class loading context, rethrowing
any of the given
+ * {@code passthroughExceptions} unchanged and wrapping everything else in a
{@link
+ * RuntimeException}.
+ *
+ * @param fn The function to be executed within the isolated class loading
context.
+ * @param passthroughExceptions Exception types to rethrow as-is instead of
wrapping.
+ * @param <T> The type of value returned by this method.
+ * @return The return value of the fn that executed within the classloader.
+ */
+ @SafeVarargs
+ public final <T> T withClassLoader(
+ ThrowableFunction<ClassLoader, T> fn,
+ Class<? extends RuntimeException>... passthroughExceptions) {
+ try {
+ return withClassLoader(fn);
+ } catch (Exception e) {
+ for (Class<? extends RuntimeException> exceptionClass :
passthroughExceptions) {
+ if (exceptionClass.isInstance(e)) {
+ throw (RuntimeException) e;
+ }
+ }
+ throw new RuntimeException(e);
+ }
+ }
+
public static IsolatedClassLoader buildClassLoader(List<String>
libAndResourcesPaths) {
// Listing all the classPath under the package path and build the isolated
class loader.
List<URL> classPathContents = Lists.newArrayList();
diff --git a/core/src/test/java/org/apache/gravitino/TestCatalog.java
b/core/src/test/java/org/apache/gravitino/TestCatalog.java
index fbea6b5e59..4e8e8ada24 100644
--- a/core/src/test/java/org/apache/gravitino/TestCatalog.java
+++ b/core/src/test/java/org/apache/gravitino/TestCatalog.java
@@ -60,6 +60,13 @@ public class TestCatalog extends BaseCatalog<TestCatalog> {
return new TestCatalogOperations(config);
}
+ @Override
+ public boolean shouldValidateOnCreate() {
+ Map<String, String> properties = entity().getProperties();
+ return properties != null
+ &&
Boolean.parseBoolean(properties.get(TestCatalogOperations.VALIDATE_ON_CREATE));
+ }
+
@Override
protected Capability newCapability() {
return new TestCatalogCapabilities();
diff --git
a/core/src/test/java/org/apache/gravitino/catalog/TestCatalogManager.java
b/core/src/test/java/org/apache/gravitino/catalog/TestCatalogManager.java
index 6d2ad36bf1..ddc57cc556 100644
--- a/core/src/test/java/org/apache/gravitino/catalog/TestCatalogManager.java
+++ b/core/src/test/java/org/apache/gravitino/catalog/TestCatalogManager.java
@@ -52,6 +52,7 @@ import org.apache.gravitino.NameIdentifier;
import org.apache.gravitino.Namespace;
import org.apache.gravitino.Schema;
import org.apache.gravitino.connector.BaseCatalog;
+import org.apache.gravitino.connector.TestCatalogOperations;
import org.apache.gravitino.connector.capability.Capability;
import org.apache.gravitino.connector.capability.CapabilityResult;
import org.apache.gravitino.exceptions.CatalogAlreadyExistsException;
@@ -388,6 +389,79 @@ public class TestCatalogManager {
Assertions.assertNull(catalogManager.getCatalogCache().getIfPresent(failedIdent));
}
+ @Test
+ public void testCreateCatalogValidatesBackendConnection() {
+ Map<String, String> okProps =
+ Maps.newHashMap(
+ ImmutableMap.of(
+ PROPERTY_KEY1,
+ "value1",
+ PROPERTY_KEY2,
+ "value2",
+ PROPERTY_KEY5_PREFIX + "1",
+ "value3",
+ TestCatalogOperations.VALIDATE_ON_CREATE,
+ "true"));
+
+ // Opted in + backend resolves: creation succeeds.
+ NameIdentifier okIdent = NameIdentifier.of("metalake", "validate-ok");
+ Assertions.assertDoesNotThrow(
+ () ->
+ catalogManager.createCatalog(
+ okIdent, Catalog.Type.RELATIONAL, provider, "comment",
okProps));
+
Assertions.assertNotNull(catalogManager.getCatalogCache().getIfPresent(okIdent));
+
+ // Opted in + backend cannot be resolved: creation fails fast and leaves
nothing behind.
+ NameIdentifier failIdent = NameIdentifier.of("metalake", "validate-fail");
+ Map<String, String> failProps =
+ Maps.newHashMap(
+ ImmutableMap.of(
+ PROPERTY_KEY1,
+ "value1",
+ PROPERTY_KEY2,
+ "value2",
+ PROPERTY_KEY5_PREFIX + "1",
+ "value3",
+ TestCatalogOperations.VALIDATE_ON_CREATE,
+ "true",
+ TestCatalogOperations.FAIL_INITIALIZE,
+ "true"));
+ Throwable failure =
+ Assertions.assertThrows(
+ IllegalArgumentException.class,
+ () ->
+ catalogManager.createCatalog(
+ failIdent, Catalog.Type.RELATIONAL, provider, "comment",
failProps));
+ Assertions.assertTrue(
+ failure.getMessage().contains("backend rejected catalog
configuration"),
+ failure.getMessage());
+
Assertions.assertNull(catalogManager.getCatalogCache().getIfPresent(failIdent));
+ // Rolled back, so the identifier is reusable.
+ Assertions.assertDoesNotThrow(
+ () ->
+ catalogManager.createCatalog(
+ failIdent, Catalog.Type.RELATIONAL, provider, "comment",
okProps));
+
+ // Not opted in: the connection is not validated at create even if it
would fail.
+ NameIdentifier skipIdent = NameIdentifier.of("metalake", "validate-skip");
+ Map<String, String> skipProps =
+ Maps.newHashMap(
+ ImmutableMap.of(
+ PROPERTY_KEY1,
+ "value1",
+ PROPERTY_KEY2,
+ "value2",
+ PROPERTY_KEY5_PREFIX + "1",
+ "value3",
+ TestCatalogOperations.FAIL_INITIALIZE,
+ "true"));
+ Assertions.assertDoesNotThrow(
+ () ->
+ catalogManager.createCatalog(
+ skipIdent, Catalog.Type.RELATIONAL, provider, "comment",
skipProps));
+
Assertions.assertNotNull(catalogManager.getCatalogCache().getIfPresent(skipIdent));
+ }
+
@Test
public void testListCatalogs() {
NameIdentifier ident = NameIdentifier.of("metalake", "test11");
diff --git
a/core/src/test/java/org/apache/gravitino/connector/TestCatalogOperations.java
b/core/src/test/java/org/apache/gravitino/connector/TestCatalogOperations.java
index b6324652ca..5fd405c62f 100644
---
a/core/src/test/java/org/apache/gravitino/connector/TestCatalogOperations.java
+++
b/core/src/test/java/org/apache/gravitino/connector/TestCatalogOperations.java
@@ -134,6 +134,10 @@ public class TestCatalogOperations
public static final String FAIL_TEST = "need-fail";
+ public static final String VALIDATE_ON_CREATE = "validate-on-create";
+
+ public static final String FAIL_INITIALIZE = "fail-initialize";
+
private static final String SLASH = "/";
public TestCatalogOperations(Map<String, String> config) {
@@ -150,7 +154,11 @@ public class TestCatalogOperations
@Override
public void initialize(
Map<String, String> config, CatalogInfo info, HasPropertyMetadata
propertyMetadata)
- throws RuntimeException {}
+ throws RuntimeException {
+ if (config != null && "true".equals(config.get(FAIL_INITIALIZE))) {
+ throw new IllegalArgumentException("Simulated: backend rejected catalog
configuration");
+ }
+ }
@Override
public void close() throws IOException {}
diff --git a/docs/open-api/catalogs.yaml b/docs/open-api/catalogs.yaml
index 54fb971bd4..f568b04c52 100644
--- a/docs/open-api/catalogs.yaml
+++ b/docs/open-api/catalogs.yaml
@@ -61,6 +61,10 @@ paths:
examples:
CatalogCreate:
$ref: "#/components/examples/CatalogCreate"
+ CatalogCreateIcebergStorageBackend:
+ $ref:
"#/components/examples/CatalogCreateIcebergStorageBackend"
+ CatalogCreateIcebergRestBackend:
+ $ref: "#/components/examples/CatalogCreateIcebergRestBackend"
responses:
"200":
@@ -93,6 +97,10 @@ paths:
examples:
CatalogCreate:
$ref: "#/components/examples/CatalogCreate"
+ CatalogCreateIcebergStorageBackend:
+ $ref:
"#/components/examples/CatalogCreateIcebergStorageBackend"
+ CatalogCreateIcebergRestBackend:
+ $ref: "#/components/examples/CatalogCreateIcebergRestBackend"
responses:
"200":
description: Test connection completed
@@ -580,6 +588,47 @@ components:
}
}
+ # Iceberg with a storage backend (hive/jdbc): `warehouse` is a physical
storage LOCATION.
+ CatalogCreateIcebergStorageBackend:
+ summary: "Iceberg catalog (hive/jdbc backend): warehouse is a storage
location"
+ description: >-
+ For the `lakehouse-iceberg` provider with `catalog-backend` set to
`hive` or `jdbc`,
+ the `warehouse` property is a physical storage location (e.g.
`s3a://bucket/path` or
+ `hdfs://ns/path`) and is required.
+ value: {
+ "name": "my_iceberg_catalog",
+ "type": "relational",
+ "provider": "lakehouse-iceberg",
+ "comment": "Iceberg catalog backed by a Hive metastore",
+ "properties": {
+ "catalog-backend": "hive",
+ "uri": "thrift://127.0.0.1:9083",
+ "warehouse": "s3a://my-bucket/warehouse"
+ }
+ }
+
+ # Iceberg with the REST backend: `warehouse` is a CATALOG SELECTOR, not a
storage location.
+ CatalogCreateIcebergRestBackend:
+ summary: "Iceberg catalog (REST backend): warehouse selects a catalog,
or omit for the default"
+ description: >-
+ For the `lakehouse-iceberg` provider with `catalog-backend` set to
`rest`, the
+ `warehouse` property is NOT a storage location. It is forwarded to the
Iceberg REST
+ server (`GET /v1/config?warehouse=...`) as a catalog selector: a
catalog name for
+ Gravitino's Iceberg REST server (omit it to use the default catalog),
or a
+ server-defined identifier such as an Amazon S3 Tables bucket ARN. A
storage URI like
+ `s3://...` here is a common mistake copied from hive/jdbc examples and
will fail to
+ resolve.
+ value: {
+ "name": "my_iceberg_rest_catalog",
+ "type": "relational",
+ "provider": "lakehouse-iceberg",
+ "comment": "Iceberg catalog backed by a remote Iceberg REST server",
+ "properties": {
+ "catalog-backend": "rest",
+ "uri": "http://127.0.0.1:9001/iceberg";
+ }
+ }
+
CatalogResponse:
value: {
"code": 0,
diff --git
a/server/src/main/java/org/apache/gravitino/server/web/rest/ExceptionHandlers.java
b/server/src/main/java/org/apache/gravitino/server/web/rest/ExceptionHandlers.java
index 2476c3f7e5..6b550fd869 100644
---
a/server/src/main/java/org/apache/gravitino/server/web/rest/ExceptionHandlers.java
+++
b/server/src/main/java/org/apache/gravitino/server/web/rest/ExceptionHandlers.java
@@ -1095,6 +1095,16 @@ public class ExceptionHandlers {
String errorMsg =
getBaseErrorMsg(formattedObject, op.name(), formattedParent,
getErrorMsg(e));
+
+ // A backend a catalog federates to being unreachable is a
downstream-dependency failure, not
+ // an internal Gravitino error: surface it as 502 Bad Gateway so callers
can tell a dependency
+ // outage from a server bug.
+ if (e instanceof ConnectionFailedException) {
+ // WARN, not ERROR: a dependency outage is not a Gravitino bug, but
still trace it here.
+ LOG.warn(errorMsg, e);
+ return Utils.connectionFailed(errorMsg, e);
+ }
+
LOG.error(errorMsg, e);
return Utils.internalError(errorMsg, e);
}
