nevzheng opened a new pull request, #11959:
URL: https://github.com/apache/gravitino/pull/11959
### What changes were proposed in this pull request?
Validate catalog properties at create time so an unresolvable Iceberg REST
`warehouse` is rejected with an actionable HTTP 400 instead of succeeding and
failing cryptically on every later operation.
- **core**: In `CatalogManager.createCatalogWrapper`, on the create path,
catalogs that opt in via a new `BaseCatalog#shouldValidateConnectionForCreate`
hook (default `false`) are eagerly initialized (`wrapper.catalog.ops()`), so a
configuration the backend cannot resolve is rejected at create time. A failed
create is rolled back as before.
- **iceberg**: `IcebergCatalog` opts in only when `warehouse` is set.
`IcebergCatalogOperations.initialize` translates a REST warehouse-resolution
failure (a `NoSuchWarehouseException` anywhere in the cause chain) into an
`IllegalArgumentException` (HTTP 400) with a hint: remove the property to use
the server's default catalog, or set it to a catalog name the server
recognizes. All other initialization failures propagate unchanged; hive/jdbc
behavior is unchanged. Resolvable URI/ARN-shaped warehouses (e.g. AWS S3 Tables
ARNs) still work, because validation resolves the value against the server
instead of pattern-matching it.
- **docs**: Contrasting OpenAPI examples for creating a lakehouse-iceberg
catalog with a storage backend (`warehouse` = storage location) vs. the REST
backend (`warehouse` = catalog selector, or omitted).
Error users now see at create time (from the integration run):
```
Failed to operate catalog(s) [...] operation [CREATE] under metalake [...],
reason [The 'warehouse' value 's3://not-a-real-catalog/' could not be resolved
by the Iceberg REST server. On the REST backend 'warehouse' selects a catalog
by name on the remote server and is not a storage location; remove 'warehouse'
to use the server's default catalog, or set it to a catalog name/identifier
that the server recognizes.]
```
### Why are the changes needed?
On the lakehouse-iceberg catalog with `catalog-backend=rest`, the
`warehouse` property is a catalog selector forwarded to the Iceberg REST server
as `GET /v1/config?warehouse=...`, not a storage location. A user copying a
hive/jdbc example sets `warehouse` to a storage URI (e.g. `s3://warehouse/`);
creation succeeds (HTTP 200) and every later operation fails with a cryptic
`NoSuchWarehouseException` that gives no hint the property is the problem.
Fix: #11943
### Does this PR introduce _any_ user-facing change?
- Creating a lakehouse-iceberg catalog with `catalog-backend=rest` and a
`warehouse` value the REST server cannot resolve now fails at create time with
HTTP 400 and an actionable message, instead of succeeding and failing on first
use. No property keys are added or removed; hive/jdbc backends and REST
catalogs without `warehouse` are unaffected.
- New `public` connector API:
`BaseCatalog#shouldValidateConnectionForCreate` (default `false`), letting
catalog implementations opt in to create-time connection validation.
- New OpenAPI examples for the two backends (docs only).
### How was this patch tested?
**Unit tests** (`./gradlew :core:test
:catalogs:catalog-lakehouse-iceberg:test -PskipITs`):
- core `TestCatalogManager#testCreateCatalogValidatesConnectionWhenOptedIn`:
create fails fast when the opted-in catalog's initialization fails, the failed
create is rolled back, and catalogs that do not opt in are not eagerly
initialized.
- iceberg
`TestIcebergCatalogOperations#testTranslateInitializationFailureRestUnresolvableWarehouse`,
`#testTranslateInitializationFailureLeavesOtherFailuresUnchanged`,
`#testTestConnectionNonRestWarehouseHasNoHint`: the translation only fires for
REST + `warehouse` + `NoSuchWarehouseException`; everything else propagates
unchanged.
- iceberg `TestIcebergCatalog#testShouldValidateConnectionForCreate`: opts
in only when a non-blank `warehouse` is configured.
**Integration test**:
`CatalogIcebergRestIT#testCreateRestCatalogWithUnresolvableWarehouseFailsAtCreate`
— creating a REST catalog with `warehouse=s3://not-a-real-catalog/` throws
`IllegalArgumentException` with the hint, and the catalog is not left behind.
Verified both with the fix (passes) and with the fix disabled (fails with
"Expected IllegalArgumentException to be thrown, but nothing was thrown"),
proving the test captures the bug.
**Manual verification**
Three cases were exercised against a server built from this branch
(memory-backed Iceberg REST service). To test a **valid named selector**, a
named catalog is registered on the IRC first:
```bash
# Register a named catalog on the built-in Iceberg REST service (for case A):
cat >> distribution/package/conf/gravitino.conf <<'CONF'
gravitino.iceberg-rest.catalog.probe.catalog-backend = memory
gravitino.iceberg-rest.catalog.probe.warehouse = /tmp/probe-warehouse
CONF
./gradlew compileDistribution -x test
./distribution/package/bin/gravitino.sh start
curl -s -X POST -H 'Content-Type: application/json' -d '{"name":"demo"}'
http://localhost:8090/api/metalakes
C='http://localhost:8090/api/metalakes/demo/catalogs';
IRC='http://localhost:9001/iceberg'
# A. VALID selector — warehouse is a catalog the IRC serves ->
200 (created)
curl -s -X POST -H 'Content-Type: application/json' -d
'{"name":"cat_named","type":"RELATIONAL","provider":"lakehouse-iceberg","properties":{"catalog-backend":"rest","uri":"'"$IRC"'","warehouse":"probe"}}'
"$C"
# B. INVALID selector — a storage URI ->
400 (rejected + hint)
curl -s -X POST -H 'Content-Type: application/json' -d
'{"name":"cat_bad","type":"RELATIONAL","provider":"lakehouse-iceberg","properties":{"catalog-backend":"rest","uri":"'"$IRC"'","warehouse":"s3://warehouse/"}}'
"$C"
# C. OMITTED — the server's default catalog ->
200 (created)
curl -s -X POST -H 'Content-Type: application/json' -d
'{"name":"cat_default","type":"RELATIONAL","provider":"lakehouse-iceberg","properties":{"catalog-backend":"rest","uri":"'"$IRC"'"}}'
"$C"
curl -s "$C" # only the resolvable catalogs exist;
cat_bad was rolled back
./distribution/package/bin/gravitino.sh stop
```
Result (responses trimmed to `code`/`type`/`message`):
```jsonc
// A. VALID selector: warehouse=probe -> HTTP 200, created (and
listing its schemas returns 200)
{"code":0,"catalog":{"name":"cat_named","type":"relational","provider":"lakehouse-iceberg","properties":{"catalog-backend":"rest","warehouse":"probe","uri":"http://localhost:9001/iceberg","in-use":"true"},
...}}
// B. INVALID selector: warehouse=s3://warehouse/ -> HTTP 400
{"code":1001,"type":"IllegalArgumentException","message":"Failed to operate
catalog(s) [cat_bad] operation [CREATE] under metalake [demo], reason [The
'warehouse' value 's3://warehouse/' could not be resolved by the Iceberg REST
server. On the REST backend 'warehouse' selects a catalog by name on the remote
server and is not a storage location; remove 'warehouse' to use the server's
default catalog, or set it to a catalog name/identifier that the server
recognizes.]"}
// C. OMITTED: no warehouse (default catalog) -> HTTP 200, created
{"code":0,"catalog":{"name":"cat_default","type":"relational","provider":"lakehouse-iceberg","properties":{"catalog-backend":"rest","uri":"http://localhost:9001/iceberg","in-use":"true"},
...}}
// final catalog list: only the resolvable catalogs; the rejected one was
rolled back
{"code":0,"identifiers":[{"namespace":["demo"],"name":"cat_default"},{"namespace":["demo"],"name":"cat_named"}]}
```
Note case **A**: a valid `warehouse` selector (including URI/ARN-shaped ones
some servers use, e.g. AWS S3 Tables) still succeeds, because validation
resolves the value against the server rather than pattern-matching it. Before
this change, case **B** returned `200 OK` and only failed later, on the first
operation, as `NoSuchWarehouseException: Couldn't find Iceberg configuration
for catalog s3://warehouse/`.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
