danhuawang opened a new issue, #11402:
URL: https://github.com/apache/gravitino/issues/11402
### Version
main branch
### Describe what's wrong
When `CALL gravitino.system.create_catalog(...)` is executed, the catalog
metadata is first persisted to the Gravitino server successfully, then the
Trino connector attempts to instantiate the catalog locally. If the connector
creation fails (e.g., due to invalid configuration properties), the catalog
metadata remains on the Gravitino server with **no rollback**. This leaves the
catalog in a "zombie" state:
- The catalog **exists** on the Gravitino server (so `create_catalog`
reports "already exists" on retry).
- The catalog **does not exist** in the Trino connector's local cache (so
`drop_catalog` reports "not exists" — see related Bug 1).
- The background scheduler repeatedly tries and fails to load the catalog,
logging errors every cycle.
The correct behavior should be: if the Trino connector fails to instantiate
the catalog, the system should either roll back the server-side metadata or
provide a clear mechanism for the user to clean up the failed catalog.
### Error message and/or stacktrace
Initial `create_catalog` call (or background scheduler attempting to load
the catalog):
```
2026-06-03 07:44:34 ERROR [Query-20260603_074434_00288_rkud3-393]
CatalogConnectorManager:428 - Failed to create connector: gt_glue_xxx1
io.trino.spi.TrinoException: Failed to create connector lakehouse
at
org.apache.gravitino.trino.connector.GravitinoConnectorPluginManager.createConnector(GravitinoConnectorPluginManager.java:299)
at
org.apache.gravitino.trino.connector.catalog.CatalogConnectorContext$Builder.build(CatalogConnectorContext.java:273)
at
org.apache.gravitino.trino.connector.catalog.CatalogConnectorManager.createCatalogConnectorContext(CatalogConnectorManager.java:422)
...
Caused by: io.airlift.bootstrap.ApplicationConfigurationException:
Configuration errors:
1) Error: Invalid value '${?aws_glue_endpoint}' for type URI (property
'hive.metastore.glue.endpoint-url')
in order to call [public GlueHiveMetastoreConfig
GlueHiveMetastoreConfig.setGlueEndpointUrl(URI)]
```
Background scheduler continuously retrying and failing:
```
2026-06-03 07:44:34 ERROR [gravitino-connector-schedule-0]
CatalogRegister:159 - Failed to register catalog gt_glue_xxx1
2026-06-03 07:44:34 ERROR [gravitino-connector-schedule-0]
CatalogConnectorManager:303 - Failed to create internal catalog connector.
The catalog is:
org.apache.gravitino.trino.connector.metadata.GravitinoCatalog@7ce2467e
io.trino.spi.TrinoException: Failed to register catalog gt_glue_xxx1
at
org.apache.gravitino.trino.connector.catalog.CatalogRegister.registerCatalog(CatalogRegister.java:160)
at
org.apache.gravitino.trino.connector.catalog.CatalogConnectorManager.loadCatalogImpl(CatalogConnectorManager.java:299)
...
Caused by: io.trino.spi.TrinoException: Failed to execute query:
CREATE CATALOG gt_glue_xxx1 USING gravitino WITH (
"__gravitino.dynamic.connector" = 'true',
"__gravitino.dynamic.connector.catalog.config" =
'{"metalake":"trino_connector_metalake","provider":"glue","name":"gt_glue_xxx1","properties":{"aws-glue-endpoint":"${?aws_glue_endpoint}","aws-access-key-id":"XXXX","aws-secret-access-key":"XXXX","warehouse":"${warehouse}","in-use":"true","aws-region":"us-east-1"},...}'
)
```
Subsequent `create_catalog` attempt:
```
trino> CALL gravitino.system.create_catalog('gt_glue_xxx1', 'glue', ...);
-- fails with: Catalog trino_connector_metalake.gt_glue_xxx1 already exists
in the server.
```
### How to reproduce
1. Deploy Gravitino + Trino with the Gravitino Trino connector (Trino 478).
2. Call `create_catalog` with properties containing invalid values, e.g.,
unresolved placeholders:
```sql
CALL gravitino.system.create_catalog(
'gt_glue_xxx1',
'glue',
MAP(
ARRAY['aws-glue-endpoint', 'aws-access-key-id',
'aws-secret-access-key', 'warehouse', 'aws-region'],
ARRAY['${?aws_glue_endpoint}', 'XXXX', 'XXXX', '${warehouse}',
'us-east-1']
)
);
```
3. The call may return an error (connector creation failed), but the catalog
metadata is already persisted on the Gravitino server.
4. Observe the background scheduler repeatedly logging `Failed to register
catalog gt_glue_xxx1` errors.
5. Attempt `drop_catalog('gt_glue_xxx1')` → fails with "not exists" (local
cache is empty).
6. Attempt `create_catalog('gt_glue_xxx1', ...)` again → fails with "already
exists" (server has the metadata).
7. The catalog is stuck in an unrecoverable zombie state from Trino's
perspective.
### Additional context
The issue occurs in the `create_catalog` flow across these components:
1. **`CreateCatalogStoredProcedure`** — persists catalog metadata to the
Gravitino server via the Gravitino client API.
2. **`CatalogConnectorManager.createCatalogConnectorContext()`** — attempts
to instantiate the Trino connector. If this fails (line 428-429), the exception
is thrown but the server-side metadata is never cleaned up.
3. **`CatalogRegister.registerCatalog()`** — the background scheduler path;
also fails and logs errors indefinitely.
**Suggested fix options**:
- **Option A (rollback)**: If connector creation fails in
`CreateCatalogStoredProcedure`, catch the exception and call
`gravitinoClient.metalake(metalake).dropCatalog(catalogName)` to remove the
server-side metadata. Re-throw the error with a clear message indicating what
went wrong.
- **Option B (allow drop of failed catalogs)**: Ensure
`DropCatalogStoredProcedure` can drop catalogs that exist on the server but not
in the local cache (see Bug 1 fix). This way, even if create leaves a zombie,
the user can clean it up.
- **Option C (validate before persist)**: Validate the catalog properties
(e.g., check that URI fields are parseable) before persisting to the Gravitino
server, failing fast with a clear error message.
Ideally, both Option A and Option B should be implemented for defense in
depth.
**Related issue**: This bug compounds with the `drop_catalog`
local-cache-only check (#11401 ) to make the zombie catalog completely
unmanageable from the Trino side.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
