danhuawang opened a new issue, #11262:
URL: https://github.com/apache/gravitino/issues/11262
### Version
main branch
### Describe what's wrong
When using the Gravitino Java client to drop a middle-layer hierarchical
schema (e.g., `A:B`) after
its child schema (e.g., `A:B:C`) has already been dropped,
`catalog.asSchemas().dropSchema("A:B", false)`
returns `false` instead of `true`, even though the schema entity still
exists in the Gravitino store.
The root cause is a mismatch between the Gravitino store state and the
underlying Iceberg catalog state:
1. User creates hierarchical schema `A:B:C` (auto-creates `A:B` and `A`).
2. User creates a table `tbl_mid` in `A:B`.
3. User drops the leaf schema `A:B:C` via Trino (`DROP SCHEMA
catalog."A.B.C"`).
4. User drops `tbl_mid` from `A:B` via Trino (`DROP TABLE
catalog."A.B".tbl_mid`).
5. User calls `catalog.asSchemas().dropSchema("A:B", false)` via Gravitino
Java client.
At step 5, the Iceberg namespace `["A", "B"]` no longer exists in the
underlying Iceberg JDBC catalog
So `IcebergCatalogOperations.dropSchema` catches `NoSuchNamespaceException`
and returns `false`.
The user expects`dropSchema` to return `true`.
**Expected behavior**: `dropSchema("A:B", false)` should return `true` and
clean up the Gravitino
store entity when the schema no longer exists in the underlying catalog but
still exists in the store.
**Actual behavior**: Returns `false`, leaving the user confused about
whether the schema was dropped.
### Error message and/or stacktrace
```
2026-05-27 11:50:59.388 INFO [iceberg-rest-38]
[org.apache.gravitino.iceberg.service.CatalogWrapperForREST.injectCredentialConfig(CatalogWrapperForREST.java:325)]
- Generate credential: s3-token for Iceberg table:
ns_e2e_27b0ac35_tc15.parent.tbl_mid
2026-05-27 11:50:59.388 INFO [iceberg-rest-38]
[org.apache.gravitino.audit.FileAuditWriter.doWrite(FileAuditWriter.java:71)] -
[2026-05-27 11:50:59] service-account-postman-client UNKNOWN_OPERATION
test.catalog_iceberg_s3.ns_e2e_27b0ac35_tc15.parent.tbl_mid SUCCESS
2026-05-27 11:50:59.397 INFO [iceberg-rest-31]
[org.apache.gravitino.iceberg.service.rest.IcebergTableOperations.dropTable(IcebergTableOperations.java:260)]
- Drop Iceberg table, catalog: catalog_iceberg_s3, namespace:
ns_e2e_27b0ac35_tc15.parent, table: tbl_mid, purgeRequested: true
2026-05-27 11:50:59.398 INFO [iceberg-rest-31]
[org.apache.iceberg.BaseMetastoreTableOperations.refreshFromMetadataLocation(BaseMetastoreTableOperations.java:190)]
- Refreshing table metadata from new version:
s3://irc-e2e-env1-metalake-b-warehouse/gravitino-e2e-env2-oauth2-auth/test/catalog_iceberg_s3/ns_e2e_27b0ac35_tc15/parent/tbl_mid-8b8c08933c3643fd9abdce29c5d02e0d/metadata/00001-da28e5c2-9279-4840-b432-3cf60858e257.metadata.json
2026-05-27 11:51:00.779 INFO [iceberg-rest-31]
[org.apache.iceberg.CatalogUtil.dropTableData(CatalogUtil.java:115)] -
Manifests to delete: GenericManifestFile{content=DATA,
path=s3://irc-e2e-env1-metalake-b-warehouse/gravitino-e2e-env2-oauth2-auth/test/catalog_iceberg_s3/ns_e2e_27b0ac35_tc15/parent/tbl_mid-8b8c08933c3643fd9abdce29c5d02e0d/metadata/4d894834-6634-4e95-83a7-6501f728561d-m0.avro,
length=7130, partition_spec_id=0, added_snapshot_id=6056422233030878467,
added_data_files_count=1, added_rows_count=1, existing_data_files_count=0,
existing_rows_count=0, deleted_data_files_count=0, deleted_rows_count=0,
partitions=[], key_metadata=null, sequence_number=2, min_sequence_number=2,
first_row_id=null}
2026-05-27 11:51:08.375 INFO [iceberg-rest-31]
[org.apache.iceberg.jdbc.JdbcCatalog.dropTable(JdbcCatalog.java:319)] - Dropped
table: ns_e2e_27b0ac35_tc15.parent.tbl_mid
2026-05-27 11:51:08.376 INFO [iceberg-rest-31]
[org.apache.gravitino.audit.FileAuditWriter.doWrite(FileAuditWriter.java:71)] -
[2026-05-27 11:51:08] service-account-postman-client UNKNOWN_OPERATION
test.catalog_iceberg_s3.ns_e2e_27b0ac35_tc15.parent.tbl_mid SUCCESS
2026-05-27 11:51:08.427 INFO [Gravitino-webserver-52]
[org.apache.gravitino.server.web.rest.SchemaOperations.dropSchema(SchemaOperations.java:245)]
- Received drop schema request:
test.catalog_iceberg_s3.ns_e2e_27b0ac35_tc15:parent
2026-05-27 11:51:08.429 INFO [Gravitino-webserver-52]
[org.apache.gravitino.audit.FileAuditWriter.doWrite(FileAuditWriter.java:71)] -
[2026-05-27 11:51:08] service-account-postman-client LOAD_CATALOG
test.catalog_iceberg_s3 SUCCESS
2026-05-27 11:51:08.430 INFO [Gravitino-webserver-52]
[org.apache.gravitino.audit.FileAuditWriter.doWrite(FileAuditWriter.java:71)] -
[2026-05-27 11:51:08] service-account-postman-client LOAD_SCHEMA
test.catalog_iceberg_s3.ns_e2e_27b0ac35_tc15:parent FAILURE
2026-05-27 11:51:08.430 WARN [Gravitino-webserver-52]
[org.apache.gravitino.authorization.AuthorizationUtils.getMetadataObjectLocation(AuthorizationUtils.java:631)]
- Failed to get location paths for metadata object
test.catalog_iceberg_s3.ns_e2e_27b0ac35_tc15:parent type SCHEMA
org.apache.gravitino.exceptions.NoSuchSchemaException: Iceberg schema
(database) does not exist: ns_e2e_27b0ac35_tc15:parent in Gravitino store
at
org.apache.gravitino.catalog.lakehouse.iceberg.IcebergCatalogOperations.loadSchema(IcebergCatalogOperations.java:274)
at
org.apache.gravitino.catalog.lakehouse.iceberg.IcebergCatalogOperations.loadSchema(IcebergCatalogOperations.java:93)
at
org.apache.gravitino.catalog.SchemaOperationDispatcher.lambda$internalLoadSchema$20(SchemaOperationDispatcher.java:456)
at
org.apache.gravitino.catalog.CatalogManager$CatalogWrapper.lambda$doWithSchemaOps$0(CatalogManager.java:144)
at
org.apache.gravitino.utils.IsolatedClassLoader.withClassLoader(IsolatedClassLoader.java:86)
at
org.apache.gravitino.catalog.CatalogManager$CatalogWrapper.doWithSchemaOps(CatalogManager.java:139)
at
org.apache.gravitino.catalog.SchemaOperationDispatcher.lambda$internalLoadSchema$21(SchemaOperationDispatcher.java:456)
at
org.apache.gravitino.catalog.OperationDispatcher.doWithCatalog(OperationDispatcher.java:98)
at
org.apache.gravitino.catalog.SchemaOperationDispatcher.internalLoadSchema(SchemaOperationDispatcher.java:454)
at
org.apache.gravitino.catalog.SchemaOperationDispatcher.lambda$loadSchema$8(SchemaOperationDispatcher.java:192)
at
org.apache.gravitino.lock.TreeLockUtils.doWithTreeLock(TreeLockUtils.java:49)
at
org.apache.gravitino.catalog.SchemaOperationDispatcher.loadSchema(SchemaOperationDispatcher.java:192)
at
org.apache.gravitino.catalog.SchemaNormalizeDispatcher.loadSchema(SchemaNormalizeDispatcher.java:71)
at
org.apache.gravitino.listener.SchemaEventDispatcher.loadSchema(SchemaEventDispatcher.java:116)
at
org.apache.gravitino.hook.SchemaHookDispatcher.loadSchema(SchemaHookDispatcher.java:160)
at
org.apache.gravitino.authorization.AuthorizationUtils.getMetadataObjectLocation(AuthorizationUtils.java:553)
at
org.apache.gravitino.hook.SchemaHookDispatcher.dropSchema(SchemaHookDispatcher.java:174)
at
org.apache.gravitino.server.web.rest.SchemaOperations.lambda$dropSchema$5(SchemaOperations.java:251)
at
java.base/java.security.AccessController.doPrivileged(AccessController.java:712)
at java.base/javax.security.auth.Subject.doAs(Subject.java:439)
at
org.apache.gravitino.utils.PrincipalUtils.doAs(PrincipalUtils.java:53)
at org.apache.gravitino.server.web.Utils.doAs(Utils.java:209)
at
org.apache.gravitino.server.web.rest.SchemaOperations.dropSchema(SchemaOperations.java:247)
at
org.apache.gravitino.server.web.rest.SchemaOperations_$$_jvst915_3._d4dropSchema(SchemaOperations_$$_jvst915_3.java)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:569)
at
org.glassfish.hk2.utilities.reflection.ReflectionHelper.invoke(ReflectionHelper.java:1268)
at
org.jvnet.hk2.internal.MethodInterceptorHandler$MethodInvocationImpl.proceed(MethodInterceptorHandler.java:164)
at
org.apache.gravitino.server.web.filter.GravitinoInterceptionService$MetadataAuthorizationMethodInterceptor.invoke(GravitinoInterceptionService.java:212)
at
org.jvnet.hk2.internal.MethodInterceptorHandler.invoke(MethodInterceptorHandler.java:97)
at
org.apache.gravitino.server.web.rest.SchemaOperations_$$_jvst915_3.dropSchema(SchemaOperations_$$_jvst915_3.java)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:569)
at
org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52)
at
org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:146)
at
org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:189)
at
org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:176)
at
org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:93)
at
org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:478)
at
org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:400)
at
org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:81)
at
org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:256)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)
at org.glassfish.jersey.internal.Errors.process(Errors.java:292)
at org.glassfish.jersey.internal.Errors.process(Errors.java:274)
at org.glassfish.jersey.internal.Errors.process(Errors.java:244)
at
org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265)
at
org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:235)
at
org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:684)
at
org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:394)
at
org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:346)
at
org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:358)
at
org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:311)
at
org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:205)
at
org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)
at
org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1656)
at
org.apache.gravitino.server.authentication.AuthenticationFilter.lambda$doFilter$0(AuthenticationFilter.java:96)
at
java.base/java.security.AccessController.doPrivileged(AccessController.java:712)
at java.base/javax.security.auth.Subject.doAs(Subject.java:439)
at
org.apache.gravitino.utils.PrincipalUtils.doAs(PrincipalUtils.java:53)
at
org.apache.gravitino.server.authentication.AuthenticationFilter.doFilter(AuthenticationFilter.java:93)
at
org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
at
org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626)
at
org.apache.gravitino.server.web.VersioningFilter.doFilter(VersioningFilter.java:111)
at
org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
at
org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626)
at
org.apache.gravitino.server.web.RequestContextFilter.doFilter(RequestContextFilter.java:59)
at
org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
at
org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626)
at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:552)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:600)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
at
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
at
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440)
at
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
at
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:505)
at
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
at
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
at
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.eclipse.jetty.server.Server.handle(Server.java:516)
at
org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487)
at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479)
at
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
at
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
at
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: org.apache.iceberg.exceptions.NoSuchNamespaceException: Namespace
does not exist: ns_e2e_27b0ac35_tc15.parent
at
org.apache.iceberg.jdbc.JdbcCatalog.loadNamespaceMetadata(JdbcCatalog.java:508)
at
org.apache.iceberg.rest.CatalogHandlers.loadNamespace(CatalogHandlers.java:175)
at
org.apache.gravitino.iceberg.common.ops.IcebergCatalogWrapper.loadNamespace(IcebergCatalogWrapper.java:182)
at
org.apache.gravitino.catalog.lakehouse.iceberg.IcebergCatalogOperations.loadSchema(IcebergCatalogOperations.java:257)
... 105 more
2026-05-27 11:51:08.432 WARN [Gravitino-webserver-52]
[org.apache.gravitino.catalog.lakehouse.iceberg.IcebergCatalogOperations.dropSchema(IcebergCatalogOperations.java:362)]
- Iceberg schema (database) ns_e2e_27b0ac35_tc15:parent does not exist
2026-05-27 11:51:08.530 INFO [Gravitino-webserver-52]
[org.apache.gravitino.audit.FileAuditWriter.doWrite(FileAuditWriter.java:71)] -
[2026-05-27 11:51:08] service-account-postman-client DROP_SCHEMA
test.catalog_iceberg_s3.ns_e2e_27b0ac35_tc15:parent SUCCESS
2026-05-27 11:51:08.530 WARN [Gravitino-webserver-52]
[org.apache.gravitino.server.web.rest.SchemaOperations.lambda$dropSchema$5(SchemaOperations.java:256)]
- Fail to drop schema ns_e2e_27b0ac35_tc15:parent under namespace
test.catalog_iceberg_s3
```
### How to reproduce
1. Use Gravitino main branch with Iceberg catalog (JDBC backend).
2. Create a 3-level hierarchical schema:
```java
catalog.asSchemas().createSchema("A:B:C", "test", Collections.emptyMap());
```
3. Create a table in the middle level via Trino:
```sql
CREATE TABLE catalog."A.B".tbl_mid (id bigint);
INSERT INTO catalog."A.B".tbl_mid VALUES (1);
```
4. Drop the leaf schema via Trino:
```sql
DROP TABLE catalog."A.B.C".some_table; -- if any
DROP SCHEMA catalog."A.B.C";
```
5. Drop the table in the middle level via Trino:
```sql
DROP TABLE catalog."A.B".tbl_mid;
```
6. Attempt to drop the middle schema via Gravitino client:
```java
boolean result = catalog.asSchemas().dropSchema("A:B", false);
// Expected: true
// Actual: false
```
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
