yuqi1129 opened a new issue, #12232:
URL: https://github.com/apache/gravitino/issues/12232
### Version
main branch
### Describe what's wrong
On an external-backed catalog (Hive, Glue, JDBC, Iceberg, …), `dropTable`
deletes the store row unconditionally, even when the external drop found
nothing (`TableOperationDispatcher.java:378-386`, `// In all situations, we try
to delete the table from the store`). A `false` return means "the external
system has no such table", which in a concurrent rename is not "nothing to
clean up" but "the table is still there under a new name".
With two servers (or after TreeLock is removed), `rename t1 -> t2` on A and
`drop t1` on B can interleave so that:
1. A's external rename lands first, so t2 exists in the external catalog;
2. B's external drop finds no t1 and returns `false`, but still runs
`store.delete(t1)`;
3. A's store update for t1 -> t2 then matches 0 rows and writes nothing.
The damage is not limited to the entity row. `TableMetaService.deleteTable`
cascades, soft-deleting the owner relation, the securable-object
(role/privilege) relations, tag relations, policy relations, statistics and
table-version rows for that table id, and appends a `DROP` to
`entity_change_log`.
`importTable` (`TableOperationDispatcher.java:474-525`) restores only the
entity row and its columns in a single `store.put`, so re-import on the next
`loadTable(t2)` does **not** bring any of that back — **not even when the id is
preserved** (Hive/Iceberg carry `gravitino.identifier` in table properties and
rename carries them to t2, so the id is reused but the relation rows stay
soft-deleted). The loss is silent and has no recovery path, and it changes
authorization state.
The caller is not told either: the failed store update throws
`NoSuchEntityException`, which `OperationDispatcher.operateOnEntity`
(`OperationDispatcher.java:204-226`) swallows with a `LOG.error`, while the
REST response is built from the connector result — so the rename is reported as
successful.
### Error message and/or stacktrace
No exception reaches the client. The only trace is a server-side log line
from `operateOnEntity`:
```
ERROR ... Entity of {} does not exist //
FormattedErrorMessages.ENTITY_NOT_FOUND
```
### How to reproduce
- Gravitino main branch, an external-backed catalog (e.g. Hive), two servers
sharing one metadata store (or a single server with the alter/drop dispatcher
paths driven concurrently — the TreeLock write lock only serializes them within
one JVM).
- Create table `db.t1`, set an owner and attach a tag/policy to it.
- Concurrently: server A `alterTable(db.t1, rename -> t2)`, server B
`dropTable(db.t1)`.
- With the interleaving above: `db.t2` exists in Hive, `loadTable(db.t2)`
succeeds (re-imported), but its owner, tags and policies are gone, and no error
was returned to either caller.
### Additional context
Proposed fix, two parts:
1. Do not touch the store when the external drop returned `false`. The
behaviour this gives up — cleaning a store row after an out-of-band drop — is
already argued to be harmless elsewhere: `loadTable` asks the external system
first so a stale row is never trusted, and a later create overwrites it with
`overwrite = true`. Leaving those rows to the reconcile path trades a benign
stale row for an irreversible loss of Gravitino-only metadata.
2. Make the failure observable: a rename whose store update matches 0 rows
should not be reduced to a log line — surface it (metric/warning) so operators
can detect "renamed externally, registration not updated".
Two related hardenings are tracked separately and are **not** part of this
issue: deleting by `(id, current_version)` rather than by name (which fixes the
drop/re-create ABA but does not fix this race), and moving the relation cascade
to a deferred GC so a wrong delete stays recoverable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
