atovk opened a new issue, #11039:
URL: https://github.com/apache/gravitino/issues/11039
### What happened
When an Iceberg table registered in Gravitino points to an Aliyun OSS
metadata JSON that no longer exists, Gravitino's Iceberg REST service can spend
a long time retrying the missing object. During that time, Gravitino
table/tag/policy requests may hold tree locks for more than 30 seconds, causing
the UI and metadata APIs for the same schema to appear stuck.
### Why this matters
A single stale or externally damaged Iceberg table can degrade a whole
schema's metadata browsing experience. This is especially visible in the web
UI, which may issue table detail, tags, policies, and existence checks close
together.
### Observed behavior
For a table similar to `metalake.catalog.schema.table`, where the JDBC
Iceberg catalog row still pointed at a missing metadata file, the Gravitino
server log repeatedly showed:
```text
Retrying task after failure: sleepTimeMs=... The specified key does not
exist.
<Key>.../metadata/00000-...metadata.json</Key>
```
The same time window also showed tree-lock watchdog warnings similar to:
```text
Thread with identifier
ThreadIdentifier{thread=Thread[Gravitino-webserver-...,5,main],
ident=metalake.catalog.schema.table} holds the lock node
TreeLockNode{ident=/,...} for more than 30s ... io-connection hangs
```
After unregistering/dropping the stale table entry, the same API returned
quickly with `NoSuchTableException`/404.
### Expected behavior
Missing Aliyun OSS metadata objects should be treated as a non-retryable
not-found condition and should fail quickly. A stale table should not cause
long retry loops that hold Gravitino tree locks and block unrelated metadata
browsing in the same schema.
### Root cause analysis
Gravitino 1.2.0 uses Iceberg 1.10.1. Iceberg's
`BaseMetastoreTableOperations.refreshFromMetadataLocation` stops retrying on
Iceberg `NotFoundException`, but the Aliyun OSS input stream path appears to
propagate Aliyun SDK `OSSException` for `NoSuchKey` directly from
`client.getObject(...)`. That exception is not classified as Iceberg
`NotFoundException`, so Iceberg's generic retry path keeps retrying.
By contrast, Iceberg's AWS S3 input stream maps `NoSuchKeyException` to
`NotFoundException`, allowing the retry loop to stop quickly.
Relevant upstream source references:
- Iceberg retry stops on `NotFoundException`:
`BaseMetastoreTableOperations.refreshFromMetadataLocation`
- Iceberg Aliyun OSS stream reads with `client.getObject(...)` without
translating `NoSuchKey`: `OSSInputStream.openStream`
- Iceberg AWS S3 stream translates `NoSuchKeyException` to
`NotFoundException`: `S3InputStream.openStream`
- Gravitino table operations hold tree locks around load/drop paths:
`TableOperationDispatcher.loadTable` and `dropTable`
### Possible fixes
1. In Gravitino's Iceberg REST wrapper, classify Aliyun OSS
`NoSuchKey`/`NoSuchBucket` failures from metadata loading as a not-found table
condition instead of allowing generic retries to continue.
2. Add a regression test around Iceberg REST `loadTable`/`tableExists`
behavior when metadata loading fails with Aliyun OSS `NoSuchKey`.
3. Longer term, consider avoiding heavyweight `loadTable` calls for shallow
existence checks used by UI authorization/tag/policy flows.
I am preparing a small PR for the fast-fail behavior.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
