raghav-reglobe commented on PR #64966:
URL: https://github.com/apache/doris/pull/64966#issuecomment-4957886286
Sure — here are the (sanitized) FE logs from the incident. The key detail:
**the 401 is on the catalog REST metadata path (`listNamespaces`), before any
FileIO or storage credential is ever involved**, so I don't think
apache/iceberg#11389 applies here. `VendedCredentialsProvider` refreshes the
*storage* credentials the catalog vends for data access (S3/GCS/ADLS), but what
expired in our case is the *catalog session token itself* — the OAuth2
client-credentials token sent in the `Authorization` header of catalog REST
calls. `listNamespaces` never constructs a FileIO; there are no storage
credentials anywhere in this code path.
```
WARN (mysql-nio-pool-4794|44432)
[IcebergMetadataOps.listDatabaseNames():153] failed to list database names in
catalog my_iceberg, root cause:
org.apache.iceberg.exceptions.NotAuthorizedException
org.apache.iceberg.exceptions.NotAuthorizedException: Not authorized:
at
org.apache.iceberg.rest.ErrorHandlers$DefaultErrorHandler.accept(ErrorHandlers.java:236)
at
org.apache.iceberg.rest.ErrorHandlers$NamespaceErrorHandler.accept(ErrorHandlers.java:192)
at org.apache.iceberg.rest.HTTPClient.throwFailure(HTTPClient.java:240)
at org.apache.iceberg.rest.HTTPClient.execute(HTTPClient.java:336)
at org.apache.iceberg.rest.BaseHTTPClient.get(BaseHTTPClient.java:77)
at
org.apache.iceberg.rest.RESTSessionCatalog.listNamespaces(RESTSessionCatalog.java:581)
at
org.apache.iceberg.catalog.BaseSessionCatalog$AsCatalog.listNamespaces(BaseSessionCatalog.java:133)
at
org.apache.iceberg.rest.RESTCatalog.listNamespaces(RESTCatalog.java:232)
at
org.apache.doris.datasource.iceberg.IcebergMetadataOps.listNestedNamespaces(IcebergMetadataOps.java:177)
at
org.apache.doris.datasource.iceberg.IcebergMetadataOps.listDatabaseNames(IcebergMetadataOps.java:151)
at
org.apache.doris.datasource.ExternalCatalog.listDatabaseNames(ExternalCatalog.java:274)
at
org.apache.doris.datasource.ExternalCatalog.getFilteredDatabaseNames(ExternalCatalog.java:524)
at
org.apache.doris.datasource.metacache.MetaCache.listNames(MetaCache.java:84)
at
org.apache.doris.datasource.ExternalCatalog.getDbNames(ExternalCatalog.java:696)
at
org.apache.doris.datasource.ExternalCatalog.buildDbForInit(ExternalCatalog.java:883)
at
org.apache.doris.datasource.ExternalCatalog.getDbNullable(ExternalCatalog.java:748)
...
```
(FE running a recent master build, so line numbers are approximate vs
today's master; caffeine/lambda frames elided. This trace predates my
restructure — it's why the PR originally patched `IcebergMetadataOps`.)
…which then surfaces to the user as the misleading
```
ERROR 1105 (HY000): errCode = 2, detailMessage = Unknown database 'db1'
```
A few data points from the incident that shaped the fix:
- The expired credential is the **OAuth2 client-credentials session token**
fetched from `oauth2-server-uri` at catalog init. Background refresh
(`token-refresh-enabled=true`) works fine in steady state — the wedge happened
when the auth endpoint was briefly overloaded at refresh time. After that
failed refresh, the session kept the stale token and never re-authenticated.
- The 401s ran **continuously for ~3 days** (2026-06-26 12:19 UTC →
2026-06-29 19:44 UTC), across every catalog call in that window, and stopped
only when we restarted the FE. Whatever refresh logic exists client-side never
recovered on its own.
- `REFRESH CATALOG` didn't help — it clears Doris's metadata caches but
reuses the same wedged HTTP client/session. Only rebuilding the client recovers.
- Because the failing calls are metadata calls, every query *and* DDL
against the catalog fails — and since DDL forwards to the master FE, one wedged
master breaks DDL for the whole cluster until restart.
I agree the root cause ("session never re-logins once a refresh
fails/expires") is ultimately an Iceberg client issue, and I'm happy to raise
it there as well. I'd still argue for this small recovery path in Doris though:
FEs are very long-lived processes holding one catalog client per catalog for
the process lifetime; Doris pins to an Iceberg release, so even a merged
upstream fix is releases away; and a bounded rebuild-and-retry on
`NotAuthorizedException` also covers server-side causes (e.g. an auth server
restart invalidating issued tokens before their TTL). Once the Iceberg client
can self-heal, this wrapper simply never fires.
Happy to attach fuller logs if useful.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
