echobeijingecho opened a new issue, #66401: URL: https://github.com/apache/doris/issues/66401
### Search before asking - [x] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues. ### Version doris-4.0.7-rc02 (commit 35854e7e92a). Not reproducible on 4.0.5: with identical catalogs and config, fe.log shows zero ugi=<process-user> HMS connections and no row-count-refresh HMS activity there. ### What's Wrong? ### Environment - HMS-type Paimon catalog with `"hadoop.username" = "hive"` (plus an HMS `hive` catalog pointing to the same metastore) - HiveMetastore has `hive.metastore.execute.setugi=true` + `AuthorizationPreEventListener` + `StorageBasedAuthorizationProvider`, so on `get_table` HMS performs HDFS `checkAccess` **as the client-reported user** - Paimon warehouse dir `/paimon` is `drwx------` (accessible only to privileged users; `hive` is allowed via Ranger) - Doris FE process runs as `root`, which has no HDFS permissions ### Symptom Queries on Paimon tables **intermittently** fail (~4% on the incident day, in bursts of a few minutes, self-healing afterwards): ``` errCode = 2, detailMessage = failed to load paimon table <ctl>.paimon.<tbl>: Failed to get Paimon table:<ctl>.paimon.<tbl>$null, because MetaException: java.security.AccessControlException: Permission denied: user=root, access=EXECUTE, inode="/paimon":paimon:hdfs:drwx------ at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(...) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkAccess(...) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.checkAccess(...) ``` Note `user=root` — the FE **process user** — even though the catalog sets `hadoop.username=hive`. ### Root cause analysis (source-verified on 4.0.7-rc02) 1. With `setugi=true`, connection identity is fixed once at the `set_ugi` handshake when the connection is **opened**. Wrapping later RPCs in doAs cannot change it. 2. The async row-count refresh path has **no ExecutionAuthenticator wrapping**: `ExternalRowCountCache.RowCountCacheLoader.doLoad` -> `loadRowCount` -> `ExternalTable.fetchRowCountWithMetaCache` -> `PaimonExternalTable.fetchRowCount` runs on `RowCountRefreshExecutor` threads with no doAs, so HMS clients created from this context carry the FE process user. fe.log evidence: every `RetryingMetaStoreClient ... ugi=root (auth:SIMPLE)` creation comes from `NotCheckpointRowCountRefreshExecutor-*` threads, while query threads (`mysql-nio-pool-*`) always log `ugi=hive`. 3. Paimon's `CachedClientPool` is a JVM-wide **static** cache whose key excludes UGI by default (clientClassName + metastore uris + identifier); clients are created lazily on the borrowing thread and evicted every 5 min (`client-pool-cache.eviction-interval-ms`). A connection created in a root context is later served to **properly wrapped** callers — the failing call in our incident was `PaimonExternalCatalog.getPaimonTable`, which *is* wrapped in `executionAuthenticator.execute(...)` (the `$null` suffix in the error message comes from its catch block `"$" + queryType`). 4. The same defect shape exists in `ThriftHMSCachedClient`: the inner `ThriftHMSClient` constructor calls `RetryingMetaStoreClient.getProxy(...)` **outside** the authenticator; only individual RPCs are wrapped in `ugiDoAs` — too late under `setugi`. ### Observed race timeline Client-pool eviction at T (5-min cycle) -> ~30s later the row-count refresh thread happened to be the next connection creator -> 2 connections opened with `ugi=root` (fe.log) -> for the next minutes, queries borrowing those pooled connections failed with `Permission denied: user=root`, then the poisoned connections were evicted and everything self-healed. The bug stayed latent for 3 days after upgrading from 4.0.5, then produced 2 bursts (32 failed queries) in one day. ### Suggested fixes 1. Wrap `ExternalRowCountCache.loadRowCount` in the owning catalog's `ExecutionAuthenticator.execute(...)`. 2. Wrap `ThriftHMSClient` creation (connection open) in the authenticator — identity must bind at connection creation, not per-RPC. 3. (Paimon side) consider including `user_name` in `CachedClientPool`'s default cache key (`client-pool-cache.keys` already supports it). ### Related - #31478 (hadoop.username not effective for filesystem Paimon catalog, fixed earlier) - #32828 (hive catalog fails to pass hadoop username) - Workaround we applied: inject `HADOOP_USER_NAME=hive` into the FE process environment so un-wrapped threads default to the intended identity. ### What You Expected? Every HMS connection created by FE should carry the catalog's `hadoop.username` identity, regardless of which thread creates it. Background/statistics threads must not create connections that authenticate as the FE process user, and identity-poisoned connections must never be shared with query execution paths. ### How to Reproduce? 1. HiveMetastore with `hive.metastore.execute.setugi=true` + `hive.metastore.pre.event.listeners=AuthorizationPreEventListener` + `hive.security.metastore.authorization.manager=StorageBasedAuthorizationProvider`; Paimon warehouse dir mode 700 owned by a dedicated user (so only the configured `hadoop.username` may pass the storage check). 2. Doris FE process running as a user WITHOUT HDFS permissions (e.g. root); create an HMS-type Paimon catalog with `"hadoop.username" = "<privileged user>"`. 3. Query Paimon tables continuously — all succeed (query threads are correctly wrapped, connections carry the configured user). 4. Wait until a client-pool eviction (default every 5 min, `client-pool-cache.eviction-interval-ms`) coincides with the async row-count refresh thread (`RowCountRefreshExecutor`) being the next connection creator. From that moment, queries borrowing the poisoned pooled connection fail with `Permission denied: user=<process user>` for a few minutes until the next eviction. Timing dependent — in our production it stayed latent for 3 days, then produced 2 bursts / 32 failed queries in one day. 5. Grep evidence: `grep "RetryingMetaStoreClient" fe.log | grep "ugi="` — creations from `NotCheckpointRowCountRefreshExecutor-*` threads log the process user, creations from `mysql-nio-pool-*` threads log the configured `hadoop.username`. On 4.0.5 the row-count-refresh HMS activity does not exist and no process-user connections ever appear. ### Anything Else? _No response_ ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
