yuqi1129 opened a new issue, #12230: URL: https://github.com/apache/gravitino/issues/12230
### Version main branch ### Describe what's wrong On a multi-node deployment with authorization enabled, a single node can permanently deny a user that its peers authorize correctly. The node keeps serving other requests normally, and only a restart clears it. The node reaches a state where `loadedRoles` records a role as loaded while the JCasbin enforcer holds none of that role's `p` rows. `versionCheckAndLoadRoles` decides whether to reload by comparing `role_meta.updated_at` against `loadedRoles` only — it never inspects the enforcer. Since `role_meta.updated_at` does not move, the reload is skipped on every subsequent request and the missing privileges never come back. Three defects combine to produce and then pin that state: 1. **A partial load is recorded as complete.** `loadPolicyByRoleEntity` silently skips a securable object whose metadata id cannot be resolved, but `versionCheckAndLoadRoles` calls `loadedRoles.put(roleId, dbUpdatedAt)` unconditionally afterwards. A role whose objects failed to resolve — even once, even transiently — is marked as loaded with zero or partial policies. 2. **Policy mutations are not serialized against cache eviction.** `SyncedEnforcer` makes each call atomic, but the `clearRolePolicies` → re-add sequence is not, and neither is it ordered against the `JcasbinLoadedRolesCache` removal listener, which clears policies from whichever thread drains Caffeine's maintenance queue. An eviction landing between another thread's policy writes and its `loadedRoles` update erases the rows that thread just wrote while leaving the marker claiming they are loaded. 3. **The TTL cannot expire the bad state away.** `JcasbinLoadedRolesCache` uses `expireAfterAccess`, unlike every other authorization cache. `versionCheckAndLoadRoles` probes the entry on every request carrying the role, so each *denied* request renews the very entry that suppresses the reload. On a node under steady traffic the entry never expires. ### Error message and/or stacktrace No exception is thrown. The authorization check simply returns `false`, and no role reload is attempted, so there is no `listSecurableObjectsByRoleId` query around the failure either. ### How to reproduce Not reproducible on demand — it depends on a metadata-id lookup failing during a role load, or on an eviction interleaving with a concurrent load. It can be reproduced deterministically at the unit level: - Stub `MetadataIdConverter.getID` to return `Optional.empty()`, invoke `versionCheckAndLoadRoles`, and observe that `loadedRoles` records the role although the enforcer holds no policy for it. - Hold a lock around the enforcer mutations and observe that `clearRolePolicies` proceeds concurrently. - Read a `JcasbinLoadedRolesCache` entry more often than its TTL and observe that it never expires. ### Additional context Same area as #12169, but a distinct defect: #12169 fixed the removal listener deleting the role's grouping rows along with its policies, whereas this one is about the enforcer losing the policies while `loadedRoles` still claims they are present. Observed on a three-pod deployment: one pod denied `USE_CATALOG` on a metalake six times over an hour while the other two pods hit the same role's policy throughout. The failing pod never logged a role reload; a peer recovered on its own only after its cache entry went idle long enough to expire. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
