henrib opened a new pull request, #6704: URL: https://github.com/apache/hive/pull/6704
### What changes were proposed in this pull request? The REST Catalog creates a fresh proxy `UserGroupInformation` per request via `UserGroupInformation.createProxyUser`. Hadoop's `FileSystem.CACHE` retains a reference to every such UGI (and its RPC/IPC resources), so under proxy authentication these short-lived UGIs accumulate and eventually exhaust memory in long-running deployments. This PR caches the proxy UGI in `ServletSecurity` using a bounded, idle-evicting Caffeine cache keyed by `(realUser, loginUser)`. Evicted entries release their resources via `FileSystem.closeAllForUGI`. Two new config vars tune the cache: - `metastore.catalog.servlet.ugi.cache.size` (default 1000) - `metastore.catalog.servlet.ugi.cache.expiry` (default 3600s, 0 disables expiry) A note in the code documents why eviction-while-in-use is not reference-counted: eviction is idle-based (`expireAfterAccess`) and both the expiry window and max size are expected to be kept well above the longest operation / peak concurrent distinct users. ### Why are the changes needed? To prevent the `OutOfMemoryError` caused by unbounded accumulation of proxy UGIs and their associated FileSystem/IPC resources in long-running REST Catalog deployments. ### Does this PR introduce _any_ user-facing change? Two new (optional) metastore configuration properties, both with sensible defaults. ### How was this patch tested? Added `TestServletSecurity` covering per-user caching, distinct proxies per user, eviction-triggered `FileSystem.closeAllForUGI` cleanup, and disabled expiry. All 4 tests pass. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
