Henri Biestro created HIVE-29818:
------------------------------------
Summary: [REST Catalog] Prevent FileSystem memory leak when
executing operations under proxy UGI
Key: HIVE-29818
URL: https://issues.apache.org/jira/browse/HIVE-29818
Project: Hive
Issue Type: Bug
Components: Hive
Affects Versions: 4.2.0
Reporter: Henri Biestro
Assignee: Henri Biestro
Fix For: 4.3.0
*Problem Statement*
When the REST Catalog service executes operations on behalf of multiple
end-users using proxy authentication (`UserGroupInformation.createProxyUser`),
Hadoop's internal `FileSystem.CACHE` retains references to both the
`FileSystem` instances and their originating `UserGroupInformation` (UGI)
objects.
Because `createProxyUser` creates un-managed UGI instances, short-lived or
high-cardinality proxy requests lead to continuous memory accumulation in
`FileSystem.CACHE`. Over time, this causes significant JVM heap pressure and
eventual `OutOfMemoryError` (OOM) failures in long-running REST catalog
deployments.
*Current Behavior*
1. The REST catalog creates a proxy `UserGroupInformation` context per
user/request.
2. File system operations executed inside `ugi.doAs(...)` register new
`FileSystem` handles inside Hadoop's static `FileSystem.CACHE`.
3. When the request or user session completes, the UGI reference remains held
by `FileSystem.CACHE`, preventing garbage collection of the UGI, associated
`Subject`/`Principal` objects, and socket/IPC connections.
*Expected Behavior*
The REST Catalog service should cleanly manage the lifecycle of proxy
`UserGroupInformation` contexts and their underlying file system resources.
Upon session termination or cache eviction of a proxy UGI, all associated file
system handles should be explicitly closed using
`FileSystem.closeAllForUGI(ugi)`.
*Proposed Solution / Implementation Details*
* *UGI Cache Management:* Introduce a bounded, time-evicting
`LoadingCache<String, UserGroupInformation>` (or leverage existing session
management) for proxy users.
* *Eviction Listener Cleanup:* Register a removal/eviction listener on the UGI
cache that explicitly calls `FileSystem.closeAllForUGI(evictedUgi)` upon
session expiration or cache displacement.
* *Concurrency Guard:* Ensure eviction timeouts are configured with
appropriate buffer margins to prevent `FileSystem closed` exceptions during
active mid-flight `doAs` execution threads.
* {*}(Alternative / Configurable option){*}: Provide an optional configuration
flag to disable Hadoop FileSystem caching (`fs.hdfs.impl.disable.cache=true`)
for environments with highly ephemeral proxy user activity.
*Impact*
* Eliminates memory leaks in long-running REST catalog server instances
handling multi-tenant workloads.
* Ensures proper cleanup of underlying RPC sockets and IPC client resources
tied to proxy credentials.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
