Indhumathi Muthumurugesh created HIVE-29708:
-----------------------------------------------
Summary: HMS OOM: FileSystem.CACHE leaks DistributedFileSystem
instances in TxnUtils.findUserToRunAs() when runAs proxy user lacks HDFS
execute permission
Key: HIVE-29708
URL: https://issues.apache.org/jira/browse/HIVE-29708
Project: Hive
Issue Type: Bug
Affects Versions: 4.2.0
Reporter: Indhumathi Muthumurugesh
h3. Background: The {{runAs}} User in Hive Compaction
When the Compactor Initiator determines that an ACID table needs compaction, it
must decide {*}which user to run the compaction job as{*}. This is done via
{{{}TxnUtils.findUserToRunAs(){}}}, which is called for every potential
compaction candidate on every Initiator cycle.
The method works as follows:
{code:java}
TxnUtils.findUserToRunAs(location, table, conf)
│
├── Step 1: Check if hive.compactor.run.as.user is configured
│ └── If set → return that user immediately (no HDFS access needed)
│
├── Step 2: Try as the HMS server user (e.g. "hive")
│ └── fs.getFileStatus(tableLocation)
│ ├── SUCCESS → return directory owner as runAs user
│ └── AccessControlException → fall through to Step 3
│
└── Step 3: Try as the table owner (proxy user)
├── UserGroupInformation.createProxyUser(table.getOwner(), loginUser)
└── ugi.doAs(() -> {
FileSystem proxyFs = p.getFileSystem(conf) ← BUG: CACHE entry
added here
proxyFs.getFileStatus(tableLocation)
├── SUCCESS → return owner, then closeAllForUGI (cleanup)
└── Exception → closeAllForUGI SKIPPED ← THE LEAK
}) {code}
The proxy user path (Step 3) is entered when:
* hive.compactor.run.as.user is NOT configured (default — no fixed runAs user)
* The HMS server user does NOT have permission to stat the table's storage
location
This is a common scenario in multi-tenant clusters where:
* ACID tables are created by end users in HDFS paths outside the managed
warehouse
* The HDFS parent directory has restrictive permissions (e.g., drwx------) not
accessible to the HMS service account
* The HMS service account is not a superuser and does not have blanket read
access to all data directories
h3. Problem Statement
HiveMetaStore (HMS) processes experience periodic OutOfMemoryError crashes over
time. The crash is caused by {{org.apache.hadoop.conf.Configuration}} objects
accumulating unboundedly in the heap via
{{{}org.apache.hadoop.fs.FileSystem.CACHE{}}}, a static {{HashMap}} that caches
{{DistributedFileSystem}} instances.
The issue is triggered by the Compactor Initiator specifically in the *proxy
user fallback path* of {{{}TxnUtils.findUserToRunAs(){}}}. Disabling compaction
({{{}hive.compactor.initiator.on=false{}}}) or configuring
{{hive.compactor.run.as.user}} prevents the OOM.
h3. Heap Dump Analysis
The {{FileSystem.CACHE}} retains the leaked {{DistributedFileSystem}} instances
as GC roots:
{code:java}
org.apache.hadoop.conf.Configuration (×N*2 instances)
└── org.apache.hadoop.hdfs.DFSClient.conf
└── org.apache.hadoop.hdfs.DistributedFileSystem.dfs (×N instances)
└── java.util.HashMap$Node.value
└── java.util.HashMap (N elements)
└── org.apache.hadoop.fs.FileSystem$Cache.map
└── static CACHE in
org.apache.hadoop.fs.FileSystem
└── [GC ROOT — never collected] {code}
Each leaked DistributedFileSystem retains approximately 300-400 KB of heap (2×
Configuration objects, DFSClient, SaslDataTransferClient,
SaslPropertiesResolver). After weeks of uptime with many affected tables, this
accumulates to gigabytes of retained heap.
h3. Steps to Reproduce
*Prerequisites:*
* ACID table under an HDFS directory where neither the HMS server user nor the
table owner has execute permission on the parent
*Setup:*
{code:java}
{code}
*# 1. Create a restricted HDFS parent directory
hdfs dfs -mkdir /test_restricted_parent
hdfs dfs -chown hdfs:hdfs /test_restricted_parent
hdfs dfs -chmod 700 /test_restricted_parent
# Now only the hdfs superuser can traverse this directory#*
*2. Create the table location inside
hdfs dfs -mkdir /test_restricted_parent/test_table
hdfs dfs -chmod 777 /test_restricted_parent/test_table*
{code:java}
{code}
*-- 3. Create ACID table pointing to restricted location
-- (or use ALTER TABLE SET LOCATION after creation)
CREATE TABLE test_leak (id INT)
STORED AS ORC
LOCATION 'hdfs:///test_restricted_parent/test_table'
TBLPROPERTIES ('transactional'='true');*
*-- 4. Insert enough rows to create delta files (exceed compaction threshold)
INSERT INTO test_leak VALUES (1);
INSERT INTO test_leak VALUES (2);
... -- repeat 11+ times to exceed hive.compactor.delta.num.threshold
(default=10)*
{code:java}
{code}
*<!-- 5. Speed up Initiator for faster reproduction -->
<property>
<name>hive.compactor.check.interval</name>
<value>30s</value>
</property>
<property>
<name>hive.compactor.initiator.on</name>
<value>true</value>
</property>*
*Observation:*
On each Initiator cycle, the following sequence occurs in
{{{}TxnUtils.findUserToRunAs(){}}}:
# {{fs.getFileStatus(tablePath)}} → {{AccessControlException}} (HMS user lacks
permission)
# {{createProxyUser(tableOwner)}} → new {{Subject}} object created
# Inside {{{}doAs(){}}}: {{p.getFileSystem(conf)}} → new
{{DistributedFileSystem}} added to {{FileSystem.CACHE}}
# {{proxyFs.getFileStatus(tablePath)}} → {{AccessControlException}} (table
owner also lacks permission)
# {{doAs()}} throws → {{closeAllForUGI(ugi)}} *never called* → cache entry
leaks permanently
{{FileSystem.CACHE}} size grows by 1 per affected table per Initiator cycle.
h3. Reproduction Results
With 500 ACID tables in a restricted directory and HMS heap set to 512 MB:
||Time (elapsed)||CACHE leaked entries||Heap used||Heap free||
|0:00|461|359 MB|121 MB|
|0:31|561|353 MB|158 MB|
|2:04|916|477 MB|35 MB|
|3:21|1,166|509 MB|3 MB|
|3:36|1,166|512 MB|0 MB → *OutOfMemoryError*|
* Growth rate: ~200 entries/minute → ~1.3 MB/second
* Time to OOM: *~3.5 minutes* (with 500 affected tables, 30s interval)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
