[
https://issues.apache.org/jira/browse/HIVE-29708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Indhumathi Muthumurugesh reassigned HIVE-29708:
-----------------------------------------------
Assignee: Indhumathi Muthumurugesh
> HMS OOM: FileSystem.CACHE leaks DistributedFileSystem instances in
> TxnUtils.findUserToRunAs() when runAs proxy user lacks HDFS execute permission
> -----------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HIVE-29708
> URL: https://issues.apache.org/jira/browse/HIVE-29708
> Project: Hive
> Issue Type: Bug
> Affects Versions: 4.2.0
> Reporter: Indhumathi Muthumurugesh
> Assignee: Indhumathi Muthumurugesh
> Priority: Major
>
> h3. Background: The {{runAs}} User in Hive Compaction
> When the Compactor Initiator determines that an ACID table needs compaction,
> it must decide {*}which user to run the compaction job as{*}. This is done
> via {{{}TxnUtils.findUserToRunAs(){}}}, which is called for every potential
> compaction candidate on every Initiator cycle.
> The method works as follows:
> {code:java}
> TxnUtils.findUserToRunAs(location, table, conf)
> │
> ├── Step 1: Check if hive.compactor.run.as.user is configured
> │ └── If set → return that user immediately (no HDFS access needed)
> │
> ├── Step 2: Try as the HMS server user (e.g. "hive")
> │ └── fs.getFileStatus(tableLocation)
> │ ├── SUCCESS → return directory owner as runAs user
> │ └── AccessControlException → fall through to Step 3
> │
> └── Step 3: Try as the table owner (proxy user)
> ├── UserGroupInformation.createProxyUser(table.getOwner(), loginUser)
> └── ugi.doAs(() -> {
> FileSystem proxyFs = p.getFileSystem(conf) ← BUG: CACHE entry
> added here
> proxyFs.getFileStatus(tableLocation)
> ├── SUCCESS → return owner, then closeAllForUGI (cleanup)
> └── Exception → closeAllForUGI SKIPPED ← THE LEAK
> }) {code}
> The proxy user path (Step 3) is entered when:
> * hive.compactor.run.as.user is NOT configured (default — no fixed runAs
> user)
> * The HMS server user does NOT have permission to stat the table's storage
> location
> This is a common scenario in multi-tenant clusters where:
> * ACID tables are created by end users in HDFS paths outside the managed
> warehouse
> * The HDFS parent directory has restrictive permissions (e.g., drwx------)
> not accessible to the HMS service account
> * The HMS service account is not a superuser and does not have blanket read
> access to all data directories
> h3. Problem Statement
> HiveMetaStore (HMS) processes experience periodic OutOfMemoryError crashes
> over time. The crash is caused by {{org.apache.hadoop.conf.Configuration}}
> objects accumulating unboundedly in the heap via
> {{{}org.apache.hadoop.fs.FileSystem.CACHE{}}}, a static {{HashMap}} that
> caches {{DistributedFileSystem}} instances.
> The issue is triggered by the Compactor Initiator specifically in the *proxy
> user fallback path* of {{{}TxnUtils.findUserToRunAs(){}}}. Disabling
> compaction ({{{}hive.compactor.initiator.on=false{}}}) or configuring
> {{hive.compactor.run.as.user}} prevents the OOM.
>
> h3. Heap Dump Analysis
> The {{FileSystem.CACHE}} retains the leaked {{DistributedFileSystem}}
> instances as GC roots:
> {code:java}
> org.apache.hadoop.conf.Configuration (×N*2 instances)
> └── org.apache.hadoop.hdfs.DFSClient.conf
> └── org.apache.hadoop.hdfs.DistributedFileSystem.dfs (×N instances)
> └── java.util.HashMap$Node.value
> └── java.util.HashMap (N elements)
> └── org.apache.hadoop.fs.FileSystem$Cache.map
> └── static CACHE in
> org.apache.hadoop.fs.FileSystem
> └── [GC ROOT — never collected] {code}
> Each leaked DistributedFileSystem retains approximately 300-400 KB of heap
> (2× Configuration objects, DFSClient, SaslDataTransferClient,
> SaslPropertiesResolver). After weeks of uptime with many affected tables,
> this accumulates to gigabytes of retained heap.
>
> h3. Steps to Reproduce
> *Prerequisites:*
> * ACID table under an HDFS directory where neither the HMS server user nor
> the table owner has execute permission on the parent
> *Setup:*
> {code:java}
> {code}
> *# 1. Create a restricted HDFS parent directory
> hdfs dfs -mkdir /test_restricted_parent
> hdfs dfs -chown hdfs:hdfs /test_restricted_parent
> hdfs dfs -chmod 700 /test_restricted_parent
> # Now only the hdfs superuser can traverse this directory#*
> *2. Create the table location inside
> hdfs dfs -mkdir /test_restricted_parent/test_table
> hdfs dfs -chmod 777 /test_restricted_parent/test_table*
> {code:java}
> {code}
> *-- 3. Create ACID table pointing to restricted location
> -- (or use ALTER TABLE SET LOCATION after creation)
> CREATE TABLE test_leak (id INT)
> STORED AS ORC
> LOCATION 'hdfs:///test_restricted_parent/test_table'
> TBLPROPERTIES ('transactional'='true');*
> *-- 4. Insert enough rows to create delta files (exceed compaction threshold)
> INSERT INTO test_leak VALUES (1);
> INSERT INTO test_leak VALUES (2);
> ... -- repeat 11+ times to exceed hive.compactor.delta.num.threshold
> (default=10)*
> {code:java}
> {code}
> *<!-- 5. Speed up Initiator for faster reproduction -->
> <property>
> <name>hive.compactor.check.interval</name>
> <value>30s</value>
> </property>
> <property>
> <name>hive.compactor.initiator.on</name>
> <value>true</value>
> </property>*
> *Observation:*
> On each Initiator cycle, the following sequence occurs in
> {{{}TxnUtils.findUserToRunAs(){}}}:
> # {{fs.getFileStatus(tablePath)}} → {{AccessControlException}} (HMS user
> lacks permission)
> # {{createProxyUser(tableOwner)}} → new {{Subject}} object created
> # Inside {{{}doAs(){}}}: {{p.getFileSystem(conf)}} → new
> {{DistributedFileSystem}} added to {{FileSystem.CACHE}}
> # {{proxyFs.getFileStatus(tablePath)}} → {{AccessControlException}} (table
> owner also lacks permission)
> # {{doAs()}} throws → {{closeAllForUGI(ugi)}} *never called* → cache entry
> leaks permanently
> {{FileSystem.CACHE}} size grows by 1 per affected table per Initiator cycle.
>
> h3. Reproduction Results
> With 500 ACID tables in a restricted directory and HMS heap set to 512 MB:
> ||Time (elapsed)||CACHE leaked entries||Heap used||Heap free||
> |0:00|461|359 MB|121 MB|
> |0:31|561|353 MB|158 MB|
> |2:04|916|477 MB|35 MB|
> |3:21|1,166|509 MB|3 MB|
> |3:36|1,166|512 MB|0 MB → *OutOfMemoryError*|
> * Growth rate: ~200 entries/minute → ~1.3 MB/second
> * Time to OOM: *~3.5 minutes* (with 500 affected tables, 30s interval)
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
