[
https://issues.apache.org/jira/browse/HIVE-29708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Indhumathi Muthumurugesh updated HIVE-29708:
--------------------------------------------
Description:
h3. Background: The {{runAs}} User in Hive Compaction
When the Compactor Initiator determines that an ACID table needs compaction, it
must decide {*}which user to run the compaction job as{*}. This is done via
{{{}TxnUtils.findUserToRunAs(){}}}, which is called for every potential
compaction candidate on every Initiator cycle.
The method works as follows:
{code:java}
TxnUtils.findUserToRunAs(location, table, conf)
│
├── Step 1: Check if hive.compactor.run.as.user is configured
│ └── If set → return that user immediately (no HDFS access needed)
│
├── Step 2: Try as the HMS server user (e.g. "hive")
│ └── fs.getFileStatus(tableLocation)
│ ├── SUCCESS → return directory owner as runAs user
│ └── AccessControlException → fall through to Step 3
│
└── Step 3: Try as the table owner (proxy user)
├── UserGroupInformation.createProxyUser(table.getOwner(), loginUser)
└── ugi.doAs(() -> {
FileSystem proxyFs = p.getFileSystem(conf) ← BUG: CACHE entry
added here
proxyFs.getFileStatus(tableLocation)
├── SUCCESS → return owner, then closeAllForUGI (cleanup)
└── Exception → closeAllForUGI SKIPPED ← THE LEAK
}) {code}
The proxy user path (Step 3) is entered when:
* hive.compactor.run.as.user is NOT configured (default — no fixed runAs user)
* The HMS server user does NOT have permission to stat the table's storage
location
This is a common scenario in multi-tenant clusters where:
* ACID tables are created by end users in HDFS paths outside the managed
warehouse
* The HDFS parent directory has restrictive permissions (e.g., drwx------) not
accessible to the HMS service account
* The HMS service account is not a superuser and does not have blanket read
access to all data directories
h3. Problem Statement
HiveMetaStore (HMS) processes experience periodic OutOfMemoryError crashes over
time. The crash is caused by {{org.apache.hadoop.conf.Configuration}} objects
accumulating unboundedly in the heap via
{{{}org.apache.hadoop.fs.FileSystem.CACHE{}}}, a static {{HashMap}} that caches
{{DistributedFileSystem}} instances.
The issue is triggered by the Compactor Initiator specifically in the *proxy
user fallback path* of {{{}TxnUtils.findUserToRunAs(){}}}. Disabling compaction
({{{}hive.compactor.initiator.on=false{}}}) or configuring
{{hive.compactor.run.as.user}} prevents the OOM.
h3. Heap Dump Analysis
The {{FileSystem.CACHE}} retains the leaked {{DistributedFileSystem}} instances
as GC roots:
{code:java}
org.apache.hadoop.conf.Configuration (×N*2 instances)
└── org.apache.hadoop.hdfs.DFSClient.conf
└── org.apache.hadoop.hdfs.DistributedFileSystem.dfs (×N instances)
└── java.util.HashMap$Node.value
└── java.util.HashMap (N elements)
└── org.apache.hadoop.fs.FileSystem$Cache.map
└── static CACHE in
org.apache.hadoop.fs.FileSystem
└── [GC ROOT — never collected] {code}
Each leaked DistributedFileSystem retains approximately 300-400 KB of heap (2×
Configuration objects, DFSClient, SaslDataTransferClient,
SaslPropertiesResolver). After weeks of uptime with many affected tables, this
accumulates to gigabytes of retained heap.
h3. Steps to Reproduce
*Prerequisites:*
* ACID table under an HDFS directory where neither the HMS server user nor the
table owner has execute permission on the parent
*Setup:*
{code:java}
# 1. Create a restricted HDFS parent directory
hdfs dfs -mkdir /test_restricted_parent
hdfs dfs -chown hdfs:hdfs /test_restricted_parent
hdfs dfs -chmod 700 /test_restricted_parent
# Now only the hdfs superuser can traverse this directory# 2. Create the table
location inside
hdfs dfs -mkdir /test_restricted_parent/test_table
hdfs dfs -chmod 777 /test_restricted_parent/test_table{code}
*
{code:java}
-- 3. Create ACID table pointing to restricted location
-- (or use ALTER TABLE SET LOCATION after creation)
CREATE TABLE test_leak (id INT)
STORED AS ORC
LOCATION 'hdfs:///test_restricted_parent/test_table'
TBLPROPERTIES ('transactional'='true');-- 4. Insert enough rows to create delta
files (exceed compaction threshold)
INSERT INTO test_leak VALUES (1);
INSERT INTO test_leak VALUES (2);
... -- repeat 11+ times to exceed hive.compactor.delta.num.threshold
(default=10){code}
{code:java}
<!-- 5. Speed up Initiator for faster reproduction -->
<property>
<name>hive.compactor.check.interval</name>
<value>30s</value>
</property>
<property>
<name>hive.compactor.initiator.on</name>
<value>true</value>
</property> {code}
*Observation:*
On each Initiator cycle, the following sequence occurs in
{{{}TxnUtils.findUserToRunAs(){}}}:
# {{fs.getFileStatus(tablePath)}} → {{AccessControlException}} (HMS user lacks
permission)
# {{createProxyUser(tableOwner)}} → new {{Subject}} object created
# Inside {{{}doAs(){}}}: {{p.getFileSystem(conf)}} → new
{{DistributedFileSystem}} added to {{FileSystem.CACHE}}
# {{proxyFs.getFileStatus(tablePath)}} → {{AccessControlException}} (table
owner also lacks permission)
# {{doAs()}} throws → {{closeAllForUGI(ugi)}} *never called* → cache entry
leaks permanently
{{FileSystem.CACHE}} size grows by 1 per affected table per Initiator cycle.
h3. Reproduction Results
With 500 ACID tables in a restricted directory and HMS heap set to 512 MB:
||Time (elapsed)||CACHE leaked entries||Heap used||Heap free||
|0:00|461|359 MB|121 MB|
|0:31|561|353 MB|158 MB|
|2:04|916|477 MB|35 MB|
|3:21|1,166|509 MB|3 MB|
|3:36|1,166|512 MB|0 MB → *OutOfMemoryError*|
* Growth rate: ~200 entries/minute → ~1.3 MB/second
* Time to OOM: *~3.5 minutes* (with 500 affected tables, 30s interval)
was:
h3. Background: The {{runAs}} User in Hive Compaction
When the Compactor Initiator determines that an ACID table needs compaction, it
must decide {*}which user to run the compaction job as{*}. This is done via
{{{}TxnUtils.findUserToRunAs(){}}}, which is called for every potential
compaction candidate on every Initiator cycle.
The method works as follows:
{code:java}
TxnUtils.findUserToRunAs(location, table, conf)
│
├── Step 1: Check if hive.compactor.run.as.user is configured
│ └── If set → return that user immediately (no HDFS access needed)
│
├── Step 2: Try as the HMS server user (e.g. "hive")
│ └── fs.getFileStatus(tableLocation)
│ ├── SUCCESS → return directory owner as runAs user
│ └── AccessControlException → fall through to Step 3
│
└── Step 3: Try as the table owner (proxy user)
├── UserGroupInformation.createProxyUser(table.getOwner(), loginUser)
└── ugi.doAs(() -> {
FileSystem proxyFs = p.getFileSystem(conf) ← BUG: CACHE entry
added here
proxyFs.getFileStatus(tableLocation)
├── SUCCESS → return owner, then closeAllForUGI (cleanup)
└── Exception → closeAllForUGI SKIPPED ← THE LEAK
}) {code}
The proxy user path (Step 3) is entered when:
* hive.compactor.run.as.user is NOT configured (default — no fixed runAs user)
* The HMS server user does NOT have permission to stat the table's storage
location
This is a common scenario in multi-tenant clusters where:
* ACID tables are created by end users in HDFS paths outside the managed
warehouse
* The HDFS parent directory has restrictive permissions (e.g., drwx------) not
accessible to the HMS service account
* The HMS service account is not a superuser and does not have blanket read
access to all data directories
h3. Problem Statement
HiveMetaStore (HMS) processes experience periodic OutOfMemoryError crashes over
time. The crash is caused by {{org.apache.hadoop.conf.Configuration}} objects
accumulating unboundedly in the heap via
{{{}org.apache.hadoop.fs.FileSystem.CACHE{}}}, a static {{HashMap}} that caches
{{DistributedFileSystem}} instances.
The issue is triggered by the Compactor Initiator specifically in the *proxy
user fallback path* of {{{}TxnUtils.findUserToRunAs(){}}}. Disabling compaction
({{{}hive.compactor.initiator.on=false{}}}) or configuring
{{hive.compactor.run.as.user}} prevents the OOM.
h3. Heap Dump Analysis
The {{FileSystem.CACHE}} retains the leaked {{DistributedFileSystem}} instances
as GC roots:
{code:java}
org.apache.hadoop.conf.Configuration (×N*2 instances)
└── org.apache.hadoop.hdfs.DFSClient.conf
└── org.apache.hadoop.hdfs.DistributedFileSystem.dfs (×N instances)
└── java.util.HashMap$Node.value
└── java.util.HashMap (N elements)
└── org.apache.hadoop.fs.FileSystem$Cache.map
└── static CACHE in
org.apache.hadoop.fs.FileSystem
└── [GC ROOT — never collected] {code}
Each leaked DistributedFileSystem retains approximately 300-400 KB of heap (2×
Configuration objects, DFSClient, SaslDataTransferClient,
SaslPropertiesResolver). After weeks of uptime with many affected tables, this
accumulates to gigabytes of retained heap.
h3. Steps to Reproduce
*Prerequisites:*
* ACID table under an HDFS directory where neither the HMS server user nor the
table owner has execute permission on the parent
*Setup:*
{code:java}
{code}
*# 1. Create a restricted HDFS parent directory
hdfs dfs -mkdir /test_restricted_parent
hdfs dfs -chown hdfs:hdfs /test_restricted_parent
hdfs dfs -chmod 700 /test_restricted_parent
# Now only the hdfs superuser can traverse this directory#*
*2. Create the table location inside
hdfs dfs -mkdir /test_restricted_parent/test_table
hdfs dfs -chmod 777 /test_restricted_parent/test_table*
{code:java}
{code}
*-- 3. Create ACID table pointing to restricted location
-- (or use ALTER TABLE SET LOCATION after creation)
CREATE TABLE test_leak (id INT)
STORED AS ORC
LOCATION 'hdfs:///test_restricted_parent/test_table'
TBLPROPERTIES ('transactional'='true');*
*-- 4. Insert enough rows to create delta files (exceed compaction threshold)
INSERT INTO test_leak VALUES (1);
INSERT INTO test_leak VALUES (2);
... -- repeat 11+ times to exceed hive.compactor.delta.num.threshold
(default=10)*
{code:java}
{code}
*<!-- 5. Speed up Initiator for faster reproduction -->
<property>
<name>hive.compactor.check.interval</name>
<value>30s</value>
</property>
<property>
<name>hive.compactor.initiator.on</name>
<value>true</value>
</property>*
*Observation:*
On each Initiator cycle, the following sequence occurs in
{{{}TxnUtils.findUserToRunAs(){}}}:
# {{fs.getFileStatus(tablePath)}} → {{AccessControlException}} (HMS user lacks
permission)
# {{createProxyUser(tableOwner)}} → new {{Subject}} object created
# Inside {{{}doAs(){}}}: {{p.getFileSystem(conf)}} → new
{{DistributedFileSystem}} added to {{FileSystem.CACHE}}
# {{proxyFs.getFileStatus(tablePath)}} → {{AccessControlException}} (table
owner also lacks permission)
# {{doAs()}} throws → {{closeAllForUGI(ugi)}} *never called* → cache entry
leaks permanently
{{FileSystem.CACHE}} size grows by 1 per affected table per Initiator cycle.
h3. Reproduction Results
With 500 ACID tables in a restricted directory and HMS heap set to 512 MB:
||Time (elapsed)||CACHE leaked entries||Heap used||Heap free||
|0:00|461|359 MB|121 MB|
|0:31|561|353 MB|158 MB|
|2:04|916|477 MB|35 MB|
|3:21|1,166|509 MB|3 MB|
|3:36|1,166|512 MB|0 MB → *OutOfMemoryError*|
* Growth rate: ~200 entries/minute → ~1.3 MB/second
* Time to OOM: *~3.5 minutes* (with 500 affected tables, 30s interval)
> HMS OOM: FileSystem.CACHE leaks DistributedFileSystem instances in
> TxnUtils.findUserToRunAs() when runAs proxy user lacks HDFS execute permission
> -----------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HIVE-29708
> URL: https://issues.apache.org/jira/browse/HIVE-29708
> Project: Hive
> Issue Type: Bug
> Affects Versions: 4.2.0
> Reporter: Indhumathi Muthumurugesh
> Assignee: Indhumathi Muthumurugesh
> Priority: Major
>
> h3. Background: The {{runAs}} User in Hive Compaction
> When the Compactor Initiator determines that an ACID table needs compaction,
> it must decide {*}which user to run the compaction job as{*}. This is done
> via {{{}TxnUtils.findUserToRunAs(){}}}, which is called for every potential
> compaction candidate on every Initiator cycle.
> The method works as follows:
> {code:java}
> TxnUtils.findUserToRunAs(location, table, conf)
> │
> ├── Step 1: Check if hive.compactor.run.as.user is configured
> │ └── If set → return that user immediately (no HDFS access needed)
> │
> ├── Step 2: Try as the HMS server user (e.g. "hive")
> │ └── fs.getFileStatus(tableLocation)
> │ ├── SUCCESS → return directory owner as runAs user
> │ └── AccessControlException → fall through to Step 3
> │
> └── Step 3: Try as the table owner (proxy user)
> ├── UserGroupInformation.createProxyUser(table.getOwner(), loginUser)
> └── ugi.doAs(() -> {
> FileSystem proxyFs = p.getFileSystem(conf) ← BUG: CACHE entry
> added here
> proxyFs.getFileStatus(tableLocation)
> ├── SUCCESS → return owner, then closeAllForUGI (cleanup)
> └── Exception → closeAllForUGI SKIPPED ← THE LEAK
> }) {code}
> The proxy user path (Step 3) is entered when:
> * hive.compactor.run.as.user is NOT configured (default — no fixed runAs
> user)
> * The HMS server user does NOT have permission to stat the table's storage
> location
> This is a common scenario in multi-tenant clusters where:
> * ACID tables are created by end users in HDFS paths outside the managed
> warehouse
> * The HDFS parent directory has restrictive permissions (e.g., drwx------)
> not accessible to the HMS service account
> * The HMS service account is not a superuser and does not have blanket read
> access to all data directories
> h3. Problem Statement
> HiveMetaStore (HMS) processes experience periodic OutOfMemoryError crashes
> over time. The crash is caused by {{org.apache.hadoop.conf.Configuration}}
> objects accumulating unboundedly in the heap via
> {{{}org.apache.hadoop.fs.FileSystem.CACHE{}}}, a static {{HashMap}} that
> caches {{DistributedFileSystem}} instances.
> The issue is triggered by the Compactor Initiator specifically in the *proxy
> user fallback path* of {{{}TxnUtils.findUserToRunAs(){}}}. Disabling
> compaction ({{{}hive.compactor.initiator.on=false{}}}) or configuring
> {{hive.compactor.run.as.user}} prevents the OOM.
>
> h3. Heap Dump Analysis
> The {{FileSystem.CACHE}} retains the leaked {{DistributedFileSystem}}
> instances as GC roots:
> {code:java}
> org.apache.hadoop.conf.Configuration (×N*2 instances)
> └── org.apache.hadoop.hdfs.DFSClient.conf
> └── org.apache.hadoop.hdfs.DistributedFileSystem.dfs (×N instances)
> └── java.util.HashMap$Node.value
> └── java.util.HashMap (N elements)
> └── org.apache.hadoop.fs.FileSystem$Cache.map
> └── static CACHE in
> org.apache.hadoop.fs.FileSystem
> └── [GC ROOT — never collected] {code}
> Each leaked DistributedFileSystem retains approximately 300-400 KB of heap
> (2× Configuration objects, DFSClient, SaslDataTransferClient,
> SaslPropertiesResolver). After weeks of uptime with many affected tables,
> this accumulates to gigabytes of retained heap.
>
> h3. Steps to Reproduce
> *Prerequisites:*
> * ACID table under an HDFS directory where neither the HMS server user nor
> the table owner has execute permission on the parent
> *Setup:*
> {code:java}
> # 1. Create a restricted HDFS parent directory
> hdfs dfs -mkdir /test_restricted_parent
> hdfs dfs -chown hdfs:hdfs /test_restricted_parent
> hdfs dfs -chmod 700 /test_restricted_parent
> # Now only the hdfs superuser can traverse this directory# 2. Create the
> table location inside
> hdfs dfs -mkdir /test_restricted_parent/test_table
> hdfs dfs -chmod 777 /test_restricted_parent/test_table{code}
> *
>
> {code:java}
> -- 3. Create ACID table pointing to restricted location
> -- (or use ALTER TABLE SET LOCATION after creation)
> CREATE TABLE test_leak (id INT)
> STORED AS ORC
> LOCATION 'hdfs:///test_restricted_parent/test_table'
> TBLPROPERTIES ('transactional'='true');-- 4. Insert enough rows to create
> delta files (exceed compaction threshold)
> INSERT INTO test_leak VALUES (1);
> INSERT INTO test_leak VALUES (2);
> ... -- repeat 11+ times to exceed hive.compactor.delta.num.threshold
> (default=10){code}
> {code:java}
> <!-- 5. Speed up Initiator for faster reproduction -->
> <property>
> <name>hive.compactor.check.interval</name>
> <value>30s</value>
> </property>
> <property>
> <name>hive.compactor.initiator.on</name>
> <value>true</value>
> </property> {code}
>
> *Observation:*
> On each Initiator cycle, the following sequence occurs in
> {{{}TxnUtils.findUserToRunAs(){}}}:
> # {{fs.getFileStatus(tablePath)}} → {{AccessControlException}} (HMS user
> lacks permission)
> # {{createProxyUser(tableOwner)}} → new {{Subject}} object created
> # Inside {{{}doAs(){}}}: {{p.getFileSystem(conf)}} → new
> {{DistributedFileSystem}} added to {{FileSystem.CACHE}}
> # {{proxyFs.getFileStatus(tablePath)}} → {{AccessControlException}} (table
> owner also lacks permission)
> # {{doAs()}} throws → {{closeAllForUGI(ugi)}} *never called* → cache entry
> leaks permanently
> {{FileSystem.CACHE}} size grows by 1 per affected table per Initiator cycle.
>
> h3. Reproduction Results
> With 500 ACID tables in a restricted directory and HMS heap set to 512 MB:
> ||Time (elapsed)||CACHE leaked entries||Heap used||Heap free||
> |0:00|461|359 MB|121 MB|
> |0:31|561|353 MB|158 MB|
> |2:04|916|477 MB|35 MB|
> |3:21|1,166|509 MB|3 MB|
> |3:36|1,166|512 MB|0 MB → *OutOfMemoryError*|
> * Growth rate: ~200 entries/minute → ~1.3 MB/second
> * Time to OOM: *~3.5 minutes* (with 500 affected tables, 30s interval)
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
