Adam Szita created MAPREDUCE-7094:
-------------------------------------
Summary: LocalDistributedCacheManager leaves classloaders open,
which leaks FDs
Key: MAPREDUCE-7094
URL: https://issues.apache.org/jira/browse/MAPREDUCE-7094
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Adam Szita
When a user starts a local mapred task from Hive's beeline, it will leave open
file descriptors on the HS2 process (which runs the mapred task).
I debugged this and saw that it is caused by LocalDistributedCacheManager
class, which creates a new URLClassLoader, with a classpath for the two jars
seen below. Somewhere down the line Loaders will be created in this
URLClassLoader for these files effectively creating the FD's on the OS level.
This is never cleaned up after execution, although LocalDistributedCacheManager
removes the files, it will not close the ClassLoader, so FDs are left open
although they point to deleted files at that time:
{code:java}
[root@host-1 ~]# lsof -p 14439 | grep hadoop-hive
java 14439 hive DEL REG 8,1 3348748
/tmp/hadoop-hive/mapred/local/1525789796610/hive-exec-core.jar
java 14439 hive DEL REG 8,1 3348750
/tmp/hadoop-hive/mapred/local/1525789796609/hive-exec-1.1.0-cdh5.13.4-SNAPSHOT-core.jar
java 14439 hive 649r REG 8,1 8112438 3348750
/tmp/hadoop-hive/mapred/local/1525789796609/hive-exec-1.1.0-cdh5.13.4-SNAPSHOT-core.jar
(deleted)
java 14439 hive 650r REG 8,1 8112438 3348748
/tmp/hadoop-hive/mapred/local/1525789796610/hive-exec-core.jar (deleted)
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org
