[
https://issues.apache.org/jira/browse/MAPREDUCE-7094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16479560#comment-16479560
]
Hudson commented on MAPREDUCE-7094:
-----------------------------------
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #14221 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/14221/])
MAPREDUCE-7094. LocalDistributedCacheManager leaves classloaders open,
(szegedim: rev a2cdffb95acbcb3625ee72ebc8aeb8bf17fa4bc7)
* (edit)
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapred/LocalDistributedCacheManager.java
* (edit)
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapred/LocalJobRunner.java
> LocalDistributedCacheManager leaves classloaders open, which leaks FDs
> ----------------------------------------------------------------------
>
> Key: MAPREDUCE-7094
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7094
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 2.6.0
> Reporter: Adam Szita
> Assignee: Adam Szita
> Priority: Major
> Fix For: 3.2.0
>
> Attachments: MAPREDUCE-7094.0.patch, MAPREDUCE-7094.1.patch,
> MAPREDUCE-7094.2.patch, MAPREDUCE-7094.3.patch, MAPREDUCE-7094.4.patch,
> MAPREDUCE-7094.5.patch, MAPREDUCE-7094.6.patch
>
>
> When a user starts a local mapred task from Hive's beeline, it will leave
> open file descriptors on the HS2 process (which runs the mapred task).
> I debugged this and saw that it is caused by LocalDistributedCacheManager
> class, which creates a new URLClassLoader, with a classpath for the two jars
> seen below. Somewhere down the line Loaders will be created in this
> URLClassLoader for these files effectively creating the FD's on the OS level.
> This is never cleaned up after execution, although
> LocalDistributedCacheManager removes the files, it will not close the
> ClassLoader, so FDs are left open although they point to deleted files at
> that time:
> {code:java}
> [root@host-1 ~]# lsof -p 14439 | grep hadoop-hive
> java 14439 hive DEL REG 8,1 3348748
> /tmp/hadoop-hive/mapred/local/1525789796610/hive-exec-core.jar
> java 14439 hive DEL REG 8,1 3348750
> /tmp/hadoop-hive/mapred/local/1525789796609/hive-exec-1.1.0-cdh5.13.4-SNAPSHOT-core.jar
> java 14439 hive 649r REG 8,1 8112438 3348750
> /tmp/hadoop-hive/mapred/local/1525789796609/hive-exec-1.1.0-cdh5.13.4-SNAPSHOT-core.jar
> (deleted)
> java 14439 hive 650r REG 8,1 8112438 3348748
> /tmp/hadoop-hive/mapred/local/1525789796610/hive-exec-core.jar (deleted)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
