[
https://issues.apache.org/jira/browse/MAPREDUCE-7094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16469771#comment-16469771
]
Wilfred Spiegelenburg commented on MAPREDUCE-7094:
--------------------------------------------------
Based on the code path there is always only one class loader created and
active. So why do we need to keep track of multiple and can we not just close
the one open class loader correctly in the close method? That would also fix
your findbugs issue.
There is only one call to the {{makeClassLoader}} method which is not inside a
loop in {{LocalJobRunner}}. There is also no loop in
{{LocalDistributedCacheManager}} to create multiple loaders at the same time.
> LocalDistributedCacheManager leaves classloaders open, which leaks FDs
> ----------------------------------------------------------------------
>
> Key: MAPREDUCE-7094
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7094
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 2.6.0
> Reporter: Adam Szita
> Priority: Major
> Attachments: MAPREDUCE-7094.0.patch, MAPREDUCE-7094.1.patch
>
>
> When a user starts a local mapred task from Hive's beeline, it will leave
> open file descriptors on the HS2 process (which runs the mapred task).
> I debugged this and saw that it is caused by LocalDistributedCacheManager
> class, which creates a new URLClassLoader, with a classpath for the two jars
> seen below. Somewhere down the line Loaders will be created in this
> URLClassLoader for these files effectively creating the FD's on the OS level.
> This is never cleaned up after execution, although
> LocalDistributedCacheManager removes the files, it will not close the
> ClassLoader, so FDs are left open although they point to deleted files at
> that time:
> {code:java}
> [root@host-1 ~]# lsof -p 14439 | grep hadoop-hive
> java 14439 hive DEL REG 8,1 3348748
> /tmp/hadoop-hive/mapred/local/1525789796610/hive-exec-core.jar
> java 14439 hive DEL REG 8,1 3348750
> /tmp/hadoop-hive/mapred/local/1525789796609/hive-exec-1.1.0-cdh5.13.4-SNAPSHOT-core.jar
> java 14439 hive 649r REG 8,1 8112438 3348750
> /tmp/hadoop-hive/mapred/local/1525789796609/hive-exec-1.1.0-cdh5.13.4-SNAPSHOT-core.jar
> (deleted)
> java 14439 hive 650r REG 8,1 8112438 3348748
> /tmp/hadoop-hive/mapred/local/1525789796610/hive-exec-core.jar (deleted)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
