[
https://issues.apache.org/jira/browse/MAPREDUCE-7094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16477609#comment-16477609
]
Miklos Szegedi commented on MAPREDUCE-7094:
-------------------------------------------
Thank you for the patch [~szita].
I noticed one more thing. If makeClassLoader is called with setup at the same
time, we create an url array with an initial size of localClassPaths. However
setup running in parallel may add more before the for iteration starts, so we
will crash with an ArrayOutOfBoundException. Either setup needs to be
synchronized as well, or you need to sample the url size and give the same
value to the iteration and the array definition.
{code:java}
try
{
try {
// Cleanup distributed cache
localDistributedCacheManager.close();
} finally {
try {
fs.delete(systemJobFile.getParent(), true); // delete submit dir
} finally {
localFs.delete(localJobFile, true); // delete local copy
}
}
} catch (IOException e) {
LOG.warn("Error cleaning up " + id + ": " + e);
}{code}
I was thinking about something more in LocalJobRunner like the code above to do
a proper cleanup of the files as well. This was not your code originally but
since we touch it, we might want to do it right.
> LocalDistributedCacheManager leaves classloaders open, which leaks FDs
> ----------------------------------------------------------------------
>
> Key: MAPREDUCE-7094
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7094
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 2.6.0
> Reporter: Adam Szita
> Assignee: Adam Szita
> Priority: Major
> Attachments: MAPREDUCE-7094.0.patch, MAPREDUCE-7094.1.patch,
> MAPREDUCE-7094.2.patch, MAPREDUCE-7094.3.patch, MAPREDUCE-7094.4.patch
>
>
> When a user starts a local mapred task from Hive's beeline, it will leave
> open file descriptors on the HS2 process (which runs the mapred task).
> I debugged this and saw that it is caused by LocalDistributedCacheManager
> class, which creates a new URLClassLoader, with a classpath for the two jars
> seen below. Somewhere down the line Loaders will be created in this
> URLClassLoader for these files effectively creating the FD's on the OS level.
> This is never cleaned up after execution, although
> LocalDistributedCacheManager removes the files, it will not close the
> ClassLoader, so FDs are left open although they point to deleted files at
> that time:
> {code:java}
> [root@host-1 ~]# lsof -p 14439 | grep hadoop-hive
> java 14439 hive DEL REG 8,1 3348748
> /tmp/hadoop-hive/mapred/local/1525789796610/hive-exec-core.jar
> java 14439 hive DEL REG 8,1 3348750
> /tmp/hadoop-hive/mapred/local/1525789796609/hive-exec-1.1.0-cdh5.13.4-SNAPSHOT-core.jar
> java 14439 hive 649r REG 8,1 8112438 3348750
> /tmp/hadoop-hive/mapred/local/1525789796609/hive-exec-1.1.0-cdh5.13.4-SNAPSHOT-core.jar
> (deleted)
> java 14439 hive 650r REG 8,1 8112438 3348748
> /tmp/hadoop-hive/mapred/local/1525789796610/hive-exec-core.jar (deleted)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
