[ 
https://issues.apache.org/jira/browse/MAPREDUCE-7094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16479497#comment-16479497
 ] 

Miklos Szegedi commented on MAPREDUCE-7094:
-------------------------------------------

Committed to trunk. Thank you for the patch [~szita] and for the reviews 
[~wilfreds] and [~pbacsko].

> LocalDistributedCacheManager leaves classloaders open, which leaks FDs
> ----------------------------------------------------------------------
>
>                 Key: MAPREDUCE-7094
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-7094
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>    Affects Versions: 2.6.0
>            Reporter: Adam Szita
>            Assignee: Adam Szita
>            Priority: Major
>         Attachments: MAPREDUCE-7094.0.patch, MAPREDUCE-7094.1.patch, 
> MAPREDUCE-7094.2.patch, MAPREDUCE-7094.3.patch, MAPREDUCE-7094.4.patch, 
> MAPREDUCE-7094.5.patch, MAPREDUCE-7094.6.patch
>
>
> When a user starts a local mapred task from Hive's beeline, it will leave 
> open file descriptors on the HS2 process (which runs the mapred task).
> I debugged this and saw that it is caused by LocalDistributedCacheManager 
> class, which creates a new URLClassLoader, with a classpath for the two jars 
> seen below. Somewhere down the line Loaders will be created in this 
> URLClassLoader for these files effectively creating the FD's on the OS level.
> This is never cleaned up after execution, although 
> LocalDistributedCacheManager removes the files, it will not close the 
> ClassLoader, so FDs are left open although they point to deleted files at 
> that time:
> {code:java}
> [root@host-1 ~]# lsof -p 14439 | grep hadoop-hive
> java    14439 hive  DEL       REG                8,1             3348748 
> /tmp/hadoop-hive/mapred/local/1525789796610/hive-exec-core.jar
> java    14439 hive  DEL       REG                8,1             3348750 
> /tmp/hadoop-hive/mapred/local/1525789796609/hive-exec-1.1.0-cdh5.13.4-SNAPSHOT-core.jar
> java    14439 hive  649r      REG                8,1   8112438   3348750 
> /tmp/hadoop-hive/mapred/local/1525789796609/hive-exec-1.1.0-cdh5.13.4-SNAPSHOT-core.jar
>  (deleted)
> java    14439 hive  650r      REG                8,1   8112438   3348748 
> /tmp/hadoop-hive/mapred/local/1525789796610/hive-exec-core.jar (deleted)
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

Reply via email to