[ https://issues.apache.org/jira/browse/MAPREDUCE-7094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16477650#comment-16477650 ]
Adam Szita commented on MAPREDUCE-7094: --------------------------------------- Thanks [~miklos.szeg...@cloudera.com], I've updated my patch: [^MAPREDUCE-7094.5.patch] > LocalDistributedCacheManager leaves classloaders open, which leaks FDs > ---------------------------------------------------------------------- > > Key: MAPREDUCE-7094 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7094 > Project: Hadoop Map/Reduce > Issue Type: Bug > Affects Versions: 2.6.0 > Reporter: Adam Szita > Assignee: Adam Szita > Priority: Major > Attachments: MAPREDUCE-7094.0.patch, MAPREDUCE-7094.1.patch, > MAPREDUCE-7094.2.patch, MAPREDUCE-7094.3.patch, MAPREDUCE-7094.4.patch, > MAPREDUCE-7094.5.patch > > > When a user starts a local mapred task from Hive's beeline, it will leave > open file descriptors on the HS2 process (which runs the mapred task). > I debugged this and saw that it is caused by LocalDistributedCacheManager > class, which creates a new URLClassLoader, with a classpath for the two jars > seen below. Somewhere down the line Loaders will be created in this > URLClassLoader for these files effectively creating the FD's on the OS level. > This is never cleaned up after execution, although > LocalDistributedCacheManager removes the files, it will not close the > ClassLoader, so FDs are left open although they point to deleted files at > that time: > {code:java} > [root@host-1 ~]# lsof -p 14439 | grep hadoop-hive > java 14439 hive DEL REG 8,1 3348748 > /tmp/hadoop-hive/mapred/local/1525789796610/hive-exec-core.jar > java 14439 hive DEL REG 8,1 3348750 > /tmp/hadoop-hive/mapred/local/1525789796609/hive-exec-1.1.0-cdh5.13.4-SNAPSHOT-core.jar > java 14439 hive 649r REG 8,1 8112438 3348750 > /tmp/hadoop-hive/mapred/local/1525789796609/hive-exec-1.1.0-cdh5.13.4-SNAPSHOT-core.jar > (deleted) > java 14439 hive 650r REG 8,1 8112438 3348748 > /tmp/hadoop-hive/mapred/local/1525789796610/hive-exec-core.jar (deleted) > {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org