[jira] [Comment Edited] (YARN-6277) Nodemanager heap memory leak
[
https://issues.apache.org/jira/browse/YARN-6277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15961822#comment-15961822
]
Feng Yuan edited comment on YARN-6277 at 4/8/17 12:44 PM:
--
[~haibochen], it may not use {{LocalFileSystem.NAME}} as key, by instead it use
key by follow:
{code}
static class Cache {
private final ClientFinalizer clientFinalizer = new ClientFinalizer();
private final Map map = new HashMap();
private final Set toAutoClose = new HashSet();
/** A variable that makes all objects in the cache unique */
private static AtomicLong unique = new AtomicLong(1);
FileSystem get(URI uri, Configuration conf) throws IOException{
Key key = new Key(uri, conf);
return getInternal(uri, conf, key);
}
==
private FileSystem getInternal(URI uri, Configuration conf, Key key) throws
IOException{
FileSystem fs;
synchronized (this) {
fs = map.get(key);
}
if (fs != null) {
return fs;
}
fs = createFileSystem(uri, conf);
{code}
potentialy a UGI is the key.
was (Author: feng yuan):
[~haibochen], it may not use {{LocalFileSystem.NAME}} as key, by instead it use
key by follow:
{code}
static class Cache {
private final ClientFinalizer clientFinalizer = new ClientFinalizer();
private final Map map = new HashMap();
private final Set toAutoClose = new HashSet();
/** A variable that makes all objects in the cache unique */
private static AtomicLong unique = new AtomicLong(1);
FileSystem get(URI uri, Configuration conf) throws IOException{
Key key = new Key(uri, conf);
return getInternal(uri, conf, key);
}
{code}
> Nodemanager heap memory leak
>
>
> Key: YARN-6277
> URL: https://issues.apache.org/jira/browse/YARN-6277
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
>Affects Versions: 2.7.3
>Reporter: Feng Yuan
>Assignee: Feng Yuan
> Attachments: YARN-6277.branch-2.8.001.patch
>
>
> Because LocalDirHandlerService@LocalDirAllocator`s mechanism,they will create
> massive LocalFileSystem.So lead to heap leak.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-6277) Nodemanager heap memory leak
[ https://issues.apache.org/jira/browse/YARN-6277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15961013#comment-15961013 ] Haibo Chen edited comment on YARN-6277 at 4/7/17 4:01 PM: -- [~Feng Yuan] If cached is enabled, there shouldn't be massive LocalFileSystem instances, unless I am missing something. For a given key, LocalFileSystem.NAME in this case, there is going to be just one instance in the cache. was (Author: haibochen): [~Feng Yuan] If cached is enabled, there shouldn't be massive LocalFileSystem instances, unless I am missing something > Nodemanager heap memory leak > > > Key: YARN-6277 > URL: https://issues.apache.org/jira/browse/YARN-6277 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 2.7.3 >Reporter: Feng Yuan >Assignee: Feng Yuan > Attachments: YARN-6277.branch-2.8.001.patch > > > Because LocalDirHandlerService@LocalDirAllocator`s mechanism,they will create > massive LocalFileSystem.So lead to heap leak. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-6277) Nodemanager heap memory leak
[ https://issues.apache.org/jira/browse/YARN-6277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15896662#comment-15896662 ] Feng Yuan edited comment on YARN-6277 at 3/6/17 3:36 AM: - Hi [~Naganarasimha],Like this: In LocalDirAllocator#confChanged will check local-dir configuration if changed. The LocalDirsHandlerService will timing change this conf because local disk breakdown. And ShuffleHandler will also use LocalDirAllocator#confChanged because the AllocatorPerContext is singleton, but the conf of shuffleHandler is a clone version,so if the conf changed is nm the sh`s conf is still old. So if shuffleHandler invoke LocalDirAllocator#confChanged once nm will create a FileSystem.. was (Author: feng yuan): Hi [~Naganarasimha],Like this: In LocalDirAllocator#confChanged will check local-dir configuration if changed. The RLS will timing change this conf because local disk breakdown. And ShuffleHandler will also use LocalDirAllocator#confChanged because the AllocatorPerContext is singleton, but the conf of shuffleHandler is a clone version,so if the conf changed is nm the sh`s conf is still old. So if shuffleHandler invoke LocalDirAllocator#confChanged once nm will create a FileSystem.. > Nodemanager heap memory leak > > > Key: YARN-6277 > URL: https://issues.apache.org/jira/browse/YARN-6277 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 2.7.3, 2.8.1, 3.0.0-alpha2 >Reporter: Feng Yuan >Assignee: Feng Yuan > > Because LocalDirHandlerService@LocalDirAllocator`s mechanism,they will create > massive LocalFileSystem.So lead to heap leak. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-6277) Nodemanager heap memory leak
[ https://issues.apache.org/jira/browse/YARN-6277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15896662#comment-15896662 ] Feng Yuan edited comment on YARN-6277 at 3/6/17 2:46 AM: - Hi [~Naganarasimha],Like this: In LocalDirAllocator#confChanged will check local-dir configuration if changed. The RLS will timing change this conf because local disk breakdown. And ShuffleHandler will also use LocalDirAllocator#confChanged because the AllocatorPerContext is singleton, but the conf of shuffleHandler is a clone version,so if the conf changed is nm the sh`s conf is still old. So if shuffleHandler invoke LocalDirAllocator#confChanged once nm will create a FileSystem.. was (Author: feng yuan): Hi [~Naganarasimha],Like this: In LocalDirAllocator#confChanged will check local-dir configuration if changed. The RLS will timing change this conf because local disk breakdown. And ShuffleHandler will also use LocalDirAllocator#confChanged because the AllocatorPerContext is singleton, but the conf of shuffleHandler is a clone version,so if the conf changed is nm the sh`s conf is still old. So if shuffleHandler invoke LocalDirAllocator#confChanged once nm will create a FileSystem.. > Nodemanager heap memory leak > > > Key: YARN-6277 > URL: https://issues.apache.org/jira/browse/YARN-6277 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 3.0.0-alpha2 >Reporter: Feng Yuan >Assignee: Feng Yuan > > Because LocalDirHandlerService@LocalDirAllocator`s mechanism,they will create > massive LocalFileSystem.So lead to heap leak. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
