Wangda Tan commented on YARN-2314:

Discussed with [~vinodkv] offline, summary of what we have discussed/confirmed,

1) There's no problem when multiple proxy to a same NM in an AM existed (Tokens 
will be simply added to Set<Token> of subject of UserGroupInformation)
2) There's no thread-leaking in MR-AM side (MRAppMaster has a 
{{yarn.app.mapreduce.am.containerlauncher.thread-count-limit}}) to limit 
maximum number of threads in ContainerLauncher
3) Basically the cache doesn't have more functionalities other than just cache 
the connection.

So it should be safe to remove the cache, but it's better to add an option for 
user to enable/disable the cache. [~jlowe], could you please add an option to 
disable/enable cache in your existing patch or you can assign it to me if you 
don't have bandwidth to do that. 


> ContainerManagementProtocolProxy can create thousands of threads for a large 
> cluster
> ------------------------------------------------------------------------------------
>                 Key: YARN-2314
>                 URL: https://issues.apache.org/jira/browse/YARN-2314
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: client
>    Affects Versions: 2.1.0-beta
>            Reporter: Jason Lowe
>            Priority: Critical
>         Attachments: disable-cm-proxy-cache.patch, 
> nmproxycachefix.prototype.patch
> ContainerManagementProtocolProxy has a cache of NM proxies, and the size of 
> this cache is configurable.  However the cache can grow far beyond the 
> configured size when running on a large cluster and blow AM address/container 
> limits.  More details in the first comment.

This message was sent by Atlassian JIRA

Reply via email to