Bikas Saha commented on YARN-2314:

Folks, this is something that would be of interest in Tez since it uses the 
ContainerManagementProtocolProxy. My summary understanding is that the default 
is to turn this proxy off and this improves things for large scale clusters. So 
when Tez moves to 2.6 then it will automatically pick the defaults (which turn 
caching off) and benefit for large clusters. Is that correct?

> ContainerManagementProtocolProxy can create thousands of threads for a large 
> cluster
> ------------------------------------------------------------------------------------
>                 Key: YARN-2314
>                 URL: https://issues.apache.org/jira/browse/YARN-2314
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: client
>    Affects Versions: 2.1.0-beta
>            Reporter: Jason Lowe
>            Assignee: Jason Lowe
>            Priority: Critical
>         Attachments: YARN-2314.patch, disable-cm-proxy-cache.patch, 
> nmproxycachefix.prototype.patch
> ContainerManagementProtocolProxy has a cache of NM proxies, and the size of 
> this cache is configurable.  However the cache can grow far beyond the 
> configured size when running on a large cluster and blow AM address/container 
> limits.  More details in the first comment.

This message was sent by Atlassian JIRA

Reply via email to