Jian He commented on YARN-2314:

Thanks Jason, I looked at the patch, looks good overall. just one thing:
- IIUC, {{mayBeCloseProxy}} can be invoked by MR/NMClient, but 
{{proxy.scheduledForClose}} is always false.  So it won’t call the following 
stopProxy. If cache is disabled, this doesn’t matter too much as the 
idleTimeout is set to 0. But if the cache is enabled, MR/NMClient, won’t be 
able to explicitly close the proxy ?

Also, Can you help me understand one point:
bq. See ClientCache.stopClient for details. Given that the whole point of the 
ContainerManagementProtocolProxy cache is to preserve at least one reference to 
the Client, the IPC Client stop method will never be called in practice and IPC 
client threads will never be explicitly torn down as a result of calling 
once {{ContainerManagementProtocolProxy#tryCloseProxy}} is called, internally 
it’ll call {{rpc.stopProxy}}, will it eventually call 
{{ClientCache#stopClient}} ?

> ContainerManagementProtocolProxy can create thousands of threads for a large 
> cluster
> ------------------------------------------------------------------------------------
>                 Key: YARN-2314
>                 URL: https://issues.apache.org/jira/browse/YARN-2314
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: client
>    Affects Versions: 2.1.0-beta
>            Reporter: Jason Lowe
>            Assignee: Jason Lowe
>            Priority: Critical
>         Attachments: YARN-2314.patch, YARN-2314v2.patch, 
> disable-cm-proxy-cache.patch, nmproxycachefix.prototype.patch, 
> tez-yarn-2314.xlsx
> ContainerManagementProtocolProxy has a cache of NM proxies, and the size of 
> this cache is configurable.  However the cache can grow far beyond the 
> configured size when running on a large cluster and blow AM address/container 
> limits.  More details in the first comment.

This message was sent by Atlassian JIRA

Reply via email to