[
https://issues.apache.org/jira/browse/YARN-2314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14070967#comment-14070967
]
Li Lu commented on YARN-2314:
-----------------------------
Hi [~jlowe], I'm interested in looking into the cache overflow side of this
issue (Sorry about the last comment that I mistyped my keyboard and sent it
out...). After checking your comments and the code, I think a quick fix would
be, when adding a new proxy into the cache and the cache is full, instead of
only relying on (and trying to delete) the least recently used item, the cache
should keep checking through the whole list to find one item that is not being
used by a RPC, and replace it at that place. There is one scenario that this
may not actually help, and that would be the whole list of cached items are
used by RPCs. I would like to check with you to see if this is a frequent case
in your cluster, and if not, if this quick fix would work for the cache
overflow problem. Thanks!
> ContainerManagementProtocolProxy can create thousands of threads for a large
> cluster
> ------------------------------------------------------------------------------------
>
> Key: YARN-2314
> URL: https://issues.apache.org/jira/browse/YARN-2314
> Project: Hadoop YARN
> Issue Type: Bug
> Components: client
> Affects Versions: 2.1.0-beta
> Reporter: Jason Lowe
> Priority: Critical
>
> ContainerManagementProtocolProxy has a cache of NM proxies, and the size of
> this cache is configurable. However the cache can grow far beyond the
> configured size when running on a large cluster and blow AM address/container
> limits. More details in the first comment.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
