[
https://issues.apache.org/jira/browse/YARN-3809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15209981#comment-15209981
]
Rohith Sharma K S commented on YARN-3809:
-----------------------------------------
Currently NMProxy wait for infinite times i.e 0 is set by default from
NMProxy. This is the reason all your AM-NM calls are hung. This can be
controlled by configuring "ipc.client.connect.max.retries.on.timeouts".
May be new configuration can be added with name
*yarn.client.connect.max.retries.on.timeouts* to control NMProxy connections
only. Can you raise new ticket for this?
> Failed to launch new attempts because ApplicationMasterLauncher's threads all
> hang
> ----------------------------------------------------------------------------------
>
> Key: YARN-3809
> URL: https://issues.apache.org/jira/browse/YARN-3809
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
> Reporter: Jun Gong
> Assignee: Jun Gong
> Fix For: 2.7.1
>
> Attachments: YARN-3809.01.patch, YARN-3809.02.patch,
> YARN-3809.03.patch
>
>
> ApplicationMasterLauncher create a thread pool whose size is 10 to deal with
> AMLauncherEventType(LAUNCH and CLEANUP).
> In our cluster, there was many NM with 10+ AM running on it, and one shut
> down for some reason. After RM found the NM LOST, it cleaned up AMs running
> on it. Then ApplicationMasterLauncher need handle these 10+ CLEANUP event.
> ApplicationMasterLauncher's thread pool would be filled up, and they all hang
> in the code containerMgrProxy.stopContainers(stopRequest) because NM was
> down, the default RPC time out is 15 mins. It means that in 15 mins
> ApplicationMasterLauncher could not handle new event such as LAUNCH, then new
> attempts will fails to launch because of time out.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
