[ 
https://issues.apache.org/jira/browse/YARN-3809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14587519#comment-14587519
 ] 

Rohith commented on YARN-3809:
------------------------------

This is interesting scenario, but am not sure why ThreadPool is set to 10 which 
is not configurable.
bq. the default RPC time out is 15 mins.. 
I see RPC timeout is 1 minute, am I missing anything?
{code}
static final int DEFAULT_COMMAND_TIMEOUT = 60000;
.....
  int expireIntvl = conf.getInt(NM_COMMAND_TIMEOUT, DEFAULT_COMMAND_TIMEOUT);
    proxy =
        (ContainerManagementProtocolPB) 
RPC.getProxy(ContainerManagementProtocolPB.class,
          clientVersion, addr, ugi, conf,
          NetUtils.getDefaultSocketFactory(conf), expireIntvl);
{code}

> Failed to launch new attempts because ApplicationMasterLauncher's threads all 
> hang
> ----------------------------------------------------------------------------------
>
>                 Key: YARN-3809
>                 URL: https://issues.apache.org/jira/browse/YARN-3809
>             Project: Hadoop YARN
>          Issue Type: Improvement
>          Components: resourcemanager
>            Reporter: Jun Gong
>            Assignee: Jun Gong
>
> ApplicationMasterLauncher create a thread pool whose size is 10 to deal with 
> AMLauncherEventType(LAUNCH and CLEANUP).
> In our cluster, there was many NM with 10+ AM running on it, and one shut 
> down for some reason. After RM found the NM LOST, it cleaned up AMs running 
> on it. Then ApplicationMasterLauncher need handle these 10+ CLEANUP event. 
> ApplicationMasterLauncher's thread pool would be filled up, and they all hang 
> in the code containerMgrProxy.stopContainers(stopRequest) because NM was 
> down, the default RPC time out is 15 mins. It means that in 15 mins 
> ApplicationMasterLauncher could not handle new event such as LAUNCH, then new 
> attempts will fails to launch because of time out.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Reply via email to