[ https://issues.apache.org/jira/browse/YARN-2359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14089307#comment-14089307 ]
Hudson commented on YARN-2359: ------------------------------ SUCCESS: Integrated in Hadoop-Mapreduce-trunk #1856 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1856/]) YARN-2359. Application hangs when it fails to launch AM container. (Zhihai Xu via kasha) (kasha: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1616375) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/RMAppAttemptImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/TestRMAppAttemptTransitions.java > Application hangs when it fails to launch AM container > ------------------------------------------------------- > > Key: YARN-2359 > URL: https://issues.apache.org/jira/browse/YARN-2359 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager > Reporter: zhihai xu > Assignee: zhihai xu > Priority: Critical > Fix For: 2.6.0 > > Attachments: YARN-2359.000.patch, YARN-2359.001.patch, > YARN-2359.002.patch > > > Application is hung without timeout and retry after DNS/network is down. > It is because right after the container is allocated for the AM, the > DNS/network is down for the node which has the AM container. > The application attempt is at state RMAppAttemptState.SCHEDULED, it receive > RMAppAttemptEventType.CONTAINER_ALLOCATED event, because the > IllegalArgumentException(due to DNS error) happened, it stay at state > RMAppAttemptState.SCHEDULED. In the state machine, only two events will be > processed at this state: > RMAppAttemptEventType.CONTAINER_ALLOCATED and RMAppAttemptEventType.KILL. > The code didn't handle the event(RMAppAttemptEventType.CONTAINER_FINISHED) > which will be generated when the node and container timeout. So even the node > is removed, the Application is still hung in this state > RMAppAttemptState.SCHEDULED. > The only way to make the application exit this state is to send > RMAppAttemptEventType.KILL event which will only be generated when you > manually kill the application from Job Client by forceKillApplication. > To fix the issue, we should add an entry in the state machine table to handle > RMAppAttemptEventType.CONTAINER_FINISHED event at state > RMAppAttemptState.SCHEDULED > add the following code in StateMachineFactory: > {code}.addTransition(RMAppAttemptState.SCHEDULED, > RMAppAttemptState.FINAL_SAVING, > RMAppAttemptEventType.CONTAINER_FINISHED, > new FinalSavingTransition( > new AMContainerCrashedBeforeRunningTransition(), > RMAppAttemptState.FAILED)){code} -- This message was sent by Atlassian JIRA (v6.2#6252)