[ 
https://issues.apache.org/jira/browse/YARN-4046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14682284#comment-14682284
 ] 

Anubhav Dhoot commented on YARN-4046:
-------------------------------------

The error in NodeManager shows 
{noformat}
2015-08-10 15:14:05,567 ERROR 
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.RecoveredContainerLaunch:
 Unable to recover container container_e45_1439244348718_0001_01_000001
java.io.IOException: Timeout while waiting for exit code from 
container_e45_1439244348718_0001_01_000001
        at 
org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor.reacquireContainer(ContainerExecutor.java:199)
        at 
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.RecoveredContainerLaunch.call(RecoveredContainerLaunch.java:83)
        at 
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.RecoveredContainerLaunch.call(RecoveredContainerLaunch.java:46)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
{noformat}

Looking under the debugger the actual shell command to check if container is 
alive fails because the kill command syntax  "kill -0 -20773" fails.
{noformat}
his = {org.apache.hadoop.util.Shell$ShellCommandExecutor@6740} "kill -0 -20773 "
builder = {java.lang.ProcessBuilder@6789} 
 command = {java.util.ArrayList@6813}  size = 3
 directory = null
 environment = null
 redirectErrorStream = false
 redirects = null
timeOutTimer = null
timeoutTimerTask = null
errReader = {java.io.BufferedReader@6830} 
inReader = {java.io.BufferedReader@6833} 
errMsg = {java.lang.StringBuffer@6836} "kill: invalid option -- '2'\n\nUsage:\n 
kill [options] <pid> [...]\n\nOptions:\n <pid> [...]            send signal to 
every <pid> listed\n -<signal>, -s, --signal <signal>\n                        
specify the <signal> to be sent\n -l, --list=[<signal>]  list all signal names, 
or convert one to a name\n -L, --table            list all signal names in a 
nice table\n\n -h, --help     display this help and exit\n -V, --version  
output version information and exit\n\nFor more details see kill(1).\n"
errThread = {org.apache.hadoop.util.Shell$1@6839} "Thread[Thread-102,5,]"
line = null
exitCode = 1
completed = {java.util.concurrent.atomic.AtomicBoolean@6806} "true"
{noformat}

This causes DefaultContainerExecutor#containerIsAlive to catch 
ExitCodeException thrown by ShellCommandExecutor.execute making it assume the 
container is lost.

> Applications fail on NM restart on some linux distro because NM container 
> recovery declares AM container as LOST
> ----------------------------------------------------------------------------------------------------------------
>
>                 Key: YARN-4046
>                 URL: https://issues.apache.org/jira/browse/YARN-4046
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: nodemanager
>            Reporter: Anubhav Dhoot
>            Assignee: Anubhav Dhoot
>            Priority: Critical
>
> On a debian machine we have seen node manager recovery of containers fail 
> because the signal syntax for process group may not work. We see errors in 
> checking if process is alive during container recovery which causes the 
> container to be declared as LOST (154) on a NodeManager restart.
> The application will fail with error
> {noformat}
> Application application_1439244348718_0001 failed 1 times due to Attempt 
> recovered after RM restartAM Container for 
> appattempt_1439244348718_0001_000001 exited with exitCode: 154
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Reply via email to