[ 
https://issues.apache.org/jira/browse/YARN-2934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14267377#comment-14267377
 ] 

Gera Shegalov commented on YARN-2934:
-------------------------------------

ContainerLaunchContext is meant for a ContainerExecutor in general. In the LCE 
case the logs may only be readable by the app user. To make it robust we can 
simply append the catting to the supplied command that runs in the executor. 
This is the hacky version for it disregarding OS diversity. It presumes that 
stderr log has "strderr" in the file name.

{code}
$ git diff
diff --git 
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java
 
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java
index a87238d..8ea2560 100644
--- 
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java
+++ 
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java
@@ -190,6 +190,11 @@ public Integer call() {
         // TODO: Should we instead work via symlinks without this grammar?
         newCmds.add(expandEnvironment(str, containerLogDir));
       }
+      newCmds.add("||");
+      newCmds.add("(cat " + containerLogDir + "/*stderr* 1>&2");
+      newCmds.add(";");
+      newCmds.add("exit -1)");
+
       launchContext.setCommands(newCmds);
 
       Map<String, String> environment = launchContext.getEnvironment();
{code}

Then we get the desired effect:
{code}
]$ hadoop org.apache.hadoop.mapreduce.SleepJob 
-Dyarn.app.mapreduce.am.env=JAVA_HOME=/no/jvm/here -m 10 
15/01/06 23:36:13 WARN util.NativeCodeLoader: Unable to load native-hadoop 
library for your platform... using builtin-java classes where applicable
15/01/06 23:36:14 INFO client.RMProxy: Connecting to ResourceManager at 
localhost/127.0.0.1:8032
15/01/06 23:36:15 INFO mapreduce.JobSubmitter: number of splits:10
15/01/06 23:36:15 INFO mapreduce.JobSubmitter: Submitting tokens for job: 
job_1420533216615_0013
15/01/06 23:36:16 INFO impl.YarnClientImpl: Submitted application 
application_1420533216615_0013
15/01/06 23:36:16 INFO mapreduce.Job: The url to track the job: 
http://localhost:8088/proxy/application_1420533216615_0013/
15/01/06 23:36:16 INFO mapreduce.Job: Running job: job_1420533216615_0013
15/01/06 23:36:21 INFO mapreduce.Job: Job job_1420533216615_0013 running in 
uber mode : false
15/01/06 23:36:21 INFO mapreduce.Job:  map 0% reduce 0%
15/01/06 23:36:21 INFO mapreduce.Job: Job job_1420533216615_0013 failed with 
state FAILED due to: Application application_1420533216615_0013 failed 2 times 
due to AM Container for appattempt_1420533216615_0013_000002 exited with  
exitCode: 255
For more detailed output, check application tracking 
page:http://localhost:8088/proxy/application_1420533216615_0013/Then, click on 
links to logs of each attempt.
Diagnostics: Exception from container-launch.
Container id: container_1420533216615_0013_02_000001
Exit code: 255
Exception message: /bin/bash: /no/jvm/here/bin/java: No such file or directory

Stack trace: ExitCodeException exitCode=255: /bin/bash: /no/jvm/here/bin/java: 
No such file or directory

        at org.apache.hadoop.util.Shell.runCommand(Shell.java:545)
        at org.apache.hadoop.util.Shell.run(Shell.java:456)
        at 
org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:722)
        at 
org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:211)
        at 
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:307)
        at 
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
{code}

> Improve handling of container's stderr 
> ---------------------------------------
>
>                 Key: YARN-2934
>                 URL: https://issues.apache.org/jira/browse/YARN-2934
>             Project: Hadoop YARN
>          Issue Type: Improvement
>            Reporter: Gera Shegalov
>            Assignee: Naganarasimha G R
>
> Most YARN applications redirect stderr to some file. That's why when 
> container launch fails with {{ExitCodeException}} the message is empty.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Reply via email to