[
https://issues.apache.org/jira/browse/YARN-1303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13795743#comment-13795743
]
Xuan Gong commented on YARN-1303:
---------------------------------
[~hitesh]
bq.Could you clarify why "ls;ls" or "ls | grep foo" does not work in the first
place? Is there a bug in the implementation that needs to be fixed to address
this basic functionality?
I am not sure whether this can be counted as implementation bug. Why those
commands does not work is on how bash read them.
For example : I give the --shell_command ls;pwd (the command pipeline has the
same issue)
The script that used to launch ApplicationMaster has something like this :
{code}
exec /bin/bash -c "$JAVA_HOME/bin/java -Xmx512m
org.apache.hadoop.yarn.applications.distributedshell.ApplicationMaster
--container_memory 128 --container_vcores 1 --num_containers 2 --priority 0
--shell_command ls;pwd
1>/Users/xuan/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/target/TestDistributedShell/TestDistributedShell-logDir-nm-0_0/application_1381875664135_0001/container_1381875664135_0001_01_000001/AppMaster.stdout
2>/Users/xuan/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/target/TestDistributedShell/TestDistributedShell-logDir-nm-0_0/application_1381875664135_0001/container_1381875664135_0001_01_000001/AppMaster.stderr
"
{code}
The bash will treat that as two separate command.
The one is
{code}
$JAVA_HOME/bin/java -Xmx512m
org.apache.hadoop.yarn.applications.distributedshell.ApplicationMaster
--container_memory 128 --container_vcores 1 --num_containers 2 --priority 0
--shell_command ls
{code}
And all the containers will execute shell_command ls.
Verify it by checking the shell script for container
{code}
exec /bin/bash -c "ls
1>/Users/xuan/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/target/TestDistributedShell/TestDistributedShell-logDir-nm-0_0/application_1381875664135_0001/container_1381875664135_0001_01_000002/stdout
2>/Users/xuan/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/target/TestDistributedShell/TestDistributedShell-logDir-nm-0_0/application_1381875664135_0001/container_1381875664135_0001_01_000002/stderr
"
{code}
The other one is :
{code}
pwd
1>/Users/xuan/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/target/TestDistributedShell/TestDistributedShell-logDir-nm-0_0/application_1381875664135_0001/container_1381875664135_0001_01_000001/AppMaster.stdout
2>/Users/xuan/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/target/TestDistributedShell/TestDistributedShell-logDir-nm-0_0/application_1381875664135_0001/container_1381875664135_0001_01_000001/AppMaster.stderr
{code}
At the AppMaster.stdout, we can only find out those message
{code}
/Users/xuan/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/target/TestDistributedShell/TestDistributedShell-localDir-nm-0_0/usercache/xuan/appcache/application_1381875664135_0001/container_1381875664135_0001_01_000001
{code}
Which the result when we do pwd
> Allow multiple commands separating with ;
> -----------------------------------------
>
> Key: YARN-1303
> URL: https://issues.apache.org/jira/browse/YARN-1303
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: applications/distributed-shell
> Reporter: Tassapol Athiapinya
> Assignee: Xuan Gong
> Fix For: 2.2.1
>
> Attachments: YARN-1303.1.patch, YARN-1303.2.patch
>
>
> In shell, we can do "ls; ls" to run 2 commands at once.
> In distributed shell, this is not working. We should improve to allow this to
> occur. There are practical use cases that I know of to run multiple commands
> or to set environment variables before a command.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
