[
https://issues.apache.org/jira/browse/YARN-7062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lingfeng Su updated YARN-7062:
------------------------------
Description:
I passed a json string like "{User: Billy, {Age: 18}}" to main method args,
when running spark jobs on yarn. It may exchange by
ContainerLaunch.expandEnvironment() to "{User: Billy, {Age: 18" ("}}" to "")
I found the final arg in launch_container.sh of yarn containers, as:
--arg '{User: Billy, {Age: 18'
{code:java}
exec /bin/bash -c
"LD_LIBRARY_PATH="$HADOOP_COMMON_HOME/../../../CDH-5.11.1-1.cdh5.11.1.p0.4/lib/hadoop/lib/native:$LD_LIBRARY_PATH"
$JAVA_HOME/bin/java -server -Xmx1024m -Djava.io.tmpdir=$PWD/tmp
-Dspark.yarn.app.container.log.dir=/var/log/hadoop-yarn/container/application_1503214867081_0015/container_1503214867081_0015_01_000001
org.apache.spark.deploy.yarn.ApplicationMaster --class
'org.apache.spark.examples.sql.hive.HiveFromSpark' --jar
file:/opt/spark-submit/spark_sql_test-1.0.jar --arg '{User: Billy, {Age: 18'
--properties-file $PWD/__spark_conf__/__spark_conf__.properties 1>
/var/log/hadoop-yarn/container/application_1503214867081_0015/container_1503214867081_0015_01_000001/stdout
2>
/var/log/hadoop-yarn/container/application_1503214867081_0015/container_1503214867081_0015_01_000001/stderr"
{code}
We could make some improvements
was:
I passed a json string like "{User: Billy, {Age: 18}}" to main method args,
when running spark jobs on yarn. It may exchange by
ContainerLaunch.expandEnvironment() to "{User: Billy, {Age: 18" ("}}" to "")
I found the final arg in launch_container.sh of yarn containers, as:
--arg '{User: Billy, {Age: 18'
{code:bash}
exec /bin/bash -c
"LD_LIBRARY_PATH="$HADOOP_COMMON_HOME/../../../CDH-5.11.1-1.cdh5.11.1.p0.4/lib/hadoop/lib/native:$LD_LIBRARY_PATH"
$JAVA_HOME/bin/java -server -Xmx1024m -Djava.io.tmpdir=$PWD/tmp
-Dspark.yarn.app.container.log.dir=/var/log/hadoop-yarn/container/application_1503214867081_0015/container_1503214867081_0015_01_000001
org.apache.spark.deploy.yarn.ApplicationMaster --class
'org.apache.spark.examples.sql.hive.HiveFromSpark' --jar
file:/opt/spark-submit/spark_sql_test-1.0.jar --arg '{User: Billy, {Age: 18'
--properties-file $PWD/__spark_conf__/__spark_conf__.properties 1>
/var/log/hadoop-yarn/container/application_1503214867081_0015/container_1503214867081_0015_01_000001/stdout
2>
/var/log/hadoop-yarn/container/application_1503214867081_0015/container_1503214867081_0015_01_000001/stderr"
{code}
We could make some improvements
> yarn job args changed by ContainerLaunch.expandEnvironment(), such as "{{"
> and "}}"
> -----------------------------------------------------------------------------------
>
> Key: YARN-7062
> URL: https://issues.apache.org/jira/browse/YARN-7062
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: applications
> Reporter: Lingfeng Su
> Assignee: Lingfeng Su
> Priority: Minor
>
> I passed a json string like "{User: Billy, {Age: 18}}" to main method args,
> when running spark jobs on yarn. It may exchange by
> ContainerLaunch.expandEnvironment() to "{User: Billy, {Age: 18" ("}}" to "")
> I found the final arg in launch_container.sh of yarn containers, as:
> --arg '{User: Billy, {Age: 18'
> {code:java}
> exec /bin/bash -c
> "LD_LIBRARY_PATH="$HADOOP_COMMON_HOME/../../../CDH-5.11.1-1.cdh5.11.1.p0.4/lib/hadoop/lib/native:$LD_LIBRARY_PATH"
> $JAVA_HOME/bin/java -server -Xmx1024m -Djava.io.tmpdir=$PWD/tmp
> -Dspark.yarn.app.container.log.dir=/var/log/hadoop-yarn/container/application_1503214867081_0015/container_1503214867081_0015_01_000001
> org.apache.spark.deploy.yarn.ApplicationMaster --class
> 'org.apache.spark.examples.sql.hive.HiveFromSpark' --jar
> file:/opt/spark-submit/spark_sql_test-1.0.jar --arg '{User: Billy, {Age: 18'
> --properties-file $PWD/__spark_conf__/__spark_conf__.properties 1>
> /var/log/hadoop-yarn/container/application_1503214867081_0015/container_1503214867081_0015_01_000001/stdout
> 2>
> /var/log/hadoop-yarn/container/application_1503214867081_0015/container_1503214867081_0015_01_000001/stderr"
> {code}
> We could make some improvements
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
