Lingfeng Su created YARN-7062: --------------------------------- Summary: yarn job args changed by ContainerLaunch.expandEnvironment(), such as "{{" and "}}" Key: YARN-7062 URL: https://issues.apache.org/jira/browse/YARN-7062 Project: Hadoop YARN Issue Type: Improvement Components: applications Reporter: Lingfeng Su Assignee: Lingfeng Su Priority: Minor
I passed a json string like "{User: Billy, {Age: 18}}" to main method args, when running spark jobs on yarn. It may exchange by ContainerLaunch.expandEnvironment() to "{User: Billy, {Age: 18" ("}}" to "") I found the final arg in launch_container.sh of yarn containers, as: --arg '{User: Billy, {Age: 18' {code:bash} exec /bin/bash -c "LD_LIBRARY_PATH="$HADOOP_COMMON_HOME/../../../CDH-5.11.1-1.cdh5.11.1.p0.4/lib/hadoop/lib/native:$LD_LIBRARY_PATH" $JAVA_HOME/bin/java -server -Xmx1024m -Djava.io.tmpdir=$PWD/tmp -Dspark.yarn.app.container.log.dir=/var/log/hadoop-yarn/container/application_1503214867081_0015/container_1503214867081_0015_01_000001 org.apache.spark.deploy.yarn.ApplicationMaster --class 'org.apache.spark.examples.sql.hive.HiveFromSpark' --jar file:/opt/spark-submit/spark_sql_test-1.0.jar --arg '{User: Billy, {Age: 18' --properties-file $PWD/__spark_conf__/__spark_conf__.properties 1> /var/log/hadoop-yarn/container/application_1503214867081_0015/container_1503214867081_0015_01_000001/stdout 2> /var/log/hadoop-yarn/container/application_1503214867081_0015/container_1503214867081_0015_01_000001/stderr" {code} We could make some improvements -- This message was sent by Atlassian JIRA (v6.4.14#64029) --------------------------------------------------------------------- To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org