[ https://issues.apache.org/jira/browse/SPARK-17814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15567964#comment-15567964 ]
Saisai Shao commented on SPARK-17814: ------------------------------------- By checking the yarn code, I think you might run into YARN's trap, Yarn will replace the environments in command from {{"{{JAVA}}"}} to {{"$JAVA"}}, so in your case last braces will be replaced with {{""}}. Please see the yarn code here: {code} @VisibleForTesting public static String expandEnvironment(String var, Path containerLogDir) { var = var.replace(ApplicationConstants.LOG_DIR_EXPANSION_VAR, containerLogDir.toString()); var = var.replace(ApplicationConstants.CLASS_PATH_SEPARATOR, File.pathSeparator); // replace parameter expansion marker. e.g. {{VAR}} on Windows is replaced // as %VAR% and on Linux replaced as "$VAR" if (Shell.WINDOWS) { var = var.replaceAll("(\\{\\{)|(\\}\\})", "%"); } else { var = var.replace(ApplicationConstants.PARAMETER_EXPANSION_LEFT, "$"); var = var.replace(ApplicationConstants.PARAMETER_EXPANSION_RIGHT, ""); } return var; } {code} Especially this line {{var = var.replace(ApplicationConstants.PARAMETER_EXPANSION_RIGHT, "");}}. The right braces will be replaced with empty String. So this is actually not a Spark issue, and may not easily be fixed by Yarn. But you can have many workarounds to avoid passing json String with arguments. > spark submit arguments are truncated in yarn-cluster mode > --------------------------------------------------------- > > Key: SPARK-17814 > URL: https://issues.apache.org/jira/browse/SPARK-17814 > Project: Spark > Issue Type: Bug > Components: Spark Submit, YARN > Affects Versions: 1.6.1 > Reporter: shreyas subramanya > Priority: Minor > > {noformat} > One of our customers is trying to pass in json through spark-submit as > follows: > spark-submit --verbose --class SimpleClass --master yarn-cluster ./simple.jar > "{\"mode\":\"wf\", \"arrays\":{\"array\":[1]}}" > The application reports the passed arguments as: {"mode":"wf", > "arrays":{"array":[1] > If the same application is submitted in yarn-client mode, as follows: > spark-submit --verbose --class SimpleClass --master yarn-client ./simple.jar > "{\"mode\":\"wf\", \"arrays\":{\"array\":[1]}}" > The application reports the passed args as: {"mode":"wf", > "arrays":{"array":[1]}} > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org