Saisai Shao commented on SPARK-17814:

By checking the yarn code, I think you might run into YARN's trap, Yarn will 
replace the environments in command from {{"{{JAVA}}"}} to {{"$JAVA"}}, so in 
your case last braces will be replaced with {{""}}. Please see the yarn code 

  public static String expandEnvironment(String var,
      Path containerLogDir) {
    var = var.replace(ApplicationConstants.LOG_DIR_EXPANSION_VAR,
    var =  var.replace(ApplicationConstants.CLASS_PATH_SEPARATOR,

    // replace parameter expansion marker. e.g. {{VAR}} on Windows is replaced
    // as %VAR% and on Linux replaced as "$VAR"
    if (Shell.WINDOWS) {
      var = var.replaceAll("(\\{\\{)|(\\}\\})", "%");
    } else {
      var = var.replace(ApplicationConstants.PARAMETER_EXPANSION_LEFT, "$");
      var = var.replace(ApplicationConstants.PARAMETER_EXPANSION_RIGHT, "");
    return var;

Especially this line {{var = 
var.replace(ApplicationConstants.PARAMETER_EXPANSION_RIGHT, "");}}. The right 
braces will be replaced with empty String.

So this is actually not a Spark issue, and may not easily be fixed by Yarn. But 
you can have many workarounds to avoid passing json String with arguments.

> spark submit arguments are truncated in yarn-cluster mode
> ---------------------------------------------------------
>                 Key: SPARK-17814
>                 URL: https://issues.apache.org/jira/browse/SPARK-17814
>             Project: Spark
>          Issue Type: Bug
>          Components: Spark Submit, YARN
>    Affects Versions: 1.6.1
>            Reporter: shreyas subramanya
>            Priority: Minor
> {noformat}
> One of our customers is trying to pass in json through spark-submit as 
> follows:
> spark-submit --verbose --class SimpleClass --master yarn-cluster ./simple.jar 
> "{\"mode\":\"wf\", \"arrays\":{\"array\":[1]}}"
> The application reports the passed arguments as: {"mode":"wf", 
> "arrays":{"array":[1]
> If the same application is submitted in yarn-client mode, as follows:
> spark-submit --verbose --class SimpleClass --master yarn-client ./simple.jar 
> "{\"mode\":\"wf\", \"arrays\":{\"array\":[1]}}"
> The application reports the passed args as: {"mode":"wf", 
> "arrays":{"array":[1]}}
> {noformat}

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

Reply via email to