[
https://issues.apache.org/jira/browse/SPARK-17814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15567964#comment-15567964
]
Saisai Shao commented on SPARK-17814:
-------------------------------------
By checking the yarn code, I think you might run into YARN's trap, Yarn will
replace the environments in command from {{"{{JAVA}}"}} to {{"$JAVA"}}, so in
your case last braces will be replaced with {{""}}. Please see the yarn code
here:
{code}
@VisibleForTesting
public static String expandEnvironment(String var,
Path containerLogDir) {
var = var.replace(ApplicationConstants.LOG_DIR_EXPANSION_VAR,
containerLogDir.toString());
var = var.replace(ApplicationConstants.CLASS_PATH_SEPARATOR,
File.pathSeparator);
// replace parameter expansion marker. e.g. {{VAR}} on Windows is replaced
// as %VAR% and on Linux replaced as "$VAR"
if (Shell.WINDOWS) {
var = var.replaceAll("(\\{\\{)|(\\}\\})", "%");
} else {
var = var.replace(ApplicationConstants.PARAMETER_EXPANSION_LEFT, "$");
var = var.replace(ApplicationConstants.PARAMETER_EXPANSION_RIGHT, "");
}
return var;
}
{code}
Especially this line {{var =
var.replace(ApplicationConstants.PARAMETER_EXPANSION_RIGHT, "");}}. The right
braces will be replaced with empty String.
So this is actually not a Spark issue, and may not easily be fixed by Yarn. But
you can have many workarounds to avoid passing json String with arguments.
> spark submit arguments are truncated in yarn-cluster mode
> ---------------------------------------------------------
>
> Key: SPARK-17814
> URL: https://issues.apache.org/jira/browse/SPARK-17814
> Project: Spark
> Issue Type: Bug
> Components: Spark Submit, YARN
> Affects Versions: 1.6.1
> Reporter: shreyas subramanya
> Priority: Minor
>
> {noformat}
> One of our customers is trying to pass in json through spark-submit as
> follows:
> spark-submit --verbose --class SimpleClass --master yarn-cluster ./simple.jar
> "{\"mode\":\"wf\", \"arrays\":{\"array\":[1]}}"
> The application reports the passed arguments as: {"mode":"wf",
> "arrays":{"array":[1]
> If the same application is submitted in yarn-client mode, as follows:
> spark-submit --verbose --class SimpleClass --master yarn-client ./simple.jar
> "{\"mode\":\"wf\", \"arrays\":{\"array\":[1]}}"
> The application reports the passed args as: {"mode":"wf",
> "arrays":{"array":[1]}}
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
