[
https://issues.apache.org/jira/browse/YARN-7277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
balloons updated YARN-7277:
---------------------------
Description:
The SPARK application I submitted always failed and I finally found that the
commands I specified to launch AM Container were changed by NM.
*The following is part of the excerpt I submitted to RM to see the command:*
{code:java}
*'{\"handler\":\"FILLER\",\"inputTable\":\"engine_arch.adult_train\",\"outputTable\":[\"ether_features_filler_\$experimentId_\$taskId_out0\"],\"params\":{\"age\":{\"param\":[\"0\"]}}}'*
{code}
*The following is an excerpt from the corresponding command used when I observe
the NM launch container:*
{code:java}
*'{\"handler\":\"FILLER\",\"inputTable\":\"engine_arch.adult_train\",\"outputTable\":[\"ether_features_filler_\$experimentId_\$taskId_out0\"],\"params\":{\"age\":{\"param\":[\"0\"]}*
{code}
Finally, I found that NM made the following transformation in launch container
which led to this situation:
{code:java}
@VisibleForTesting
public static String expandEnvironment(String var,
Path containerLogDir) {
var = var.replace(ApplicationConstants.LOG_DIR_EXPANSION_VAR,
containerLogDir.toString());
var = var.replace(ApplicationConstants.CLASS_PATH_SEPARATOR,
File.pathSeparator);
// replace parameter expansion marker. e.g. {{VAR}} on Windows is replaced
// as %VAR% and on Linux replaced as "$VAR"
if (Shell.WINDOWS) {
var = var.replaceAll("(\\{\\{)|(\\}\\})", "%");
} else {
var = var.replace(ApplicationConstants.PARAMETER_EXPANSION_LEFT, "$");
*var = var.replace(ApplicationConstants.PARAMETER_EXPANSION_RIGHT, "");*
}
return var;
}
{code}
I think this is a Bug that doesn't even consider the pairing of
"PARAMETER_EXPANSION_LEFT" and "*PARAMETER_EXPANSION_RIGHT*" when substituting.
But simply substituting for simple violence.
was:
The SPARK application I submitted always failed and I finally found that the
commands I specified to launch AM Container were changed by NM.
*The following is part of the excerpt I submitted to RM to see the command:*
{code:java}
*'{\"handler\":\"FILLER\",\"inputTable\":\"engine_arch.adult_train\",\"outputTable\":[\"ether_features_filler_\$experimentId_\$taskId_out0\"],\"params\":{\"age\":{\"param\":[\"0\"]}}}'*
{code}
*The following is an excerpt from the corresponding command used when I observe
the NM launch container:*
{code:java}
*'{\"handler\":\"FILLER\",\"inputTable\":\"engine_arch.adult_train\",\"outputTable\":[\"ether_features_filler_\$experimentId_\$taskId_out0\"],\"params\":{\"age\":{\"param\":[\"0\"]}*
{code}
Finally, I found that NM made the following transformation in launch container
which led to this situation:
{code:java}
@VisibleForTesting
public static String expandEnvironment(String var,
Path containerLogDir) {
var = var.replace(ApplicationConstants.LOG_DIR_EXPANSION_VAR,
containerLogDir.toString());
var = var.replace(ApplicationConstants.CLASS_PATH_SEPARATOR,
File.pathSeparator);
// replace parameter expansion marker. e.g. {{VAR}} on Windows is replaced
// as %VAR% and on Linux replaced as "$VAR"
if (Shell.WINDOWS) {
var = var.replaceAll("(\\{\\{)|(\\}\\})", "%");
} else {
var = var.replace(ApplicationConstants.PARAMETER_EXPANSION_LEFT, "$");
*var = var.replace(ApplicationConstants.PARAMETER_EXPANSION_RIGHT, "");*
}
return var;
}
{code}
I think this is a Bug that doesn't even consider the pairing of "*{{*" and
"*}}*" when substituting, but simply substituting for simple violence.
> Container Launch expand environment needs to consider bracket matching
> ----------------------------------------------------------------------
>
> Key: YARN-7277
> URL: https://issues.apache.org/jira/browse/YARN-7277
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
> Reporter: balloons
> Priority: Critical
>
> The SPARK application I submitted always failed and I finally found that the
> commands I specified to launch AM Container were changed by NM.
> *The following is part of the excerpt I submitted to RM to see the command:*
> {code:java}
> *'{\"handler\":\"FILLER\",\"inputTable\":\"engine_arch.adult_train\",\"outputTable\":[\"ether_features_filler_\$experimentId_\$taskId_out0\"],\"params\":{\"age\":{\"param\":[\"0\"]}}}'*
> {code}
> *The following is an excerpt from the corresponding command used when I
> observe the NM launch container:*
> {code:java}
> *'{\"handler\":\"FILLER\",\"inputTable\":\"engine_arch.adult_train\",\"outputTable\":[\"ether_features_filler_\$experimentId_\$taskId_out0\"],\"params\":{\"age\":{\"param\":[\"0\"]}*
> {code}
> Finally, I found that NM made the following transformation in launch
> container which led to this situation:
> {code:java}
> @VisibleForTesting
> public static String expandEnvironment(String var,
> Path containerLogDir) {
> var = var.replace(ApplicationConstants.LOG_DIR_EXPANSION_VAR,
> containerLogDir.toString());
> var = var.replace(ApplicationConstants.CLASS_PATH_SEPARATOR,
> File.pathSeparator);
> // replace parameter expansion marker. e.g. {{VAR}} on Windows is replaced
> // as %VAR% and on Linux replaced as "$VAR"
> if (Shell.WINDOWS) {
> var = var.replaceAll("(\\{\\{)|(\\}\\})", "%");
> } else {
> var = var.replace(ApplicationConstants.PARAMETER_EXPANSION_LEFT, "$");
> *var = var.replace(ApplicationConstants.PARAMETER_EXPANSION_RIGHT, "");*
> }
> return var;
> }
> {code}
> I think this is a Bug that doesn't even consider the pairing of
> "PARAMETER_EXPANSION_LEFT" and "*PARAMETER_EXPANSION_RIGHT*" when
> substituting. But simply substituting for simple violence.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
