[
https://issues.apache.org/jira/browse/YARN-4727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Terence Yim updated YARN-4727:
------------------------------
Description:
Given the default config of "yarn.nodemanager.env-whitelist", application
should be able to set the env variable $HADOOP_CONF_DIR to value other than the
one in the NodeManager system environment. However, I believe due to a bug in
the
{{org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch}}
class, it is not possible so.
>From the {{sanitizeEnv()}} method in the ContainerLaunch class
>(https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java#L977)
{noformat}
putEnvIfNotNull(environment,
Environment.HADOOP_CONF_DIR.name(),
System.getenv(Environment.HADOOP_CONF_DIR.name())
);
if (!Shell.WINDOWS) {
environment.put("JVM_PID", "$$");
}
String[] whitelist = conf.get(YarnConfiguration.NM_ENV_WHITELIST,
YarnConfiguration.DEFAULT_NM_ENV_WHITELIST).split(",");
for(String whitelistEnvVariable : whitelist) {
putEnvIfAbsent(environment, whitelistEnvVariable.trim());
}
...
private static void putEnvIfAbsent(
Map<String, String> environment, String variable) {
if (environment.get(variable) == null) {
putEnvIfNotNull(environment, variable, System.getenv(variable));
}
}
{noformat}
So there two issues here.
1. the environment is already set with the system environment of the NM in the
{{putEnvIfNotNull}} call, hence the {{putEnvIfAbsent}} call will never set it
to some new value
2. Inside the {{putEnvIfAbsent}} call, it uses the system environment of the
NM, which it should be using the one from the {{launchContext}} instead.
was:
Given the default config of "yarn.nodemanager.env-whitelist", application
should be able to set the env variable $HADOOP_CONF_DIR to value other than the
one in the NodeManager system environment. However, I believe due to a bug in
the
{{org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch}}
class, it is not possible so.
>From the {{sanitizeEnv()}} method in the ContainerLaunch class
>(https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java#L991)
{noformat}
String[] whitelist = conf.get(YarnConfiguration.NM_ENV_WHITELIST,
YarnConfiguration.DEFAULT_NM_ENV_WHITELIST).split(",");
for(String whitelistEnvVariable : whitelist) {
putEnvIfAbsent(environment, whitelistEnvVariable.trim());
}
...
private static void putEnvIfAbsent(
Map<String, String> environment, String variable) {
if (environment.get(variable) == null) {
putEnvIfNotNull(environment, variable, System.getenv(variable));
}
}
{noformat}
The {{putEnvIfAbsent}} call simply use the {{whitelistEnvVariable}} to call
{{System.getenv(whitelistEnvVariable)}} and set it to the environment map. It
should be using the one from the {{launchContext}} instead.
> Unable to override the $HADOOP_CONF_DIR env variable for container
> ------------------------------------------------------------------
>
> Key: YARN-4727
> URL: https://issues.apache.org/jira/browse/YARN-4727
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
> Affects Versions: 2.4.1, 2.5.2, 2.7.2, 2.6.4
> Reporter: Terence Yim
>
> Given the default config of "yarn.nodemanager.env-whitelist", application
> should be able to set the env variable $HADOOP_CONF_DIR to value other than
> the one in the NodeManager system environment. However, I believe due to a
> bug in the
> {{org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch}}
> class, it is not possible so.
> From the {{sanitizeEnv()}} method in the ContainerLaunch class
> (https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java#L977)
> {noformat}
> putEnvIfNotNull(environment,
> Environment.HADOOP_CONF_DIR.name(),
> System.getenv(Environment.HADOOP_CONF_DIR.name())
> );
> if (!Shell.WINDOWS) {
> environment.put("JVM_PID", "$$");
> }
> String[] whitelist = conf.get(YarnConfiguration.NM_ENV_WHITELIST,
> YarnConfiguration.DEFAULT_NM_ENV_WHITELIST).split(",");
>
> for(String whitelistEnvVariable : whitelist) {
> putEnvIfAbsent(environment, whitelistEnvVariable.trim());
> }
> ...
> private static void putEnvIfAbsent(
> Map<String, String> environment, String variable) {
> if (environment.get(variable) == null) {
> putEnvIfNotNull(environment, variable, System.getenv(variable));
> }
> }
> {noformat}
> So there two issues here.
> 1. the environment is already set with the system environment of the NM in
> the {{putEnvIfNotNull}} call, hence the {{putEnvIfAbsent}} call will never
> set it to some new value
> 2. Inside the {{putEnvIfAbsent}} call, it uses the system environment of the
> NM, which it should be using the one from the {{launchContext}} instead.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
