[
https://issues.apache.org/jira/browse/YARN-4727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Lowe reassigned YARN-4727:
--------------------------------
Assignee: Jason Lowe
Affects Version/s: 2.8.1
Target Version/s: 2.8.3, 2.7.5
Sorry for missing this when it was filed. I just recently ran into the same
issue.
bq. the environment is already set with the system environment of the NM in the
putEnvIfNotNull call, hence the putEnvIfAbsent call will never set it to some
new value
This is the crux of the issue. Looks like the intent was to call
putEnvIfAbsent, as other whitelist processing does, rather than putEnvIfNotNull.
bq. Inside the putEnvIfAbsent call, it uses the system environment of the NM,
which it should be using the one from the launchContext instead.
This should be OK. If the user specified the variable in their environment
then it will not be absent, so it will not stuff the NM's version into the
environment. If the user did _not_ specify the variable then it will pick up
the NM's version.
> Unable to override the $HADOOP_CONF_DIR env variable for container
> ------------------------------------------------------------------
>
> Key: YARN-4727
> URL: https://issues.apache.org/jira/browse/YARN-4727
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
> Affects Versions: 2.4.1, 2.5.2, 2.7.2, 2.6.4, 2.8.1
> Reporter: Terence Yim
> Assignee: Jason Lowe
>
> Given the default config of "yarn.nodemanager.env-whitelist", application
> should be able to set the env variable $HADOOP_CONF_DIR to value other than
> the one in the NodeManager system environment. However, I believe due to a
> bug in the
> {{org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch}}
> class, it is not possible so.
> From the {{sanitizeEnv()}} method in the ContainerLaunch class
> (https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java#L977)
> {noformat}
> putEnvIfNotNull(environment,
> Environment.HADOOP_CONF_DIR.name(),
> System.getenv(Environment.HADOOP_CONF_DIR.name())
> );
> if (!Shell.WINDOWS) {
> environment.put("JVM_PID", "$$");
> }
> String[] whitelist = conf.get(YarnConfiguration.NM_ENV_WHITELIST,
> YarnConfiguration.DEFAULT_NM_ENV_WHITELIST).split(",");
>
> for(String whitelistEnvVariable : whitelist) {
> putEnvIfAbsent(environment, whitelistEnvVariable.trim());
> }
> ...
> private static void putEnvIfAbsent(
> Map<String, String> environment, String variable) {
> if (environment.get(variable) == null) {
> putEnvIfNotNull(environment, variable, System.getenv(variable));
> }
> }
> {noformat}
> So there two issues here.
> 1. the environment is already set with the system environment of the NM in
> the {{putEnvIfNotNull}} call, hence the {{putEnvIfAbsent}} call will never
> set it to some new value
> 2. Inside the {{putEnvIfAbsent}} call, it uses the system environment of the
> NM, which it should be using the one from the {{launchContext}} instead.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
