Jim Brennan commented on YARN-7677:

I've put up another patch for this that addresses the ordering issues.

The new patch writes environment variables to the launch script in the 
following order:
 # Whitelisted variables that are coming from the NM environment (they were not 
set in the ContainerLaunchContext).  These are written using the default value 
syntax, so that whitelisted variables that are also set in the docker container 
will use the docker container setting.  These are written in the order they are 
listed in the NM_ENV_WHITELIST yarn config.
 # Variables that are explicitly set by the nodemanager (mostly in 
ContainerLaunch.sanitizeEnv()).  These are tracked so that they are written to 
the launch script in the order they are added in code.
 # Remaining variables from the ContainerLaunchContext.  Note that these are 
still not ordered - they are just written in hash-map order.  Changing these to 
ordered is covered by YARN-5714.  This patch just ensures that they come after 
whitelisted and NM vars.

 A couple notes about this patch:
 * Variables that are listed in NM_ENV_WHITELIST, but are also set in the 
ContainerLaunchContext() are written along with the rest of the 
ContainerLaunchContext environment variables (after NM variables).
 * I am currently tracking as an NM variable the CLASSPATH that is touched in 
sanitizeWindowsEnv(), but I think this is incorrect.  sanitizeWindowsEnv() only 
modifies CLASSPATH if it's already in the ContainerLaunchContext().
 * The variables defined in NM_ADMIN_USER_ENV are tracked along with those set 
by NM.  I'm not sure this is correct.

The patch includes a new test that verifies this ordering.


> Docker image cannot set HADOOP_CONF_DIR
> ---------------------------------------
>                 Key: YARN-7677
>                 URL: https://issues.apache.org/jira/browse/YARN-7677
>             Project: Hadoop YARN
>          Issue Type: Bug
>    Affects Versions: 3.0.0
>            Reporter: Eric Badger
>            Assignee: Jim Brennan
>            Priority: Major
>         Attachments: YARN-7677.001.patch, YARN-7677.002.patch, 
> YARN-7677.003.patch
> Currently, {{HADOOP_CONF_DIR}} is being put into the task environment whether 
> it's set by the user or not. It completely bypasses the whitelist and so 
> there is no way for a task to not have {{HADOOP_CONF_DIR}} set. This causes 
> problems in the Docker use case where Docker containers will set up their own 
> environment and have their own {{HADOOP_CONF_DIR}} preset in the image 
> itself. 

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

Reply via email to