[
https://issues.apache.org/jira/browse/YARN-7677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16361418#comment-16361418
]
Jim Brennan commented on YARN-7677:
-----------------------------------
I've put up another patch for this that addresses the ordering issues.
The new patch writes environment variables to the launch script in the
following order:
# Whitelisted variables that are coming from the NM environment (they were not
set in the ContainerLaunchContext). These are written using the default value
syntax, so that whitelisted variables that are also set in the docker container
will use the docker container setting. These are written in the order they are
listed in the NM_ENV_WHITELIST yarn config.
# Variables that are explicitly set by the nodemanager (mostly in
ContainerLaunch.sanitizeEnv()). These are tracked so that they are written to
the launch script in the order they are added in code.
# Remaining variables from the ContainerLaunchContext. Note that these are
still not ordered - they are just written in hash-map order. Changing these to
ordered is covered by YARN-5714. This patch just ensures that they come after
whitelisted and NM vars.
A couple notes about this patch:
* Variables that are listed in NM_ENV_WHITELIST, but are also set in the
ContainerLaunchContext() are written along with the rest of the
ContainerLaunchContext environment variables (after NM variables).
* I am currently tracking as an NM variable the CLASSPATH that is touched in
sanitizeWindowsEnv(), but I think this is incorrect. sanitizeWindowsEnv() only
modifies CLASSPATH if it's already in the ContainerLaunchContext().
* The variables defined in NM_ADMIN_USER_ENV are tracked along with those set
by NM. I'm not sure this is correct.
The patch includes a new test that verifies this ordering.
> Docker image cannot set HADOOP_CONF_DIR
> ---------------------------------------
>
> Key: YARN-7677
> URL: https://issues.apache.org/jira/browse/YARN-7677
> Project: Hadoop YARN
> Issue Type: Bug
> Affects Versions: 3.0.0
> Reporter: Eric Badger
> Assignee: Jim Brennan
> Priority: Major
> Attachments: YARN-7677.001.patch, YARN-7677.002.patch,
> YARN-7677.003.patch
>
>
> Currently, {{HADOOP_CONF_DIR}} is being put into the task environment whether
> it's set by the user or not. It completely bypasses the whitelist and so
> there is no way for a task to not have {{HADOOP_CONF_DIR}} set. This causes
> problems in the Docker use case where Docker containers will set up their own
> environment and have their own {{HADOOP_CONF_DIR}} preset in the image
> itself.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
