[
https://issues.apache.org/jira/browse/YARN-7226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16176376#comment-16176376
]
Jason Lowe commented on YARN-7226:
----------------------------------
bq. For example, an image used for running MR tasks could specify its own
JAVA_HOME that needs to be used instead of the JAVA_HOME that is specified in
the nodemanager's environment.
But that's not what the whitelist is for. The whitelist is for the
*nodemanager's environment variables* not the variable associated with the
Docker image that the user requested to run with their job. That's not a
nodemanager thing, that's a user's Docker image thing. In all cases, a
whitelist should never prevent the container from overriding the variable, but
that appears to be what this is doing for the Docker case. Any variable in the
whitelist will not be set to the user's value _unless_ the Docker image doesn't
specify it, correct? If so, that's going to be surprising to users.
If Docker wants/needs the ability to "screen" certain variables from ever being
set if they are specified in the image, whether that setting comes from the
container context directly or the NM's environment via the whitelist, then IMHO
that should be a separate config. Otherwise in a mixed-mode environment where
some containers are running outside of Docker and some are running in Docker,
how is an admin supposed to properly set yarn.nodemanager.env-whitelist
properly? The set of variables in a Docker image and the set of NM variables
that need to peek through to non-Docker containers may not be the same.
> Whitelisted variables do not support delayed variable expansion
> ---------------------------------------------------------------
>
> Key: YARN-7226
> URL: https://issues.apache.org/jira/browse/YARN-7226
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
> Affects Versions: 2.9.0, 2.8.1, 3.0.0-alpha4
> Reporter: Jason Lowe
> Assignee: Jason Lowe
> Attachments: YARN-7226.001.patch, YARN-7226.002.patch
>
>
> The nodemanager supports a configurable list of environment variables, via
> yarn.nodemanager.env-whitelist, that will be propagated to the container's
> environment unless those variables were specified in the container launch
> context. Unfortunately the handling of these whitelisted variables prevents
> using delayed variable expansion. For example, if a user shipped their own
> version of hadoop with their job via the distributed cache and specified:
> {noformat}
> HADOOP_COMMON_HOME={{PWD}}/my-private-hadoop/
> {noformat}
> as part of their job, the variable will be set as the *literal* string:
> {noformat}
> $PWD/my-private-hadoop/
> {noformat}
> rather than having $PWD expand to the container's current directory as it
> does for any other, non-whitelisted variable being set to the same value.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
