[
https://issues.apache.org/jira/browse/YARN-7226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16173232#comment-16173232
]
Jason Lowe commented on YARN-7226:
----------------------------------
Maybe I'm missing something, but the whitelisted variable support seems to be a
lot more complicated than it needs to be. As I understand it, a whitelisted
variable is simply an environment variable that will be propagated from the
nodemanager's environment to the container's environment _unless_ the container
has specified it's own value for that variable. Rather than have special
whitelisted support when writing the shell scripts, etc., it seems all we need
to do is:
# Set all the environment variables in the env variable map that the container
specified as we do today.
# Do a subsequent pass over the whitelisted variables and simply leverage
{{putEnvIfAbsent}} to propagate the variable into the container's env if it
wasn't set in the previous step
...and that's it. No special script handling of these variables and no need
for {{var:-default}} variable expansion syntax. The latter causes the problem
because the value is always set in the environment of the launch_container.sh
process before the script executes, so it never takes the default value.
> Whitelisted variables do not support delayed variable expansion
> ---------------------------------------------------------------
>
> Key: YARN-7226
> URL: https://issues.apache.org/jira/browse/YARN-7226
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
> Affects Versions: 2.9.0, 2.8.1, 3.0.0-alpha4
> Reporter: Jason Lowe
> Assignee: Jason Lowe
>
> The nodemanager supports a configurable list of environment variables, via
> yarn.nodemanager.env-whitelist, that will be propagated to the container's
> environment unless those variables were specified in the container launch
> context. Unfortunately the handling of these whitelisted variables prevents
> using delayed variable expansion. For example, if a user shipped their own
> version of hadoop with their job via the distributed cache and specified:
> {noformat}
> HADOOP_COMMON_HOME={{PWD}}/my-private-hadoop/
> {noformat}
> as part of their job, the variable will be set as the *literal* string:
> {noformat}
> $PWD/my-private-hadoop/
> {noformat}
> rather than having $PWD expand to the container's current directory as it
> does for any other, non-whitelisted variable being set to the same value.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
