[
https://issues.apache.org/jira/browse/YARN-7226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16176987#comment-16176987
]
Jason Lowe commented on YARN-7226:
----------------------------------
bq. So my vote would be for putting this patch in and then filing a followup
jira to add a NM configurable list of variables that cannot be changed by the
user (empty/disabled by default)
I'm not sure I can come up with a great reason to suppress any variables for
the non-Docker case. Besides, if a user really wants to get around that
behavior they can just run a shell script as their command that in turn smashes
the env vars the way they need it and runs the "real" command. IMHO this
appears to call for a config specific to the Docker runtime that says "here are
the list of variables I do not want to be overridden" or maybe relaxed to "here
are the list of variables to preclude from the NM's whitelist".
I do see the problem where a container did _not_ ask for any setting of some
variable before entering the Docker container, but it gets a setting anyway
because of NM whitelist processing (e.g.: HADOOP_COMMON_HOME or
HADOOP_CONF_DIR). That whitelist-inherited setting can end up being
problematic for the Docker image, e.g.: Hadoop is installed in a different
place in the image compared to where the NM is picking it up. However I would
counter-argue that since the user picked the Docker image, the user knows what
values to set for these variables and can override them explicitly in the
container startup. Basically I see it as a "would be nice to have" feature
rather than "Docker containers do not work without it" feature. Am I missing
the use case where it's really necessary?
> Whitelisted variables do not support delayed variable expansion
> ---------------------------------------------------------------
>
> Key: YARN-7226
> URL: https://issues.apache.org/jira/browse/YARN-7226
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
> Affects Versions: 2.9.0, 2.8.1, 3.0.0-alpha4
> Reporter: Jason Lowe
> Assignee: Jason Lowe
> Attachments: YARN-7226.001.patch, YARN-7226.002.patch
>
>
> The nodemanager supports a configurable list of environment variables, via
> yarn.nodemanager.env-whitelist, that will be propagated to the container's
> environment unless those variables were specified in the container launch
> context. Unfortunately the handling of these whitelisted variables prevents
> using delayed variable expansion. For example, if a user shipped their own
> version of hadoop with their job via the distributed cache and specified:
> {noformat}
> HADOOP_COMMON_HOME={{PWD}}/my-private-hadoop/
> {noformat}
> as part of their job, the variable will be set as the *literal* string:
> {noformat}
> $PWD/my-private-hadoop/
> {noformat}
> rather than having $PWD expand to the container's current directory as it
> does for any other, non-whitelisted variable being set to the same value.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
