Eric Badger created YARN-10607:
----------------------------------
Summary: User environment is unable to prepend PATH when
mapreduce.admin.user.env also sets PATH
Key: YARN-10607
URL: https://issues.apache.org/jira/browse/YARN-10607
Project: Hadoop YARN
Issue Type: Bug
Reporter: Eric Badger
Assignee: Eric Badger
When using the tarball approach to ship relevant Hadoop jars to containers, it
is helpful to set {{mapreduce.admin.user.env}} to something like
{{PATH=./hadoop-tarball:\{\{PATH\}\}}} to make sure that all of the Hadoop
binaries are on the PATH. This way you can call {{hadoop}} instead of
{{./hadoop-tarball/hadoop}}. The intention here is to force prepend
{{./hadoop-tarball}} and then append the set {{PATH}} afterwards. But if a user
would like to override the appended portion of {{PATH}} in their environment,
they are unable to do so. This is because {{PATH}} ends up getting parsed
twice. Initially it is set via {{mapreduce.admin.user.env}} to
{{PATH=./hadoop-tarball:$SYS_PATH}}}. In this case {{SYS_PATH}} is what I'll
refer to as the normal system path. E.g. {{/usr/local/bin:/usr/bin}}, etc.
After this, the user env parsing happens. For example, let's say the user sets
their {{PATH}} to {{PATH=.:$PATH}}. We have already parsed {{PATH}} from the
admin.user.env. Then we go to parse the user environment and find the user also
specified {{PATH}}. So {{$PATH}} ends up getting getting expanded to
{{./hadoop-tarball:$SYS_PATH}}, which leads to the user's {{PATH}} being
{{PATH=.:./hadoop-tarball:$SYS_PATH}}. We then append this to {{PATH}}, which
has already been set in the environment map via the admin.user.env. So we
finally end up with
{{PATH=./hadoop-tarball:$SYS_PATH:.:./hadoop-tarball:$SYS_PATH}}
This normally isn't a huge deal, but if you want to override the version of
python, perl, etc. that is on the host or in the container, you need to prepend
it to your {{PATH}} or you will need to refer to it by its full path. This is a
pain and it should be possible to prepend to {{PATH}} to override the
system/container {{PATH}} even if you have admin.user.env set to prepend a
{{PATH}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
