[
https://issues.apache.org/jira/browse/YARN-4789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15191345#comment-15191345
]
Sangjin Lee commented on YARN-4789:
-----------------------------------
Thanks for filing the issue and the patch [~jira.shegalov]! This definitely is
a bug.
I also find the descriptions of the admin env properties very confusing:
{noformat}
<property>
<name>mapreduce.admin.user.env</name>
<value></value>
<description>
Expert: Additional execution environment entries for
map and reduce task processes. This is not an additive property.
You must preserve the original value if you want your map and
reduce tasks to have access to native libraries (compression, etc).
When this value is empty, the command to set execution
envrionment will be OS dependent:
For linux, use LD_LIBRARY_PATH=$HADOOP_COMMON_HOME/lib/native.
For windows, use PATH = %PATH%;%HADOOP_COMMON_HOME%\\bin.
</description>
</property>
<property>
<name>yarn.app.mapreduce.am.admin.user.env</name>
<value></value>
<description> Environment variables for the MR App Master
processes for admin purposes. These values are set first and can be
overridden by the user env (yarn.app.mapreduce.am.env) Example :
1) A=foo This will set the env variable A to foo
2) B=$B:c This is inherit app master's B env variable.
</description>
</property>
{noformat}
The statement about it being non-additive is confusing, as for PATH-like
variables they are very much additive. Also,
{{yarn.app.mapreduce.am.admin.user.env}} says
bq. These values are set first and can be overridden by the user env
(yarn.app.mapreduce.am.env)
So it seems to me that we should revisit these descriptions and improve them,
especially if we are going to reject user values for non-PATH environment
variables.
I only took a quick look at the patch, but it appears the exception will be
thrown every time {{Apps.addToEnvironment()}} is called for a non-PATH
environment variable? That would cause exceptions for more types of situations
than we're targeting, right? For example, I suspect even the attempt to set
{{yarn.app.mapreduce.am.admin.user.env}} would fail:
{code}
// Setup the environment variables for Admin first
MRApps.setEnvFromInputString(environment,
conf.get(MRJobConfig.MR_AM_ADMIN_USER_ENV,
MRJobConfig.DEFAULT_MR_AM_ADMIN_USER_ENV), conf);
{code}
I think it'd be better to target this change more narrowly.
> Provide helpful exception for non-PATH-like conflict with admin.user.env
> ------------------------------------------------------------------------
>
> Key: YARN-4789
> URL: https://issues.apache.org/jira/browse/YARN-4789
> Project: Hadoop YARN
> Issue Type: Bug
> Affects Versions: 2.7.2
> Reporter: Gera Shegalov
> Assignee: Gera Shegalov
> Attachments: YARN-4789.001.patch
>
>
> Environment variables specified in mapreduce.admin.user.env are supposed to
> be paths (class, shell, library) and they can be merged with the
> user-provided values. However, it's also possible that the cluster admins
> specify some non-PATH-like variable such as JAVA_HOME. In this case if there
> is the same variable provided by the user, we'll get a concatenation that
> does not make sense and is difficult to debug.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
