[
https://issues.apache.org/jira/browse/YARN-5877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15727893#comment-15727893
]
Sunil G commented on YARN-5877:
-------------------------------
We have checked various cases for this issue in detail and had some offline
discussions with [~vvasudev] and [~bibinchundatt] also.
Now coming to the issue mentioned. MAPREDUCE-6704 has come in to a common
consensus that HADOOP_MAPRED_HOME could be published with nm-white-list so that
even if this variable is not present in env, we can take the configured value.
This will solve the container launch problem mentioned there.
As of today, if we publish HADOOP_MAPRED_HOME through nm-whitelist-env, it will
look like below in launch_container.sh.
{{export HADOOP_MAPRED_HOME="/home/hadoopbuild"}}
As mentioned above, this will be fine in case of normal container launch. But
in docker scenario, the path value given may not be valid. In such case, we
will see the failure as mentioned in this jira. Ideally if this environment
variable were published as whitelist env, then it could try to take from env
itself first.
{code}
export HADOOP_MAPRED_HOME=${HADOOP_MAPRED_HOME:-"/home/hadoopbuild"}
{code}
Here container will try to get the value from *HADOOP_MAPRED_HOME* if its
available in env first. If not, it will take value which is given as
"/home/hadoopbuild". In case of docker, this value was set in its system env
itself (i think Bibin used sequence iq docker image here which had this value).
So in conclusion, all variables mentioned in {{yarn.nodemanager.env-whitelist}}
could be whitelisted. This can solve the problem. If some docker images doesn't
have this env variable set in its env, and if the default path to
HADOOP_MAPRED_HOME is also invalid, then user has to set correct mapred home
path in {{yarn.app.mapreduce.am.env}}.
I think YARN-5877.0002.patch is good one to go. [~bibinchundatt] could you
rebase the patch and give a latest version number to compile against trunk. If
possible, could you please share test results with this latest patch too in
docker.
> Allow all nm-whitelist-env to get overridden during launch
> ----------------------------------------------------------
>
> Key: YARN-5877
> URL: https://issues.apache.org/jira/browse/YARN-5877
> Project: Hadoop YARN
> Issue Type: Bug
> Reporter: Bibin A Chundatt
> Assignee: Bibin A Chundatt
> Attachments: Dockerfile, YARN-5877.0001.patch, YARN-5877.0002.patch,
> YARN-5877.0003.patch, bootstrap.sh, yarn-site.xml
>
>
> As per the {{yarn.nodemanager.env-whitelist}} for the configured values
> should containers may override rather than use NodeManager's default.
> {code}
> <property>
> <description>Environment variables that containers may override rather
> than use NodeManager's default.</description>
> <name>yarn.nodemanager.env-whitelist</name>
>
> <value>JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME</value>
> </property>
> {code}
> But only the following containers can override
> {code}
> whitelist.add(ApplicationConstants.Environment.HADOOP_YARN_HOME.name());
> whitelist.add(ApplicationConstants.Environment.HADOOP_COMMON_HOME.name());
> whitelist.add(ApplicationConstants.Environment.HADOOP_HDFS_HOME.name());
> whitelist.add(ApplicationConstants.Environment.HADOOP_CONF_DIR.name());
> whitelist.add(ApplicationConstants.Environment.JAVA_HOME.name());
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
