NING DING created MAPREDUCE-6590:
------------------------------------
Summary: Update MRJobConfig.DEFAULT_MAPRED_ADMIN_USER_ENV value
for loading correct navite lib when using ubertask feature
Key: MAPREDUCE-6590
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6590
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mr-am
Affects Versions: 2.7.1
Reporter: NING DING
Assignee: NING DING
By default, the MR AM unable to load native library without
MR_AM_ADMIN_USER_ENV set.
If yarn.app.mapreduce.am.admin.user.env (or yarn.app.mapreduce.am.env) is not
configured to set LD_LIBRARY_PATH, MR AM will fail to load the native library,
then you can find the error message as below.
{panel}
org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop library
for your platform... using builtin-java classes where applicable.
{panel}
I found the patch in https://issues.apache.org/jira/browse/MAPREDUCE-5799
update YARNRunner code and use MRJobConfig.DEFAULT_MAPRED_ADMIN_USER_ENV as the
default value. The code is as below.
{code:title=YARNRunner.java|borderStyle=solid}
MRApps.setEnvFromInputString(environment,
conf.get(MRJobConfig.MR_AM_ADMIN_USER_ENV,
MRJobConfig.DEFAULT_MAPRED_ADMIN_USER_ENV), conf);
{code}
In fact the parameters yarn.app.mapreduce.am.env and
yarn.app.mapreduce.am.admin.user.env are resolved in YARNRunner running on
hadoop client host.
But their values, e.g. LD_LIBRARY_PATH=$HADOOP_COMMON_HOME/lib/native, will be
used on hadoop cluster hosts.
If the hadoop client host installs hadoop on a different path from that on
hadoop cluster hosts,
the $HADOOP_COMMON_HOME/lib/native will be resolved to a path which is not
existed on hadoop cluster hosts.
Then MRAppMaster running on hadoop cluster host cannot load native lib.
If don't use MAPREDUCE-5799 patch, I set my hadoop client's mapred-site.xml
with the following content.
{code:xml}
<property>
<name>yarn.app.mapreduce.am.admin.user.env</name>
<value>LD_LIBRARY_PATH={{HADOOP_COMMON_HOME}}/lib/native</value>
</property>
{code}
In this way, the YARNRunner will put
LD_LIBRARY_PATH={{HADOOP_COMMON_HOME}}/lib/native into environment.
The method ContainerLaunch.expandEnvironment running in NodeManager can
translate LD_LIBRARY_PATH={{HADOOP_COMMON_HOME}}/lib/native
to LD_LIBRARY_PATH=$HADOOP_COMMON_HOME/lib/native.
The host running NodeManager can find its $HADOOP_COMMON_HOME/lib/native path.
I suggest MRJobConfig.DEFAULT_MAPRED_ADMIN_USER_ENV should be defined as below.
{code:title=MRJobConfig.java|borderStyle=solid}
public final String DEFAULT_MAPRED_ADMIN_USER_ENV =
Shell.WINDOWS ?
"PATH={{PATH}};{{HADOOP_COMMON_HOME}}\\bin":
"LD_LIBRARY_PATH={{HADOOP_COMMON_HOME}}/lib/native";
{code}
Please see my patch.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
