[
https://issues.apache.org/jira/browse/MAPREDUCE-7152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17903041#comment-17903041
]
wuchang commented on MAPREDUCE-7152:
------------------------------------
[~pbacsko] May I know more details for this? I exactly meet the same issue on
my Hadoop Yarn cluster in which different NodeManagers have different running
location. I configure my mapred-site.xml as below:
{code:java}
<property>
<name>mapreduce.reduce.env</name>
<value>LD_LIBRARY_PATH=$HADOOP_COMMON_HOME/lib/native:$JAVA_LIBRARY_PATH:/usr/lib/x86_64-linux-gnu</value>
</property>
<property>
<name>mapreduce.map.env</name>
<value>LD_LIBRARY_PATH=$HADOOP_COMMON_HOME/lib/native:$JAVA_LIBRARY_PATH:/usr/lib/x86_64-linux-gnu</value>
</property>{code}
And all the mappers which run on the same host with ApplicationMaster
succeeded, all the mappers which run on the different hosts with
ApplicationMaster failed for Snappy not found issue.
I find that seems that all mappers's LD_LIBRARY_PATH is using the expanded
value of $HADOOP_COMMON_HOME on ApplicationMaster, then mappers running on
different hosts with ApplicationMaster will always fail.
> LD_LIBRARY_PATH is always passed from MR AM to tasks
> ----------------------------------------------------
>
> Key: MAPREDUCE-7152
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7152
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Peter Bacsko
> Assignee: Peter Bacsko
> Priority: Major
> Attachments: MAPREDUCE-7152-NMAdminEnvPOC_POC01.patch,
> MAPREDUCE-7152-lazyEval_POC01.patch
>
>
> {{LD_LIBRARY_PATH}} is set to {{$HADOOP_COMMON_HOME/lib/native}} by default
> in Hadoop (as part of {{mapreduce.admin.user.env}} and
> {{yarn.app.mapreduce.am.user.env}}), and passed as an environment variable
> from AM container to task containers in the container launch context.
> In cases where {{HADOOP_COMMON_HOME}} is different in AM node and task node,
> tasks will fail to load native library. A reliable way to fix this is to add
> {{LD_LIBRARY_PATH}} in {{yarn.nodemanager.admin-env}} instead.
> Another approach is to perform a lazy evaluation of {{LD_LIBRARY_PATH}} on
> the NM side.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
