[
https://issues.apache.org/jira/browse/HADOOP-19001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17792541#comment-17792541
]
Zilong Zhu edited comment on HADOOP-19001 at 12/3/23 3:30 PM:
--------------------------------------------------------------
About hadoop-functions.sh, the Hadoop3 script has been rewritten by
HADOOP-9902. I can't get any more information. I think it makes no sense to
export LD_LIBRARY_PATH directly.
After
https://github.com/apache/hadoop/commit/9d4d30243b0fc9630da51a2c17b543ef671d035c
env not in yarn.nodemanager.env-whitelist(include LD_LIBRARY_PATH) is not able
to get from builder.environment() since
DefaultContainerExecutor#buildCommandExecutor sets inherit to false.
So I'm confused as to whether this is an issue. But it does make it impossible
for spark and flink tasks to directly load native hadoop lib on yarn.
was (Author: JIRAUSER287487):
About hadoop-functions.sh, the Hadoop3 script has been rewritten by
HADOOP-9902. I can't get any more information. I think it makes no sense to
export LD_LIBRARY_PATH directly.
After
https://github.com/apache/hadoop/commit/9d4d30243b0fc9630da51a2c17b543ef671d035c
env not in yarn.nodemanager.env-whitelist(include LD_LIBRARY_PATH) is not able
to get from builder.environment() since
DefaultContainerExecutor#buildCommandExecutor sets inherit to false.
So I'm confused as to whether this is an issue.
> LD_LIBRARY_PATH is missing HADOOP_COMMON_LIB_NATIVE_DIR
> -------------------------------------------------------
>
> Key: HADOOP-19001
> URL: https://issues.apache.org/jira/browse/HADOOP-19001
> Project: Hadoop Common
> Issue Type: Bug
> Affects Versions: 3.2.4
> Reporter: Zilong Zhu
> Priority: Major
>
> When we run a spark job, we find that it cannot load the native library
> successfully.
> We found a difference between hadoop2 and hadoop3.
> hadoop2-Spark-System Properties：
> |java.library.path|:/hadoop/lib/native:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib|
> hadoop3-Spark-System Properties：
> |java.library.path|:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib|
> The key point is：
> hadoop2-hadoop-config.sh：
> HADOOP_OPTS="$HADOOP_OPTS -Djava.library.path=$JAVA_LIBRARY_PATH" <--267
> export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$JAVA_LIBRARY_PATH <--268
>
> hadoop3-hadoop-functions.sh:
> hadoop_add_param HADOOP_OPTS java.library.path \
> "-Djava.library.path=${JAVA_LIBRARY_PATH}"
> export LD_LIBRARY_PATH <--1484
>
> At the same time, the hadoop3 will clear all non-whitelisted environment
> variables.
> I'm not sure if it was intentional. But it makes our spark job unable to find
> the native library on hadoop3.
> Maybe we should modify hadoop-functions.sh(1484) and add LD_LIBRARY_PATH to
> the default configuration item yarn.nodemanager.env-whitelist.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
