[
https://issues.apache.org/jira/browse/YARN-729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13670712#comment-13670712
]
Chuan Liu commented on YARN-729:
--------------------------------
Hi [~mostafae], how did you run into this exception?
I agree mapreduce.admin.user.env default value should be OS dependent.
But I think this is not a fix to the exception you saw.
LD_LIBRARY_PATH and PATH are used for OS functions to dynamically load
libraries, i.e. they are for dlopen() and LoadLibrary() on Linux and Windows
respectively.
On the other hand, the JNI libraries or DLLs load path is configured via the
Java property -Djava.library.path, which is used by Java API System.loadLibrary
(not to be confused with Windows LoadLibrary()).
I think the exception you saw was caused by the second API failure, i.e. native
library is not loaded, instead of a foreign library missing on PATH.
To confirm, you could check if you have following warnings in your log entries
prior to the exception.
{noformat}
Unable to load native-hadoop library for your platform...
{noformat}
> MR uses LD_LIBRARY_PATH which doesn't mean anything in Windows
> --------------------------------------------------------------
>
> Key: YARN-729
> URL: https://issues.apache.org/jira/browse/YARN-729
> Project: Hadoop YARN
> Issue Type: Bug
> Affects Versions: 3.0.0
> Environment: Windows
> Reporter: Mostafa Elhemali
>
> In order to set the path for loading native libraries, MR relies on the
> default value of the mapreduce.admin.user.env configuration setting the
> LD_LIBRARY_PATH environment entry. There are two problems with this setting
> in Windows:
> a) LD_LIBRARY_PATH doesn't mean anything in Windows.
> b) It sets it using $HADOOP_COMMON_HOME, instead of %HADOOP_COMMON_HOME%.
> The default value here should be platform-dependent (use the PATH variable in
> Windows instead of LD_LIBRARY_PATH), or we should rely on another mechanism.
> The net effect is that in Windows unless this configuration is over-ridden MR
> jobs fail with this error:
> {code}
> 2013-05-29 13:51:41,049 FATAL [main] org.apache.hadoop.mapred.YarnChild:
> Error running child : java.lang.UnsatisfiedLinkError:
> org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Ljava/lang/String;I)Z
> at org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Native Method)
> at
> org.apache.hadoop.io.nativeio.NativeIO$Windows.access(NativeIO.java:393)
> at org.apache.hadoop.fs.FileUtil.canRead(FileUtil.java:928)
> at
> org.apache.hadoop.util.DiskChecker.checkAccessByFileMethods(DiskChecker.java:177)
> at
> org.apache.hadoop.util.DiskChecker.checkDirAccess(DiskChecker.java:164)
> at org.apache.hadoop.util.DiskChecker.checkDir(DiskChecker.java:98)
> at
> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.confChanged(LocalDirAllocator.java:288)
> at
> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathToRead(LocalDirAllocator.java:431)
> at
> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathToRead(LocalDirAllocator.java:164)
> at
> org.apache.hadoop.mapred.YarnChild.configureLocalDirs(YarnChild.java:235)
> at org.apache.hadoop.mapred.YarnChild.configureTask(YarnChild.java:294)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:143)
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
