[
https://issues.apache.org/jira/browse/MAPREDUCE-4072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13239882#comment-13239882
]
Kihwal Lee commented on MAPREDUCE-4072:
---------------------------------------
It look like MAPREDUCE-2880 got rid of merging of java.library.path and
introduced setEnvFromInputString(). However this does not take care of the
cases for java.library.path. When launching a container, "-Djava.library.path="
should contain both the system jni lib path (i.e. the location of libhadoop.so)
and whatever user specified.
Please note that setting LD_LIBRARY_PATH alone is not enough to load JNI
libraries. A custom LD_LIBRARY_PATH needs to be used when the user supplies
custom native libraries, which the JNI library depends on, but the base system
does not provide.
setEnvFromInputString() does path munging on LD_LIBRARY_PATH.
> User set java.library.path seems to overwrite default creating problems
> native lib loading
> ------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-4072
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4072
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Reporter: Anupam Seth
> Assignee: Anupam Seth
>
> This was found by Peeyush Bishnoi.
> While running a distributed cache example with Hadoop-0.23,
> tasks are failing as follows:
> ------------------------------------------------------------------------------------------------------------
> Exception from container-launch:
> org.apache.hadoop.util.Shell$ExitCodeException: at
> org.apache.hadoop.util.Shell.runCommand(Shell.java:261) at
> org.apache.hadoop.util.Shell.run(Shell.java:188) at
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:381) at
> org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.launchContainer(LinuxContainerExecutor.java:207)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:241)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:68)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at
> java.util.concurrent.FutureTask.run(FutureTask.java:138) at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619) main : command provided 1 main : user
> is <user>
> ------------------------------------------------------------------------------------------------------------
> Same Pig script and command work successfully on 0.20
> See this in the stderr:
> Exception in thread "main" java.lang.ExceptionInInitializerError
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:247)
> at
> org.apache.hadoop.conf.Configuration.getClassByNameOrNull(Configuration.java:1179)
> at
> org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1149)
> at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1238)
> at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1264)
> at org.apache.hadoop.security.Groups.(Groups.java:54)
> at
> org.apache.hadoop.security.Groups.getUserToGroupsMappingService(Groups.java:178)
> at
> org.apache.hadoop.security.UserGroupInformation.initUGI(UserGroupInformation.java:252)
> at
> org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:223)
> at
> org.apache.hadoop.security.UserGroupInformation.setConfiguration(UserGroupInformation.java:265)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:75)
> Caused by: java.lang.RuntimeException: Bailing out since native library
> couldn't be loaded
> at
> org.apache.hadoop.security.JniBasedUnixGroupsMapping.(JniBasedUnixGroupsMapping.java:48)
> ... 12 more
> Pig command:
> $ pig -Dmapred.job.queue.name=<queue> -Dmapred.cache.archives=<archives>
> -Dmapred.child.java.opts="-Djava.library.path=./ygeo/lib
> -Dip2geo.preLoadLibraries=<some other libs>" -Djava.io.tmpdir=/grid/0/tmp
> -Dmapred.create.symlink=yes -Dmapred.job.map.memory.mb=3072 piggeoscript.pig
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
