[ https://issues.apache.org/jira/browse/HADOOP-7825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Matt Foley updated HADOOP-7825: ------------------------------- Target Version/s: (was: 1.2.0) > Hadoop wrapper script not picking up native libs correctly > ---------------------------------------------------------- > > Key: HADOOP-7825 > URL: https://issues.apache.org/jira/browse/HADOOP-7825 > Project: Hadoop Common > Issue Type: Bug > Components: scripts > Affects Versions: 0.20.205.0 > Environment: Debian 6.0 x64_64 > java version "1.6.0_26" > Java(TM) SE Runtime Environment (build 1.6.0_26-b03) > Java HotSpot(TM) 64-Bit Server VM (build 20.1-b02, mixed mode) > Reporter: stephen mulcahy > > Originally discussed in > https://mail-archives.apache.org/mod_mbox/hadoop-common-user/201111.mbox/%3C4EC3A3AE.7060402%40deri.org%3E > I'm testing out native lib support on our test amd64 test cluster > running 0.20.205 running the following > ./bin/hadoop jar hadoop-test-0.20.205.0.jar testsequencefile -seed 0 > -count 1000 -compressType RECORD xxx -codec > org.apache.hadoop.io.compress.GzipCodec -check 2 > it fails with > WARN util.NativeCodeLoader: Unable to load native-hadoop library for > your platform... using builtin-java classes where applicable > Looking at > bin/hadoop > it seems to successfully detect that the native libs are available (they > seem to come pre-compiled with 0.20.205 which is nice) > if [ -d "${HADOOP_HOME}/lib/native" ]; then > if [ "x$JAVA_LIBRARY_PATH" != "x" ]; then > > JAVA_LIBRARY_PATH=${JAVA_LIBRARY_PATH}:${HADOOP_HOME}/lib/native/${JAVA_PLATFORM} > else > JAVA_LIBRARY_PATH=${HADOOP_HOME}/lib/native/${JAVA_PLATFORM} > fi > fi > and sets JAVA_LIBRARY_PATH to contain them. > Then in the following line, if ${HADOOP_HOME}/lib contains libhadoop.a > (which is seems to in the stock tar) then it proceeds to ignore the > native libs > if [ -e "${HADOOP_PREFIX}/lib/libhadoop.a" ]; then > JAVA_LIBRARY_PATH=${HADOOP_PREFIX}/lib > fi > The libhadoop.a in ${HADOOP_HOME}/lib seems to be a copy of the > lib/native/Linux-i386-32 going from the filesizes (and also noted by > https://mail-archives.apache.org/mod_mbox/hadoop-common-user/201111.mbox/%3ccaocnvr2azudnn0lfhmtqumayujytvhfkmmm_j0r-bmxw2wu...@mail.gmail.com%3E) > hadoop@testhbase01:~$ ls -la hadoop/lib/libhadoop.* > -rw-r--r-- 1 hadoop hadoop 237244 Oct 7 08:20 hadoop/lib/libhadoop.a > -rw-r--r-- 1 hadoop hadoop 877 Oct 7 08:20 hadoop/lib/libhadoop.la > -rw-r--r-- 1 hadoop hadoop 160438 Oct 7 08:20 hadoop/lib/libhadoop.so > -rw-r--r-- 1 hadoop hadoop 160438 Oct 7 08:19 hadoop/lib/libhadoop.so.1.0.0 > hadoop@testhbase01:~$ ls -la hadoop/lib/native/Linux-i386-32/ > total 728 > drwxr-xr-x 3 hadoop hadoop 4096 Nov 15 14:05 . > drwxr-xr-x 5 hadoop hadoop 4096 Oct 7 08:24 .. > -rw-r--r-- 1 hadoop hadoop 237244 Oct 7 08:20 libhadoop.a > -rw-r--r-- 1 hadoop hadoop 877 Oct 7 08:20 libhadoop.la > -rw-r--r-- 1 hadoop hadoop 160438 Oct 7 08:20 libhadoop.so > -rw-r--r-- 1 hadoop hadoop 160438 Oct 7 08:20 libhadoop.so.1 > -rw-r--r-- 1 hadoop hadoop 160438 Oct 7 08:20 libhadoop.so.1.0.0 > A possible solution includes removing libhadoop.a and friends from > ${HADOOP_HOME}/lib and possibly also modifying the hadoop wrapper to remove > if [ -e "${HADOOP_PREFIX}/lib/libhadoop.a" ]; then > JAVA_LIBRARY_PATH=${HADOOP_PREFIX}/lib > fi > unless there is some other reason for this to exist. > This was also noted in a comment to HADOOP-6453 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira