See: https://issues.apache.org/jira/browse/HADOOP-9911
Can you recompile snappy for 64-bit system ? Cheers On Wed, Dec 10, 2014 at 9:22 AM, peterm_second <[email protected]> wrote: > Hi Fabio , > Thanks for the reply, but unfortunately it didn't work. I am using vanilla > hadoop 2.4 with vanilla hive 0.14 and so on, I am using the vanilla distros. > I did set the HADOOP_COMMON_LIB_NATIVE_DIR but that didn't make any > change. What version were you using ? > > Peter > > > > On 10.12.2014 16:23, Fabio wrote: > >> Not sure it will help, but if the problem is native library loading, I >> spent a loooong time trying anything to make it work. >> I may suggest to try also: >> export JAVA_LIBRARY_PATH=/opt/yarn/hadoop-2.5.0/lib/native >> export HADOOP_COMMON_LIB_NATIVE_DIR=/opt/yarn/hadoop-2.5.0/lib >> I have this both in the bash "init" script (/etc/profile.p/...) and in >> /opt/yarn/hadoop-2.5.0/etc/hadoop/hadoop-env.sh; quite sure it's >> redundant, but as long as it works I don't change it. >> I see here I commented out my attempts to set HADOOP_OPTS, so maybe it's >> not necessary. >> I don't see anything in my .xml config files. >> Also, someone says to compile the libraries under your 64 bit system, >> since the ones in Hadoop are for a 32bit architecture. >> >> Good luck >> >> Fabio >> >> On 12/10/2014 02:57 PM, peterm_second wrote: >> >>> Hi guys, >>> I have a hadoop + hbase + hive application, >>> For some reason my cluster is unable to find the snappy native library >>> Here is the exception : >>> org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy()Z >>> at org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy(Native >>> Method) >>> at >>> org.apache.hadoop.io.compress.SnappyCodec.checkNativeCodeLoaded(SnappyCodec.java:63) >>> >>> at >>> org.apache.hadoop.io.compress.SnappyCodec.getCompressorType(SnappyCodec.java:132) >>> >>> at org.apache.hadoop.io.compress.CodecPool.getCompressor( >>> CodecPool.java:148) >>> at org.apache.hadoop.io.compress.CodecPool.getCompressor( >>> CodecPool.java:163) >>> at org.apache.hadoop.mapred.IFile$Writer.<init>(IFile.java:115) >>> at org.apache.hadoop.mapred.MapTask$MapOutputBuffer. >>> sortAndSpill(MapTask.java:1583) >>> at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush( >>> MapTask.java:1462) >>> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:437) >>> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342) >>> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167) >>> at java.security.AccessController.doPrivileged(Native Method) >>> at javax.security.auth.Subject.doAs(Subject.java:422) >>> at org.apache.hadoop.security.UserGroupInformation.doAs( >>> UserGroupInformation.java:1548) >>> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162) >>> >>> >>> I am working with a 64bit ubuntu 14.04LTS. I've installed snappy on my >>> os and added the coppied the libs to hadoop_home/lib/native >>> I've also added the libs to the JRE, but it still fails as if nothing is >>> present. >>> I've added HADOOP_OPTS="-Djava.net.preferIPv4Stack=true $GC_DEBUG_OPTS >>> -Djava.library.path=/usr/local/hadoop-2.4.0/lib/native $HADOOP_OPTS" >>> in my yarn xml I have >>> <property> >>> <name>yarn.app.mapreduce.am.env</name> >>> <value>LD_LIBRARY_PATH=$HADOOP_HOME/lib/native</value> >>> </property> >>> >>> in my mapred-site.xml i have >>> <property> >>> <name>mapred.child.java.opts</name> >>> <value> -Djava.library.path=/usr/local/hadoop-2.4.0/lib/native< >>> /value> >>> </property> >>> <property> >>> <name>mapreduce.reduce.java.opts</name> >>> <value>-Djava.library.path=/usr/local/hadoop-2.4.0/lib/native</value> >>> </property> >>> >>> The last two were a desperation move. >>> The result is always the same. Any ideas would be welcomed. >>> >>> Thanks, >>> Peter >>> >>> >>> >>> >>> >>> >>> >>> >> >
