Looks like you are having different versions of snappy library. Here's a similar discussion if you haven't seen it already http://stackoverflow.com/questions/22150417/hadoop-mapreduce-java-lang-unsatisfiedlinkerror-org-apache-hadoop-util-nativec
Thanks Best Regards On Mon, Sep 7, 2015 at 7:41 AM, dong.yajun <dongt...@gmail.com> wrote: > hi all, > > I met problem that can't read the file with snappy encoding from HDFS in > Spark1.4.1, > > I have configured the SPARK_LIBRARY_PATH property in conf/spark-env.sh to > the native path of Hadoop and restarted the spark cluster > > SPARK_LIBRARY_PATH=$SPARK_LIBRARY_PATH:/opt/app/install/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop/lib/native > > > the partial exception: > > Caused by: org.apache.hadoop.hbase.io.hfile.CorruptHFileException: Problem > reading HFile Trailer from file > hdfs://nameservice1/hbase/data/default/IM_ItemBase/02296539242087aea77877dced9ba3d5/BaseInfo/9fe36f74334c4d30ba1bfc17bbd717f5 > > at > org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:478) > > at > org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:521) > > at > com.newegg.ec.bigdata.dump.CombineHFileRecordReader.<init>(CombineHFileRecordReader.java:33) > > ... 19 more > > Caused by: java.lang.UnsatisfiedLinkError: > org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy()Z > > at > org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy(Native Method) > > at > org.apache.hadoop.io.compress.SnappyCodec.checkNativeCodeLoaded(SnappyCodec.java:63) > > at > org.apache.hadoop.io.compress.SnappyCodec.getDecompressorType(SnappyCodec.java:192) > > at > org.apache.hadoop.io.compress.CodecPool.getDecompressor(CodecPool.java:176) > > at > org.apache.hadoop.hbase.io.compress.Compression$Algorithm.getDecompressor(Compression.java:328) > > at > org.apache.hadoop.hbase.io.compress.Compression.decompress(Compression.java:423) > > at > org.apache.hadoop.hbase.io.encoding.HFileBlockDefaultDecodingContext.prepareDecoding(HFileBlockDefaultDecodingContext.java:90) > > -- > *Ric Dong* > >
