Hi,
I'm having trouble reading Bzip2 compressed sequence files after I
enabled hadoop native libraries in spark.
Running
LD_LIBRARY_PATH=$HADOOP_HOME/lib/native/ $SPARK_HOME/bin/spark-submit
--class .... gives the following error
5/06/26 00:48:02 INFO CodecPool: Got brand-new decompressor [.bz2]
15/06/26 00:48:02 ERROR Executor: Exception in task 3.0 in stage 0.0 (TID 3)
java.lang.UnsupportedOperationException
at
org.apache.hadoop.io.compress.bzip2.BZip2DummyDecompressor.decompress(BZip2DummyDecompressor.java:32)
at
org.apache.hadoop.io.compress.DecompressorStream.decompress(DecompressorStream.java:91)
at
org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:85)
at java.io.DataInputStream.readFully(DataInputStream.java:195)
at java.io.DataInputStream.readLong(DataInputStream.java:416)
removing the LD_LIBRARY_PATH makes spark run fine but it gives the
following warning
WARN NativeCodeLoader: Unable to load native-hadoop library for your
platform... using builtin-java classes where applicable
Has anyone else run into this issue ? Any help is welcome.
