Re: java.io.IOException: Compression algorithm 'snappy' previously failed test

Mon, 23 Apr 2012 08:24:22 -0700 

Hi Nathaniel,
My memory's a little hazy (we ended going back to CDH3), but I believe the key to fixing this problem for me was the following log message: 


WARN util.NativeCodeLoader: Unable to load native-hadoop library for
your platform... using builtin-java classes where applicable



I had set HBASE_LIBRARY_PATH to hadoop's library path since that was where  
snappy was. Once I set the path ( or linked created symbolic links ) so  
that HBASE_LIBRARY_PATH included both HBase's and Hadoop's the snappy  
issue was resolved.



Also, in general, I ended up having a lot better luck with the RPMs  
instead of the tarballs.


~ Peter


On Mon, 23 Apr 2012 11:01:19 -0400, Nathaniel Cook <[email protected]>  
wrote:



Was there any resolution to this? I am experiencing the same issue.

Nathaniel


On Wed, Feb 29, 2012 at 10:52 AM, Peter Naudus <[email protected]>  
wrote:

Thanks for your help :)

To make sure I manually set LD_LIBRARY_PATH, LIBRARY_PATH, and
HBASE_LIBRARY_PATH

bash-3.2$ export
LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/opt/dataraker/software/cdh4/hadoop-0.23.0-cdh4b1/lib/native
bash-3.2$ export
LIBRARY_PATH=$LIBRARY_PATH:/opt/dataraker/software/cdh4/hadoop-0.23.0-cdh4b1/lib/native
bash-3.2$ export
HBASE_LIBRARY_PATH=/opt/dataraker/software/cdh4/hadoop-0.23.0-cdh4b1/lib/native

But running the compression test failed with "native snappy library not
available"

bash-3.2$ ./hbase org.apache.hadoop.hbase.util.CompressionTest
file:///tmp/test.txt snappy
log4j:WARN No appenders could be found for logger
(org.apache.hadoop.conf.Configuration).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for
more info.

Exception in thread "main" java.lang.RuntimeException: native snappy  
library

not available
       at
org.apache.hadoop.io.compress.SnappyCodec.getCompressorType(SnappyCodec.java:121)
       at
org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:104)
       at
org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:118)
       at
org.apache.hadoop.hbase.io.hfile.Compression$Algorithm.getCompressor(Compression.java:236)
       at
org.apache.hadoop.hbase.io.hfile.HFileBlock$Writer.<init>(HFileBlock.java:588)
       at
org.apache.hadoop.hbase.io.hfile.HFileWriterV2.finishInit(HFileWriterV2.java:178)
       at
org.apache.hadoop.hbase.io.hfile.HFileWriterV2.<init>(HFileWriterV2.java:150)
       at
org.apache.hadoop.hbase.io.hfile.HFileWriterV2.<init>(HFileWriterV2.java:140)
       at
org.apache.hadoop.hbase.io.hfile.HFileWriterV2$WriterFactoryV2.createWriter(HFileWriterV2.java:104)
       at
org.apache.hadoop.hbase.util.CompressionTest.doSmokeTest(CompressionTest.java:108)
       at
org.apache.hadoop.hbase.util.CompressionTest.main(CompressionTest.java:137)

I verified that libsnappy is indeed, installed

bash-3.2$ ls -al $HBASE_LIBRARY_PATH
total 1412
drwxr-xr-x 2 1106 592   4096 Feb 11 01:06 .
drwxr-xr-x 3 1106 592   4096 Feb 11 01:06 ..
-rw-r--r-- 1 1106 592 616862 Feb 11 01:06 libhadoop.a
-rwxr-xr-x 1 1106 592   1051 Feb 11 01:06 libhadoop.la

lrwxrwxrwx 1 1106 592     18 Feb 27 18:12 libhadoop.so ->  
libhadoop.so.1.0.0

lrwxrwxrwx 1 1106 592     18 Feb 27 18:12 libhadoop.so.1 ->
libhadoop.so.1.0.0
-rwxr-xr-x 1 1106 592 340361 Feb 11 01:06 libhadoop.so.1.0.0
-rw-r--r-- 1 1106 592 184418 Feb 11 01:06 libhdfs.a
-rwxr-xr-x 1 1106 592   1034 Feb 11 01:06 libhdfs.la
lrwxrwxrwx 1 1106 592     16 Feb 27 18:12 libhdfs.so -> libhdfs.so.0.0.0

lrwxrwxrwx 1 1106 592     16 Feb 27 18:12 libhdfs.so.0 ->  
libhdfs.so.0.0.0

-rwxr-xr-x 1 1106 592 125455 Feb 11 01:06 libhdfs.so.0.0.0
-rw-r--r-- 1 1106 592  37392 Feb 11 01:06 libsnappy.a

lrwxrwxrwx 1 1106 592     18 Feb 27 18:12 libsnappy.so ->  
libsnappy.so.1.1.1

lrwxrwxrwx 1 1106 592     18 Feb 27 18:12 libsnappy.so.1 ->
libsnappy.so.1.1.1
-rw-r--r-- 1 1106 592  26824 Feb 11 01:06 libsnappy.so.1.1.1

Just for grins and giggles I re-ran this as root


In addition to the Exception mentioned above, I also got following  
Warning:
       WARN util.NativeCodeLoader: Unable to load native-hadoop library  
for

your platform... using builtin-java classes where applicable

Any ideas?


On Tue, 28 Feb 2012 20:02:38 -0500, Stack <[email protected]> wrote:


On Tue, Feb 28, 2012 at 1:52 PM, Peter Naudus <[email protected]>
wrote:


What else can I do to fix / diagnose this problem?



Does our little compression tool help?
http://hbase.apache.org/book.html#compression.test

St.Ack




--






















					Reply via email to