Hi JM,
I had experienced similar error when I was installing LZO compression to
RegionServers:
Below is from my record about installing LZO:
Issue:
java.lang.UnsatisfiedLinkError: no gplcompression in java.library.path
...
12/11/23 19:03:14 ERROR lzo.LzoCodec: Cannot load native-lzo without
native-hadoop
Exception in thread "main" java.lang.RuntimeException: native-lzo library not
available
Solution:
(compiled lzo from source), the do these EXTRA steps:
1) cp
{working_folder_of_lzo}/hadoop-lzo-master/build/native/Linux-amd64-64/lib/*
/usr/local/lib/
2) echo export HBASE_LIBRARY_PATH=/usr/local/lib/ >>
$HBASE_HOME/conf/hbase-env.sh
3) mkdir -p $HBASE_HOME/build
4) cp -r {working_folder_of_lzo}/hadoop-lzo-master/build/native
$HBASE_HOME/build/native
When looking at your email below, I saw your log also has "The error I'm
getting is java.lang.UnsatisfiedLinkError: no hadoopsnappy in
java.library.path."
I think you could try these steps:
1) cp {working_folder_your_snappy}/build/native/Linux-amd64-64/lib/*
/usr/local/lib/
2) echo export HBASE_LIBRARY_PATH=/usr/local/lib/ >>
$HBASE_HOME/conf/hbase-env.sh
3) mkdir -p $HBASE_HOME/build
4) cp -r {working_folder_your_snappy}/build/native $HBASE_HOME/build/native
Good luck.
Thanks
TC
On 3 Dec 2012, at 9:47 PM, Jean-Marc Spaggiari wrote:
> Ok....
>
> This: http://code.google.com/p/hadoop-snappy/issues/detail?id=2 helped
> me and my test program is now working. I'm able to load both
> libraries. Fine.
>
> But the CompressionTest is still not working.
>
> What is very strange is that:
> 12/12/03 08:44:24 WARN snappy.LoadSnappy: Snappy native library is available
> 12/12/03 08:44:24 WARN snappy.LoadSnappy: Snappy native library not loaded
>
> It's available, but not loaded.
>
> But from the code:
> static {
> try {
> System.loadLibrary("snappy");
> System.loadLibrary("hadoopsnappy");
> LOG.warn("Snappy native library is available");
> AVAILABLE = true;
> } catch (UnsatisfiedLinkError ex) {
> //NOP
> }
> LOADED = AVAILABLE;
> if (LOADED) {
> LOG.info("Snappy native library loaded");
> } else {
> LOG.warn("Snappy native library not loaded");
> }
> }
> If "Snappy native library is available" is displayed, that mean
> AVAILABLE = true... And if AVAILABLE = true, then LOADED is set to
> true and Snappy native library loaded must be displayed... But it's
> not... How is this possible?
>
> I have not expected Snappy installation to be such a challenge...
>
> I will continue to dig and summarize the steps when I will be done (If
> I'm able to finish...)
>
> JM
>
> 2012/12/3, Jean-Marc Spaggiari <[email protected]>:
>> Thanks all for your replies.
>>
>> So, to reply to all in one.
>>
>> I'm not using CD3. I'm using Hadoop 1.0.3 and HBase 0.94.2 directly
>> from the JARs.
>>
>> Here are all the places where I have put the lib:
>> /home/hadoop/hadoop-1.0.3/lib/native/Linux-amd64-64/libsnappy.so
>> /home/hadoop/hadoop-1.0.3/lib/native/Linux-amd64-64/libsnappy.so.1
>> /home/hadoop/hadoop-1.0.3/lib/native/Linux-amd64-64/libsnappy.so.1.1.3
>> /home/hadoop/hadoop-1.0.3/lib/native/Linux-i386-32/libsnappy.so
>> /home/hadoop/hadoop-1.0.3/lib/native/Linux-i386-32/libsnappy.so.1
>> /home/hadoop/hadoop-1.0.3/lib/native/Linux-i386-32/libsnappy.so.1.1.3
>> /home/hbase/hbase-0.94.2/lib/native/libsnappy.so
>> /home/hbase/hbase-0.94.2/lib/native/libsnappy.so.1
>> /home/hbase/hbase-0.94.2/lib/native/libsnappy.so.1.1.3
>> /home/hbase/hbase-0.94.2/lib/native/Linux-amd64-64/libsnappy.so
>> /home/hbase/hbase-0.94.2/lib/native/Linux-amd64-64/libsnappy.so.1
>> /home/hbase/hbase-0.94.2/lib/native/Linux-amd64-64/libsnappy.so.1.1.3
>> /home/hbase/hbase-0.94.2/lib/native/Linux-i386-32/libsnappy.so
>> /home/hbase/hbase-0.94.2/lib/native/Linux-i386-32/libsnappy.so.1
>> /home/hbase/hbase-0.94.2/lib/native/Linux-i386-32/libsnappy.so.1.1.3
>> /lib/x86_64-linux-gnu/libsnappy.so
>> /usr/lib/libsnappy.so
>> /usr/lib/libsnappy.so.1
>> /usr/lib/libsnappy.so.1.1.3
>> /usr/local/lib/libsnappy.so
>> /usr/local/lib/libsnappy.so.1
>> /usr/local/lib/libsnappy.so.1.1.3
>>
>> I tried to add this on my hbase-env.xml:
>> export
>> HBASE_LIBRARY_PATH=/home/hbase/hbase-0.94.2/lib/native/Linux-amd64-64
>>
>> Before I was trying with doing export on the command line directly
>> since it seems the hbase script is taking that into consideration too.
>>
>> I have not yet put the hbase.regionserver.codecs line since I still
>> need to use my cluster until I get snappy working. On the hbase/lib
>> directory I have snappy-java-1.0.3.2.jar.
>>
>>
>> Should snappy be installed within hbase? Or should it be in hadoop?
>> I'm not sure anymore.
>>
>> But it's still not working. So I tried the small code below:
>>
>> import java.util.StringTokenizer;
>>
>> public class Test
>> {
>> static {
>> try {
>> System.loadLibrary("snappy");
>> System.loadLibrary("hadoopsnappy");
>> System.out.println ("Snappy native library is available");
>> } catch (UnsatisfiedLinkError ex) {
>> ex.printStackTrace();
>> }
>> }
>>
>> public static void main (String [] args)
>> {
>> System.out.println ("Coucou");
>> String property = System.getProperty("java.library.path");
>> StringTokenizer parser = new StringTokenizer(property, ";");
>> while (parser.hasMoreTokens()) {
>> System.err.println(parser.nextToken());
>> }
>> }
>> }
>>
>>
>> This code is from org.apache.hadoop.io.compress.snappy.LoadSnappy.
>> The error I'm getting is java.lang.UnsatisfiedLinkError: no
>> hadoopsnappy in java.library.path.
>>
>> So the issue is not the snappy lib. It' there and working fine. The
>> issue is the hadoopsnappy lib which I don't have...
>>
>> I found it there: http://code.google.com/p/hadoop-snappy/
>>
>> So I have extracted it with svn checkout
>> http://hadoop-snappy.googlecode.com/svn/trunk/
>> hadoop-snappy-read-only, tried to built it with mvn package but it's
>> failing with something saying "cannot find -ljvm"
>>
>> So seems my challenge will be to build hadoop-snappy and not to
>> install snappy which is already there and working...
>>
>> JM
>>
>> 2012/12/3, surfer <[email protected]>:
>>> hope it helps. this is what I do on apache hadoop 1.0.x and hbase 0.92.y:
>>> in hbase-site.xml add:
>>>
>>> <property>
>>> <name>hbase.regionserver.codecs</name>
>>> <value>snappy</value>
>>> </property>
>>>
>>> copy that file into the hadoop conf directory.
>>>
>>> in hbase-env.sh:
>>> export HBASE_LIBRARY_PATH=/pathtoyourhadoop/lib/native/Linux-amd64-64
>>>
>>> ( In hbase-env.sh I set also HBASE_HOME, HBASE_CONF_DIR, HADOOP_HOME,
>>> HADOOP_CONF_DIR but I don't know if they contribute to make snappy
>>> working...)
>>>
>>> in /pathtoyourhadoop/lib/native/Linux-amd64-64 I have:
>>> libsnappy.a
>>> libsnappy.so
>>> libsnappy.so.1
>>> libsnappy.so.1.1.2
>>>
>>> good luck
>>> giovanni
>>>
>>>
>>>
>>>
>>>
>>>
>>> On 12/02/2012 02:25 PM, Jean-Marc Spaggiari wrote:
>>>> So. I spent few hours on that yesterday with no luck.
>>>>
>>>> Here is what I did:
>>>> - Install the google tar, untared, configured, maked and installed it.
>>>> - Copied the .so files all over my fs in the os lib dir,
>>>> HBase/lib/native and subdirs, Hadoop/lib/native and subdirs.
>>>> - Installed all debian packages with snappy in the name:
>>>> python-snappy, libsnappy-dev, libsnappy1, libsnappy-java
>>>>
>>>> But still exactly the same issue as above. And I don't have any clue
>>>> where to dig. There is nothing on internet about that.
>>>>
>>>> Anyone faced that already while installing Snappy?
>>>>
>>>> JM
>>>>
>>>> 2012/12/1, Jean-Marc Spaggiari <[email protected]>:
>>>>> Sorry, I forgot to paste few maybe useful lines. I have the lib in
>>>>> /usr/local/lib copied properly, and I have the HBASE_LIBRARY_PATH set
>>>>> correctly. Do I need to restart HBase to run this test?
>>>>>
>>>>> hbase@node3:~/hbase-0.94.2$ export HBASE_LIBRARY_PATH=/usr/local/lib/
>>>>> hbase@node3:~/hbase-0.94.2$ bin/hbase
>>>>> org.apache.hadoop.hbase.util.CompressionTest /tmp/test.txt snappy
>>>>> 12/12/01 18:55:29 INFO util.ChecksumType:
>>>>> org.apache.hadoop.util.PureJavaCrc32 not available.
>>>>> 12/12/01 18:55:29 INFO util.ChecksumType: Checksum can use
>>>>> java.util.zip.CRC32
>>>>> 12/12/01 18:55:29 INFO util.ChecksumType:
>>>>> org.apache.hadoop.util.PureJavaCrc32C not available.
>>>>> 12/12/01 18:55:29 DEBUG util.FSUtils: Creating file:/tmp/test.txtwith
>>>>> permission:rwxrwxrwx
>>>>> 12/12/01 18:55:29 WARN util.NativeCodeLoader: Unable to load
>>>>> native-hadoop library for your platform... using builtin-java classes
>>>>> where applicable
>>>>> 12/12/01 18:55:29 WARN metrics.SchemaConfigured: Could not determine
>>>>> table and column family of the HFile path /tmp/test.txt. Expecting at
>>>>> least 5 path components.
>>>>> 12/12/01 18:55:29 WARN snappy.LoadSnappy: Snappy native library is
>>>>> available
>>>>> 12/12/01 18:55:29 WARN snappy.LoadSnappy: Snappy native library not
>>>>> loaded
>>>>> Exception in thread "main" java.lang.RuntimeException: native snappy
>>>>> library not available
>>>>> at
>>>>> org.apache.hadoop.io.compress.SnappyCodec.getCompressorType(SnappyCodec.java:123)
>>>>> at
>>>>> org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:100)
>>>>> at
>>>>> org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:112)
>>>>> at
>>>>> org.apache.hadoop.hbase.io.hfile.Compression$Algorithm.getCompressor(Compression.java:264)
>>>>> at
>>>>> org.apache.hadoop.hbase.io.hfile.HFileBlock$Writer.<init>(HFileBlock.java:739)
>>>>> at
>>>>> org.apache.hadoop.hbase.io.hfile.HFileWriterV2.finishInit(HFileWriterV2.java:127)
>>>>> at
>>>>> org.apache.hadoop.hbase.io.hfile.HFileWriterV2.<init>(HFileWriterV2.java:118)
>>>>> at
>>>>> org.apache.hadoop.hbase.io.hfile.HFileWriterV2$WriterFactoryV2.createWriter(HFileWriterV2.java:101)
>>>>> at
>>>>> org.apache.hadoop.hbase.io.hfile.HFile$WriterFactory.create(HFile.java:394)
>>>>> at
>>>>> org.apache.hadoop.hbase.util.CompressionTest.doSmokeTest(CompressionTest.java:108)
>>>>> at
>>>>> org.apache.hadoop.hbase.util.CompressionTest.main(CompressionTest.java:138)
>>>>> hbase@node3:~/hbase-0.94.2$ ll /usr/local/lib/
>>>>> total 572
>>>>> -rw-r--r-- 1 root staff 391614 déc 1 18:33 libsnappy.a
>>>>> -rwxr-xr-x 1 root staff 957 déc 1 18:33 libsnappy.la
>>>>> lrwxrwxrwx 1 root staff 18 déc 1 18:33 libsnappy.so ->
>>>>> libsnappy.so.1.1.3
>>>>> lrwxrwxrwx 1 root staff 18 déc 1 18:33 libsnappy.so.1 ->
>>>>> libsnappy.so.1.1.3
>>>>> -rwxr-xr-x 1 root staff 178210 déc 1 18:33 libsnappy.so.1.1.3
>>>>> drwxrwsr-x 4 root staff 4096 jui 13 10:06 python2.6
>>>>> drwxrwsr-x 4 root staff 4096 jui 13 10:06 python2.7
>>>>> hbase@node3:~/hbase-0.94.2$
>>>>>
>>>>>
>>>>> 2012/12/1, Jean-Marc Spaggiari <[email protected]>:
>>>>>> Hi,
>>>>>>
>>>>>> I'm currently using GZip and want to move to Snappy.
>>>>>>
>>>>>> I have downloaded the tar file, extracted, build, make install, make
>>>>>> check, everything is working fine.
>>>>>>
>>>>>> However, I'm not able to get this working:
>>>>>> bin/hbase org.apache.hadoop.hbase.util.CompressionTest /tmp/test.txt
>>>>>> snappy
>>>>>> 12/12/01 18:46:21 WARN snappy.LoadSnappy: Snappy native library not
>>>>>> loaded
>>>>>> Exception in thread "main" java.lang.RuntimeException: native snappy
>>>>>> library not available
>>>>>> at
>>>>>> org.apache.hadoop.io.compress.SnappyCodec.getCompressorType(SnappyCodec.java:123)
>>>>>>
>>>>>> Sound like HBase is not able to find the native library. How can I
>>>>>> tell HBase where the library is?
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> JM
>>>>>>
>>>
>>>
>>
