There is a compression test JIRA right now. What are you seeing? On Mon, Dec 3, 2012 at 8:47 AM, Jean-Marc Spaggiari <[email protected] > wrote:
> Ok.... > > This: http://code.google.com/p/hadoop-snappy/issues/detail?id=2 helped > me and my test program is now working. I'm able to load both > libraries. Fine. > > But the CompressionTest is still not working. > > What is very strange is that: > 12/12/03 08:44:24 WARN snappy.LoadSnappy: Snappy native library is > available > 12/12/03 08:44:24 WARN snappy.LoadSnappy: Snappy native library not loaded > > It's available, but not loaded. > > But from the code: > static { > try { > System.loadLibrary("snappy"); > System.loadLibrary("hadoopsnappy"); > LOG.warn("Snappy native library is available"); > AVAILABLE = true; > } catch (UnsatisfiedLinkError ex) { > //NOP > } > LOADED = AVAILABLE; > if (LOADED) { > LOG.info("Snappy native library loaded"); > } else { > LOG.warn("Snappy native library not loaded"); > } > } > If "Snappy native library is available" is displayed, that mean > AVAILABLE = true... And if AVAILABLE = true, then LOADED is set to > true and Snappy native library loaded must be displayed... But it's > not... How is this possible? > > I have not expected Snappy installation to be such a challenge... > > I will continue to dig and summarize the steps when I will be done (If > I'm able to finish...) > > JM > > 2012/12/3, Jean-Marc Spaggiari <[email protected]>: > > Thanks all for your replies. > > > > So, to reply to all in one. > > > > I'm not using CD3. I'm using Hadoop 1.0.3 and HBase 0.94.2 directly > > from the JARs. > > > > Here are all the places where I have put the lib: > > /home/hadoop/hadoop-1.0.3/lib/native/Linux-amd64-64/libsnappy.so > > /home/hadoop/hadoop-1.0.3/lib/native/Linux-amd64-64/libsnappy.so.1 > > /home/hadoop/hadoop-1.0.3/lib/native/Linux-amd64-64/libsnappy.so.1.1.3 > > /home/hadoop/hadoop-1.0.3/lib/native/Linux-i386-32/libsnappy.so > > /home/hadoop/hadoop-1.0.3/lib/native/Linux-i386-32/libsnappy.so.1 > > /home/hadoop/hadoop-1.0.3/lib/native/Linux-i386-32/libsnappy.so.1.1.3 > > /home/hbase/hbase-0.94.2/lib/native/libsnappy.so > > /home/hbase/hbase-0.94.2/lib/native/libsnappy.so.1 > > /home/hbase/hbase-0.94.2/lib/native/libsnappy.so.1.1.3 > > /home/hbase/hbase-0.94.2/lib/native/Linux-amd64-64/libsnappy.so > > /home/hbase/hbase-0.94.2/lib/native/Linux-amd64-64/libsnappy.so.1 > > /home/hbase/hbase-0.94.2/lib/native/Linux-amd64-64/libsnappy.so.1.1.3 > > /home/hbase/hbase-0.94.2/lib/native/Linux-i386-32/libsnappy.so > > /home/hbase/hbase-0.94.2/lib/native/Linux-i386-32/libsnappy.so.1 > > /home/hbase/hbase-0.94.2/lib/native/Linux-i386-32/libsnappy.so.1.1.3 > > /lib/x86_64-linux-gnu/libsnappy.so > > /usr/lib/libsnappy.so > > /usr/lib/libsnappy.so.1 > > /usr/lib/libsnappy.so.1.1.3 > > /usr/local/lib/libsnappy.so > > /usr/local/lib/libsnappy.so.1 > > /usr/local/lib/libsnappy.so.1.1.3 > > > > I tried to add this on my hbase-env.xml: > > export > > HBASE_LIBRARY_PATH=/home/hbase/hbase-0.94.2/lib/native/Linux-amd64-64 > > > > Before I was trying with doing export on the command line directly > > since it seems the hbase script is taking that into consideration too. > > > > I have not yet put the hbase.regionserver.codecs line since I still > > need to use my cluster until I get snappy working. On the hbase/lib > > directory I have snappy-java-1.0.3.2.jar. > > > > > > Should snappy be installed within hbase? Or should it be in hadoop? > > I'm not sure anymore. > > > > But it's still not working. So I tried the small code below: > > > > import java.util.StringTokenizer; > > > > public class Test > > { > > static { > > try { > > System.loadLibrary("snappy"); > > System.loadLibrary("hadoopsnappy"); > > System.out.println ("Snappy native library is available"); > > } catch (UnsatisfiedLinkError ex) { > > ex.printStackTrace(); > > } > > } > > > > public static void main (String [] args) > > { > > System.out.println ("Coucou"); > > String property = System.getProperty("java.library.path"); > > StringTokenizer parser = new StringTokenizer(property, ";"); > > while (parser.hasMoreTokens()) { > > System.err.println(parser.nextToken()); > > } > > } > > } > > > > > > This code is from org.apache.hadoop.io.compress.snappy.LoadSnappy. > > The error I'm getting is java.lang.UnsatisfiedLinkError: no > > hadoopsnappy in java.library.path. > > > > So the issue is not the snappy lib. It' there and working fine. The > > issue is the hadoopsnappy lib which I don't have... > > > > I found it there: http://code.google.com/p/hadoop-snappy/ > > > > So I have extracted it with svn checkout > > http://hadoop-snappy.googlecode.com/svn/trunk/ > > hadoop-snappy-read-only, tried to built it with mvn package but it's > > failing with something saying "cannot find -ljvm" > > > > So seems my challenge will be to build hadoop-snappy and not to > > install snappy which is already there and working... > > > > JM > > > > 2012/12/3, surfer <[email protected]>: > >> hope it helps. this is what I do on apache hadoop 1.0.x and hbase > 0.92.y: > >> in hbase-site.xml add: > >> > >> <property> > >> <name>hbase.regionserver.codecs</name> > >> <value>snappy</value> > >> </property> > >> > >> copy that file into the hadoop conf directory. > >> > >> in hbase-env.sh: > >> export HBASE_LIBRARY_PATH=/pathtoyourhadoop/lib/native/Linux-amd64-64 > >> > >> ( In hbase-env.sh I set also HBASE_HOME, HBASE_CONF_DIR, HADOOP_HOME, > >> HADOOP_CONF_DIR but I don't know if they contribute to make snappy > >> working...) > >> > >> in /pathtoyourhadoop/lib/native/Linux-amd64-64 I have: > >> libsnappy.a > >> libsnappy.so > >> libsnappy.so.1 > >> libsnappy.so.1.1.2 > >> > >> good luck > >> giovanni > >> > >> > >> > >> > >> > >> > >> On 12/02/2012 02:25 PM, Jean-Marc Spaggiari wrote: > >>> So. I spent few hours on that yesterday with no luck. > >>> > >>> Here is what I did: > >>> - Install the google tar, untared, configured, maked and installed it. > >>> - Copied the .so files all over my fs in the os lib dir, > >>> HBase/lib/native and subdirs, Hadoop/lib/native and subdirs. > >>> - Installed all debian packages with snappy in the name: > >>> python-snappy, libsnappy-dev, libsnappy1, libsnappy-java > >>> > >>> But still exactly the same issue as above. And I don't have any clue > >>> where to dig. There is nothing on internet about that. > >>> > >>> Anyone faced that already while installing Snappy? > >>> > >>> JM > >>> > >>> 2012/12/1, Jean-Marc Spaggiari <[email protected]>: > >>>> Sorry, I forgot to paste few maybe useful lines. I have the lib in > >>>> /usr/local/lib copied properly, and I have the HBASE_LIBRARY_PATH set > >>>> correctly. Do I need to restart HBase to run this test? > >>>> > >>>> hbase@node3:~/hbase-0.94.2$ export HBASE_LIBRARY_PATH=/usr/local/lib/ > >>>> hbase@node3:~/hbase-0.94.2$ bin/hbase > >>>> org.apache.hadoop.hbase.util.CompressionTest /tmp/test.txt snappy > >>>> 12/12/01 18:55:29 INFO util.ChecksumType: > >>>> org.apache.hadoop.util.PureJavaCrc32 not available. > >>>> 12/12/01 18:55:29 INFO util.ChecksumType: Checksum can use > >>>> java.util.zip.CRC32 > >>>> 12/12/01 18:55:29 INFO util.ChecksumType: > >>>> org.apache.hadoop.util.PureJavaCrc32C not available. > >>>> 12/12/01 18:55:29 DEBUG util.FSUtils: Creating file:/tmp/test.txtwith > >>>> permission:rwxrwxrwx > >>>> 12/12/01 18:55:29 WARN util.NativeCodeLoader: Unable to load > >>>> native-hadoop library for your platform... using builtin-java classes > >>>> where applicable > >>>> 12/12/01 18:55:29 WARN metrics.SchemaConfigured: Could not determine > >>>> table and column family of the HFile path /tmp/test.txt. Expecting at > >>>> least 5 path components. > >>>> 12/12/01 18:55:29 WARN snappy.LoadSnappy: Snappy native library is > >>>> available > >>>> 12/12/01 18:55:29 WARN snappy.LoadSnappy: Snappy native library not > >>>> loaded > >>>> Exception in thread "main" java.lang.RuntimeException: native snappy > >>>> library not available > >>>> at > >>>> > org.apache.hadoop.io.compress.SnappyCodec.getCompressorType(SnappyCodec.java:123) > >>>> at > >>>> > org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:100) > >>>> at > >>>> > org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:112) > >>>> at > >>>> > org.apache.hadoop.hbase.io.hfile.Compression$Algorithm.getCompressor(Compression.java:264) > >>>> at > >>>> > org.apache.hadoop.hbase.io.hfile.HFileBlock$Writer.<init>(HFileBlock.java:739) > >>>> at > >>>> > org.apache.hadoop.hbase.io.hfile.HFileWriterV2.finishInit(HFileWriterV2.java:127) > >>>> at > >>>> > org.apache.hadoop.hbase.io.hfile.HFileWriterV2.<init>(HFileWriterV2.java:118) > >>>> at > >>>> > org.apache.hadoop.hbase.io.hfile.HFileWriterV2$WriterFactoryV2.createWriter(HFileWriterV2.java:101) > >>>> at > >>>> > org.apache.hadoop.hbase.io.hfile.HFile$WriterFactory.create(HFile.java:394) > >>>> at > >>>> > org.apache.hadoop.hbase.util.CompressionTest.doSmokeTest(CompressionTest.java:108) > >>>> at > >>>> > org.apache.hadoop.hbase.util.CompressionTest.main(CompressionTest.java:138) > >>>> hbase@node3:~/hbase-0.94.2$ ll /usr/local/lib/ > >>>> total 572 > >>>> -rw-r--r-- 1 root staff 391614 déc 1 18:33 libsnappy.a > >>>> -rwxr-xr-x 1 root staff 957 déc 1 18:33 libsnappy.la > >>>> lrwxrwxrwx 1 root staff 18 déc 1 18:33 libsnappy.so -> > >>>> libsnappy.so.1.1.3 > >>>> lrwxrwxrwx 1 root staff 18 déc 1 18:33 libsnappy.so.1 -> > >>>> libsnappy.so.1.1.3 > >>>> -rwxr-xr-x 1 root staff 178210 déc 1 18:33 libsnappy.so.1.1.3 > >>>> drwxrwsr-x 4 root staff 4096 jui 13 10:06 python2.6 > >>>> drwxrwsr-x 4 root staff 4096 jui 13 10:06 python2.7 > >>>> hbase@node3:~/hbase-0.94.2$ > >>>> > >>>> > >>>> 2012/12/1, Jean-Marc Spaggiari <[email protected]>: > >>>>> Hi, > >>>>> > >>>>> I'm currently using GZip and want to move to Snappy. > >>>>> > >>>>> I have downloaded the tar file, extracted, build, make install, make > >>>>> check, everything is working fine. > >>>>> > >>>>> However, I'm not able to get this working: > >>>>> bin/hbase org.apache.hadoop.hbase.util.CompressionTest /tmp/test.txt > >>>>> snappy > >>>>> 12/12/01 18:46:21 WARN snappy.LoadSnappy: Snappy native library not > >>>>> loaded > >>>>> Exception in thread "main" java.lang.RuntimeException: native snappy > >>>>> library not available > >>>>> at > >>>>> > org.apache.hadoop.io.compress.SnappyCodec.getCompressorType(SnappyCodec.java:123) > >>>>> > >>>>> Sound like HBase is not able to find the native library. How can I > >>>>> tell HBase where the library is? > >>>>> > >>>>> Thanks, > >>>>> > >>>>> JM > >>>>> > >> > >> > > > -- Kevin O'Dell Customer Operations Engineer, Cloudera
