Hey JM, Sorry for the quick message earlier. I tracked down the JIRA I was referring to: https://issues.apache.org/jira/browse/HBASE-7080
Does this look like what you are seeing in Compression test? On Mon, Dec 3, 2012 at 9:09 AM, Kevin O'dell <[email protected]>wrote: > There is a compression test JIRA right now. What are you seeing? > > > On Mon, Dec 3, 2012 at 8:47 AM, Jean-Marc Spaggiari < > [email protected]> wrote: > >> Ok.... >> >> This: http://code.google.com/p/hadoop-snappy/issues/detail?id=2 helped >> me and my test program is now working. I'm able to load both >> libraries. Fine. >> >> But the CompressionTest is still not working. >> >> What is very strange is that: >> 12/12/03 08:44:24 WARN snappy.LoadSnappy: Snappy native library is >> available >> 12/12/03 08:44:24 WARN snappy.LoadSnappy: Snappy native library not loaded >> >> It's available, but not loaded. >> >> But from the code: >> static { >> try { >> System.loadLibrary("snappy"); >> System.loadLibrary("hadoopsnappy"); >> LOG.warn("Snappy native library is available"); >> AVAILABLE = true; >> } catch (UnsatisfiedLinkError ex) { >> //NOP >> } >> LOADED = AVAILABLE; >> if (LOADED) { >> LOG.info("Snappy native library loaded"); >> } else { >> LOG.warn("Snappy native library not loaded"); >> } >> } >> If "Snappy native library is available" is displayed, that mean >> AVAILABLE = true... And if AVAILABLE = true, then LOADED is set to >> true and Snappy native library loaded must be displayed... But it's >> not... How is this possible? >> >> I have not expected Snappy installation to be such a challenge... >> >> I will continue to dig and summarize the steps when I will be done (If >> I'm able to finish...) >> >> JM >> >> 2012/12/3, Jean-Marc Spaggiari <[email protected]>: >> > Thanks all for your replies. >> > >> > So, to reply to all in one. >> > >> > I'm not using CD3. I'm using Hadoop 1.0.3 and HBase 0.94.2 directly >> > from the JARs. >> > >> > Here are all the places where I have put the lib: >> > /home/hadoop/hadoop-1.0.3/lib/native/Linux-amd64-64/libsnappy.so >> > /home/hadoop/hadoop-1.0.3/lib/native/Linux-amd64-64/libsnappy.so.1 >> > /home/hadoop/hadoop-1.0.3/lib/native/Linux-amd64-64/libsnappy.so.1.1.3 >> > /home/hadoop/hadoop-1.0.3/lib/native/Linux-i386-32/libsnappy.so >> > /home/hadoop/hadoop-1.0.3/lib/native/Linux-i386-32/libsnappy.so.1 >> > /home/hadoop/hadoop-1.0.3/lib/native/Linux-i386-32/libsnappy.so.1.1.3 >> > /home/hbase/hbase-0.94.2/lib/native/libsnappy.so >> > /home/hbase/hbase-0.94.2/lib/native/libsnappy.so.1 >> > /home/hbase/hbase-0.94.2/lib/native/libsnappy.so.1.1.3 >> > /home/hbase/hbase-0.94.2/lib/native/Linux-amd64-64/libsnappy.so >> > /home/hbase/hbase-0.94.2/lib/native/Linux-amd64-64/libsnappy.so.1 >> > /home/hbase/hbase-0.94.2/lib/native/Linux-amd64-64/libsnappy.so.1.1.3 >> > /home/hbase/hbase-0.94.2/lib/native/Linux-i386-32/libsnappy.so >> > /home/hbase/hbase-0.94.2/lib/native/Linux-i386-32/libsnappy.so.1 >> > /home/hbase/hbase-0.94.2/lib/native/Linux-i386-32/libsnappy.so.1.1.3 >> > /lib/x86_64-linux-gnu/libsnappy.so >> > /usr/lib/libsnappy.so >> > /usr/lib/libsnappy.so.1 >> > /usr/lib/libsnappy.so.1.1.3 >> > /usr/local/lib/libsnappy.so >> > /usr/local/lib/libsnappy.so.1 >> > /usr/local/lib/libsnappy.so.1.1.3 >> > >> > I tried to add this on my hbase-env.xml: >> > export >> > HBASE_LIBRARY_PATH=/home/hbase/hbase-0.94.2/lib/native/Linux-amd64-64 >> > >> > Before I was trying with doing export on the command line directly >> > since it seems the hbase script is taking that into consideration too. >> > >> > I have not yet put the hbase.regionserver.codecs line since I still >> > need to use my cluster until I get snappy working. On the hbase/lib >> > directory I have snappy-java-1.0.3.2.jar. >> > >> > >> > Should snappy be installed within hbase? Or should it be in hadoop? >> > I'm not sure anymore. >> > >> > But it's still not working. So I tried the small code below: >> > >> > import java.util.StringTokenizer; >> > >> > public class Test >> > { >> > static { >> > try { >> > System.loadLibrary("snappy"); >> > System.loadLibrary("hadoopsnappy"); >> > System.out.println ("Snappy native library is available"); >> > } catch (UnsatisfiedLinkError ex) { >> > ex.printStackTrace(); >> > } >> > } >> > >> > public static void main (String [] args) >> > { >> > System.out.println ("Coucou"); >> > String property = System.getProperty("java.library.path"); >> > StringTokenizer parser = new StringTokenizer(property, ";"); >> > while (parser.hasMoreTokens()) { >> > System.err.println(parser.nextToken()); >> > } >> > } >> > } >> > >> > >> > This code is from org.apache.hadoop.io.compress.snappy.LoadSnappy. >> > The error I'm getting is java.lang.UnsatisfiedLinkError: no >> > hadoopsnappy in java.library.path. >> > >> > So the issue is not the snappy lib. It' there and working fine. The >> > issue is the hadoopsnappy lib which I don't have... >> > >> > I found it there: http://code.google.com/p/hadoop-snappy/ >> > >> > So I have extracted it with svn checkout >> > http://hadoop-snappy.googlecode.com/svn/trunk/ >> > hadoop-snappy-read-only, tried to built it with mvn package but it's >> > failing with something saying "cannot find -ljvm" >> > >> > So seems my challenge will be to build hadoop-snappy and not to >> > install snappy which is already there and working... >> > >> > JM >> > >> > 2012/12/3, surfer <[email protected]>: >> >> hope it helps. this is what I do on apache hadoop 1.0.x and hbase >> 0.92.y: >> >> in hbase-site.xml add: >> >> >> >> <property> >> >> <name>hbase.regionserver.codecs</name> >> >> <value>snappy</value> >> >> </property> >> >> >> >> copy that file into the hadoop conf directory. >> >> >> >> in hbase-env.sh: >> >> export HBASE_LIBRARY_PATH=/pathtoyourhadoop/lib/native/Linux-amd64-64 >> >> >> >> ( In hbase-env.sh I set also HBASE_HOME, HBASE_CONF_DIR, HADOOP_HOME, >> >> HADOOP_CONF_DIR but I don't know if they contribute to make snappy >> >> working...) >> >> >> >> in /pathtoyourhadoop/lib/native/Linux-amd64-64 I have: >> >> libsnappy.a >> >> libsnappy.so >> >> libsnappy.so.1 >> >> libsnappy.so.1.1.2 >> >> >> >> good luck >> >> giovanni >> >> >> >> >> >> >> >> >> >> >> >> >> >> On 12/02/2012 02:25 PM, Jean-Marc Spaggiari wrote: >> >>> So. I spent few hours on that yesterday with no luck. >> >>> >> >>> Here is what I did: >> >>> - Install the google tar, untared, configured, maked and installed it. >> >>> - Copied the .so files all over my fs in the os lib dir, >> >>> HBase/lib/native and subdirs, Hadoop/lib/native and subdirs. >> >>> - Installed all debian packages with snappy in the name: >> >>> python-snappy, libsnappy-dev, libsnappy1, libsnappy-java >> >>> >> >>> But still exactly the same issue as above. And I don't have any clue >> >>> where to dig. There is nothing on internet about that. >> >>> >> >>> Anyone faced that already while installing Snappy? >> >>> >> >>> JM >> >>> >> >>> 2012/12/1, Jean-Marc Spaggiari <[email protected]>: >> >>>> Sorry, I forgot to paste few maybe useful lines. I have the lib in >> >>>> /usr/local/lib copied properly, and I have the HBASE_LIBRARY_PATH set >> >>>> correctly. Do I need to restart HBase to run this test? >> >>>> >> >>>> hbase@node3:~/hbase-0.94.2$ export >> HBASE_LIBRARY_PATH=/usr/local/lib/ >> >>>> hbase@node3:~/hbase-0.94.2$ bin/hbase >> >>>> org.apache.hadoop.hbase.util.CompressionTest /tmp/test.txt snappy >> >>>> 12/12/01 18:55:29 INFO util.ChecksumType: >> >>>> org.apache.hadoop.util.PureJavaCrc32 not available. >> >>>> 12/12/01 18:55:29 INFO util.ChecksumType: Checksum can use >> >>>> java.util.zip.CRC32 >> >>>> 12/12/01 18:55:29 INFO util.ChecksumType: >> >>>> org.apache.hadoop.util.PureJavaCrc32C not available. >> >>>> 12/12/01 18:55:29 DEBUG util.FSUtils: Creating file:/tmp/test.txtwith >> >>>> permission:rwxrwxrwx >> >>>> 12/12/01 18:55:29 WARN util.NativeCodeLoader: Unable to load >> >>>> native-hadoop library for your platform... using builtin-java classes >> >>>> where applicable >> >>>> 12/12/01 18:55:29 WARN metrics.SchemaConfigured: Could not determine >> >>>> table and column family of the HFile path /tmp/test.txt. Expecting at >> >>>> least 5 path components. >> >>>> 12/12/01 18:55:29 WARN snappy.LoadSnappy: Snappy native library is >> >>>> available >> >>>> 12/12/01 18:55:29 WARN snappy.LoadSnappy: Snappy native library not >> >>>> loaded >> >>>> Exception in thread "main" java.lang.RuntimeException: native snappy >> >>>> library not available >> >>>> at >> >>>> >> org.apache.hadoop.io.compress.SnappyCodec.getCompressorType(SnappyCodec.java:123) >> >>>> at >> >>>> >> org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:100) >> >>>> at >> >>>> >> org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:112) >> >>>> at >> >>>> >> org.apache.hadoop.hbase.io.hfile.Compression$Algorithm.getCompressor(Compression.java:264) >> >>>> at >> >>>> >> org.apache.hadoop.hbase.io.hfile.HFileBlock$Writer.<init>(HFileBlock.java:739) >> >>>> at >> >>>> >> org.apache.hadoop.hbase.io.hfile.HFileWriterV2.finishInit(HFileWriterV2.java:127) >> >>>> at >> >>>> >> org.apache.hadoop.hbase.io.hfile.HFileWriterV2.<init>(HFileWriterV2.java:118) >> >>>> at >> >>>> >> org.apache.hadoop.hbase.io.hfile.HFileWriterV2$WriterFactoryV2.createWriter(HFileWriterV2.java:101) >> >>>> at >> >>>> >> org.apache.hadoop.hbase.io.hfile.HFile$WriterFactory.create(HFile.java:394) >> >>>> at >> >>>> >> org.apache.hadoop.hbase.util.CompressionTest.doSmokeTest(CompressionTest.java:108) >> >>>> at >> >>>> >> org.apache.hadoop.hbase.util.CompressionTest.main(CompressionTest.java:138) >> >>>> hbase@node3:~/hbase-0.94.2$ ll /usr/local/lib/ >> >>>> total 572 >> >>>> -rw-r--r-- 1 root staff 391614 déc 1 18:33 libsnappy.a >> >>>> -rwxr-xr-x 1 root staff 957 déc 1 18:33 libsnappy.la >> >>>> lrwxrwxrwx 1 root staff 18 déc 1 18:33 libsnappy.so -> >> >>>> libsnappy.so.1.1.3 >> >>>> lrwxrwxrwx 1 root staff 18 déc 1 18:33 libsnappy.so.1 -> >> >>>> libsnappy.so.1.1.3 >> >>>> -rwxr-xr-x 1 root staff 178210 déc 1 18:33 libsnappy.so.1.1.3 >> >>>> drwxrwsr-x 4 root staff 4096 jui 13 10:06 python2.6 >> >>>> drwxrwsr-x 4 root staff 4096 jui 13 10:06 python2.7 >> >>>> hbase@node3:~/hbase-0.94.2$ >> >>>> >> >>>> >> >>>> 2012/12/1, Jean-Marc Spaggiari <[email protected]>: >> >>>>> Hi, >> >>>>> >> >>>>> I'm currently using GZip and want to move to Snappy. >> >>>>> >> >>>>> I have downloaded the tar file, extracted, build, make install, make >> >>>>> check, everything is working fine. >> >>>>> >> >>>>> However, I'm not able to get this working: >> >>>>> bin/hbase org.apache.hadoop.hbase.util.CompressionTest /tmp/test.txt >> >>>>> snappy >> >>>>> 12/12/01 18:46:21 WARN snappy.LoadSnappy: Snappy native library not >> >>>>> loaded >> >>>>> Exception in thread "main" java.lang.RuntimeException: native snappy >> >>>>> library not available >> >>>>> at >> >>>>> >> org.apache.hadoop.io.compress.SnappyCodec.getCompressorType(SnappyCodec.java:123) >> >>>>> >> >>>>> Sound like HBase is not able to find the native library. How can I >> >>>>> tell HBase where the library is? >> >>>>> >> >>>>> Thanks, >> >>>>> >> >>>>> JM >> >>>>> >> >> >> >> >> > >> > > > > -- > Kevin O'Dell > Customer Operations Engineer, Cloudera > -- Kevin O'Dell Customer Operations Engineer, Cloudera
