Sure I will. JM
2012/12/3, Stack <[email protected]>: > Any chance of an update to > http://hbase.apache.org/book.html#snappy.compression ? If someone writes > it up, I'll stitch it in. Thanks, > St.Ack > > > On Mon, Dec 3, 2012 at 6:29 AM, [email protected] <[email protected]> wrote: > >> Hi, >> >> Something more about my workaround last time: >> >> I used the following steps to test my workaround: >> >> 1) cd $HBASE_HOME >> ./bin/hbase org.apache.hadoop.hbase.util.CompressionTest >> file:///tmp/testfile lzo >> >> >> 2) hbase shell >> create 't1', {NAME => 'cf1', COMPRESSION => 'LZO'} >> >> >> You could modify above for your test cases. >> >> Good luck >> Thanks >> AC >> >> >> On 3 Dec 2012, at 10:22 PM, [email protected] wrote: >> >> > Hi JM, >> > >> > I had experienced similar error when I was installing LZO compression >> > to >> RegionServers: >> > >> > Below is from my record about installing LZO: >> > Issue: >> > java.lang.UnsatisfiedLinkError: no gplcompression in java.library.path >> > ... >> > 12/11/23 19:03:14 ERROR lzo.LzoCodec: Cannot load native-lzo without >> native-hadoop >> > Exception in thread "main" java.lang.RuntimeException: native-lzo >> library not available >> > Solution: >> > (compiled lzo from source), then do these EXTRA steps: >> > 1) cp >> {working_folder_of_lzo}/hadoop-lzo-master/build/native/Linux-amd64-64/lib/* >> /usr/local/lib/ >> > 2) echo export HBASE_LIBRARY_PATH=/usr/local/lib/ >> >> $HBASE_HOME/conf/hbase-env.sh >> > 3) mkdir -p $HBASE_HOME/build >> > 4) cp -r {working_folder_of_lzo}/hadoop-lzo-master/build/native >> $HBASE_HOME/build/native >> > >> > >> > >> > When looking at your email below, I saw your log also has "The error >> I'm getting is java.lang.UnsatisfiedLinkError: no hadoopsnappy in >> java.library.path." >> > I think you could try these steps: >> > 1) cp {working_folder_your_snappy}/build/native/Linux-amd64-64/lib/* >> /usr/local/lib/ >> > 2) echo export HBASE_LIBRARY_PATH=/usr/local/lib/ >> >> $HBASE_HOME/conf/hbase-env.sh >> > 3) mkdir -p $HBASE_HOME/build >> > 4) cp -r {working_folder_your_snappy}/build/native >> $HBASE_HOME/build/native >> > >> > >> > Good luck. >> > Thanks >> > AC >> > >> > >> > >> > On 3 Dec 2012, at 9:47 PM, Jean-Marc Spaggiari wrote: >> > >> >> Ok.... >> >> >> >> This: http://code.google.com/p/hadoop-snappy/issues/detail?id=2 helped >> >> me and my test program is now working. I'm able to load both >> >> libraries. Fine. >> >> >> >> But the CompressionTest is still not working. >> >> >> >> What is very strange is that: >> >> 12/12/03 08:44:24 WARN snappy.LoadSnappy: Snappy native library is >> available >> >> 12/12/03 08:44:24 WARN snappy.LoadSnappy: Snappy native library not >> loaded >> >> >> >> It's available, but not loaded. >> >> >> >> But from the code: >> >> static { >> >> try { >> >> System.loadLibrary("snappy"); >> >> System.loadLibrary("hadoopsnappy"); >> >> LOG.warn("Snappy native library is available"); >> >> AVAILABLE = true; >> >> } catch (UnsatisfiedLinkError ex) { >> >> //NOP >> >> } >> >> LOADED = AVAILABLE; >> >> if (LOADED) { >> >> LOG.info("Snappy native library loaded"); >> >> } else { >> >> LOG.warn("Snappy native library not loaded"); >> >> } >> >> } >> >> If "Snappy native library is available" is displayed, that mean >> >> AVAILABLE = true... And if AVAILABLE = true, then LOADED is set to >> >> true and Snappy native library loaded must be displayed... But it's >> >> not... How is this possible? >> >> >> >> I have not expected Snappy installation to be such a challenge... >> >> >> >> I will continue to dig and summarize the steps when I will be done (If >> >> I'm able to finish...) >> >> >> >> JM >> >> >> >> 2012/12/3, Jean-Marc Spaggiari <[email protected]>: >> >>> Thanks all for your replies. >> >>> >> >>> So, to reply to all in one. >> >>> >> >>> I'm not using CD3. I'm using Hadoop 1.0.3 and HBase 0.94.2 directly >> >>> from the JARs. >> >>> >> >>> Here are all the places where I have put the lib: >> >>> /home/hadoop/hadoop-1.0.3/lib/native/Linux-amd64-64/libsnappy.so >> >>> /home/hadoop/hadoop-1.0.3/lib/native/Linux-amd64-64/libsnappy.so.1 >> >>> /home/hadoop/hadoop-1.0.3/lib/native/Linux-amd64-64/libsnappy.so.1.1.3 >> >>> /home/hadoop/hadoop-1.0.3/lib/native/Linux-i386-32/libsnappy.so >> >>> /home/hadoop/hadoop-1.0.3/lib/native/Linux-i386-32/libsnappy.so.1 >> >>> /home/hadoop/hadoop-1.0.3/lib/native/Linux-i386-32/libsnappy.so.1.1.3 >> >>> /home/hbase/hbase-0.94.2/lib/native/libsnappy.so >> >>> /home/hbase/hbase-0.94.2/lib/native/libsnappy.so.1 >> >>> /home/hbase/hbase-0.94.2/lib/native/libsnappy.so.1.1.3 >> >>> /home/hbase/hbase-0.94.2/lib/native/Linux-amd64-64/libsnappy.so >> >>> /home/hbase/hbase-0.94.2/lib/native/Linux-amd64-64/libsnappy.so.1 >> >>> /home/hbase/hbase-0.94.2/lib/native/Linux-amd64-64/libsnappy.so.1.1.3 >> >>> /home/hbase/hbase-0.94.2/lib/native/Linux-i386-32/libsnappy.so >> >>> /home/hbase/hbase-0.94.2/lib/native/Linux-i386-32/libsnappy.so.1 >> >>> /home/hbase/hbase-0.94.2/lib/native/Linux-i386-32/libsnappy.so.1.1.3 >> >>> /lib/x86_64-linux-gnu/libsnappy.so >> >>> /usr/lib/libsnappy.so >> >>> /usr/lib/libsnappy.so.1 >> >>> /usr/lib/libsnappy.so.1.1.3 >> >>> /usr/local/lib/libsnappy.so >> >>> /usr/local/lib/libsnappy.so.1 >> >>> /usr/local/lib/libsnappy.so.1.1.3 >> >>> >> >>> I tried to add this on my hbase-env.xml: >> >>> export >> >>> HBASE_LIBRARY_PATH=/home/hbase/hbase-0.94.2/lib/native/Linux-amd64-64 >> >>> >> >>> Before I was trying with doing export on the command line directly >> >>> since it seems the hbase script is taking that into consideration >> >>> too. >> >>> >> >>> I have not yet put the hbase.regionserver.codecs line since I still >> >>> need to use my cluster until I get snappy working. On the hbase/lib >> >>> directory I have snappy-java-1.0.3.2.jar. >> >>> >> >>> >> >>> Should snappy be installed within hbase? Or should it be in hadoop? >> >>> I'm not sure anymore. >> >>> >> >>> But it's still not working. So I tried the small code below: >> >>> >> >>> import java.util.StringTokenizer; >> >>> >> >>> public class Test >> >>> { >> >>> static { >> >>> try { >> >>> System.loadLibrary("snappy"); >> >>> System.loadLibrary("hadoopsnappy"); >> >>> System.out.println ("Snappy native library is available"); >> >>> } catch (UnsatisfiedLinkError ex) { >> >>> ex.printStackTrace(); >> >>> } >> >>> } >> >>> >> >>> public static void main (String [] args) >> >>> { >> >>> System.out.println ("Coucou"); >> >>> String property = System.getProperty("java.library.path"); >> >>> StringTokenizer parser = new StringTokenizer(property, ";"); >> >>> while (parser.hasMoreTokens()) { >> >>> System.err.println(parser.nextToken()); >> >>> } >> >>> } >> >>> } >> >>> >> >>> >> >>> This code is from org.apache.hadoop.io.compress.snappy.LoadSnappy. >> >>> The error I'm getting is java.lang.UnsatisfiedLinkError: no >> >>> hadoopsnappy in java.library.path. >> >>> >> >>> So the issue is not the snappy lib. It' there and working fine. The >> >>> issue is the hadoopsnappy lib which I don't have... >> >>> >> >>> I found it there: http://code.google.com/p/hadoop-snappy/ >> >>> >> >>> So I have extracted it with svn checkout >> >>> http://hadoop-snappy.googlecode.com/svn/trunk/ >> >>> hadoop-snappy-read-only, tried to built it with mvn package but it's >> >>> failing with something saying "cannot find -ljvm" >> >>> >> >>> So seems my challenge will be to build hadoop-snappy and not to >> >>> install snappy which is already there and working... >> >>> >> >>> JM >> >>> >> >>> 2012/12/3, surfer <[email protected]>: >> >>>> hope it helps. this is what I do on apache hadoop 1.0.x and hbase >> 0.92.y: >> >>>> in hbase-site.xml add: >> >>>> >> >>>> <property> >> >>>> <name>hbase.regionserver.codecs</name> >> >>>> <value>snappy</value> >> >>>> </property> >> >>>> >> >>>> copy that file into the hadoop conf directory. >> >>>> >> >>>> in hbase-env.sh: >> >>>> export >> >>>> HBASE_LIBRARY_PATH=/pathtoyourhadoop/lib/native/Linux-amd64-64 >> >>>> >> >>>> ( In hbase-env.sh I set also HBASE_HOME, HBASE_CONF_DIR, >> >>>> HADOOP_HOME, >> >>>> HADOOP_CONF_DIR but I don't know if they contribute to make snappy >> >>>> working...) >> >>>> >> >>>> in /pathtoyourhadoop/lib/native/Linux-amd64-64 I have: >> >>>> libsnappy.a >> >>>> libsnappy.so >> >>>> libsnappy.so.1 >> >>>> libsnappy.so.1.1.2 >> >>>> >> >>>> good luck >> >>>> giovanni >> >>>> >> >>>> >> >>>> >> >>>> >> >>>> >> >>>> >> >>>> On 12/02/2012 02:25 PM, Jean-Marc Spaggiari wrote: >> >>>>> So. I spent few hours on that yesterday with no luck. >> >>>>> >> >>>>> Here is what I did: >> >>>>> - Install the google tar, untared, configured, maked and installed >> it. >> >>>>> - Copied the .so files all over my fs in the os lib dir, >> >>>>> HBase/lib/native and subdirs, Hadoop/lib/native and subdirs. >> >>>>> - Installed all debian packages with snappy in the name: >> >>>>> python-snappy, libsnappy-dev, libsnappy1, libsnappy-java >> >>>>> >> >>>>> But still exactly the same issue as above. And I don't have any >> >>>>> clue >> >>>>> where to dig. There is nothing on internet about that. >> >>>>> >> >>>>> Anyone faced that already while installing Snappy? >> >>>>> >> >>>>> JM >> >>>>> >> >>>>> 2012/12/1, Jean-Marc Spaggiari <[email protected]>: >> >>>>>> Sorry, I forgot to paste few maybe useful lines. I have the lib in >> >>>>>> /usr/local/lib copied properly, and I have the HBASE_LIBRARY_PATH >> set >> >>>>>> correctly. Do I need to restart HBase to run this test? >> >>>>>> >> >>>>>> hbase@node3:~/hbase-0.94.2$ export >> HBASE_LIBRARY_PATH=/usr/local/lib/ >> >>>>>> hbase@node3:~/hbase-0.94.2$ bin/hbase >> >>>>>> org.apache.hadoop.hbase.util.CompressionTest /tmp/test.txt snappy >> >>>>>> 12/12/01 18:55:29 INFO util.ChecksumType: >> >>>>>> org.apache.hadoop.util.PureJavaCrc32 not available. >> >>>>>> 12/12/01 18:55:29 INFO util.ChecksumType: Checksum can use >> >>>>>> java.util.zip.CRC32 >> >>>>>> 12/12/01 18:55:29 INFO util.ChecksumType: >> >>>>>> org.apache.hadoop.util.PureJavaCrc32C not available. >> >>>>>> 12/12/01 18:55:29 DEBUG util.FSUtils: Creating >> file:/tmp/test.txtwith >> >>>>>> permission:rwxrwxrwx >> >>>>>> 12/12/01 18:55:29 WARN util.NativeCodeLoader: Unable to load >> >>>>>> native-hadoop library for your platform... using builtin-java >> classes >> >>>>>> where applicable >> >>>>>> 12/12/01 18:55:29 WARN metrics.SchemaConfigured: Could not >> >>>>>> determine >> >>>>>> table and column family of the HFile path /tmp/test.txt. Expecting >> at >> >>>>>> least 5 path components. >> >>>>>> 12/12/01 18:55:29 WARN snappy.LoadSnappy: Snappy native library is >> >>>>>> available >> >>>>>> 12/12/01 18:55:29 WARN snappy.LoadSnappy: Snappy native library >> >>>>>> not >> >>>>>> loaded >> >>>>>> Exception in thread "main" java.lang.RuntimeException: native >> >>>>>> snappy >> >>>>>> library not available >> >>>>>> at >> >>>>>> >> org.apache.hadoop.io.compress.SnappyCodec.getCompressorType(SnappyCodec.java:123) >> >>>>>> at >> >>>>>> >> org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:100) >> >>>>>> at >> >>>>>> >> org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:112) >> >>>>>> at >> >>>>>> >> org.apache.hadoop.hbase.io.hfile.Compression$Algorithm.getCompressor(Compression.java:264) >> >>>>>> at >> >>>>>> >> org.apache.hadoop.hbase.io.hfile.HFileBlock$Writer.<init>(HFileBlock.java:739) >> >>>>>> at >> >>>>>> >> org.apache.hadoop.hbase.io.hfile.HFileWriterV2.finishInit(HFileWriterV2.java:127) >> >>>>>> at >> >>>>>> >> org.apache.hadoop.hbase.io.hfile.HFileWriterV2.<init>(HFileWriterV2.java:118) >> >>>>>> at >> >>>>>> >> org.apache.hadoop.hbase.io.hfile.HFileWriterV2$WriterFactoryV2.createWriter(HFileWriterV2.java:101) >> >>>>>> at >> >>>>>> >> org.apache.hadoop.hbase.io.hfile.HFile$WriterFactory.create(HFile.java:394) >> >>>>>> at >> >>>>>> >> org.apache.hadoop.hbase.util.CompressionTest.doSmokeTest(CompressionTest.java:108) >> >>>>>> at >> >>>>>> >> org.apache.hadoop.hbase.util.CompressionTest.main(CompressionTest.java:138) >> >>>>>> hbase@node3:~/hbase-0.94.2$ ll /usr/local/lib/ >> >>>>>> total 572 >> >>>>>> -rw-r--r-- 1 root staff 391614 déc 1 18:33 libsnappy.a >> >>>>>> -rwxr-xr-x 1 root staff 957 déc 1 18:33 libsnappy.la >> >>>>>> lrwxrwxrwx 1 root staff 18 déc 1 18:33 libsnappy.so -> >> >>>>>> libsnappy.so.1.1.3 >> >>>>>> lrwxrwxrwx 1 root staff 18 déc 1 18:33 libsnappy.so.1 -> >> >>>>>> libsnappy.so.1.1.3 >> >>>>>> -rwxr-xr-x 1 root staff 178210 déc 1 18:33 libsnappy.so.1.1.3 >> >>>>>> drwxrwsr-x 4 root staff 4096 jui 13 10:06 python2.6 >> >>>>>> drwxrwsr-x 4 root staff 4096 jui 13 10:06 python2.7 >> >>>>>> hbase@node3:~/hbase-0.94.2$ >> >>>>>> >> >>>>>> >> >>>>>> 2012/12/1, Jean-Marc Spaggiari <[email protected]>: >> >>>>>>> Hi, >> >>>>>>> >> >>>>>>> I'm currently using GZip and want to move to Snappy. >> >>>>>>> >> >>>>>>> I have downloaded the tar file, extracted, build, make install, >> make >> >>>>>>> check, everything is working fine. >> >>>>>>> >> >>>>>>> However, I'm not able to get this working: >> >>>>>>> bin/hbase org.apache.hadoop.hbase.util.CompressionTest >> /tmp/test.txt >> >>>>>>> snappy >> >>>>>>> 12/12/01 18:46:21 WARN snappy.LoadSnappy: Snappy native library >> >>>>>>> not >> >>>>>>> loaded >> >>>>>>> Exception in thread "main" java.lang.RuntimeException: native >> snappy >> >>>>>>> library not available >> >>>>>>> at >> >>>>>>> >> org.apache.hadoop.io.compress.SnappyCodec.getCompressorType(SnappyCodec.java:123) >> >>>>>>> >> >>>>>>> Sound like HBase is not able to find the native library. How can >> >>>>>>> I >> >>>>>>> tell HBase where the library is? >> >>>>>>> >> >>>>>>> Thanks, >> >>>>>>> >> >>>>>>> JM >> >>>>>>> >> >>>> >> >>>> >> >>> >> > >> >> >
