Hi all,
I got stuck trying to enable the LZO compression on a table. I installed
the native-lzo library on each tablet server (sudo apt-get install
liblzo2-dev) and set the compression codecs libraries in the Hadoop
core-site.xml:
<property>
<name>io.compression.codecs</name>
<value>org.apache.hadoop.io.compress.GzipCodec,
org.apache.hadoop.io.compress.DefaultCodec,
org.apache.hadoop.io.compress.BZip2Codec,
com.hadoop.compression.lzo.LzoCodec,
com.hadoop.compression.lzo.LzopCodec
</value>
</property>
<property>
<name>io.compression.codec.lzo.class</name>
<value>com.hadoop.compression.lzo.LzoCodec</value>
</property>
I also added the path to the library to the general.classpaths property
of accumulo-site.xml:
<property>
<name>general.classpaths</name>
<value>
....
/usr/hdp/current/hadoop-client/lib/hadoop-lzo-0.6.0.2.5.0.0-1245.jar,
/usr/hdp/current/hadoop-client/lib/native/Linux-amd64-64/libgplcompression.so,
/lib/x86_64-linux-gnu/liblzo2.so.2
</value>
</property>
I tested that the lzo library is working correctly compressing a file via
a MapReduce Job, but when I asked Accumulo to compact the table on which I
enabled the LZO compression I got this error on each tablet server:
MajC Unexpected exception, extent = 1z;???????????<;???????????.
java.lang.RuntimeException: native-lzo library not available
at
com.hadoop.compression.lzo.LzoCodec.getCompressorType(LzoCodec.java:155)
at
org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:150)
at
org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:165)
at
org.apache.accumulo.core.file.rfile.bcfile.Compression$Algorithm.getCompressor(Compression.java:502)
at
org.apache.accumulo.core.file.rfile.bcfile.BCFile$Writer$WBlockState.<init>(BCFile.java:192)
at
org.apache.accumulo.core.file.rfile.bcfile.BCFile$Writer.prepareDataBlock(BCFile.java:487)
at
org.apache.accumulo.core.file.blockfile.impl.CachableBlockFile$Writer.prepareDataBlock(CachableBlockFile.java:89)
at
org.apache.accumulo.core.file.rfile.RFile$LocalityGroupWriter.append(RFile.java:429)
at
org.apache.accumulo.core.file.rfile.RFile$Writer.append(RFile.java:596)
at
org.apache.accumulo.core.file.BloomFilterLayer$Writer.append(BloomFilterLayer.java:141)
at
org.apache.accumulo.tserver.tablet.Compactor.compactLocalityGroup(Compactor.java:367)
at
org.apache.accumulo.tserver.tablet.Compactor.call(Compactor.java:223)
at
org.apache.accumulo.tserver.tablet.Tablet._majorCompact(Tablet.java:1850)
at
org.apache.accumulo.tserver.tablet.Tablet.majorCompact(Tablet.java:1967)
at
org.apache.accumulo.tserver.tablet.CompactionRunner.run(CompactionRunner.java:44)
at
org.apache.htrace.wrappers.TraceRunnable.run(TraceRunnable.java:57)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at
org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
at java.lang.Thread.run(Thread.java:745)
What am I missing?
Thanks
Regards,
Max
