Does your HADOOP_CONF_DIR/core-site.xml have its io.compression.codecs property configured to include LZO? Can you ensure your Flume picks up the same configuration? You could also try to make it <final> to enforce it and prevent overriding.
On Tue, Aug 2, 2011 at 2:46 PM, Eran Kutner <[email protected]> wrote: > Hi, > I'm trying to enable the LZO codec in flume. > I'm using flume 0.9.4 from CDH3 > I placed the hadopp-lzo.jar file in flume's lib directory. > I added the native directory to the java lib path in flume-env.sh > I verified, using "flume classpath" that the codec jar file is indeed > included in the flume classpath. > > This command is working fine: > java -classpath "/usr/lib/flume/lib/*" > -Djava.library.path=/usr/lib/hadoop/lib/native/Linux-amd64-64 > com.hadoop.compression.lzo.LzoIndexer /tmp/test.lzo > > > However, when I try to enable LzopCodec I get this error: > 11/08/02 04:45:35 INFO connector.DirectDriver: Connector logicalNode > collector1-44 exited with error: Unsupported compression codec LzopCodec. > Please choose from: [None, DefaultCodec, GzipCodec, BZip2Codec, > DeflateCodec, SnappyCodec] > java.lang.IllegalArgumentException: Unsupported compression codec > LzopCodec. Please choose from: [None, DefaultCodec, GzipCodec, BZip2Codec, > DeflateCodec, SnappyCodec] > at > com.cloudera.flume.handlers.hdfs.CustomDfsSink.getCodec(CustomDfsSink.java:180) > at > com.cloudera.flume.handlers.hdfs.CustomDfsSink.open(CustomDfsSink.java:109) > at > com.cloudera.flume.handlers.hdfs.EscapedCustomDfsSink.openWriter(EscapedCustomDfsSink.java:104) > at > com.cloudera.flume.handlers.hdfs.EscapedCustomDfsSink.append(EscapedCustomDfsSink.java:119) > at > com.cloudera.flume.core.CompositeSink.append(CompositeSink.java:61) > at > com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:60) > at > com.cloudera.flume.handlers.rolling.RollSink.synchronousAppend(RollSink.java:234) > at > com.cloudera.flume.handlers.rolling.RollSink$1.call(RollSink.java:183) > at > com.cloudera.flume.handlers.rolling.RollSink$1.call(RollSink.java:181) > at > java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) > at java.util.concurrent.FutureTask.run(FutureTask.java:138) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > > > Note that it doesn't even list the LzoCodec in available codecs list. Any > idea why? > > -eran > > -- Harsh J
