Thanks Harsh, that did the trick. I didn't realize Flume was reading the hadoop configuration, iand it's not mentioned anywhere.
However, now I have a different problem. I'm trying to write compressed sequence files to be read by hadoop, however the file produced by flume is not valid. After looking into it, it seems that flume is compressing the entire file, not just the data. This means the file doesn't have a valid sequence file header, and so hadoop can't read it. It's not a LZO problem I tried it with the gzip codec and got the same result. After I ran gunzip on the file, it became a valid sequence file with the "SEQ" header, etc. Something I'm doing wrong? -eran On Tue, Aug 2, 2011 at 16:12, Harsh J <[email protected]> wrote: > Does your HADOOP_CONF_DIR/core-site.xml have its io.compression.codecs > property configured to include LZO? Can you ensure your Flume picks up > the same configuration? You could also try to make it <final> to > enforce it and prevent overriding. > > On Tue, Aug 2, 2011 at 2:46 PM, Eran Kutner <[email protected]> wrote: > > Hi, > > I'm trying to enable the LZO codec in flume. > > I'm using flume 0.9.4 from CDH3 > > I placed the hadopp-lzo.jar file in flume's lib directory. > > I added the native directory to the java lib path in flume-env.sh > > I verified, using "flume classpath" that the codec jar file is indeed > > included in the flume classpath. > > > > This command is working fine: > > java -classpath "/usr/lib/flume/lib/*" > > -Djava.library.path=/usr/lib/hadoop/lib/native/Linux-amd64-64 > > com.hadoop.compression.lzo.LzoIndexer /tmp/test.lzo > > > > > > However, when I try to enable LzopCodec I get this error: > > 11/08/02 04:45:35 INFO connector.DirectDriver: Connector logicalNode > > collector1-44 exited with error: Unsupported compression codec LzopCodec. > > Please choose from: [None, DefaultCodec, GzipCodec, BZip2Codec, > > DeflateCodec, SnappyCodec] > > java.lang.IllegalArgumentException: Unsupported compression codec > > LzopCodec. Please choose from: [None, DefaultCodec, GzipCodec, > BZip2Codec, > > DeflateCodec, SnappyCodec] > > at > > > com.cloudera.flume.handlers.hdfs.CustomDfsSink.getCodec(CustomDfsSink.java:180) > > at > > > com.cloudera.flume.handlers.hdfs.CustomDfsSink.open(CustomDfsSink.java:109) > > at > > > com.cloudera.flume.handlers.hdfs.EscapedCustomDfsSink.openWriter(EscapedCustomDfsSink.java:104) > > at > > > com.cloudera.flume.handlers.hdfs.EscapedCustomDfsSink.append(EscapedCustomDfsSink.java:119) > > at > > com.cloudera.flume.core.CompositeSink.append(CompositeSink.java:61) > > at > > > com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:60) > > at > > > com.cloudera.flume.handlers.rolling.RollSink.synchronousAppend(RollSink.java:234) > > at > > com.cloudera.flume.handlers.rolling.RollSink$1.call(RollSink.java:183) > > at > > com.cloudera.flume.handlers.rolling.RollSink$1.call(RollSink.java:181) > > at > > java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) > > at java.util.concurrent.FutureTask.run(FutureTask.java:138) > > at > > > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > > at > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > > at java.lang.Thread.run(Thread.java:662) > > > > > > Note that it doesn't even list the LzoCodec in available codecs list. Any > > idea why? > > > > -eran > > > > > > > > -- > Harsh J >
