Hi Mike, Actually your timing is pretty good. Just today we have committed two major features to Flume 1.x (NG) that can enable your use case:
https://issues.apache.org/jira/browse/FLUME-1157 - Support for Interceptors, which are plugins that can write e.g. timestamp headers https://issues.apache.org/jira/browse/FLUME-1183 - HBase sink These two features are now available on Flume trunk. I encourage you to adopt Flume 1.x in favor of Flume 0.9.x … there are some serious issues with 0.9.x that no one seems to be working on, and the momentum behind Flume 1.x is growing day by day. Mike On Tuesday, May 15, 2012 at 1:54 PM, Mike M wrote: > I appreciate the response, Mike. It wan't clear from the docs that > having a timestamp header was required. Is it possible to make the > exec source provide one (I'm trying to use Flume to `tail -F` a > logfile)? > > If not, would Flume OG provide the functionality I'm looking for? It > already has an HBase sink, which is tempting for my use case... > > Thanks! > > Mike > > On Tue, May 15, 2012 at 3:16 PM, Mike Percy <[email protected] > (mailto:[email protected])> wrote: > > Hi Mike, you have to have events each containing a "timestamp" header for > > that functionality to work. > > > > Mike > > > > On Tuesday, May 15, 2012 at 8:37 AM, Mike M wrote: > > > > I'm running into an issue where trying to use an escape sequence with > > an HDFS sink causes an exception. For example, if I have the sink > > configured like: > > > > tail-nginx.sinks.HDFS.hdfs.path = hdfs://localhost:8020/flume/%Y/%m/%d/ > > > > I see the following exception showing up thousands of times: > > > > 2012-05-14 16:20:15,730 INFO source.ExecSource: Exec source starting > > with command:tail -F /var/log/nginx/localhost.access.log > > 2012-05-14 16:20:15,743 ERROR hdfs.HDFSEventSink: process failed > > java.lang.NumberFormatException: null > > at java.lang.Long.parseLong(Long.java:404) > > at java.lang.Long.valueOf(Long.java:540) > > at > > org.apache.flume.formatter.output.BucketPath.replaceShorthand(BucketPath.java:185) > > at > > org.apache.flume.formatter.output.BucketPath.escapeString(BucketPath.java:219) > > at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:344) > > at > > org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68) > > at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147) > > at java.lang.Thread.run(Thread.java:722) > > 2012-05-14 16:20:15,745 ERROR flume.SinkRunner: Unable to deliver > > event. Exception follows. > > org.apache.flume.EventDeliveryException: java.lang.NumberFormatException: > > null > > at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:407) > > at > > org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68) > > at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147) > > at java.lang.Thread.run(Thread.java:722) > > Caused by: java.lang.NumberFormatException: null > > at java.lang.Long.parseLong(Long.java:404) > > at java.lang.Long.valueOf(Long.java:540) > > at > > org.apache.flume.formatter.output.BucketPath.replaceShorthand(BucketPath.java:185) > > at > > org.apache.flume.formatter.output.BucketPath.escapeString(BucketPath.java:219) > > at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:344) > > ... 3 more > > > > I've tried various methods of quoting the escape sequences, to no > > avail. When the escape sequences are removed, Flume works as > > expected. Is this just a configuration issue on my side, or is there > > possibly a bug? I'm running Flume-NG 1.1.0 packaged with CDH3u4. > > > > Thanks! > > > > Mike
