I appreciate the response, Mike. It wan't clear from the docs that having a timestamp header was required. Is it possible to make the exec source provide one (I'm trying to use Flume to `tail -F` a logfile)?
If not, would Flume OG provide the functionality I'm looking for? It already has an HBase sink, which is tempting for my use case... Thanks! Mike On Tue, May 15, 2012 at 3:16 PM, Mike Percy <[email protected]> wrote: > Hi Mike, you have to have events each containing a "timestamp" header for > that functionality to work. > > Mike > > On Tuesday, May 15, 2012 at 8:37 AM, Mike M wrote: > > I'm running into an issue where trying to use an escape sequence with > an HDFS sink causes an exception. For example, if I have the sink > configured like: > > tail-nginx.sinks.HDFS.hdfs.path = hdfs://localhost:8020/flume/%Y/%m/%d/ > > I see the following exception showing up thousands of times: > > 2012-05-14 16:20:15,730 INFO source.ExecSource: Exec source starting > with command:tail -F /var/log/nginx/localhost.access.log > 2012-05-14 16:20:15,743 ERROR hdfs.HDFSEventSink: process failed > java.lang.NumberFormatException: null > at java.lang.Long.parseLong(Long.java:404) > at java.lang.Long.valueOf(Long.java:540) > at > org.apache.flume.formatter.output.BucketPath.replaceShorthand(BucketPath.java:185) > at > org.apache.flume.formatter.output.BucketPath.escapeString(BucketPath.java:219) > at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:344) > at > org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68) > at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147) > at java.lang.Thread.run(Thread.java:722) > 2012-05-14 16:20:15,745 ERROR flume.SinkRunner: Unable to deliver > event. Exception follows. > org.apache.flume.EventDeliveryException: java.lang.NumberFormatException: > null > at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:407) > at > org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68) > at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147) > at java.lang.Thread.run(Thread.java:722) > Caused by: java.lang.NumberFormatException: null > at java.lang.Long.parseLong(Long.java:404) > at java.lang.Long.valueOf(Long.java:540) > at > org.apache.flume.formatter.output.BucketPath.replaceShorthand(BucketPath.java:185) > at > org.apache.flume.formatter.output.BucketPath.escapeString(BucketPath.java:219) > at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:344) > ... 3 more > > I've tried various methods of quoting the escape sequences, to no > avail. When the escape sequences are removed, Flume works as > expected. Is this just a configuration issue on my side, or is there > possibly a bug? I'm running Flume-NG 1.1.0 packaged with CDH3u4. > > Thanks! > > Mike > >
