It does http://flume.apache.org/releases/1.1.0.html but the root of the issue is that the source doesn't add the timestamp header to the event. Also 1.2 has just been released which has Interceptors which I think can add the timestamp for you.
Brock On Thu, Jul 26, 2012 at 12:59 PM, khadar basha <[email protected]> wrote: > Hi Brock. > > flume-1.1.0 doesn't support this escape sequence options? > > -Khadar > > On Thu, Jul 26, 2012 at 10:34 PM, Brock Noland <[email protected]> wrote: > >> The flume user doc here http://flume.apache.org/releases/1.2.0.html has >> many examples. >> >> Brock >> >> >> On Thu, Jul 26, 2012 at 11:37 AM, khadar basha <[email protected]>wrote: >> >>> Hi Brock, >>> >>> could you please provide me the sample configuration? >>> >>> Thanks, >>> Khadar >>> >>> >>> On Thu, Jul 26, 2012 at 7:23 PM, Brock Noland <[email protected]>wrote: >>> >>>> If you want to use a bucketed paths your events need to have a >>>> "timestamp" header which tail source does not appear to add. >>>> >>>> Brock >>>> >>>> On Thu, Jul 26, 2012 at 6:32 AM, khadar basha <[email protected]>wrote: >>>> >>>>> >>>>> Hi >>>>> >>>>> we are using flume-1.1.0-cdh4.0.1 . we have the following >>>>> configurations in flume-conf.properties. >>>>> >>>>> agent1Test1.sources = seqGenSrc >>>>> agent1Test1.channels = memoryChannel >>>>> agent1Test1.sinks = loggerSink >>>>> >>>>> # For each one of the sources, the type is defined >>>>> agent1Test1.sources.seqGenSrc.type = exec >>>>> agent1Test1.sources.seqGenSrc.command= tail -f >>>>> /tomcat/logs/catalina.out >>>>> >>>>> >>>>> # The channel can be defined as follows. >>>>> agent1Test1.sources.seqGenSrc.channels = memoryChannel >>>>> >>>>> # Each sink's type must be defined >>>>> agent1Test1.sinks.loggerSink.type = hdfs >>>>> agent1Test1.sinks.loggerSink.hdfs.path = >>>>> hdfs://hdfshost:port/data/%Y/%m/%{host}/mylogs >>>>> >>>>> agent1Test1.sinks.loggerSink.hdfs.fileType = DataStream >>>>> >>>>> #Specify the channel the sink should use >>>>> agent1Test1.sinks.loggerSink.channel = memoryChannel >>>>> >>>>> # Each channel's type is defined. >>>>> agent1Test1.channels.memoryChannel.type = memory >>>>> >>>>> # Other config values specific to each type of channel(sink or source) >>>>> # can be defined as well >>>>> # In this case, it specifies the capacity of the memory channel >>>>> agent1Test1.channels.memoryChannel.capacity = 1000 >>>>> >>>>> >>>>> I am getting the following exception. Anybody facing this issue. >>>>> >>>>> >>>>> *2012-07-26 13:07:52,002 ERROR hdfs.HDFSEventSink: process failed* >>>>> *java.lang.NumberFormatException: null* >>>>> * at java.lang.Long.parseLong(Long.java:375)* >>>>> * at java.lang.Long.valueOf(Long.java:525)* >>>>> * at >>>>> org.apache.flume.formatter.output.BucketPath.replaceShorthand(BucketPath.java:220) >>>>> * >>>>> * at >>>>> org.apache.flume.formatter.output.BucketPath.escapeString(BucketPath.java:310) >>>>> * >>>>> * at >>>>> org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:402) >>>>> * >>>>> * at >>>>> org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68) >>>>> * >>>>> * at >>>>> org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)* >>>>> * at java.lang.Thread.run(Thread.java:662)* >>>>> *2012-07-26 13:07:52,003 ERROR flume.SinkRunner: Unable to deliver >>>>> event. Exception follows.* >>>>> *org.apache.flume.EventDeliveryException: >>>>> java.lang.NumberFormatException: null* >>>>> * at >>>>> org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:469) >>>>> * >>>>> * at >>>>> org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68) >>>>> * >>>>> * at >>>>> org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)* >>>>> * at java.lang.Thread.run(Thread.java:662)* >>>>> *Caused by: java.lang.NumberFormatException: null* >>>>> * at java.lang.Long.parseLong(Long.java:375)* >>>>> * at java.lang.Long.valueOf(Long.java:525)* >>>>> * at >>>>> org.apache.flume.formatter.output.BucketPath.replaceShorthand(BucketPath.java:220) >>>>> * >>>>> * at >>>>> org.apache.flume.formatter.output.BucketPath.escapeString(BucketPath.java:310) >>>>> * >>>>> * at >>>>> org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:402) >>>>> * >>>>> * ... 3 more* >>>>> >>>>> -- >>>>> Thanks, >>>>> Khadar >>>>> >>>>> >>>> >>>> >>>> -- >>>> Apache MRUnit - Unit testing MapReduce - >>>> http://incubator.apache.org/mrunit/ >>>> >>> >>> >>> >>> -- >>> Thanks, >>> Khadar >>> >>> >> >> >> -- >> Apache MRUnit - Unit testing MapReduce - >> http://incubator.apache.org/mrunit/ >> > > > > -- > Thanks, > Khadar > > -- Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/
