Thanks! I didn't remember the source does that :) Actually, the best way to insert your own timestamp is using an interceptor - because the custom code will need to get the information from within the Kafka message. Your interceptor can overwrite the default header with the timestamp you prefer.
On Tue, Apr 14, 2015 at 10:15 AM, Tao Li <[email protected]> wrote: > @Gwen Shapira, I think the feature customarize timestamp is useful, > especialy with HDFSEventSink. Do you consider to add this feature in the > future? > > 2015-04-15 0:55 GMT+08:00 Tao Li <[email protected]>: >> >> Code like this, KafkaSource will simply put "System.currentTimeMillis()" >> as "timestamp" >> >> 2015-04-15 0:52 GMT+08:00 Gwen Shapira <[email protected]>: >>> >>> Are you sure KafkaSource does that? Or do you have an Interceptor? >>> >>> On Tue, Apr 14, 2015 at 9:50 AM, Tao Li <[email protected]> wrote: >>> > Hi all: >>> > >>> > I found that KafkaSource will put "System.currentTimeMillis()" as >>> > "timestamp" field in header. If we use HDFSEventSink, the HDFS log can >>> > be >>> > rotate by the timestamp. >>> > >>> > But the web server log usually contains "timestamp" string, and it >>> > indicate >>> > the true timestamp, other than "System.currentTimeMillis()". So I think >>> > if >>> > have a feature to allow to customarize the "timestamp" field, will be >>> > better. >> >> >
