Hey Konstantin, According to KIP-32 the timestamp is also used for log rolling and log retention. Therefore, unless broker is configured to never delete any message based on time, messages produced with negative timestamp in your use-case will be deleted by the broker anyway. Do you actually plan to use Kafka as a persistent storage system that never delete messages?
Thanks, Dong On Tue, Dec 5, 2017 at 1:24 PM, Konstantin Chukhlomin <chuhlo...@gmail.com> wrote: > Hi Dong, > > Currently we are storing historical timestamp in the message. > > What we are trying to achieve is to make it possible to do Kafka lookup > by timestamp. Ideally I would do `offsetsForTimes` to find articles > published > in 1910s (if we are storing articles on the log). > > So first two suggestions aren't really covering our use-case. > > We could create a new timestamp type like "HistoricalTimestamp" or > "MaybeNegativeTimestamp". > And the only difference between this one and CreateTime is that it could > be negative. > I tend to use CreateTime for this purpose because it's easier to > understand from > user perspective as a timestamp which publisher can set. > > Thanks, > Konstantin > > > On Dec 5, 2017, at 3:47 PM, Dong Lin <lindon...@gmail.com> wrote: > > > > Hey Konstantin, > > > > Thanks for the KIP. I have a few questions below. > > > > Strictly speaking Kafka actually allows you to store historical data. And > > user are free to encode arbitrary timestamp field in their Kafka message. > > For example, your Kafka message can currently have Json or Avro format > and > > you can put a timestamp field there. Do you think that could address your > > use-case? > > > > Alternatively, KIP-82 introduced Record Header in Kafka and you can also > > define your customized key/value pair in the header. Do you think this > can > > address your use-case? > > > > Also, currently there are two types of timestamp according to KIP-32. If > > the type is LogAppendTime then the timestamp value is the time when > broker > > receives the message. If the type is CreateTime then the timestamp value > is > > determined when producer produces message. With these two definitions, > the > > timestamp should always be positive. We probably need a new type here if > we > > can not put timestamp in the Record Header or the message payload. Does > > this sound reasonable? > > > > Thanks, > > Dong > > > > > > > > On Tue, Dec 5, 2017 at 8:40 AM, Konstantin Chukhlomin < > chuhlo...@gmail.com> > > wrote: > > > >> Hi all, > >> > >> I have created a KIP to support negative timestamp: > >> https://cwiki.apache.org/confluence/display/KAFKA/KIP- > >> 228+Negative+record+timestamp+support <https://cwiki.apache.org/ > >> confluence/display/KAFKA/KIP-228+Negative+record+timestamp+support> > >> > >> Here are proposed changes: https://github.com/apache/ > >> kafka/compare/trunk...chuhlomin:trunk <https://github.com/apache/ > >> kafka/compare/trunk...chuhlomin:trunk> > >> > >> I'm pretty sure that not cases are covered, so comments and suggestions > >> are welcome. > >> > >> Thank you, > >> Konstantin > >
