What if the negative timestamp is stored this way ? Long.MIN_VALUE + delta (where delta is positvie)
and calculated this way when used: 1/1/1970 - delta This approach avoids the ambiguity of -1 timestamp since -1 would be stored as Long.MIN_VALUE+1 Log retention can handle such format with minor modification. Just my two cents. On Tue, Dec 5, 2017 at 4:35 PM, Dong Lin <lindon...@gmail.com> wrote: > Hey Konstantin, > > According to KIP-32 the timestamp is also used for log rolling and log > retention. Therefore, unless broker is configured to never delete any > message based on time, messages produced with negative timestamp in your > use-case will be deleted by the broker anyway. Do you actually plan to use > Kafka as a persistent storage system that never delete messages? > > Thanks, > Dong > > > > > On Tue, Dec 5, 2017 at 1:24 PM, Konstantin Chukhlomin <chuhlo...@gmail.com > > > wrote: > > > Hi Dong, > > > > Currently we are storing historical timestamp in the message. > > > > What we are trying to achieve is to make it possible to do Kafka lookup > > by timestamp. Ideally I would do `offsetsForTimes` to find articles > > published > > in 1910s (if we are storing articles on the log). > > > > So first two suggestions aren't really covering our use-case. > > > > We could create a new timestamp type like "HistoricalTimestamp" or > > "MaybeNegativeTimestamp". > > And the only difference between this one and CreateTime is that it could > > be negative. > > I tend to use CreateTime for this purpose because it's easier to > > understand from > > user perspective as a timestamp which publisher can set. > > > > Thanks, > > Konstantin > > > > > On Dec 5, 2017, at 3:47 PM, Dong Lin <lindon...@gmail.com> wrote: > > > > > > Hey Konstantin, > > > > > > Thanks for the KIP. I have a few questions below. > > > > > > Strictly speaking Kafka actually allows you to store historical data. > And > > > user are free to encode arbitrary timestamp field in their Kafka > message. > > > For example, your Kafka message can currently have Json or Avro format > > and > > > you can put a timestamp field there. Do you think that could address > your > > > use-case? > > > > > > Alternatively, KIP-82 introduced Record Header in Kafka and you can > also > > > define your customized key/value pair in the header. Do you think this > > can > > > address your use-case? > > > > > > Also, currently there are two types of timestamp according to KIP-32. > If > > > the type is LogAppendTime then the timestamp value is the time when > > broker > > > receives the message. If the type is CreateTime then the timestamp > value > > is > > > determined when producer produces message. With these two definitions, > > the > > > timestamp should always be positive. We probably need a new type here > if > > we > > > can not put timestamp in the Record Header or the message payload. Does > > > this sound reasonable? > > > > > > Thanks, > > > Dong > > > > > > > > > > > > On Tue, Dec 5, 2017 at 8:40 AM, Konstantin Chukhlomin < > > chuhlo...@gmail.com> > > > wrote: > > > > > >> Hi all, > > >> > > >> I have created a KIP to support negative timestamp: > > >> https://cwiki.apache.org/confluence/display/KAFKA/KIP- > > >> 228+Negative+record+timestamp+support <https://cwiki.apache.org/ > > >> confluence/display/KAFKA/KIP-228+Negative+record+timestamp+support> > > >> > > >> Here are proposed changes: https://github.com/apache/ > > >> kafka/compare/trunk...chuhlomin:trunk <https://github.com/apache/ > > >> kafka/compare/trunk...chuhlomin:trunk> > > >> > > >> I'm pretty sure that not cases are covered, so comments and > suggestions > > >> are welcome. > > >> > > >> Thank you, > > >> Konstantin > > > > >
