bq. you can add X to the timestamp before you produce Kafka message This assumes the earliest timestamp (for user application) is known beforehand.
However, what if this earliest timestamp shifts even earlier (e.g. due to some discovery) ? Cheers On Tue, Dec 5, 2017 at 6:36 PM, Dong Lin <lindon...@gmail.com> wrote: > Hey Boerge, > > Thanks for the blog link. I will read this blog later. > > Here is another alternative solution which may be worth thinking. We know > that the Unix time 0 corresponds to January 1, 1970. Let's say the earliest > time you may want to use as the timestamp of the Kafka message is within X > milliseconds before the January 1, 1970. Then you can add X to the > timestamp before you produce Kafka message. And you can also make similar > conversion when you use `offsetsForTimes()` or after you consume messages. > This seems to address your use-case without introducing negative timestamp. > > IMO, this solution requires a bit more logic in your application code. But > it keeps the Kafka timestamp logic simple and we reserve the capability to > use timestamp -1 for messages without timestamp for most Kafka users who do > not need negative timestamp. Do you think this would be a good alternative > solution? > > Thanks, > Dong > > > On Tue, Dec 5, 2017 at 5:39 PM, Boerge Svingen <bsvin...@borkdal.com> > wrote: > > > > > Yes. To provide a little more detail, we are using Kafka to store > > everything ever published by The New York Times, and to make this content > > available to a range of systems and applications. Assets are published to > > Kafka chronologically, so that consumers can seek to any point in time > and > > start consuming from there, like Konstantin is describing, all the way > back > > to our beginning in 1851. > > > > https://www.confluent.io/blog/publishing-apache-kafka-new-york-times/ < > > https://www.confluent.io/blog/publishing-apache-kafka-new-york-times/> > > has more information on the use case. > > > > > > Thanks, > > Boerge. > > > > > > -- > > > > Boerge Svingen > > Director of Engineering > > The New York Times > > > > > > > > > > > On 2017-12-05, at 19:35, Dong Lin <lindon...@gmail.com> wrote: > > > > > > Hey Konstantin, > > > > > > According to KIP-32 the timestamp is also used for log rolling and log > > > retention. Therefore, unless broker is configured to never delete any > > > message based on time, messages produced with negative timestamp in > your > > > use-case will be deleted by the broker anyway. Do you actually plan to > > use > > > Kafka as a persistent storage system that never delete messages? > > > > > > Thanks, > > > Dong > > > > > > > > > > > > > > > On Tue, Dec 5, 2017 at 1:24 PM, Konstantin Chukhlomin < > > chuhlo...@gmail.com> > > > wrote: > > > > > >> Hi Dong, > > >> > > >> Currently we are storing historical timestamp in the message. > > >> > > >> What we are trying to achieve is to make it possible to do Kafka > lookup > > >> by timestamp. Ideally I would do `offsetsForTimes` to find articles > > >> published > > >> in 1910s (if we are storing articles on the log). > > >> > > >> So first two suggestions aren't really covering our use-case. > > >> > > >> We could create a new timestamp type like "HistoricalTimestamp" or > > >> "MaybeNegativeTimestamp". > > >> And the only difference between this one and CreateTime is that it > could > > >> be negative. > > >> I tend to use CreateTime for this purpose because it's easier to > > >> understand from > > >> user perspective as a timestamp which publisher can set. > > >> > > >> Thanks, > > >> Konstantin > > >> > > >>> On Dec 5, 2017, at 3:47 PM, Dong Lin <lindon...@gmail.com> wrote: > > >>> > > >>> Hey Konstantin, > > >>> > > >>> Thanks for the KIP. I have a few questions below. > > >>> > > >>> Strictly speaking Kafka actually allows you to store historical data. > > And > > >>> user are free to encode arbitrary timestamp field in their Kafka > > message. > > >>> For example, your Kafka message can currently have Json or Avro > format > > >> and > > >>> you can put a timestamp field there. Do you think that could address > > your > > >>> use-case? > > >>> > > >>> Alternatively, KIP-82 introduced Record Header in Kafka and you can > > also > > >>> define your customized key/value pair in the header. Do you think > this > > >> can > > >>> address your use-case? > > >>> > > >>> Also, currently there are two types of timestamp according to KIP-32. > > If > > >>> the type is LogAppendTime then the timestamp value is the time when > > >> broker > > >>> receives the message. If the type is CreateTime then the timestamp > > value > > >> is > > >>> determined when producer produces message. With these two > definitions, > > >> the > > >>> timestamp should always be positive. We probably need a new type here > > if > > >> we > > >>> can not put timestamp in the Record Header or the message payload. > Does > > >>> this sound reasonable? > > >>> > > >>> Thanks, > > >>> Dong > > >>> > > >>> > > >>> > > >>> On Tue, Dec 5, 2017 at 8:40 AM, Konstantin Chukhlomin < > > >> chuhlo...@gmail.com> > > >>> wrote: > > >>> > > >>>> Hi all, > > >>>> > > >>>> I have created a KIP to support negative timestamp: > > >>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP- > > >>>> 228+Negative+record+timestamp+support <https://cwiki.apache.org/ > > >>>> confluence/display/KAFKA/KIP-228+Negative+record+timestamp+support> > > >>>> > > >>>> Here are proposed changes: https://github.com/apache/ > > >>>> kafka/compare/trunk...chuhlomin:trunk <https://github.com/apache/ > > >>>> kafka/compare/trunk...chuhlomin:trunk> > > >>>> > > >>>> I'm pretty sure that not cases are covered, so comments and > > suggestions > > >>>> are welcome. > > >>>> > > >>>> Thank you, > > >>>> Konstantin > > >> > > >> > > > > >
