[
https://issues.apache.org/jira/browse/KAFKA-6048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16221014#comment-16221014
]
Børge Svingen commented on KAFKA-6048:
--------------------------------------
@james.c:
We are trying to resolve the same problem at The New York Times.
Maybe we can work together on this?
> Support negative record timestamps
> ----------------------------------
>
> Key: KAFKA-6048
> URL: https://issues.apache.org/jira/browse/KAFKA-6048
> Project: Kafka
> Issue Type: Improvement
> Components: clients, core, streams
> Affects Versions: 1.0.0
> Reporter: Matthias J. Sax
> Assignee: james chien
> Labels: needs-kip
>
> Kafka does not support negative record timestamps, and this prevents the
> storage of historical data in Kafka. In general, negative timestamps are
> supported by UNIX system time stamps:
> From https://en.wikipedia.org/wiki/Unix_time
> {quote}
> The Unix time number is zero at the Unix epoch, and increases by exactly
> 86,400 per day since the epoch. Thus 2004-09-16T00:00:00Z, 12,677 days after
> the epoch, is represented by the Unix time number 12,677 × 86,400 =
> 1095292800. This can be extended backwards from the epoch too, using negative
> numbers; thus 1957-10-04T00:00:00Z, 4,472 days before the epoch, is
> represented by the Unix time number −4,472 × 86,400 = −386380800.
> {quote}
> Allowing for negative timestamps would require multiple changes:
> - while brokers in general do support negative timestamps, broker use {{-1}}
> as default value if a producer uses an old message format (this would not be
> compatible with supporting negative timestamps "end-to-end" as {{-1}} cannot
> be used as "unknown" anymore): we could introduce a message flag indicating a
> missing timestamp (and let producer throw an exception if
> {{ConsumerRecord#timestamp()}} is called. Another possible solution might be,
> to require topics that are used by old producers to be configured with
> {{LogAppendTime}} semantics and rejecting writes to topics with
> {{CreateTime}} semantics for older message formats
> - {{KafkaProducer}} does not allow to send records with negative timestamp
> and thus this would need to be fixed
> - Streams API does drop records with negative timestamps (or fails by
> default) -- also, some internal store implementation for windowed stores
> assume that there are not negative timestamps to do range queries
> There might be other gaps we need to address. This is just a summary of issue
> coming to my mind atm.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
