[
https://issues.apache.org/jira/browse/KAFKA-4340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16023668#comment-16023668
]
Magnus Edenhill commented on KAFKA-4340:
----------------------------------------
Generally I would agree, but in this case I don't think there is a practical
difference to discarding outdated messages, or removing them with the retention
cleaner, it is just a matter of time frame - discarding and not appending
outdated messages to the logs is immediate, while the retention cleaner might
kick in immediately or in whatever max interval it is configured to.
So from the producer and consumer perspectives the end result is pretty much
the same: there is no guarantee that an outdated message will be seen by a
consumer.
However, rejecting the entire batch means there will be guaranteed data loss in
the message stream: the producer will not try to re-send those failed messages,
even if all but one message in the batch were actually okay. I strongly feel
this is undesired behaviour from the application's point of view.
> Change the default value of log.message.timestamp.difference.max.ms to the
> same as log.retention.ms
> ---------------------------------------------------------------------------------------------------
>
> Key: KAFKA-4340
> URL: https://issues.apache.org/jira/browse/KAFKA-4340
> Project: Kafka
> Issue Type: Improvement
> Components: core
> Affects Versions: 0.10.1.0
> Reporter: Jiangjie Qin
> Assignee: Jiangjie Qin
> Fix For: 0.11.0.0
>
>
> [~junrao] brought up the following scenario:
> If users are pumping data with timestamp already passed log.retention.ms into
> Kafka, the messages will be appended to the log but will be immediately
> rolled out by log retention thread when it kicks in and the messages will be
> deleted.
> To avoid this produce-and-deleted scenario, we can set the default value of
> log.message.timestamp.difference.max.ms to be the same as log.retention.ms.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
