[
https://issues.apache.org/jira/browse/KAFKA-8522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16874578#comment-16874578
]
Jun Rao commented on KAFKA-8522:
--------------------------------
[~EeveeB], thanks for reporting this. This is related to KAFKA-4545.
Basically, the tombstone needs to be retained for at least delete.retention.ms
after the cleaning point (say T) has passed the offset of the tombstone.
However, the current estimation of time T is based on the retained last
modified time of the log segment, which is not accurate. This can cause the
tombstone to be retained too short or too long (even forever).
One way to fix that is to write the timestamp of time T in the tombstone record
when the cleaning point first passes the tombstone offset. We can then clean
the tombstone at a more accurate time.
> Tombstones can survive forever
> ------------------------------
>
> Key: KAFKA-8522
> URL: https://issues.apache.org/jira/browse/KAFKA-8522
> Project: Kafka
> Issue Type: Bug
> Components: log cleaner
> Reporter: Evelyn Bayes
> Priority: Minor
>
> This is a bit grey zone as to whether it's a "bug" but it is certainly
> unintended behaviour.
>
> Under specific conditions tombstones effectively survive forever:
> * Small amount of throughput;
> * min.cleanable.dirty.ratio near or at 0; and
> * Other parameters at default.
> What happens is all the data continuously gets cycled into the oldest
> segment. Old records get compacted away, but the new records continuously
> update the timestamp of the oldest segment reseting the countdown for
> deleting tombstones.
> So tombstones build up in the oldest segment forever.
>
> While you could "fix" this by reducing the segment size, this can be
> undesirable as a sudden change in throughput could cause a dangerous number
> of segments to be created.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
