Marc created KAFKA-13695:
----------------------------
Summary: Low-traffic topics don't roll (and therefore compact) nor
delete tombstones
Key: KAFKA-13695
URL: https://issues.apache.org/jira/browse/KAFKA-13695
Project: Kafka
Issue Type: Improvement
Affects Versions: 3.1.0, 2.5.0
Reporter: Marc
I set up a testbed with some partitions and inspected carefully the behaviour
on the Kafka /data folder.
It looks like when active segment qualifies for rolling it is not effectively
being closed until a new record arrives. Thus, it cannot be compacted in a
timely and deterministic manner by means of max.compaction.lag.ms, for instance.
Again the problem occurs when setting up delete.retention.ms. Once compaction
happened and the canonical latest state of a key is a unique tombstone on the
compacted tail, we must wait for an arbitrary record arrival in order for
deletion to be triggered, just as before.
I expected log.preallocate property to create a new segment file once we marked
the current one for rolling and we still got no new records, hoping it can be
rolled at last, but it seems it has nothing to do.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
