[
https://issues.apache.org/jira/browse/KAFKA-10760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17619908#comment-17619908
]
Jun Rao commented on KAFKA-10760:
---------------------------------
A potential solution is to implement the time based segment rolling based on
the difference between the current time and the segment creation time. Since
java 7, we could use
[https://docs.oracle.com/javase/8/docs/api/java/nio/file/attribute/BasicFileAttributes.html#creationTime--]
to get segment create time.
> In compacted topic with max.compaction.lag.ms, the segments are not rolled
> until new messages arrive
> ----------------------------------------------------------------------------------------------------
>
> Key: KAFKA-10760
> URL: https://issues.apache.org/jira/browse/KAFKA-10760
> Project: Kafka
> Issue Type: Bug
> Components: core
> Reporter: Sarwar Bhuiyan
> Assignee: Brajesh Kumar
> Priority: Major
>
> Currently, if a compacted topic has min.cleanable.dirty.ratio set to
> something low and max.compaction.lag.ms set to a small time, according to KIP
> [https://cwiki.apache.org/confluence/display/KAFKA/KIP-354] the expectation
> is that the active segment will be rolled regardless or segment.ms or whether
> new data has come in to "advance" the time. However, in practice, the current
> implementation only rolls the segment when new data which means that there
> are situations where the topic is not fully compacted until new data arrives
> which may not be until a while later. The implementation can be improved by
> rolling the segment just purely based on the max.compaction.lag.ms setting.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
