lucliu1108 commented on code in PR #22489: URL: https://github.com/apache/kafka/pull/22489#discussion_r3364852958
########## docs/design/design.md: ########## @@ -454,7 +454,11 @@ This can be used to prevent messages newer than a minimum message age from being log.cleaner.max.compaction.lag.ms ``` -This can be used to prevent log with low produce rate from remaining ineligible for compaction for an unbounded duration. If not set, logs that do not exceed min.cleanable.dirty.ratio are not compacted. Note that this compaction deadline is not a hard guarantee since it is still subjected to the availability of log cleaner threads and the actual compaction time. You will want to monitor the uncleanable-partitions-count, max-clean-time-secs and max-compaction-delay-secs metrics. +This can be used to prevent log with low produce rate from remaining ineligible for compaction for an unbounded duration. If not set, logs that do not exceed min.cleanable.dirty.ratio are not compacted. + +Because the active segment is never compacted (as noted above), records become eligible for compaction only through active segment rolling. For a compacted topic the active segment is rolled when the first of these is reached: it grows to segment.bytes, or its age reaches the smaller of segment.ms and max.compaction.lag.ms. So max.compaction.lag.ms governs two distinct things. First, it triggers active segment rolling by lowering the effective time-based roll deadline to the smaller of segment.ms and max.compaction.lag.ms, moving older records out of the active segment. This active segment rolling is evaluated when records are appended, so a partition that has stopped receiving writes will not roll its active segment until the next append. Second, max.compaction.lag.ms then makes the rolled records eligible for compaction even when the log does not exceed min.cleanable.dirty.ratio. Review Comment: > This active segment rolling is evaluated when records are appended The size/time roll check in maybeRoll is indeed only invoked from the append path, but the active segment can still be rolled from the retention path without an append. Illustrating all these situations might be too much for the doc We could consider dropping the sentence, or narrowing it to the operator-facing consequence, e.g. "lowering max.compaction.lag.ms won't force-roll an idle partition; a new produce is needed before the dirty records become eligible for compaction." -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
