Hi all, I’d like to request a review for a small Tiered Storage bug fix.
*JIRA:* https://issues.apache.org/jira/browse/KAFKA-20977 *PR:* https://github.com/apache/kafka/pull/23246 *Summary* On Tiered Storage partitions with low enough throughput that a rolled segment contains a single record (baseOffset == endOffset), RemoteCopyLagSegments can get stuck reporting 1 (and RemoteCopyLagBytes reports that segment’s size), even though every closed segment has already been copied to remote storage and only the active segment remains local. *Root cause* UnifiedLog.onlyLocalLogSegmentsCount() / onlyLocalLogSegmentsSize() filter segments using baseOffset() >= highestOffsetInRemoteStorage(). Since highestOffsetInRemoteStorage represents the end offset of the last segment copied to remote storage, a single-record segment whose base offset equals that value is incorrectly counted as local. RemoteLogManager derives the metric as onlyLocalLogSegmentsCount() - 1 and only refreshes it after a copy, causing idle partitions to remain latched at a phantom lag of 1. This is a residual off-by-one issue on top of KAFKA-16895 and is related to KAFKA-19995 / KAFKA-20732. *Fix* Use a strict > comparison so the already-copied boundary segment is excluded. The PR includes: - A UnifiedLogTest regression test that fails with the old >= behavior. - A multi-record guard test. - Clean checkstyle and SpotBugs results. Would appreciate a review when someone has bandwidth. Happy to iterate on any feedback. Thanks, Avishek Das GitHub/JIRA: imavishek -- Thanks, *Avishek Das* <https://www.linkedin.com/in/imavishek/> Member Of Technical Staff At Salesforce <https://www.linkedin.com/company/salesforce/> *[email protected] <[email protected]>* Mobile: +91-7008383890, +91-8908904383
