junrao commented on code in PR #22873:
URL: https://github.com/apache/kafka/pull/22873#discussion_r4106463939
##########
storage/src/main/java/org/apache/kafka/storage/internals/log/UnifiedLog.java:
##########
@@ -2000,39 +2000,34 @@ public interface DeletionCondition {
}
private int deleteRetentionMsBreachedSegments() throws IOException {
- long retentionMs = UnifiedLog.localRetentionMs(config(),
remoteLogEnabledAndRemoteCopyEnabled());
+ boolean remoteLogEnabledAndRemoteCopyEnabled =
remoteLogEnabledAndRemoteCopyEnabled();
+ long retentionMs = UnifiedLog.localRetentionMs(config(),
remoteLogEnabledAndRemoteCopyEnabled);
if (retentionMs < 0) return 0;
long startMs = time().milliseconds();
DeletionCondition shouldDelete = (segment, nextSegmentOpt) -> {
- if (startMs < segment.largestTimestamp()) {
- futureTimestampLogger.warn("{} contains future timestamp(s),
making it ineligible to be deleted", segment);
+ long anchorTimestamp = segment.largestTimestamp();
+ if (startMs < anchorTimestamp) {
+ if (remoteLogEnabledAndRemoteCopyEnabled) {
+ anchorTimestamp = segment.lastModified();
Review Comment:
The problem with using lastModified is that it can still delay the deletion
of local segments and it feels a bit hacky.
It would be useful to think through the semantic of local retention time.
Currently, the semantic is based on the event time. We could change the
semantic to "local segment is retained for local retention time after it's
uploaded to tier storage". Does that solve the problem more completely?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
