[
https://issues.apache.org/jira/browse/KAFKA-20091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
fujian updated KAFKA-20091:
---------------------------
Description:
When currentTime- largestTimestamp of segment = retentionMs, the behavior is
different:
# Local Segment delete: *Not delete*
org.apache.kafka.storage.internals.log.UnifiedLog#deleteRetentionMsBreachedSegments:
boolean delete = startMs - segment.largestTimestamp() > retentionMs;
2. Remote segement delete: *Delete*
org.apache.kafka.server.log.remote.storage.RemoteLogManager.RLMExpirationTask.RemoteLogRetentionHandler#isSegmentBreachedByRetentionTime
shouldDeleteSegment = metadata.maxTimestampMs() <=
retentionTimeData.get().cleanupUntilMs;
//cleanupUntilMs is time.milliseconds() - retentionMs;
private Optional<RetentionTimeData> buildRetentionTimeData(long retentionMs) {
long cleanupUntilMs = time.milliseconds() - retentionMs;
return retentionMs > -1 && cleanupUntilMs >= 0
? Optional.of(new RetentionTimeData(retentionMs, cleanupUntilMs))
: Optional.empty();
}
was:
When currentTime- largestTimestamp of segment = retentionMs, the behavior is
different:
# Local Segment delete: *Not delete*
org.apache.kafka.storage.internals.log.UnifiedLog#deleteRetentionMsBreachedSegments:
boolean delete = startMs - segment.largestTimestamp() > retentionMs;
# Remote segement delete: *Delete*
org.apache.kafka.server.log.remote.storage.RemoteLogManager.RLMExpirationTask.RemoteLogRetentionHandler#isSegmentBreachedByRetentionTime
shouldDeleteSegment = metadata.maxTimestampMs() <=
retentionTimeData.get().cleanupUntilMs;
//cleanupUntilMs is time.milliseconds() - retentionMs;
private Optional<RetentionTimeData> buildRetentionTimeData(long retentionMs) \{
long cleanupUntilMs = time.milliseconds() - retentionMs;
return retentionMs > -1 && cleanupUntilMs >= 0
? Optional.of(new RetentionTimeData(retentionMs, cleanupUntilMs))
: Optional.empty();
}{{}}
> Fix inconsistency in time-based retention checks between remote and local
> segment deletion logic.
> --------------------------------------------------------------------------------------------------
>
> Key: KAFKA-20091
> URL: https://issues.apache.org/jira/browse/KAFKA-20091
> Project: Kafka
> Issue Type: Bug
> Components: Tiered-Storage
> Reporter: fujian
> Assignee: fujian
> Priority: Minor
>
> When currentTime- largestTimestamp of segment = retentionMs, the behavior is
> different:
> # Local Segment delete: *Not delete*
> org.apache.kafka.storage.internals.log.UnifiedLog#deleteRetentionMsBreachedSegments:
> boolean delete = startMs - segment.largestTimestamp() > retentionMs;
> 2. Remote segement delete: *Delete*
> org.apache.kafka.server.log.remote.storage.RemoteLogManager.RLMExpirationTask.RemoteLogRetentionHandler#isSegmentBreachedByRetentionTime
> shouldDeleteSegment = metadata.maxTimestampMs() <=
> retentionTimeData.get().cleanupUntilMs;
> //cleanupUntilMs is time.milliseconds() - retentionMs;
> private Optional<RetentionTimeData> buildRetentionTimeData(long retentionMs) {
> long cleanupUntilMs = time.milliseconds() - retentionMs;
> return retentionMs > -1 && cleanupUntilMs >= 0
> ? Optional.of(new RetentionTimeData(retentionMs, cleanupUntilMs))
> : Optional.empty();
> }
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
