[
https://issues.apache.org/jira/browse/KAFKA-16414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17847876#comment-17847876
]
Kamal Chandraprakash edited comment on KAFKA-16414 at 5/20/24 1:14 PM:
-----------------------------------------------------------------------
We may need a discussion before changing the behavior. One issue that can come
is if the user sets the below configs:
{code:java}
segment.bytes = 1 GB
local.retention.bytes = 1 KB
{code}
Going by this JIRA, we will rotate the active segment and mark it eligible for
local-log deletion. This can lead to huge number of smaller-size segments which
impacts the RemoteLogMetadataManager as we have to maintain the metadata for
all the segments. Impact on the Remote Storage Manager can be increase in the
number of file descriptors etc.
was (Author: ckamal):
We may need a discussion before changing the behavior. One issue that can come
is if the user sets the below configs:
{code:java}
segment.bytes = 1 GB
local.retention.bytes = 1 KB
{code}
Going by this JIRA, we will rotate the active segment and mark it eligible for
local-log deletion. This can lead to huge number of smaller-size segments which
impacts the RemoteLogMetadataManager as we have to maintain the metadata for
all the segments.
> Inconsistent active segment expiration behavior between retention.ms and
> retention.bytes
> ----------------------------------------------------------------------------------------
>
> Key: KAFKA-16414
> URL: https://issues.apache.org/jira/browse/KAFKA-16414
> Project: Kafka
> Issue Type: Improvement
> Affects Versions: 3.6.1
> Reporter: Kuan Po Tseng
> Assignee: Kuan Po Tseng
> Priority: Major
>
> This is a follow up issue on KAFKA-16385.
> Currently, there's a difference between how retention.ms and retention.bytes
> handle active segment expiration:
> - retention.ms always expire active segment when max segment timestamp
> matches the condition.
> - retention.bytes only expire active segment when retention.bytes is
> configured to zero.
> The behavior should be either rotate active segments for both retention
> configurations or none at all.
> For more details, see
> https://issues.apache.org/jira/browse/KAFKA-16385?focusedCommentId=17829682&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17829682
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
