[
https://issues.apache.org/jira/browse/KAFKA-19948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhangtongr updated KAFKA-19948:
-------------------------------
Description:
- Kafka version: 2.7.1
- Topic: `mytopic01`, 48 partitions, replication factor 2
- Topic config: `cleanup.policy=delete`, `retention.ms=86400000` (1 day)
- Observed behavior:
- Each partition has data {*}older than 7 days{*}{*}, Each partition{*}
*40GB* , and segments are *{*}never deleted{*}*.
- Other topics in the same cluster (including `__consumer_offsets`) are
deleted as expected under the delete policy.
- `kafka-topics.sh --describe --topic mytopic01` shows the ISR is *{*}fully
in-sync{*}*
1. what are the possible conditions under which a specific topic’s segments
might never become eligible for deletion?
2. For Kafka 2.7.1 specifically, what are the recommended debugging and
verification steps to diagnose this?
I am mainly looking for a list of likely causes and practical troubleshooting
steps to apply in production when a single topic does not respect delete-based
retention while others do.
was:
- Kafka version: 2.7.1
- Topic: `mytopic01`, 48 partitions, replication factor 2
- Topic config: `cleanup.policy=delete`, `retention.ms=86400000` (1 day)
- Observed behavior:
- Each partition has data **older than 7 days**, ~**40GB** in total, and
segments are **never deleted**.
- Other topics in the same cluster (including `__consumer_offsets`) are
deleted as expected under the delete policy.
- `kafka-topics.sh --describe --topic mytopic01` shows the ISR is **fully
in-sync**
1. what are the possible conditions under which a specific topic’s segments
might never become eligible for deletion?
2. For Kafka 2.7.1 specifically, what are the recommended debugging and
verification steps to diagnose this?
I am mainly looking for a list of likely causes and practical troubleshooting
steps to apply in production when a single topic does not respect delete-based
retention while others do.
> Single topic not deleting old segments with cleanup.policy=delete +
> retention.ms applied
> ----------------------------------------------------------------------------------------
>
> Key: KAFKA-19948
> URL: https://issues.apache.org/jira/browse/KAFKA-19948
> Project: Kafka
> Issue Type: Bug
> Components: log cleaner
> Affects Versions: 2.7.1
> Reporter: zhangtongr
> Priority: Major
>
> - Kafka version: 2.7.1
> - Topic: `mytopic01`, 48 partitions, replication factor 2
> - Topic config: `cleanup.policy=delete`, `retention.ms=86400000` (1 day)
> - Observed behavior:
> - Each partition has data {*}older than 7 days{*}{*}, Each partition{*}
> *40GB* , and segments are *{*}never deleted{*}*.
> - Other topics in the same cluster (including `__consumer_offsets`) are
> deleted as expected under the delete policy.
> - `kafka-topics.sh --describe --topic mytopic01` shows the ISR is *{*}fully
> in-sync{*}*
> 1. what are the possible conditions under which a specific topic’s segments
> might never become eligible for deletion?
> 2. For Kafka 2.7.1 specifically, what are the recommended debugging and
> verification steps to diagnose this?
> I am mainly looking for a list of likely causes and practical troubleshooting
> steps to apply in production when a single topic does not respect
> delete-based retention while others do.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
