[jira] [Updated] (KAFKA-19949) Single topic not deleting old segments with cleanup.policy=delete + retention.ms applied
[
https://issues.apache.org/jira/browse/KAFKA-19949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhangtongr updated KAFKA-19949:
---
Description:
- Kafka version: 2.7.1
- Topic: `mytopic01`, 48 partitions, replication factor 2
- Topic config: `cleanup.policy=delete`, `retention.ms=8640` (1 day)
- Observed behavior:
- Each partition has data {*}older than 7 days{*}{*}, Each partition{*}
*40GB* , and segments are never deleted.
- Other topics in the same cluster (including `__consumer_offsets`) are
deleted as expected under the delete policy.
- `kafka-topics.sh --describe --topic mytopic01` shows the ISR is fully
in-sync
1. what are the possible conditions under which a specific topic’s segments
might never become eligible for deletion?
2. For Kafka 2.7.1 specifically, what are the recommended debugging and
verification steps to diagnose this?
I am looking for possible causes of this issue and recommended troubleshooting
steps. Any experience or advice on why a single topic might not respect the
delete policy while other topics behave normally would be greatly appreciated.
> Single topic not deleting old segments with cleanup.policy=delete +
> retention.ms applied
>
>
> Key: KAFKA-19949
> URL: https://issues.apache.org/jira/browse/KAFKA-19949
> Project: Kafka
> Issue Type: Bug
> Components: log cleaner
>Affects Versions: 2.7.1
>Reporter: zhangtongr
>Priority: Major
>
> - Kafka version: 2.7.1
> - Topic: `mytopic01`, 48 partitions, replication factor 2
> - Topic config: `cleanup.policy=delete`, `retention.ms=8640` (1 day)
> - Observed behavior:
> - Each partition has data {*}older than 7 days{*}{*}, Each partition{*}
> *40GB* , and segments are never deleted.
> - Other topics in the same cluster (including `__consumer_offsets`) are
> deleted as expected under the delete policy.
> - `kafka-topics.sh --describe --topic mytopic01` shows the ISR is fully
> in-sync
> 1. what are the possible conditions under which a specific topic’s segments
> might never become eligible for deletion?
> 2. For Kafka 2.7.1 specifically, what are the recommended debugging and
> verification steps to diagnose this?
> I am looking for possible causes of this issue and recommended
> troubleshooting steps. Any experience or advice on why a single topic might
> not respect the delete policy while other topics behave normally would be
> greatly appreciated.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (KAFKA-19949) Single topic not deleting old segments with cleanup.policy=delete + retention.ms applied
[
https://issues.apache.org/jira/browse/KAFKA-19949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhangtongr updated KAFKA-19949:
---
Description: (was: - Kafka version: 2.7.1
- Topic: `mytopic01`, 48 partitions, replication factor 2
- Topic config: `cleanup.policy=delete`, `retention.ms=8640` (1 day)
- Observed behavior:
- Each partition has data {*}older than 7 days{*}{*}, Each partition{*}
{*}40GB{*}, and segments are {*}never deleted{*}.
- Other topics in the same cluster (including __consumer_offsets) are deleted
as expected under the delete policy.
- `kafka-topics.sh --describe --topic mytopic01` shows the ISR is *fully
in-sync*
1. what are the possible conditions under which a specific topic’s segments
might never become eligible for deletion?
2. For Kafka 2.7.1 specifically, what are the recommended debugging and
verification steps to diagnose this?
I am mainly looking for a list of likely causes and practical troubleshooting
steps to apply in production when a single topic does not respect delete-based
retention while others do.)
> Single topic not deleting old segments with cleanup.policy=delete +
> retention.ms applied
>
>
> Key: KAFKA-19949
> URL: https://issues.apache.org/jira/browse/KAFKA-19949
> Project: Kafka
> Issue Type: Bug
> Components: log cleaner
>Affects Versions: 2.7.1
>Reporter: zhangtongr
>Priority: Major
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (KAFKA-19949) Single topic not deleting old segments with cleanup.policy=delete + retention.ms applied
[
https://issues.apache.org/jira/browse/KAFKA-19949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhangtongr updated KAFKA-19949:
---
Description:
- Kafka version: 2.7.1
- Topic: `mytopic01`, 48 partitions, replication factor 2
- Topic config: `cleanup.policy=delete`, `retention.ms=8640` (1 day)
- Observed behavior:
- Each partition has data {*}older than 7 days{*}{*}, Each partition{*}
{*}40GB{*}, and segments are {*}never deleted{*}.
- Other topics in the same cluster (including __consumer_offsets) are deleted
as expected under the delete policy.
- `kafka-topics.sh --describe --topic mytopic01` shows the ISR is *fully
in-sync*
1. what are the possible conditions under which a specific topic’s segments
might never become eligible for deletion?
2. For Kafka 2.7.1 specifically, what are the recommended debugging and
verification steps to diagnose this?
I am mainly looking for a list of likely causes and practical troubleshooting
steps to apply in production when a single topic does not respect delete-based
retention while others do.
was:
- Kafka version: 2.7.1
- Topic: `mytopic01`, 48 partitions, replication factor 2
- Topic config: `cleanup.policy=delete`, `retention.ms=8640` (1 day)
- Observed behavior:
- Each partition has data **older than 7 days**, ~**40GB** in total, and
segments are **never deleted**.
- Other topics in the same cluster (including `__consumer_offsets`) are
deleted as expected under the delete policy.
- `kafka-topics.sh --describe --topic mytopic01` shows the ISR is **fully
in-sync**
1. what are the possible conditions under which a specific topic’s segments
might never become eligible for deletion?
2. For Kafka 2.7.1 specifically, what are the recommended debugging and
verification steps to diagnose this?
I am mainly looking for a list of likely causes and practical troubleshooting
steps to apply in production when a single topic does not respect delete-based
retention while others do.
> Single topic not deleting old segments with cleanup.policy=delete +
> retention.ms applied
>
>
> Key: KAFKA-19949
> URL: https://issues.apache.org/jira/browse/KAFKA-19949
> Project: Kafka
> Issue Type: Bug
> Components: log cleaner
>Affects Versions: 2.7.1
>Reporter: zhangtongr
>Priority: Major
>
> - Kafka version: 2.7.1
> - Topic: `mytopic01`, 48 partitions, replication factor 2
> - Topic config: `cleanup.policy=delete`, `retention.ms=8640` (1 day)
> - Observed behavior:
> - Each partition has data {*}older than 7 days{*}{*}, Each partition{*}
> {*}40GB{*}, and segments are {*}never deleted{*}.
> - Other topics in the same cluster (including __consumer_offsets) are
> deleted as expected under the delete policy.
> - `kafka-topics.sh --describe --topic mytopic01` shows the ISR is *fully
> in-sync*
> 1. what are the possible conditions under which a specific topic’s segments
> might never become eligible for deletion?
> 2. For Kafka 2.7.1 specifically, what are the recommended debugging and
> verification steps to diagnose this?
> I am mainly looking for a list of likely causes and practical troubleshooting
> steps to apply in production when a single topic does not respect
> delete-based retention while others do.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
