For those who is interested - I've posted a detailed explanation of the reasons
for these issues here:
https://stackoverflow.com/questions/67625641/kafka-segments-are-deleted-too-often-or-not-at-all/67827549#67827549
In brief - it was an issue with a low=volume topic and segment rotation for
Thank you, Matthias, for the ideas to verify next!
Here is what I see:
Topic 1 - that is not being cleaned up for 3 days already, but has retention
set to 4 hrs: (I've truncated the payload but left the important details):
11:32:11 confluent@qv-ckafkama7 [~] $ /usr/bin/kafka-dump-log
Kafka's retention policy is a little bit tricky, because it determines
if data is subject to deletion by comparing the embedded record
timestamps to the broker wall-clock time.
Can you verify your record timestamps as well as broker clocks?
If you write "future data", ie, data with timestamps
Thank you, Ran,
yes, I understand about the segment size impact - however, if you look at the
configuration of those topics (see below) - you will see they have exactly the
same config, with the same segment.bytes setting (to 1G, default), and all
other settings are also the same - except for
Sorry I did not see all the info at first, what do you mean by topic
getting cleaned, you have setting to check retention every 5 minutes, the
data that getting "cleaned" is the older data which is 30 days old... am I
missing something?
בתאריך יום ג׳, 25 במאי 2021, 23:04, מאת Ran Lupovich :
>
By the segment size you are "delete" after 1 giga bytes is full , per
partition, you need to remmber the retention is done when segments closed
, per partition
בתאריך יום ג׳, 25 במאי 2021, 22:59, מאת Ran Lupovich :
> Have you checked the segment size? Did you decribe the topic
>
Have you checked the segment size? Did you decribe the topic
configuration?maybe you created it with some settings you dont remember
בתאריך יום ג׳, 25 במאי 2021, 19:51, מאת Marina Popova
:
>
> Any idea what is wrong here? I have restarted Kafka brokers a few times,
> and all other Confluent
Any idea what is wrong here? I have restarted Kafka brokers a few times, and
all other Confluent services like KSQL - but I see exactly the same behavior -
one topic gets its logs cleaned up every 5 minutes, while the other one - does
not get cleaned up at all
Is there anything else I
Hi, I have posted this question on SO:
https://stackoverflow.com/questions/67625641/kafka-segments-are-deleted-too-often-or-not-at-all
but wanted to re-post here as well in case someone spots the issue right away
Thank you for your help!
>
We have two topics on our Kafka cluster that
