Re: Kafka retention policy per topic not working as expected

2020-05-27 Thread Don Ky
Thanks Christopher, Hrm, I checked the logs and retention.ms has been enabled somewhere around October last year. {"log":"[2019-10-20 03:01:41,818] INFO Processing override for entityPath: topics/ETD-TEST with config: Map(retention.ms -\u003e 121000)

Re: Kafka retention policy per topic not working as expected

2020-05-24 Thread Christopher Smith
It's a bit weird having the default policy for your brokers being compact, but yes, the policy for the topic overrides the broker policy. What you are seeing on with your example of the segment files makes sense to me. With a 121000ms retention policy you've actually got 400 seconds beyond

Re: Kafka retention policy per topic not working as expected

2020-05-24 Thread Don Ky
Hello all, any thoughts on this? Just pinging if anyone out there has any pointers on where to go from here? I was wondering if it could be that the topic is actually being compacted and not a retention policy of delete? Could this be because that the topics were initially created with a compact