Hi Karan, Just to clarify, with `--time -1` you are getting back the latest offset of the partition. If you do `--time -2` you'll get the earliest valid offset.
So, let's say the latest offset of partition 0 of topic 'test' is 100. When you publish 5 messages to the partition, and before retention policy kicks in, - with `--time -1` you should get test:0:105 - with `--time -2` you should get test:0:100 But after retention policy kicks in and old messages are removed, - with `--time -1` you should get test:0:105 - with `--time -2` you should get test:0:105 Could you please advise whether you're seeing a different behavior? Thanks. --Vahid From: "Vahid S Hashemian" <vahidhashem...@us.ibm.com> To: users@kafka.apache.org Date: 06/22/2017 06:43 PM Subject: Re: Deleting/Purging data from Kafka topics (Kafka 0.10) Hi Karan, I think the issue is in verification step. Because the start and end offsets are not going to be reset when messages are deleted. Have you checked whether a consumer would see the messages that are supposed to be deleted? Thanks. --Vahid From: karan alang <karan.al...@gmail.com> To: users@kafka.apache.org Date: 06/22/2017 06:09 PM Subject: Re: Deleting/Purging data from Kafka topics (Kafka 0.10) Hi Vahid, somehow, the changes suggested don't seem to be taking effect, and i dont see the data being purged from the topic. Here are the steps i followed - 1) topic is set with param -- retention.ms=1000 $KAFKA10_HOME/bin/kafka-topics.sh --describe --topic topicPurge --zookeeper localhost:2161 Topic:topicPurge PartitionCount:3 ReplicationFactor:3 Configs:retention.ms =1000 Topic: topicPurge Partition: 0 Leader: 3 Replicas: 3,1,2 Isr: 3,1,2 Topic: topicPurge Partition: 1 Leader: 0 Replicas: 0,2,3 Isr: 0,2,3 Topic: topicPurge Partition: 2 Leader: 1 Replicas: 1,3,0 Isr: 1,3,0 2) There are 4 brokers, and in the server.properties (for each of the brokers), i've modified the following property log.retention.check.interval.ms=30000 I am expecting the data to be purged every 30 secs based on property - log.retention.check.interval.ms, however, that does not seem to be happening. 3) Here is the command to check the offsets $KAFKA10_HOME/bin/kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list localhost:6092,localhost:6093,localhost:6094,localhost:6095 --topic topicPurge --time -1 --partitions 0,1,2 topicPurge:0:67 topicPurge:1:67 topicPurge:2:66 Any ideas on what the issue might be ? On Thu, Jun 22, 2017 at 1:31 PM, Vahid S Hashemian < vahidhashem...@us.ibm.com> wrote: > Hi Karan, > > The other broker config that plays a role here is > "log.retention.check.interval.ms". > For a low log retention time like in your example if this broker config > value is much higher, then the broker doesn't delete old logs regular > enough. > > --Vahid > > > > From: karan alang <karan.al...@gmail.com> > To: users@kafka.apache.org > Date: 06/22/2017 12:27 PM > Subject: Deleting/Purging data from Kafka topics (Kafka 0.10) > > > > Hi All - > How do i go about deleting data from Kafka Topics ? I've Kafka 0.10 > installed. > > I tried setting the parameter of the topic as shown below -> > > $KAFKA10_HOME/bin/kafka-topics.sh --zookeeper localhost:2161 --alter > --topic mmtopic6 --config retention.ms=1000 > I was expecting to have the data purged in about a min or so .. however, > i > dont see that happening .. > any ideas on what needs to be done ? > > > > >
