Thanks a lot Gwen and Raju. This information was really helpful. I used the following example:- kafka-topics.sh --zookeeper localhost:13003 --alter --topic MyTopic --config retention.ms=1000
And it works as expected. By setting its limit to a particular value, it prunes messages after that limit. I just want to make sure I am looking at right property. Kindly provide me with feedback, if I am on right direction. Thanks once again. Your help is much appreciated. Best regards, Dilpreet On 11/13/15, 1:24 AM, "Raju Bairishetti" <raju....@gmail.com> wrote: >Adding some more info inline. > >On Fri, Nov 13, 2015 at 10:43 AM, Gwen Shapira <g...@confluent.io> wrote: > >> See answers inline >> >> On Thu, Nov 12, 2015 at 2:53 PM, Sandhu, Dilpreet <dsan...@illumina.com> >> wrote: >> >> > Hi all, >> > I am new to Kafka usage. Here are some questions that I have in >> > mind. Kindly help me understand it better. If some questions make no >> sense >> > feel free to call it out. >> > 1. Is it possible to prune log offsets (messages)older than certain >>date >> > in a partition? >> > >> >> You can configure the retention time for a topic, all older messages >>will >> be pruned. This configuration is at topic level, not for a specific >> partition. >> > >*IMO, we can set the retention in bytes(**retention.bytes**) at partition >level also. Whenever the partition log grows more than configured >retention >bytes then it deletes the older messages from the partition log. **Please >correct me if I am wrong.* > >> >> >> > 2. Will Kafka delete a topic regardless of unconsumed messages, if log >> > retention hours limit is met? >> > >> >> Yes >> > > Kafka does not care & does not know whether the message is consumed or >not. Kafka provides a mechanism to store all messages from multiple >producer machines to a central location. > >> >> >> > 3. Is there a way to just delete messages in Kafka? >> > >> >> Normally, we are only deleting messages when they are old or when size >> limit is exceeded. >> There is a special "compaction" mode where the last message for any key >>is >> stored forever. In this mode you can delete messages by sending a new >> message with that key and a "null" value. However, this also just takes >> place after a certain amount of time. >> >> >> > >> > Any help will be much appreciated. >> > >> > Thanks. >> > Best regards, >> > Dilpreet >> > >> > > > >-- >Thanks >Raju Bairishetti, > >www.inmobi.com > > > > >*2012 ME Best Mobile Ad Network Award Winner > >[image: http://www.inmobi.com/press-releases/files/2013/02/MIT.png]
