Since Metron is the only consumer for the entire Kafka bus, I think I should be able to set a low retention policy for the Kafka internal "__consumer_offsets" topic to clear it. In addition to the "__consumer_offsets" I guess, removing all of the consumers from the zookeeper will help.
On Wed, Apr 5, 2017 at 11:31 PM, Casey Stella <ceste...@gmail.com> wrote: > Sorry, misspoke. I meant to say "the new consumer API" not "consumer > groups" in the previous email. > > On Wed, Apr 5, 2017 at 9:26 AM, Casey Stella <ceste...@gmail.com> wrote: > >> To my knowledge wiping the topic offsets in zookeeper through the zkcli >> is the only way and yep, it's painful. It's unclear whether the move to >> the new kafka spout which uses consumer groups that can be stored in kafka >> rather than zookeeper will help in this. Someone who is more of a kafka >> expert than me should speak, but my understanding is that should make >> deleting topics less of a pain going forward. >> >> On Wed, Apr 5, 2017 at 5:27 AM, Ali Nazemian <alinazem...@gmail.com> >> wrote: >> >>> Is there any way to reset the offset in an automatic way? It is very >>> painful to set them one by one through zookeeper CLI! >>> >>> On Wed, Apr 5, 2017 at 7:04 PM, Ali Nazemian <alinazem...@gmail.com> >>> wrote: >>> >>>> Hi Simon, >>>> >>>> Yeah. I should have applied the retention strategy that is much safer. >>>> Do you think my current problem will be solved by resetting the offset for >>>> those topics? How can I force the storm topologies to use the new topics? >>>> >>>> Cheers, >>>> Ali >>>> >>>> On Wed, Apr 5, 2017 at 6:44 PM, Simon Elliston Ball < >>>> si...@simonellistonball.com> wrote: >>>> >>>>> It's generally not a good idea to delete these topics, it would be >>>>> better to lower retention, or change the offset rules to latest on the >>>>> storm topologies and ignore the problematic messages until they age off >>>>> naturally. >>>>> >>>>> If you really have to delete and purge the logs to recreate be careful >>>>> about partitioning, and also make sure you bounce all the metron storm >>>>> topologies to force connection against the new topics and offset reset. >>>>> >>>>> Simon >>>>> >>>>> On 5 Apr 2017, at 10:32, Ali Nazemian <alinazem...@gmail.com> wrote: >>>>> >>>>> Hi all, >>>>> >>>>> I have had some issue to clear the "enrichments" and "indexing" Kafka >>>>> topics, so I have tried to delete and recreate these topics. I have had >>>>> some troubles to clear those topics. I tried to remove those topics and >>>>> create them again. Unfortunately, Kafka did not remove the topic >>>>> gracefully >>>>> so I tried to remove partitions from disk manually and remove the related >>>>> config files from zookeeper by following this instruction: >>>>> https://github.com/darrenfu/bigdata/issues/6 >>>>> >>>>> After I have managed to remove those topics, Metron has been broken >>>>> and give me the following warning: >>>>> >>>>> 2017-04-05 18:28:37.063 o.a.s.k.KafkaUtils [WARN] >>>>> Partition{host=gsoclabsn02.test.telstra.com:6667, topic=indexing, >>>>> partition=10} Got fetch request with offset out of range: [1227] >>>>> >>>>> >>>>> I can see the messages inside parser topics, but there is not message >>>>> inside "enrichments" and "indexing" topics. Do you know what is the >>>>> problem >>>>> and how can I fix it? >>>>> >>>>> Thanks, >>>>> Ali >>>>> >>>>> >>>> >>>> >>>> -- >>>> A.Nazemian >>>> >>> >>> >>> >>> -- >>> A.Nazemian >>> >> >> > -- A.Nazemian
