Sorry, misspoke. I meant to say "the new consumer API" not "consumer groups" in the previous email.
On Wed, Apr 5, 2017 at 9:26 AM, Casey Stella <ceste...@gmail.com> wrote: > To my knowledge wiping the topic offsets in zookeeper through the zkcli is > the only way and yep, it's painful. It's unclear whether the move to the > new kafka spout which uses consumer groups that can be stored in kafka > rather than zookeeper will help in this. Someone who is more of a kafka > expert than me should speak, but my understanding is that should make > deleting topics less of a pain going forward. > > On Wed, Apr 5, 2017 at 5:27 AM, Ali Nazemian <alinazem...@gmail.com> > wrote: > >> Is there any way to reset the offset in an automatic way? It is very >> painful to set them one by one through zookeeper CLI! >> >> On Wed, Apr 5, 2017 at 7:04 PM, Ali Nazemian <alinazem...@gmail.com> >> wrote: >> >>> Hi Simon, >>> >>> Yeah. I should have applied the retention strategy that is much safer. >>> Do you think my current problem will be solved by resetting the offset for >>> those topics? How can I force the storm topologies to use the new topics? >>> >>> Cheers, >>> Ali >>> >>> On Wed, Apr 5, 2017 at 6:44 PM, Simon Elliston Ball < >>> si...@simonellistonball.com> wrote: >>> >>>> It's generally not a good idea to delete these topics, it would be >>>> better to lower retention, or change the offset rules to latest on the >>>> storm topologies and ignore the problematic messages until they age off >>>> naturally. >>>> >>>> If you really have to delete and purge the logs to recreate be careful >>>> about partitioning, and also make sure you bounce all the metron storm >>>> topologies to force connection against the new topics and offset reset. >>>> >>>> Simon >>>> >>>> On 5 Apr 2017, at 10:32, Ali Nazemian <alinazem...@gmail.com> wrote: >>>> >>>> Hi all, >>>> >>>> I have had some issue to clear the "enrichments" and "indexing" Kafka >>>> topics, so I have tried to delete and recreate these topics. I have had >>>> some troubles to clear those topics. I tried to remove those topics and >>>> create them again. Unfortunately, Kafka did not remove the topic gracefully >>>> so I tried to remove partitions from disk manually and remove the related >>>> config files from zookeeper by following this instruction: >>>> https://github.com/darrenfu/bigdata/issues/6 >>>> >>>> After I have managed to remove those topics, Metron has been broken and >>>> give me the following warning: >>>> >>>> 2017-04-05 18:28:37.063 o.a.s.k.KafkaUtils [WARN] >>>> Partition{host=gsoclabsn02.test.telstra.com:6667, topic=indexing, >>>> partition=10} Got fetch request with offset out of range: [1227] >>>> >>>> >>>> I can see the messages inside parser topics, but there is not message >>>> inside "enrichments" and "indexing" topics. Do you know what is the problem >>>> and how can I fix it? >>>> >>>> Thanks, >>>> Ali >>>> >>>> >>> >>> >>> -- >>> A.Nazemian >>> >> >> >> >> -- >> A.Nazemian >> > >
