To my knowledge wiping the topic offsets in zookeeper through the zkcli is the only way and yep, it's painful. It's unclear whether the move to the new kafka spout which uses consumer groups that can be stored in kafka rather than zookeeper will help in this. Someone who is more of a kafka expert than me should speak, but my understanding is that should make deleting topics less of a pain going forward.
On Wed, Apr 5, 2017 at 5:27 AM, Ali Nazemian <alinazem...@gmail.com> wrote: > Is there any way to reset the offset in an automatic way? It is very > painful to set them one by one through zookeeper CLI! > > On Wed, Apr 5, 2017 at 7:04 PM, Ali Nazemian <alinazem...@gmail.com> > wrote: > >> Hi Simon, >> >> Yeah. I should have applied the retention strategy that is much safer. Do >> you think my current problem will be solved by resetting the offset for >> those topics? How can I force the storm topologies to use the new topics? >> >> Cheers, >> Ali >> >> On Wed, Apr 5, 2017 at 6:44 PM, Simon Elliston Ball < >> si...@simonellistonball.com> wrote: >> >>> It's generally not a good idea to delete these topics, it would be >>> better to lower retention, or change the offset rules to latest on the >>> storm topologies and ignore the problematic messages until they age off >>> naturally. >>> >>> If you really have to delete and purge the logs to recreate be careful >>> about partitioning, and also make sure you bounce all the metron storm >>> topologies to force connection against the new topics and offset reset. >>> >>> Simon >>> >>> On 5 Apr 2017, at 10:32, Ali Nazemian <alinazem...@gmail.com> wrote: >>> >>> Hi all, >>> >>> I have had some issue to clear the "enrichments" and "indexing" Kafka >>> topics, so I have tried to delete and recreate these topics. I have had >>> some troubles to clear those topics. I tried to remove those topics and >>> create them again. Unfortunately, Kafka did not remove the topic gracefully >>> so I tried to remove partitions from disk manually and remove the related >>> config files from zookeeper by following this instruction: >>> https://github.com/darrenfu/bigdata/issues/6 >>> >>> After I have managed to remove those topics, Metron has been broken and >>> give me the following warning: >>> >>> 2017-04-05 18:28:37.063 o.a.s.k.KafkaUtils [WARN] >>> Partition{host=gsoclabsn02.test.telstra.com:6667, topic=indexing, >>> partition=10} Got fetch request with offset out of range: [1227] >>> >>> >>> I can see the messages inside parser topics, but there is not message >>> inside "enrichments" and "indexing" topics. Do you know what is the problem >>> and how can I fix it? >>> >>> Thanks, >>> Ali >>> >>> >> >> >> -- >> A.Nazemian >> > > > > -- > A.Nazemian >
