Hi Simon, Yeah. I should have applied the retention strategy that is much safer. Do you think my current problem will be solved by resetting the offset for those topics? How can I force the storm topologies to use the new topics?
Cheers, Ali On Wed, Apr 5, 2017 at 6:44 PM, Simon Elliston Ball < si...@simonellistonball.com> wrote: > It's generally not a good idea to delete these topics, it would be better > to lower retention, or change the offset rules to latest on the storm > topologies and ignore the problematic messages until they age off > naturally. > > If you really have to delete and purge the logs to recreate be careful > about partitioning, and also make sure you bounce all the metron storm > topologies to force connection against the new topics and offset reset. > > Simon > > On 5 Apr 2017, at 10:32, Ali Nazemian <alinazem...@gmail.com> wrote: > > Hi all, > > I have had some issue to clear the "enrichments" and "indexing" Kafka > topics, so I have tried to delete and recreate these topics. I have had > some troubles to clear those topics. I tried to remove those topics and > create them again. Unfortunately, Kafka did not remove the topic gracefully > so I tried to remove partitions from disk manually and remove the related > config files from zookeeper by following this instruction: > https://github.com/darrenfu/bigdata/issues/6 > > After I have managed to remove those topics, Metron has been broken and > give me the following warning: > > 2017-04-05 18:28:37.063 o.a.s.k.KafkaUtils [WARN] > Partition{host=gsoclabsn02.test.telstra.com:6667, topic=indexing, > partition=10} Got fetch request with offset out of range: [1227] > > > I can see the messages inside parser topics, but there is not message > inside "enrichments" and "indexing" topics. Do you know what is the problem > and how can I fix it? > > Thanks, > Ali > > -- A.Nazemian
