It's generally not a good idea to delete these topics, it would be better to lower retention, or change the offset rules to latest on the storm topologies and ignore the problematic messages until they age off naturally.
If you really have to delete and purge the logs to recreate be careful about partitioning, and also make sure you bounce all the metron storm topologies to force connection against the new topics and offset reset. Simon > On 5 Apr 2017, at 10:32, Ali Nazemian <alinazem...@gmail.com> wrote: > > Hi all, > > I have had some issue to clear the "enrichments" and "indexing" Kafka topics, > so I have tried to delete and recreate these topics. I have had some troubles > to clear those topics. I tried to remove those topics and create them again. > Unfortunately, Kafka did not remove the topic gracefully so I tried to remove > partitions from disk manually and remove the related config files from > zookeeper by following this instruction: > https://github.com/darrenfu/bigdata/issues/6 > > After I have managed to remove those topics, Metron has been broken and give > me the following warning: > > 2017-04-05 18:28:37.063 o.a.s.k.KafkaUtils [WARN] > Partition{host=gsoclabsn02.test.telstra.com:6667, topic=indexing, > partition=10} Got fetch request with offset out of range: [1227] > > I can see the messages inside parser topics, but there is not message inside > "enrichments" and "indexing" topics. Do you know what is the problem and how > can I fix it? > > Thanks, > Ali
