Re: Metron has been broken due to issue with Kafka

[Ali Nazemian]

Since Metron is the only consumer for the entire Kafka bus, I think I
should be able to set a low retention policy for the Kafka internal
"__consumer_offsets" topic to clear it. In addition to
the "__consumer_offsets" I guess, removing all of the consumers from the
zookeeper will help.

On Wed, Apr 5, 2017 at 11:31 PM, Casey Stella  wrote:

> Sorry, misspoke.  I meant to say "the new consumer API" not "consumer
> groups" in the previous email.
>
> On Wed, Apr 5, 2017 at 9:26 AM, Casey Stella  wrote:
>
>> To my knowledge wiping the topic offsets in zookeeper through the zkcli
>> is the only way and yep, it's painful.  It's unclear whether the move to
>> the new kafka spout which uses consumer groups that can be stored in kafka
>> rather than zookeeper will help in this.  Someone who is more of a kafka
>> expert than me should speak, but my understanding is that should make
>> deleting topics less of a pain going forward.
>>
>> On Wed, Apr 5, 2017 at 5:27 AM, Ali Nazemian 
>> wrote:
>>
>>> Is there any way to reset the offset in an automatic way? It is very
>>> painful to set them one by one through zookeeper CLI!
>>>
>>> On Wed, Apr 5, 2017 at 7:04 PM, Ali Nazemian 
>>> wrote:
>>>
 Hi Simon,

 Yeah. I should have applied the retention strategy that is much safer.
 Do you think my current problem will be solved by resetting the offset for
 those topics? How can I force the storm topologies to use the new topics?

 Cheers,
 Ali

 On Wed, Apr 5, 2017 at 6:44 PM, Simon Elliston Ball <
 si...@simonellistonball.com> wrote:

> It's generally not a good idea to delete these topics, it would be
> better to lower retention, or change the offset rules to latest on the
> storm topologies and ignore the problematic messages until they age off
> naturally.
>
> If you really have to delete and purge the logs to recreate be careful
> about partitioning, and also make sure you bounce all the metron storm
> topologies to force connection against the new topics and offset reset.
>
> Simon
>
> On 5 Apr 2017, at 10:32, Ali Nazemian  wrote:
>
> Hi all,
>
> I have had some issue to clear the "enrichments" and "indexing" Kafka
> topics, so I have tried to delete and recreate these topics. I have had
> some troubles to clear those topics. I tried to remove those topics and
> create them again. Unfortunately, Kafka did not remove the topic 
> gracefully
> so I tried to remove partitions from disk manually and remove the related
> config files from zookeeper by following this instruction:
> https://github.com/darrenfu/bigdata/issues/6
>
> After I have managed to remove those topics, Metron has been broken
> and give me the following warning:
>
> 2017-04-05 18:28:37.063 o.a.s.k.KafkaUtils [WARN] 
> Partition{host=gsoclabsn02.test.telstra.com:6667, topic=indexing, 
> partition=10} Got fetch request with offset out of range: [1227]
>
>
> I can see the messages inside parser topics, but there is not message
> inside "enrichments" and "indexing" topics. Do you know what is the 
> problem
> and how can I fix it?
>
> Thanks,
> Ali
>
>


 --
 A.Nazemian

>>>
>>>
>>>
>>> --
>>> A.Nazemian
>>>
>>
>>
>


-- 
A.Nazemian

Metron has been broken due to issue with Kafka

[Ali Nazemian]

Hi all,

I have had some issue to clear the "enrichments" and "indexing" Kafka
topics, so I have tried delete and remove these topics. I have some
troubles to clear those topics. I tried to remove those topics and create
them again. Unfortunately, Kafka did not remove the topic gracefully so I
tried to remove partitions from disk manually and remove the related config
files from zookeeper by following this instruction:


-- 
A.Nazemian