Re: Broker brought down and under replicated partitions

2014-10-16 Thread Jean-Pascal Billaud
The only thing that I find very weird is the fact that brokers that are dead are still part of the ISR set for hours... and are basically not removed. Note this is not constantly the case, most of the dead brokers are properly removed and it is really just in a few cases. I am not sure why this

Re: Broker brought down and under replicated partitions

2014-10-16 Thread Neha Narkhede
Is there a known issue in the 0.8.0 version that was fixed later on? What can I do to diagnose/fix the situation? Yes, quite a few bugs related to this have been fixed since 0.8.0. I'd suggest upgrading to 0.8.1.1 On Wed, Oct 15, 2014 at 11:09 PM, Jean-Pascal Billaud j...@tellapart.com wrote:

Re: Broker brought down and under replicated partitions

2014-10-15 Thread Neha Narkhede
Which version of Kafka are you using? The current stable one is 0.8.1.1 On Tue, Oct 14, 2014 at 5:51 PM, Jean-Pascal Billaud j...@tellapart.com wrote: Hey Neha, so I removed another broker like 30mn ago and since then basically the Producer is dying with: Event queue is full of unsent

Re: Broker brought down and under replicated partitions

2014-10-15 Thread Jean-Pascal Billaud
So I am using 0.8.0. I think I found the issue actually. It turns out that some partitions only had a single replica and the leaders of those partitions would basically refuse new writes. As soon as I reassigned replicas to those partitions things kicked off again. Not sure if that's expected...

Re: Broker brought down and under replicated partitions

2014-10-14 Thread Neha Narkhede
Regarding (1), I am assuming that it is expected that brokers going down will be brought back up soon. At which point, they will pick up from the current leader and get back into the ISR. Am I right? The broker will be added back to the ISR once it is restarted, but it never goes out of the

Re: Broker brought down and under replicated partitions

2014-10-14 Thread Jean-Pascal Billaud
Hey Neha, so I removed another broker like 30mn ago and since then basically the Producer is dying with: Event queue is full of unsent messages, could not send event: KeyedMessage(my_topic,[B@1b71b7a6,[B@35fdd1e7) kafka.common.QueueFullException: Event queue is full of unsent messages, could not