Re: Kafka High level consumer rebalancing

2015-07-24 Thread Mayuresh Gharat
This sounds like an interesting problem. What is your consumerTimeout set to? Thanks, Mayuresh On Tue, Jul 21, 2015 at 6:12 PM, Pranay Agarwal agarwalpran...@gmail.com wrote: Thanks Mayuresh, Can I at least control the rebalance of consumers? Currently consumes die after specific partition

Re: Kafka High level consumer rebalancing

2015-07-21 Thread Pranay Agarwal
Any ideas? On Mon, Jul 20, 2015 at 2:34 PM, Pranay Agarwal agarwalpran...@gmail.com wrote: Hi all, Is there any way I can force Zookeeper/Kafka to rebalance new consumers only for subset of total number of partitions. I have a situation where out of 120 partitions 60 have been already

Re: Kafka High level consumer rebalancing

2015-07-21 Thread Pranay Agarwal
Thanks Mayuresh, Can I at least control the rebalance of consumers? Currently consumes die after specific partition has no more messages, and there is rebalance of consumes triggered, which causes more consumers to die who get assigned to empty partition(because zookeeper treat empty partition no

Re: Kafka High level consumer rebalancing

2015-07-21 Thread Mayuresh Gharat
Not sure if you can do that with High level consumer. Thanks, Mayuresh On Tue, Jul 21, 2015 at 10:53 AM, Pranay Agarwal agarwalpran...@gmail.com wrote: Any ideas? On Mon, Jul 20, 2015 at 2:34 PM, Pranay Agarwal agarwalpran...@gmail.com wrote: Hi all, Is there any way I can force

Kafka High level consumer rebalancing

2015-07-20 Thread Pranay Agarwal
Hi all, Is there any way I can force Zookeeper/Kafka to rebalance new consumers only for subset of total number of partitions. I have a situation where out of 120 partitions 60 have been already consumed, but the zookeeper also assigns these empty/inactive partitions as well for the re-balancing,