Re: Kafka High level consumer rebalancing

2015-07-21 Thread Pranay Agarwal
Any ideas? On Mon, Jul 20, 2015 at 2:34 PM, Pranay Agarwal agarwalpran...@gmail.com wrote: Hi all, Is there any way I can force Zookeeper/Kafka to rebalance new consumers only for subset of total number of partitions. I have a situation where out of 120 partitions 60 have been already

Re: Kafka High level consumer rebalancing

2015-07-21 Thread Pranay Agarwal
, Jul 21, 2015 at 10:53 AM, Pranay Agarwal agarwalpran...@gmail.com wrote: Any ideas? On Mon, Jul 20, 2015 at 2:34 PM, Pranay Agarwal agarwalpran...@gmail.com wrote: Hi all, Is there any way I can force Zookeeper/Kafka to rebalance new consumers only for subset of total

Kafka High level consumer rebalancing

2015-07-20 Thread Pranay Agarwal
Hi all, Is there any way I can force Zookeeper/Kafka to rebalance new consumers only for subset of total number of partitions. I have a situation where out of 120 partitions 60 have been already consumed, but the zookeeper also assigns these empty/inactive partitions as well for the re-balancing,

Re: Kafka Out of Memory error

2015-01-26 Thread Pranay Agarwal
there are other stuff on the server). Gwen On Mon, Jan 19, 2015 at 3:06 PM, Pranay Agarwal agarwalpran...@gmail.com wrote: Thanks a lot Natty. I am using this Ruby gem on the client side with all the default config https://github.com/joekiller/jruby-kafka/blob/master/lib/jruby-kafka

Re: Kafka Out of Memory error

2015-01-19 Thread Pranay Agarwal
http://www.linkedin.com/in/nattyice On Mon, Jan 19, 2015 at 2:34 PM, Pranay Agarwal agarwalpran...@gmail.com wrote: Thanks Natty. Is there any config which I need to change on the client side as well? Also, currently I am trying with only 1 consumer thread. Does the equation changes

Re: Kafka Out of Memory error

2015-01-19 Thread Pranay Agarwal
, Natty Jonathan Natty Natkins StreamSets | Customer Engagement Engineer mobile: 609.577.1600 | linkedin http://www.linkedin.com/in/nattyice On Mon, Jan 19, 2015 at 2:10 PM, Pranay Agarwal agarwalpran...@gmail.com wrote: Hi All, I have a kafka cluster setup which has 2 topics

Kafka Out of Memory error

2015-01-19 Thread Pranay Agarwal
Hi All, I have a kafka cluster setup which has 2 topics topic1 with 10 partitions topic2 with 1000 partitions. While, I am able to consume messages from topic1 just fine, I get following error from the topic2. There is a resolved issue here on the same thing