Thanks Cees and Abhinav, will give this trick a try and update if it helped
for my case.
Regards,
Sahitya Agrawal
On Fri, May 13, 2016 at 9:36 PM, Cees de Groot <c...@pagerduty.com> wrote:
> What Abhinav said. To give some context: the common cause of frequent
> rebalances is that your consumer takes too long to process batches. As
> long as you don't call into the consumer library, heartbeats aren't
> sent so if you take too long working through a batch, the broker
> things your consumer is gone and starts re-balancing. The message
> batch under processing never gets marked as done, so after
> rebalancing, things start over from the same spot.
>
> So the solution is to either make the batches smaller or the heartbeat
> interval longer. There are fancier solutions for when this doesn't
> work, but it should do the trick for most normal cases.
>
> On Fri, May 13, 2016 at 10:20 AM, Abhinav Solan <abhinav.so...@gmail.com>
> wrote:
> > Hi Sahitya,
> >
> > Try reducing max.partition.fetch.bytes in your consumer.
> > Then also increase heartbeat.interval.ms, this might help in to delay
> the
> > consumer rebalance of your inbound process is taking more time than this
> >
> > - Abhinav
> >
> > On Fri, May 13, 2016 at 5:42 AM sahitya agrawal <sahitya2...@gmail.com>
> > wrote:
> >
> >> Hi,
> >>
> >> I am using new Kafka consumer API ( 0.9.0.0) . I created 100
> partitions of
> >> a topic and started only one consumer to consume. Many of times , In
> >> consumer logs I see lot of rebalancing activity and no object is
> consumed
> >> due to that.
> >>
> >> Is this a known issue? Please let me know if some body can help with
> regard
> >> to this.
> >>
> >> My Consumer config:
> >> props.put("zookeeper.session.timeout.ms", "10000");
> >> props.put("rebalance.backoff.ms","10000");
> >> props.put("zookeeper.sync.time.ms","200");
> >> props.put("rebalance.max.retries","10");
> >> props.put("enable.auto.commit", "false");
> >> props.put("consumer.timeout.ms","20000");
> >> props.put("auto.offset.reset", "smallest");
> >>
> >> Thanks,
> >> Sahitya
> >>
>
>
>
> --
> Cees de Groot
> Principal Software Engineer
> PagerDuty, Inc.
>
