Hello Shantanu, It is also important to consider your consumer code. You should not spend to much time in between two calls to "poll" method. Otherwise, the consumer not calling poll will be considered dead by the group, triggering a rebalancing.
Best On Thu, May 24, 2018 at 1:45 PM M. Manna <manme...@gmail.com> wrote: > Set your rebalance.backoff.ms=10000 and zookeeper.session.timeout.ms=30000 > in addition to what Manikumar said. > > > > On 24 May 2018 at 12:41, Shantanu Deshmukh <shantanu...@gmail.com> wrote: > > > Hello, > > > > There was a type in my first mail. session.timeout.ms is actually 60000 > > not > > 6000. So it is less than heartbeat.interval.ms. > > > > On Thu, May 24, 2018 at 2:46 PM Manikumar <manikumar.re...@gmail.com> > > wrote: > > > > > heartbeat.interval.ms should be lower than session.timeout.ms. > > > > > > Check here: > > > http://kafka.apache.org/0101/documentation.html#newconsumerconfigs > > > > > > > > > On Thu, May 24, 2018 at 2:39 PM, Shantanu Deshmukh < > > shantanu...@gmail.com> > > > wrote: > > > > > > > Someone please help me. I am suffering due to this issue since a long > > > time > > > > and not finding any solution. > > > > > > > > On Wed, May 23, 2018 at 3:48 PM Shantanu Deshmukh < > > shantanu...@gmail.com > > > > > > > > wrote: > > > > > > > > > We have a 3 broker Kafka 0.10.0.1 cluster. There we have 3 topics > > with > > > 10 > > > > > partitions each. We have an application which spawns threads as > > > > consumers. > > > > > We spawn 5 consumers for each topic. I am observing that consider > > group > > > > > randomly keeps rebalancing. Then many times we see logs saying > > > "Revoking > > > > > partitions for". This happens almost every 10 minutes. Consumption > > > during > > > > > this time completely stops. > > > > > > > > > > I have applied this configuration > > > > > max.poll.records 20 > > > > > heartbeat.interval.ms 10000 > > > > > Session.timeout.ms 6000 > > > > > > > > > > Still this did not help. Strange thing is I observed consumer > writing > > > > logs > > > > > saying "auto commit failed because poll() loop spent too much time > > > > > processing records" even when there was no data in partition to > > > process. > > > > We > > > > > have polling interval of 500 ms, specified as argument in poll(). > > > > Initially > > > > > I had set same consumer group for all three topics' consumers. > Then I > > > > > specified different CGs for different topics' consumers. Even this > is > > > not > > > > > helping. > > > > > > > > > > I am trying to search over the web, checked my code, tried many > > > > > combinations of configuration but still no luck. Please help me. > > > > > > > > > > *Thanks & Regards,* > > > > > > > > > > *Shantanu Deshmukh* > > > > > > > > > > > > > > >
