Hi Vincent, Yes I reduced max.poll.records to get that same effect. I reduced it all the way down to 5 records still I am seeing same error. What else can be done? For one topic I can see that a single message processing is taking about 20 seconds. So 5 of them will take 1 minute. So I set session.timeout.ms to 5 minutes, max.poll.interval.ms to 10 minutes. But it is not helping still.
On Thu, May 24, 2018 at 6:15 PM Vincent Maurin <vincent.mau...@glispa.com> wrote: > Hello Shantanu, > > It is also important to consider your consumer code. You should not spend > to much time in between two calls to "poll" method. Otherwise, the consumer > not calling poll will be considered dead by the group, triggering a > rebalancing. > > Best > > On Thu, May 24, 2018 at 1:45 PM M. Manna <manme...@gmail.com> wrote: > > > Set your rebalance.backoff.ms=10000 and zookeeper.session.timeout.ms > =30000 > > in addition to what Manikumar said. > > > > > > > > On 24 May 2018 at 12:41, Shantanu Deshmukh <shantanu...@gmail.com> > wrote: > > > > > Hello, > > > > > > There was a type in my first mail. session.timeout.ms is actually > 60000 > > > not > > > 6000. So it is less than heartbeat.interval.ms. > > > > > > On Thu, May 24, 2018 at 2:46 PM Manikumar <manikumar.re...@gmail.com> > > > wrote: > > > > > > > heartbeat.interval.ms should be lower than session.timeout.ms. > > > > > > > > Check here: > > > > http://kafka.apache.org/0101/documentation.html#newconsumerconfigs > > > > > > > > > > > > On Thu, May 24, 2018 at 2:39 PM, Shantanu Deshmukh < > > > shantanu...@gmail.com> > > > > wrote: > > > > > > > > > Someone please help me. I am suffering due to this issue since a > long > > > > time > > > > > and not finding any solution. > > > > > > > > > > On Wed, May 23, 2018 at 3:48 PM Shantanu Deshmukh < > > > shantanu...@gmail.com > > > > > > > > > > wrote: > > > > > > > > > > > We have a 3 broker Kafka 0.10.0.1 cluster. There we have 3 topics > > > with > > > > 10 > > > > > > partitions each. We have an application which spawns threads as > > > > > consumers. > > > > > > We spawn 5 consumers for each topic. I am observing that consider > > > group > > > > > > randomly keeps rebalancing. Then many times we see logs saying > > > > "Revoking > > > > > > partitions for". This happens almost every 10 minutes. > Consumption > > > > during > > > > > > this time completely stops. > > > > > > > > > > > > I have applied this configuration > > > > > > max.poll.records 20 > > > > > > heartbeat.interval.ms 10000 > > > > > > Session.timeout.ms 6000 > > > > > > > > > > > > Still this did not help. Strange thing is I observed consumer > > writing > > > > > logs > > > > > > saying "auto commit failed because poll() loop spent too much > time > > > > > > processing records" even when there was no data in partition to > > > > process. > > > > > We > > > > > > have polling interval of 500 ms, specified as argument in poll(). > > > > > Initially > > > > > > I had set same consumer group for all three topics' consumers. > > Then I > > > > > > specified different CGs for different topics' consumers. Even > this > > is > > > > not > > > > > > helping. > > > > > > > > > > > > I am trying to search over the web, checked my code, tried many > > > > > > combinations of configuration but still no luck. Please help me. > > > > > > > > > > > > *Thanks & Regards,* > > > > > > > > > > > > *Shantanu Deshmukh* > > > > > > > > > > > > > > > > > > > > >
