Another observation is that when I restart my application. Consumption doesn't start till 5-6 minutes. In kafka consumer logs I see
ConsumerCoordinator.333 - Revoking previously assigned partitions [] for group notifications-consumer AbstractCoordinator:381 - (Re-)joining group notifications-consumer Then nothing. After 5-6 minutes activities start. On Thu, May 24, 2018 at 6:49 PM Shantanu Deshmukh <shantanu...@gmail.com> wrote: > Hi Vincent, > > Yes I reduced max.poll.records to get that same effect. I reduced it all > the way down to 5 records still I am seeing same error. What else can be > done? For one topic I can see that a single message processing is taking > about 20 seconds. So 5 of them will take 1 minute. So I set > session.timeout.ms to 5 minutes, max.poll.interval.ms to 10 minutes. But > it is not helping still. > > On Thu, May 24, 2018 at 6:15 PM Vincent Maurin <vincent.mau...@glispa.com> > wrote: > >> Hello Shantanu, >> >> It is also important to consider your consumer code. You should not spend >> to much time in between two calls to "poll" method. Otherwise, the >> consumer >> not calling poll will be considered dead by the group, triggering a >> rebalancing. >> >> Best >> >> On Thu, May 24, 2018 at 1:45 PM M. Manna <manme...@gmail.com> wrote: >> >> > Set your rebalance.backoff.ms=10000 and zookeeper.session.timeout.ms >> =30000 >> > in addition to what Manikumar said. >> > >> > >> > >> > On 24 May 2018 at 12:41, Shantanu Deshmukh <shantanu...@gmail.com> >> wrote: >> > >> > > Hello, >> > > >> > > There was a type in my first mail. session.timeout.ms is actually >> 60000 >> > > not >> > > 6000. So it is less than heartbeat.interval.ms. >> > > >> > > On Thu, May 24, 2018 at 2:46 PM Manikumar <manikumar.re...@gmail.com> >> > > wrote: >> > > >> > > > heartbeat.interval.ms should be lower than session.timeout.ms. >> > > > >> > > > Check here: >> > > > http://kafka.apache.org/0101/documentation.html#newconsumerconfigs >> > > > >> > > > >> > > > On Thu, May 24, 2018 at 2:39 PM, Shantanu Deshmukh < >> > > shantanu...@gmail.com> >> > > > wrote: >> > > > >> > > > > Someone please help me. I am suffering due to this issue since a >> long >> > > > time >> > > > > and not finding any solution. >> > > > > >> > > > > On Wed, May 23, 2018 at 3:48 PM Shantanu Deshmukh < >> > > shantanu...@gmail.com >> > > > > >> > > > > wrote: >> > > > > >> > > > > > We have a 3 broker Kafka 0.10.0.1 cluster. There we have 3 >> topics >> > > with >> > > > 10 >> > > > > > partitions each. We have an application which spawns threads as >> > > > > consumers. >> > > > > > We spawn 5 consumers for each topic. I am observing that >> consider >> > > group >> > > > > > randomly keeps rebalancing. Then many times we see logs saying >> > > > "Revoking >> > > > > > partitions for". This happens almost every 10 minutes. >> Consumption >> > > > during >> > > > > > this time completely stops. >> > > > > > >> > > > > > I have applied this configuration >> > > > > > max.poll.records 20 >> > > > > > heartbeat.interval.ms 10000 >> > > > > > Session.timeout.ms 6000 >> > > > > > >> > > > > > Still this did not help. Strange thing is I observed consumer >> > writing >> > > > > logs >> > > > > > saying "auto commit failed because poll() loop spent too much >> time >> > > > > > processing records" even when there was no data in partition to >> > > > process. >> > > > > We >> > > > > > have polling interval of 500 ms, specified as argument in >> poll(). >> > > > > Initially >> > > > > > I had set same consumer group for all three topics' consumers. >> > Then I >> > > > > > specified different CGs for different topics' consumers. Even >> this >> > is >> > > > not >> > > > > > helping. >> > > > > > >> > > > > > I am trying to search over the web, checked my code, tried many >> > > > > > combinations of configuration but still no luck. Please help me. >> > > > > > >> > > > > > *Thanks & Regards,* >> > > > > > >> > > > > > *Shantanu Deshmukh* >> > > > > > >> > > > > >> > > > >> > > >> > >> >
