Have a consumer group with one consumer for the topic .. by misunderstanding I have two partitions on the topic .. Due to having no key set for the record - I think having several consumers making no sense, or am I wrong.
Is there any possibility to work around that? Cause for example on lagging topic is put to a external REST service, which takes around 300ms to be handled. So is lowering the max.poll.records an option? Anyhow, I could probably not avoid situations like that. Sounds to me like a pretty common scenario? So how to deal with them? Having a health check that crush the app if no data is appearing anymore? Regards -----Ursprüngliche Nachricht----- Von: M. Manna <manme...@gmail.com> Gesendet: Donnerstag, 7. November 2019 23:35 An: users@kafka.apache.org Betreff: Re: Consumer Lags and receive no records anymore Consuming not fast/frequent enough is one of the most common reasons for it. Have you you checked how fast/much message you’re churning out vs. how many consumers you have in the group the handle the workload? Also, what are your partition setup for consumer groups? Regards, On Thu, 7 Nov 2019 at 22:03, Oliver Eckle <iee1...@gmx.de> wrote: > Using kafka-consumer-groups.sh --bootstrap-server localhost:9092 > --describe -group my-app .. > put the output within the logs .. also its pretty obvious, cause no > data will flow anymore > > Regards > > -----Ursprüngliche Nachricht----- > Von: M. Manna <manme...@gmail.com> > Gesendet: Donnerstag, 7. November 2019 22:10 > An: users@kafka.apache.org > Betreff: Re: Consumer Lags and receive no records anymore > > Have you checked your Kafka consumer group status ? How did you > determine that your consumers are lagging ? > > Thanks, > > On Thu, 7 Nov 2019 at 20:55, Oliver Eckle <iee1...@gmx.de> wrote: > > > Hi there, > > > > > > > > have pretty strange behaviour questioned here already: > > https://stackoverflow.com/q/58650416/7776688 > > > > > > > > As you could see from the logs: https://pastebin.com/yrSytSHD at a > > specific point the client is stopping to receive records. > > > > I have a strong suspicion that it relates to performance on handling > > the records - so that I run into kind of timeout. > > > > What seems to be strange, is that the client is not getting back and > > heartbeats are processed successfully. > > > > Even the consumer will be returned on inspecting the consumer group. > > Any idea .. kafka log has no error in it. > > > > > > > > Running a cluster with 3 broker inside a Kubernetes cluster, using > > the bitnami helm chart. > > > > > > > > Kind Regards > > > > Oliver > > > > > > > > > > > > > >
