Hi, slow consumers - that could be the case. But why is that an issue? I mean I try to use kafka exactly for that and the ability to recover. So e.g if there is some burst scenario where a lot of data arrives and has to be processed, a "slow consumer" will be the default case. What I could understand, that having constantly slow consumers will be an issue, e.g. if there is some compaction on the topic or data will be erased, without having been read.
This is what I think about the "lagging topic" The scenario is like that: Producer --- Topic C ---> Consumer --- Processing ---> external REST Endpoint Sending a Record to the external REST Endpoint takes around 300ms. So if I have the "burst scenario" I mentioned above, there is maybe a lag of 1000-2000 records. So consumer is pulling 500 and process them, which means it takes around 150s for the app to process the records. This could create some timeouts I guess ... so that’s the reason why I try to lower the poll records to 50 e.g. cause then is takes only 15s until the poll is committed. Yeah having some liveness probe sounds pretty elegant .. give that a try ... Anyway, I need to understand why that is happening to deal with the scenario the correct way.. killing the consumer after he stops to consume messages, seems to me more like a workaround. Regards -----Ursprüngliche Nachricht----- Von: M. Manna <manme...@gmail.com> Gesendet: Freitag, 8. November 2019 00:24 An: users@kafka.apache.org Betreff: Re: Consumer Lags and receive no records anymore Hi, > On 7 Nov 2019, at 22:39, Oliver Eckle <iee1...@gmx.de> wrote: > > Have a consumer group with one consumer for the topic .. by misunderstanding > I have two partitions on the topic .. > Due to having no key set for the record - I think having several consumers > making no sense, or am I wrong. > I am not sure why that would be an issue. If you have 1 consumer your cgroup, yes all the topic partitions will be assigned to that consumer. Slow consumer means your consumers aren’t consuming messages as fast as you are producing (or, fast enough). > Is there any possibility to work around that? > Cause for example on lagging topic is put to a external REST service, which > takes around 300ms to be handled. What do you mean by “Lagging topic is put to an external REST service”? > So is lowering the max.poll.records an option? Polling will keep blocking continuously until minimum bytes of records available. Also, it sends a heartbeat per call of poll(). > Anyhow, I could probably not avoid situations like that. Sounds to me like a > pretty common scenario? > So how to deal with them? Having a health check that crush the app if no data > is appearing anymore? In K8s world, you can tie this with liveness probe, if you consumers aren’t live and then you may chose to destroy the pod and bring them back up. Provided that your offset commits are adhering to how technical requirements, you should be able to recover based on the last committed offset. Try that and see how it goes. > > Regards > > -----Ursprüngliche Nachricht----- > Von: M. Manna <manme...@gmail.com> > Gesendet: Donnerstag, 7. November 2019 23:35 > An: users@kafka.apache.org > Betreff: Re: Consumer Lags and receive no records anymore > > Consuming not fast/frequent enough is one of the most common reasons for it. > Have you you checked how fast/much message you’re churning out vs. how many > consumers you have in the group the handle the workload? > > Also, what are your partition setup for consumer groups? > > > Regards, > > On Thu, 7 Nov 2019 at 22:03, Oliver Eckle <iee1...@gmx.de> wrote: > >> Using kafka-consumer-groups.sh --bootstrap-server localhost:9092 >> --describe -group my-app .. >> put the output within the logs .. also its pretty obvious, cause no >> data will flow anymore >> >> Regards >> >> -----Ursprüngliche Nachricht----- >> Von: M. Manna <manme...@gmail.com> >> Gesendet: Donnerstag, 7. November 2019 22:10 >> An: users@kafka.apache.org >> Betreff: Re: Consumer Lags and receive no records anymore >> >> Have you checked your Kafka consumer group status ? How did you >> determine that your consumers are lagging ? >> >> Thanks, >> >> On Thu, 7 Nov 2019 at 20:55, Oliver Eckle <iee1...@gmx.de> wrote: >> >>> Hi there, >>> >>> >>> >>> have pretty strange behaviour questioned here already: >>> https://stackoverflow.com/q/58650416/7776688 >>> >>> >>> >>> As you could see from the logs: https://pastebin.com/yrSytSHD at a >>> specific point the client is stopping to receive records. >>> >>> I have a strong suspicion that it relates to performance on handling >>> the records - so that I run into kind of timeout. >>> >>> What seems to be strange, is that the client is not getting back and >>> heartbeats are processed successfully. >>> >>> Even the consumer will be returned on inspecting the consumer group. >>> Any idea .. kafka log has no error in it. >>> >>> >>> >>> Running a cluster with 3 broker inside a Kubernetes cluster, using >>> the bitnami helm chart. >>> >>> >>> >>> Kind Regards >>> >>> Oliver >>> >>> >>> >>> >>> >>> >> >> >
