Hi Philip, Please find logs of Kafka, Zookeeper, and Service logs from the below link. https://github.com/santhoshadityam/Kafka_consumer_issues
Thanks, Santhosh Aditya From: Marigowda, Santhosh Aditya Sent: Friday, February 9, 2024 11:55 AM To: Philip Nee <philip...@gmail.com>; Jain, Ankit <ankit.j...@unisys.com> Cc: dev@kafka.apache.org; Namboodiri, Vishnu <vishnu.nambood...@in.unisys.com>; Mudlapur, Rajesh <rajesh.mudla...@au.unisys.com>; Reddy, Thanmai <thanmai.re...@unisys.com> Subject: RE: Kafka consumer group crashing and not able to consume once service is up Hi Philip, Are you expecting logs from subscriber from our service please do let me know. Thanks, Santhosh Aditya From: Marigowda, Santhosh Aditya Sent: Friday, February 9, 2024 11:32 AM To: Philip Nee <philip...@gmail.com<mailto:philip...@gmail.com>>; Jain, Ankit <ankit.j...@unisys.com<mailto:ankit.j...@unisys.com>> Cc: dev@kafka.apache.org<mailto:dev@kafka.apache.org>; Namboodiri, Vishnu <vishnu.nambood...@in.unisys.com<mailto:vishnu.nambood...@in.unisys.com>>; Mudlapur, Rajesh <rajesh.mudla...@au.unisys.com<mailto:rajesh.mudla...@au.unisys.com>>; Reddy, Thanmai <thanmai.re...@unisys.com<mailto:thanmai.re...@unisys.com>> Subject: RE: Kafka consumer group crashing and not able to consume once service is up Hi Philip, Please find the service logs(Subscriber logs). We don’t see any logs-related issues with consumer. Kafka consumer configuration kafka-consumer = { server= "10.221.104.150:30092" deserializer="org.apache.kafka.common.serialization.StringDeserializer" auto_offset_reset="latest" enable_auto_commit="true" maxRequestSize=20971520 } Thanks, Santhosh Aditya From: Philip Nee <philip...@gmail.com<mailto:philip...@gmail.com>> Sent: Thursday, February 8, 2024 11:27 PM To: Jain, Ankit <ankit.j...@unisys.com<mailto:ankit.j...@unisys.com>> Cc: Marigowda, Santhosh Aditya <santhosh.marigo...@unisys.com<mailto:santhosh.marigo...@unisys.com>>; dev@kafka.apache.org<mailto:dev@kafka.apache.org>; Namboodiri, Vishnu <vishnu.nambood...@in.unisys.com<mailto:vishnu.nambood...@in.unisys.com>>; Mudlapur, Rajesh <rajesh.mudla...@au.unisys.com<mailto:rajesh.mudla...@au.unisys.com>>; Reddy, Thanmai <thanmai.re...@unisys.com<mailto:thanmai.re...@unisys.com>> Subject: Re: Kafka consumer group crashing and not able to consume once service is up Hi, I believe what you sent was the broker log not the client log - client log would be useful to debug the issue. Also, the configuration you provided seems to be your application log - there are a few configuration I'm not sure what they Hi, I believe what you sent was the broker log not the client log - client log would be useful to debug the issue. Also, the configuration you provided seems to be your application log - there are a few configuration I'm not sure what they actually mean, e.g. delay, timeout. P On Wed, Feb 7, 2024 at 11:49 PM Jain, Ankit <ankit.j...@unisys.com<mailto:ankit.j...@unisys.com>> wrote: Update the below response. From: Marigowda, Santhosh Aditya <santhosh.marigo...@unisys.com<mailto:santhosh.marigo...@unisys.com>> Sent: Thursday, February 8, 2024 1:14 PM To: Philip Nee <philip...@gmail.com<mailto:philip...@gmail.com>>; dev@kafka.apache.org<mailto:dev@kafka.apache.org> Cc: Jain, Ankit <ankit.j...@unisys.com<mailto:ankit.j...@unisys.com>>; Namboodiri, Vishnu <vishnu.nambood...@in.unisys.com<mailto:vishnu.nambood...@in.unisys.com>>; Mudlapur, Rajesh <rajesh.mudla...@au.unisys.com<mailto:rajesh.mudla...@au.unisys.com>>; Reddy, Thanmai <thanmai.re...@unisys.com<mailto:thanmai.re...@unisys.com>>; Marigowda, Santhosh Aditya <santhosh.marigo...@unisys.com<mailto:santhosh.marigo...@unisys.com>> Subject: RE: Kafka consumer group crashing and not able to consume once service is up Hi Philip, Thanks for your queries, Please find my response. Do you have a single-member consumer group? Yes, We have multiple consumer groups (CG1, CG2, CG3, CG4) with single member. Did you also mean after restarting the "service", the service was not able to resume consuming the messages? Yes, Not able to resume. could you also include the client version : 3.5.0 [cid:image001.png@01DA5B5C.18AA6530] Logs would be very helpful to see why these consumers stopped consuming messages: If any message is available in the topic will be subscribed and printed below in the below logs "Entered ReplicationDataListener.onMessage method". In our case below logs are not at all present in the service logs. I have attached Zookeper and Kafka logs for reference. --Code to consume messages public void onMessage(Vertx vertx, KafkaConsumer consumer, String jsonObject) { logger.info("Entered ReplicationDataListener.onMessage method"); initialize(vertx); service = SyncServiceImpl.getInstance(vertx); try { } The setup looks like below. [cid:image002.png@01DA5B5C.18AA6530] Please refer below image when service1 shuts down/crashes down. After bringing up Service1 none of the messages being consumed by service1. [cid:image003.png@01DA5B5C.18AA6530] Workaround solution, Rename consumer name from CG1 to CG5 then restart Service1, messages are started consuming. [cid:image004.png@01DA5B5C.18AA6530] Thanks, Santhosh Aditya From: Reddy, Thanmai <thanmai.re...@unisys.com<mailto:thanmai.re...@unisys.com>> Sent: Thursday, February 8, 2024 11:33 AM To: Marigowda, Santhosh Aditya <santhosh.marigo...@unisys.com<mailto:santhosh.marigo...@unisys.com>> Subject: FW: Kafka consumer group crashing and not able to consume once service is up From: Philip Nee <philip...@gmail.com<mailto:philip...@gmail.com>> Sent: Thursday, February 8, 2024 11:26 AM To: dev@kafka.apache.org<mailto:dev@kafka.apache.org> Cc: Jain, Ankit <ankit.j...@unisys.com<mailto:ankit.j...@unisys.com>>; Namboodiri, Vishnu <vishnu.nambood...@in.unisys.com<mailto:vishnu.nambood...@in.unisys.com>>; Mudlapur, Rajesh <rajesh.mudla...@au.unisys.com<mailto:rajesh.mudla...@au.unisys.com>>; Reddy, Thanmai <thanmai.re...@unisys.com<mailto:thanmai.re...@unisys.com>> Subject: Re: Kafka consumer group crashing and not able to consume once service is up Hi Santhosh, Your problem statement confuses me a bit (apologize). You mentioned "if one of the kafka consumer(Service)" - Do you have a single member consumer group? Could you elaborate on the setup a bit? Did you also mean after Hi Santhosh, Your problem statement confuses me a bit (apologize). You mentioned "if one of the kafka consumer(Service)" - Do you have a single member consumer group? Could you elaborate on the setup a bit? Did you also mean after restarting the "service", the service was not able to resume from consuming the messages? Logs would be very helpful to see why these consumers stopped consuming messages. If you include the logs (that would be really helpful), could you also include the client version. P On Wed, Feb 7, 2024 at 9:45 PM Marigowda, Santhosh Aditya <santhosh.marigo...@unisys.com.invalid<mailto:santhosh.marigo...@unisys.com.invalid>> wrote: Hi Dev team, Could you Please let me know if any updates on my query. Thanks, Santhosh Aditya From: Marigowda, Santhosh Aditya Sent: Wednesday, January 31, 2024 8:50 AM To: dev@kafka.apache.org<mailto:dev@kafka.apache.org> Cc: Jain, Ankit <ankit.j...@unisys.com<mailto:ankit.j...@unisys.com>>; Namboodiri, Vishnu <vishnu.nambood...@in.unisys.com<mailto:vishnu.nambood...@in.unisys.com>>; Mudlapur, Rajesh <rajesh.mudla...@au.unisys.com<mailto:rajesh.mudla...@au.unisys.com>>; Reddy, Thanmai <thanmai.re...@unisys.com<mailto:thanmai.re...@unisys.com>> Subject: RE: Kafka consumer group crashing and not able to consume once service is up Thanks Matthias for your response. Sorry, my bad please find the updated Image. • We have 1 topic and 4 partitions, Each consumer is pointing to the topic. There is no partition assignment specific to the consumer group from our software code. • Please let me know if you need Kafka server logs or consumer service logs, Post restart of our service no logs are printed in onMessage method. • We are using consumer.subscribe(topic) Thanks, Santhosh Aditya From: Matthias J. Sax <mj...@apache.org<mailto:mj...@apache.org>> Sent: Wednesday, January 31, 2024 12:13 AM To: dev@kafka.apache.org<mailto:dev@kafka.apache.org> Subject: Re: Kafka consumer group crashing and not able to consume once service is up I am a not sure if I can follow completely. From the figures you show, you have a topic with 4 partitions, and 4 consumer groups. Thus, each consumer group should read all 4 partitions, but the figure indicate that each group would read a single I am a not sure if I can follow completely. From the figures you show, you have a topic with 4 partitions, and 4 consumer groups. Thus, each consumer group should read all 4 partitions, but the figure indicate that each group would read a single partition only? Can you clarify? Are you using `consumer.subscribe` or `consumer.assign`? In general, might be good too collect some INFO (or DEBUG) level logs for the crashing service after restart to see what it's doing. -Matthias On 1/30/24 7:17 AM, Marigowda, Santhosh Aditya wrote: > Hi Kafka Dev Team, > > Could you please help us with our problem. > > In our POC, if one of the kafka consumer(Service) shuts down or crashes > then post restart of service none of the messages are getting consumed > by the crashed Service. > > Other services are consuming without any issues. > > One of service crash/Shutdown > > If we rename the Kafka consumer group name and start the service then > messages start consuming. > > Consumer configuration : > > { > > delay:1000 > > timeout:0 > > topic-name= test > > handlers=["Listener"] > > source="kafka-consumer" > > enable_auto_commit="false" > > group="Consumer-Group-1" > > } > > local-kafka-consumer = { > > server= > "{kafka-hostname}:{kafka-port}" > > > deserializer="org.apache.kafka.common.serialization.StringDeserializer" > > auto_offset_reset="latest" > > enable_auto_commit="true" > > maxRequestSize=20971520 > > } > > Thanks, > > Santhosh Aditya >
