...the classpath is required to make the formatter available to the console producer consumer...
On Thu, Jun 25, 2020 at 9:44 AM Liam Clarke-Hutchinson < liam.cla...@adscale.co.nz> wrote: > To investigate, you can read the messages on the topic to figure out how > consumer groups are committing. They're stored in a binary format, but you > can make them human readable using a formatter. Here's what I did last time > to do this - the classpath is required to make the formatter available to > the console producer, but you could do this in code with Kafka as a > dependency: > > export CLASSPATH=/home/development/kafka_2.12-2.4.1/libs/* \ > && /home/development/kafka_2.12-2.4.1/bin/kafka-console-consumer.sh \ > --bootstrap-server kafka01:9092 \ > --topic __consumer_offsets \ > --from-beginning \ > --formatter > "kafka.coordinator.group.GroupMetadataManager\$OffsetsMessageFormatter" > > offsets.txt > > In case the email formatting garbles the above, here it is in a gist also > <https://gist.github.com/LiamClarkeFMG/4a0f038a8d1782b1348e34b3ec471c21>. You > can then parse and analyse the output data using your preferred scripting > language. > > Hope that helps, > > Kind regards, > > Liam Clarke-Hutchinson > > > > > On Thu, Jun 25, 2020 at 3:25 AM Karolis Pocius > <karolis.poc...@sentiance.com.invalid> wrote: > >> Check if any of your consumers have auto commit turned off and instead >> commit after processing each message. >> >> Also, even if all consumers are using auto commit, maybe some of them have >> the interval set to something crazy low like 1 ms. >> >> On Sat, Jun 20, 2020 at 8:31 PM Ashutosh singh <getas...@gmail.com> >> wrote: >> >> > Hi Guys, >> > >> > Hope you all are doing well. >> > >> > All of sudden I see very high throughput for _consumer_offsets topic. >> it >> > is around 30-40K per second. What could be the reason for such a high >> > rate ? Do I need to be concerned around this ? >> > >> > [image: image.png] >> > >> > >> > This happens for an hour and after that it goes down. Now it is >> happening >> > once or twice daily. >> > >> > I have 8 node cluster , 1000+ topics and 644 consumer groups. >> > All nodes have almost equal number of lead partition across nodes and >> > almost equal partition on all nodes. >> > >> > Kafka version : 2.1.1 >> > >> > If you see above graph, there are no other topics where messages are >> more >> > than 300 Message/sec. only _consumer_offsets is having so high through >> > put. >> > >> > I know this is internal topic ans store metadata of topics and consumer >> > information. But I don't see anything abnormal in my cluster then why >> > _consumer_offset topic is going crazy. What is going on here ? >> > >> > Any help will be appreciated. >> > >> > -- >> > Thanx & Regard >> > Ashutosh Singh >> > 08151945559 >> > >> > >> >
