To investigate, you can read the messages on the topic to figure out how
consumer groups are committing. They're stored in a binary format, but you
can make them human readable using a formatter. Here's what I did last time
to do this - the classpath is required to make the formatter available to
the console producer, but you could do this in code with Kafka as a
dependency:
export CLASSPATH=/home/development/kafka_2.12-2.4.1/libs/* \
&& /home/development/kafka_2.12-2.4.1/bin/kafka-console-consumer.sh \
--bootstrap-server kafka01:9092 \
--topic __consumer_offsets \
--from-beginning \
--formatter
"kafka.coordinator.group.GroupMetadataManager\$OffsetsMessageFormatter"
> offsets.txt
In case the email formatting garbles the above, here it is in a gist
also <https://gist.github.com/LiamClarkeFMG/4a0f038a8d1782b1348e34b3ec471c21>.
You can then parse and analyse the output data using your preferred
scripting language.
Hope that helps,
Kind regards,
Liam Clarke-Hutchinson
On Thu, Jun 25, 2020 at 3:25 AM Karolis Pocius
<[email protected]> wrote:
> Check if any of your consumers have auto commit turned off and instead
> commit after processing each message.
>
> Also, even if all consumers are using auto commit, maybe some of them have
> the interval set to something crazy low like 1 ms.
>
> On Sat, Jun 20, 2020 at 8:31 PM Ashutosh singh <[email protected]> wrote:
>
> > Hi Guys,
> >
> > Hope you all are doing well.
> >
> > All of sudden I see very high throughput for _consumer_offsets topic.
> it
> > is around 30-40K per second. What could be the reason for such a high
> > rate ? Do I need to be concerned around this ?
> >
> > [image: image.png]
> >
> >
> > This happens for an hour and after that it goes down. Now it is
> happening
> > once or twice daily.
> >
> > I have 8 node cluster , 1000+ topics and 644 consumer groups.
> > All nodes have almost equal number of lead partition across nodes and
> > almost equal partition on all nodes.
> >
> > Kafka version : 2.1.1
> >
> > If you see above graph, there are no other topics where messages are more
> > than 300 Message/sec. only _consumer_offsets is having so high through
> > put.
> >
> > I know this is internal topic ans store metadata of topics and consumer
> > information. But I don't see anything abnormal in my cluster then why
> > _consumer_offset topic is going crazy. What is going on here ?
> >
> > Any help will be appreciated.
> >
> > --
> > Thanx & Regard
> > Ashutosh Singh
> > 08151945559
> >
> >
>
