As far as I know, such metric does not exist. Strictly speaking, consumer lag
can be defined as the difference between the last produced offset (high
watermark) and the last committed offset by the group, but such metric has very
little value without considering the time dimension.
It'd be tricky for the broker to report on consumer 'lag', as the concept of
lag itself varies. You already know about Burrow (and I recall reading about
Uber's uGroup), and you already see that it considers a consumer lagging if it
is not making enough progress in a sliding time window (10 mins?). But other
tools/use cases can define lags using a different criteria (e.g. number of
messages exceeds a threshold).
I think because of these variances, it kinda makes sense for tools like Burrow
(and others) to be used for this purpose, instead of having the broker
dictating when consumers are lagging.
Just my two cents
From: dev@kafka.apache.org At: 08/16/22 15:06:16 UTC-4:00To:
us...@kafka.apache.org, show...@gmail.com, mmcfarl...@cavulus.com,
dev@kafka.apache.org, scante...@gmail.com, ranlupov...@gmail.com,
israele...@gmail.com
Subject: Re: Consumer Lag-Apache_kafka_JMX metrics
Hello Experts, Any info or pointers on my query please.
On Mon, Aug 15, 2022 at 11:36 PM Kafka Life wrote:
> Dear Kafka Experts
> we need to monitor the consumer lag in kafka clusters 2.5.1 and 2.8.0
> versions of kafka in Grafana.
>
> 1/ What is the correct path for JMX metrics to evaluate Consumer Lag in
> kafka cluster.
>
> 2/ I had thought it is FetcherLag but it looks like it is not as per the
> link below.
>
>
https://www.instaclustr.com/support/documentation/kafka/monitoring-information/f
etcher-lag-metrics/#:~:text=Aggregated%20Fetcher%20Consumer%20Lag%20This%20metri
c%20aggregates%20lag,in%20sync%20with%20partitions%20that%20it%20is%20replicatin
g
> .
>
> Could one of you experts please guide on which JMX i should use for
> consumer lag apart from kafka burrow or such intermediate tools
>
> Thanking you in advance
>
>