I am one of the maintainers of prometheus-kafka-consumer-group-exporter[1],
which exports consumer group offsets and lag to Prometheus. The way we
currently scrape this information is by periodically executing
`kafka-consumer-groups.sh --describe` for each group and parse the output.

Recently the output from `kafka-consumer-groups.sh --describe` was
changed[2]. While I am working on a patch[3] to accomodate for the new
output format I was wondering if there is an easier, possibly more
stable[4] and more future proof, way for our project to extract the
information we are interested in. Does anyone know of a Go library that
could extract the metrics we need? Or would it make sense to refactor
`kafka-consumer-groups.sh` to support a more structured output? I'd love to
hear your input.

Also, if Kafka exported the same metrics through JMX our project would not
exist, but maybe that's another story...


[1] https://github.com/kawamuray/prometheus-kafka-consumer-group-exporter
[4] We've also encountered `kafka-consumer-groups.sh` hanging a few times
in production. There's a race condition somewhere in the script, most
likely when a topic is rebalancing. Currently we kill the process if it
doesn't finish within a timeout. See

Want to communicate with me securely? You can find my PGP key here

Reply via email to