On each broker, we have a process (scheduled with cron) that polls the kafka jmx api every 60 seconds. It sends the metrics data to graphite (https://graphiteapp.org). We have graphite configured as a data source for grafana (https://grafana.com) and use it to build various dashboards to present the metrics we’re interested in.
There are various jmx-to-graphite tools available. We use one written in house, but this one looks like it’ll do the job: https://github.com/logzio/jmx2graphite > On Jan 16, 2019, at 2:15 PM, Amitav Mohanty <amitavmohant...@gmail.com> wrote: > > Peter, > > Thanks for the inputs. I am interested in aggregate bytes published into a > topic. The approach of metrics collector along with graphing tool seems > appealing. I can volume ingested over arbitrary periods of time which is > exactly what I am looking for. Can you please point to some metrics > collector that I can use? Is it sort of a cron-job that notes the rate > every minute or every 15 mins? > > Regards, > Amitav > > On Thu, Jan 17, 2019 at 3:23 AM Peter Bukowinski <pmb...@gmail.com> wrote: > >> Amitav, >> >> When you say total volume, do you want a topic’s size on disk, taking into >> account replication and retention, or do you want the aggregate bytes >> published into a topic? If you have a metrics collector and a graphing tool >> such as grafana, you can transform the rate metrics to a byte sum by >> applying an integral function, but those will always grow and not take into >> account deletion after the retention period. >> >> If you want metrics on how much space a topic occupies on disk, I’d >> suggest using collectd and this plugin: >> https://github.com/HubSpot/collectd-kafka-disk >> >> — >> Peter >> >>> On Jan 16, 2019, at 1:12 PM, Amitav Mohanty <amitavmohant...@gmail.com> >> wrote: >>> >>> Hi >>> >>> I am interested in getting total volume of data that a topic ingested in >> a >>> period of time. Does Kafka collect any such metrics? I check JMX console >>> but I only found rate metrics. >>> >>> Regards, >>> Amitav >> >>
