Hi, Earlier today I finished work on a library that adds an additional metrics reporter (https://github.com/pingles/kafka-riemann-reporter). I did it because although we were writing the CSV metrics we had nothing to display the data.
It pushes metrics to Riemann (http://riemann.io) a very cool tool for aggregating and processing event streams. We've experimented with it a little in the past but this is the first production integration we've had. Streams are processed by Riemann and then plotted by Librato ( http://librato.com). Installing requires building the project and copying the JAR to Kafka's ./libs dir and then putting the following in the server.properties: kafka.metrics.reporters=org.pingles.kafka.KafkaRiemannReporter kafka.riemann.metrics.reporter.enabled=true kafka.riemann.metrics.reporter.publisher=riemann kafka.riemann.metrics.reporter.publisher.host=127.0.0.1 kafka.riemann.metrics.reporter.publisher.port=5555 It lets you push directly from the broker to a Riemann service. However, we actually push messages (the Riemann events- protocol buffer messages) onto a Kafka topic to be later consumed by a bridged and forwarded to Riemann. This has the added benefit of allowing metrics to be flowed into our existing Redshift data load pipeline as well as being monitored in realtime. It's pretty early days but I deployed it to our small Kafka cluster on EC2 earlier and have been watching the graphs all day. Although we were writing metrics to CSVs before we had nothing to visualise it, this makes it pretty easy to pump the broker metrics out. I thought I'd mention it here in case other people are in a similar position to us and wanted an easy (alebit still a little experimental :) way to get the metrics out of Kafka. I'd really appreciate some feedback, and of course I'd be delighted if other people use it and help fix my mistakes :) At the moment it's built against an earlier 0.8 that we're running in production (9bd2a) but I'm keen to start working (and deploying) an officially released build. Paul
