Paul, Thanks for sharing.
Jun On Thu, Jul 18, 2013 at 2:00 PM, Paul Ingles <[email protected]> wrote: > Hi, > > Earlier today I finished work on a library that adds an additional metrics > reporter (https://github.com/pingles/kafka-riemann-reporter). I did it > because although we were writing the CSV metrics we had nothing to display > the data. > > It pushes metrics to Riemann (http://riemann.io) a very cool tool for > aggregating and processing event streams. We've experimented with it a > little in the past but this is the first production integration we've had. > Streams are processed by Riemann and then plotted by Librato ( > http://librato.com). > > Installing requires building the project and copying the JAR to Kafka's > ./libs dir and then putting the following in the server.properties: > > kafka.metrics.reporters=org.pingles.kafka.KafkaRiemannReporter > kafka.riemann.metrics.reporter.enabled=true > kafka.riemann.metrics.reporter.publisher=riemann > kafka.riemann.metrics.reporter.publisher.host=127.0.0.1 > kafka.riemann.metrics.reporter.publisher.port=5555 > > It lets you push directly from the broker to a Riemann service. However, we > actually push messages (the Riemann events- protocol buffer messages) onto > a Kafka topic to be later consumed by a bridged and forwarded to Riemann. > This has the added benefit of allowing metrics to be flowed into our > existing Redshift data load pipeline as well as being monitored in > realtime. > > It's pretty early days but I deployed it to our small Kafka cluster on EC2 > earlier and have been watching the graphs all day. Although we were writing > metrics to CSVs before we had nothing to visualise it, this makes it pretty > easy to pump the broker metrics out. > > I thought I'd mention it here in case other people are in a similar > position to us and wanted an easy (alebit still a little experimental :) > way to get the metrics out of Kafka. > > I'd really appreciate some feedback, and of course I'd be delighted if > other people use it and help fix my mistakes :) > > At the moment it's built against an earlier 0.8 that we're running in > production (9bd2a) but I'm keen to start working (and deploying) an > officially released build. > > Paul >
