This post from Gil Tene sums up the Metrics package's issues with measuring latency well: https://groups.google.com/forum/#!msg/mechanical-sympathy/I4JfZQ1GYi8/LKc3MYXRGWUJ
Date: Mon, 16 Jun 2014 19:00:30 -0600 Subject: RE: Extracting Performance Metrics From: [email protected] To: [email protected] What kind of issues does Metrics have that leads you to recommend HdrHistogram? On Jun 16, 2014 6:57 PM, "Dan" <[email protected]> wrote: Be careful when using Coda Hale's Metrics package when measuring latency. Consider using Gil Tene'sHigh Dynamic Range Histogram instead: http://hdrhistogram.github.io/HdrHistogram/ -Dan From: [email protected] Date: Mon, 16 Jun 2014 18:20:11 -0400 Subject: Re: Extracting Performance Metrics To: [email protected] Also, I came across this presentation by Visible Measures which actually walks through a lot of great options covering most of what you want to know about: http://files.meetup.com/5809742/storm%20monitoring.pdf One other thing to be aware of is that in Storm 0.9.2 (forthcoming release), there is a new REST API used by the Storm UI for gathering some of these metrics: https://github.com/apache/incubator-storm/pull/101 https://issues.apache.org/jira/browse/STORM-205 On Mon, Jun 16, 2014 at 6:13 PM, Andrew Montalenti <[email protected]> wrote: I haven't used it yet, but a lot of people get pointed to metrics_storm: https://github.com/ooyala/metrics_storm With this blog post that discusses it: http://engineering.ooyala.com/blog/open-sourcing-metrics-storm Michael Noll also has a nice blog post about streaming Storm 0.9 metrics to Graphite: http://www.michael-noll.com/blog/2013/11/06/sending-metrics-from-storm-to-graphite/ Currently, when we use Storm, we do a lot of custom metrics in Graphite using statsd, as described in this post (not about Storm, but about Graphite/statsd): http://codeascraft.com/2011/02/15/measure-anything-measure-everything/ On Mon, Jun 16, 2014 at 4:37 PM, Anis Nasir <[email protected]> wrote: Dear all, I am running a cluster with 1 kafka + 1 nimbus + 10 supervisor + 1 zookeeper nodes. I am executing multiple topologies on the cluster and I want to extract different metrics that I am mentioning below. Can someone help me by recommending tools that I can use to extract this information. Per Topology - Throughput - Latency Per Spout or Bolt - Throughput - Latency - Execution Time - Queuing Time - Number of Messages Processed RegardsAnis
