I used a library from Codahale that took care of the pushing to graphite. Ultimately all it consists of a is a background thread that scans a ConcurrentHashMap and sends it over the desired protocol (TCP/UDP/HTTP).
I did have to do some segmentation per JVM as I run topologies on multiple workers. Something to keep in mind. On Wed, Jan 22, 2014 at 5:23 AM, bijoy deb <[email protected]>wrote: > Hi, > The approach to send metrics from Storm to something else will vary from > one charting service to another. > Did you use Ganglia as the charting service? To send the metrics to > Ganglia from Storm,I am looking for some api that can be used to push the > metrics to Ganglia server. > > Thanks > > > On Tue, Jan 21, 2014 at 7:52 PM, Mark Greene <[email protected]> wrote: > >> Ah sorry. So I use a singleton that runs a ScheduledThreadExecutor that >> ships the counters to my charting service. >> >> >> On Tue, Jan 21, 2014 at 12:05 AM, bijoy deb <[email protected]>wrote: >> >>> Hi Mark, >>> >>> I have the required code already in place that gathers the Storm metrics >>> via Nimbus api calls. >>> What I am looking for is a way to forward those to Ganglia to display on >>> Ganglia web screen. >>> >>> Thanks >>> Bijoy >>> >>> >>> On Tue, Jan 21, 2014 at 6:31 AM, Mark Greene <[email protected]> wrote: >>> >>>> Hey Bijoy, >>>> >>>> Not sure how you would go about capturing number of spouts/bolts but >>>> we've done some custom metrics capturing with respect to latencies inside >>>> spouts and bolts. >>>> >>>> For spouts, when you override the open method, you have access to a >>>> TopologyContext object. This is where you can add a TaskHook. We use this >>>> as a callback to record latencies for spout. >>>> >>>> For bolts similar situation but for the prepare method. >>>> >>>> You just also need to be aware that the sampling rate of these callback >>>> hooks are controlled from within the Topology Config object via conf. >>>> setStatsSampleRate(double). >>>> >>>> >>>> On Mon, Jan 20, 2014 at 12:05 PM, bijoy deb >>>> <[email protected]>wrote: >>>> >>>>> Hi, >>>>> >>>>> I am trying to use Ganglia to capture all the metrics that are >>>>> displayed in Storm UI. I am not quite sure on how to do that.I was able to >>>>> integrate Cassandra with Ganglia,where I just need to specify the metric >>>>> name I want to capture in a cassandra config file and Ganglia displays >>>>> that >>>>> metric.Is there anything similar for Storm as well? I know its possible to >>>>> capture the JVM,Memory stats of the Storm cluster,but I am more interested >>>>> in Storm specific parameters such as Nimbus config,Number of >>>>> spouts/bolts,Tuples emitted,Tuples acked,Latency,etc. >>>>> >>>>> Any help in this regard will be highly appreciated. >>>>> >>>> >>>> >>> >> >
