Thanks Mark with custom consumer metric however spout can not record anything and beans are not serialized like JMX but for bolt operation and graphs certainly ok
So if there is a stat in a spout that never made to a bolt that metric can not be recorded in the consumer at least in the most recent version of storm Using JMX will slow down the whole topology leaving one option a Stats bolt with decent parallelism to which to send counters for example grouped in serialized java bean such a bolt can be emitted to from anywhere in topology believe emit Direct might to do it need to check... Storm needs "counter" feature like hadoop has but if there is a better way excluding JMX (but maybe JMX is the right thing for counters ? Please advice on any solid implementation of counters that does not compromise topology speed! Andrew On Saturday, February 15, 2014, Mark Hu <[email protected]> wrote: > One option includes statsd and graphite for safe metrics gathering and > easy graphing over time. > On Feb 15, 2014 4:09 PM, "Andrew Milkowski" > <[email protected]<javascript:_e(%7B%7D,'cvml','[email protected]');>> > wrote: > >> Hi >> >> I have a question regarding collecting of stats data from running topology >> >> for example in a simple scenario >> >> having let say counters >> >> storm_read >> bolt_A_read >> bolt_B_read >> >> and topology >> >> Spout -> Bolt A -> Bolt B >> >> Spout -> increments storm_read counter on every nextTuple >> >> Bolt A -> increments bolt_A_read counter on every execute >> Bolt B -> increments bolt_B_read counter on every execute >> >> and it is all accumulative >> >> Then to have an app to access these global statistics and display these >> counters >> >> What would be best way to go about this ? >> >> I did look into "metrics" but two things >> >> 1. don't seem to work with spout (I followed instructions at >> http://www.endgame.com/news/storm-metrics-how-to.html) >> >> 2. work with bolt but counters are not accumulative and reset (they act >> as what they are metrics data, for graphs etc) >> >> highly appreciate any advice on the subject! >> >> >> >> >>
