Hello there -
On my current project we would like to accumulate usage stats for our
spouts. We want total number of tuples transfered and TOTAL NUMBER OF
BYTES TRANSFERED for all the spouts we are running.
It seems like Storm's built in metric's will give
us information on tuples emitted for spouts and bolts. But we could
not find anything in the Storm UI, or the docs that gave us any
information on total bytes transfered for spouts or bolts.
So,
First Question:
----------------
Is there any 'out of the box' way to
get metrics on total bytes x-fered by a spout ?
Our Contemplated Approach (If no 'out of the box' solution)
--------------------------------------------------------------
If we need to roll our own, we'll likely do the following:
* define subclasses of backtype.storm.task Class OutputCollector
( com.acme.MetricEnabledCollector )
* define constructor of MetricEnabledCollector to
i) take an existing OutputCollector instance as an argument
ii) set up a backtype.storm.metric.api.CountMetric to collect
our 'bytes transfered' metric.
* for every emit* method of MetricEnabledCollector we will
call a method to run over the tuple and compute the total size
of its payload. then we call countMetric.incr( calculatedPayloadSize )
* declare a sublcass of BaseRichSpout, AbstractAcmeSpout, whose
prepare method does the following:
@Override
public void open(Map conf, TopologyContext context,
SpoutOutputCollector collector) {
this.collector = new MetricEnabledCollector (collector );
}
* Any spout we implement will inherit from AbstractAcmeSpout-- that way
none of our
developers will forget and leave out metrics collection
And of course, we would define and register (with the topology) an instance
of
backtype.storm.metric.api.IMetricsConsumer to squirrel away
our metrics in some interesting place.
Feedback and advice most welcome...
Chris
--
Chris Bedford
Founder & Lead Lackey
Build Lackey Labs: http://buildlackey.com
Go Grails!: http://blog.buildlackey.com
