Hi Martin, Do you need the metric information to be written to logs? If that is not a hard constraint, replacing the 'LoggingMetricsConsumer' with a component that sends the metric data to a metric aggregation daemon like StatsD can solve your issue. All you need to make sure is that every metric corresponding to a task is uniquely identified across the Topology and StatsD will take care of the aggregation for you.
Regards, Yash On Sat, Feb 14, 2015 at 4:30 AM, Martin Illecker <[email protected]> wrote: > Hello, > > 1) I would like to measure and aggregate the tuples per second for a bolt, > which is running on multiple workers and multiple executors. > > Therefore I used the CountMetric [1] together with a > LoggingMetricsConsumer according to [2]. > But the results were spread among multiple worker logs and its executor. > How can I aggregate this data and get the average number of tuples per > second every 10 seconds? > > 2) Furthermore, I would also like to measure the end-to-end delay of the > whole topology. > Is there a better way than propagating the emitting time from the spout to > the last bolt? > And similar to 1), how can I finally aggregate the calculated end-to-end > delay among multiple workers and supervisors? > > What would be the best solution to get these aggregated measurements of > tuples per second and end-to-end delay without impacting the performance? > I would prefer one global LoggingMetricsConsumer. > > Thanks! > Best regards > Martin > > [1] > https://github.com/nathanmarz/storm/blob/master/storm-core/src/jvm/backtype/storm/metric/api/CountMetric.java > [2] https://www.endgame.com/blog/storm-metrics-how-to.html >
