Hi devs, While Storm publishes topology metrics by metrics consumers, Storm doesn't publish cluster metrics any way.
Therefore, I'd like to introduce the feature of publishing cluster stats to pluggable consumers so that users can also push to external storages and query on it or even configure dashboard. (For topology stats it's supported via MetricsConsumer.) I'm seeing some workarounds like having their own reporter process polling cluster information from Nimbus or REST API. (For example, Ambari has cluster metrics reporter for Storm <https://github.com/apache/ambari/blob/trunk/ambari-metrics/ambari-metrics-storm-sink/src/main/java/org/apache/hadoop/metrics2/sink/storm/StormTimelineMetricsReporter.java> but it relies on custom build of Storm.) Yes it works anyway but if Storm provides the feature naturally, with pluggable way like MetricsConsumer it would be great for users who want to configure dashboard regarding this. I also saw that STORM-1158 <http://issues.apache.org/jira/browse/STORM-1158> adds metrics reporter for internal actions, but it has some limitation on it. - It focuses how many requests the daemon receives, not cluster stats. - It requires reporter to use codahale metrics. Moreover it shades codahale-metrics so other implementations of reporter may not work properly if it uses other codahale-metrics plugin like ganglia. So I'm planning to add metrics feature of cluster stat by not relying on daemon metrics reporter, but introduce new pluggable consumer/reporter. What do you think? Do you have opinion that we need to add cluster stats to daemon metrics reporter? Thanks, Jungtaek Lim (HeartSaVioR)
