I'm +1 for publishing cluster stats, but don't quite understand your implementation. Do you mean that a topology which wants cluster stats can get cluster stats by implementing the pluggable consumer/reporter? Personally I would prefer to put this role into nimbus (can be pluggable too) since it's the responsibility of nimbus to take care of all topologies, naturally all topology stats. Then we expose API's from nimbus for external queries.
On Tue, Apr 19, 2016 at 9:47 AM, Jungtaek Lim <[email protected]> wrote: > Hi devs, > > While Storm publishes topology metrics by metrics consumers, Storm doesn't > publish cluster metrics any way. > > Therefore, I'd like to introduce the feature of publishing cluster stats to > pluggable consumers so that users can also push to external storages and > query on it or even configure dashboard. > (For topology stats it's supported via MetricsConsumer.) > > I'm seeing some workarounds like having their own reporter process polling > cluster information from Nimbus or REST API. > (For example, Ambari has cluster metrics reporter for Storm > < > https://github.com/apache/ambari/blob/trunk/ambari-metrics/ambari-metrics-storm-sink/src/main/java/org/apache/hadoop/metrics2/sink/storm/StormTimelineMetricsReporter.java > > > but it relies on custom build of Storm.) > Yes it works anyway but if Storm provides the feature naturally, with > pluggable way like MetricsConsumer it would be great for users who want to > configure dashboard regarding this. > > I also saw that STORM-1158 < > http://issues.apache.org/jira/browse/STORM-1158> > adds metrics reporter for internal actions, but it has some limitation on > it. > > - It focuses how many requests the daemon receives, not cluster stats. > - It requires reporter to use codahale metrics. Moreover it shades > codahale-metrics so other implementations of reporter may not work properly > if it uses other codahale-metrics plugin like ganglia. > > So I'm planning to add metrics feature of cluster stat by not relying on > daemon metrics reporter, but introduce new pluggable consumer/reporter. > > What do you think? Do you have opinion that we need to add cluster stats to > daemon metrics reporter? > > Thanks, > Jungtaek Lim (HeartSaVioR) >
