Jungtaek, Thanks for your explanation, still +1 for your proposal. Moreover, I'd like to hear storm users about this, especially use cases for metrics so that we can see what's best to do this.
On Tue, Apr 19, 2016 at 11:19 AM, Jungtaek Lim <[email protected]> wrote: > Cody, > Thanks for giving an opinion. > > I guess there was a miscommunication between you and me. > This shouldn't be related to topology. Plugin should be set up via cluster > configuration, and singular daemon like Nimbus (master) or UI should > publish cluster metrics to the plugin. > I agree Nimbus should have this feature, but only master should publish the > cluster metrics so we should handle it. > > Btw, since we don't have built-in metrics storage, we can't provide query > API with time-series fashion. > I guess publishing (pushing) seems the better way what we can address > without huge modification. > (Did you mean Nimbus has query API which queries the external storage?) > > Please note that it's an idea for improvement against 1.x, not 2.0. > In phase 2 we should evaluate and replace metrics feature with JStorm or > another after porting to Java. > > Thanks, > Jungtaek Lim (HeartSaVioR) > > 2016년 4월 19일 (화) 오전 11:59, Cody Innowhere <[email protected]>님이 작성: > > > I'm +1 for publishing cluster stats, but don't quite understand your > > implementation. > > Do you mean that a topology which wants cluster stats can get cluster > stats > > by implementing the pluggable consumer/reporter? > > Personally I would prefer to put this role into nimbus (can be pluggable > > too) since it's the responsibility of nimbus to take care of all > > topologies, naturally all topology stats. Then we expose API's from > nimbus > > for external queries. > > > > On Tue, Apr 19, 2016 at 9:47 AM, Jungtaek Lim <[email protected]> wrote: > > > > > Hi devs, > > > > > > While Storm publishes topology metrics by metrics consumers, Storm > > doesn't > > > publish cluster metrics any way. > > > > > > Therefore, I'd like to introduce the feature of publishing cluster > stats > > to > > > pluggable consumers so that users can also push to external storages > and > > > query on it or even configure dashboard. > > > (For topology stats it's supported via MetricsConsumer.) > > > > > > I'm seeing some workarounds like having their own reporter process > > polling > > > cluster information from Nimbus or REST API. > > > (For example, Ambari has cluster metrics reporter for Storm > > > < > > > > > > https://github.com/apache/ambari/blob/trunk/ambari-metrics/ambari-metrics-storm-sink/src/main/java/org/apache/hadoop/metrics2/sink/storm/StormTimelineMetricsReporter.java > > > > > > > but it relies on custom build of Storm.) > > > Yes it works anyway but if Storm provides the feature naturally, with > > > pluggable way like MetricsConsumer it would be great for users who want > > to > > > configure dashboard regarding this. > > > > > > I also saw that STORM-1158 < > > > http://issues.apache.org/jira/browse/STORM-1158> > > > adds metrics reporter for internal actions, but it has some limitation > on > > > it. > > > > > > - It focuses how many requests the daemon receives, not cluster stats. > > > - It requires reporter to use codahale metrics. Moreover it shades > > > codahale-metrics so other implementations of reporter may not work > > properly > > > if it uses other codahale-metrics plugin like ganglia. > > > > > > So I'm planning to add metrics feature of cluster stat by not relying > on > > > daemon metrics reporter, but introduce new pluggable consumer/reporter. > > > > > > What do you think? Do you have opinion that we need to add cluster > stats > > to > > > daemon metrics reporter? > > > > > > Thanks, > > > Jungtaek Lim (HeartSaVioR) > > > > > >
