Cody, Great idea! Let's try it out. I'll send the mail to user@ regarding use cases or concerns of metrics on Apache Storm. We may have a luck to find brilliant ideas or use cases we didn't know.
2016년 4월 19일 (화) 오후 2:07, Cody Innowhere <[email protected]>님이 작성: > Jungtaek, > Thanks for your explanation, still +1 for your proposal. > Moreover, I'd like to hear storm users about this, especially use cases for > metrics so that we can see what's best to do this. > > On Tue, Apr 19, 2016 at 11:19 AM, Jungtaek Lim <[email protected]> wrote: > > > Cody, > > Thanks for giving an opinion. > > > > I guess there was a miscommunication between you and me. > > This shouldn't be related to topology. Plugin should be set up via > cluster > > configuration, and singular daemon like Nimbus (master) or UI should > > publish cluster metrics to the plugin. > > I agree Nimbus should have this feature, but only master should publish > the > > cluster metrics so we should handle it. > > > > Btw, since we don't have built-in metrics storage, we can't provide query > > API with time-series fashion. > > I guess publishing (pushing) seems the better way what we can address > > without huge modification. > > (Did you mean Nimbus has query API which queries the external storage?) > > > > Please note that it's an idea for improvement against 1.x, not 2.0. > > In phase 2 we should evaluate and replace metrics feature with JStorm or > > another after porting to Java. > > > > Thanks, > > Jungtaek Lim (HeartSaVioR) > > > > 2016년 4월 19일 (화) 오전 11:59, Cody Innowhere <[email protected]>님이 작성: > > > > > I'm +1 for publishing cluster stats, but don't quite understand your > > > implementation. > > > Do you mean that a topology which wants cluster stats can get cluster > > stats > > > by implementing the pluggable consumer/reporter? > > > Personally I would prefer to put this role into nimbus (can be > pluggable > > > too) since it's the responsibility of nimbus to take care of all > > > topologies, naturally all topology stats. Then we expose API's from > > nimbus > > > for external queries. > > > > > > On Tue, Apr 19, 2016 at 9:47 AM, Jungtaek Lim <[email protected]> > wrote: > > > > > > > Hi devs, > > > > > > > > While Storm publishes topology metrics by metrics consumers, Storm > > > doesn't > > > > publish cluster metrics any way. > > > > > > > > Therefore, I'd like to introduce the feature of publishing cluster > > stats > > > to > > > > pluggable consumers so that users can also push to external storages > > and > > > > query on it or even configure dashboard. > > > > (For topology stats it's supported via MetricsConsumer.) > > > > > > > > I'm seeing some workarounds like having their own reporter process > > > polling > > > > cluster information from Nimbus or REST API. > > > > (For example, Ambari has cluster metrics reporter for Storm > > > > < > > > > > > > > > > https://github.com/apache/ambari/blob/trunk/ambari-metrics/ambari-metrics-storm-sink/src/main/java/org/apache/hadoop/metrics2/sink/storm/StormTimelineMetricsReporter.java > > > > > > > > > but it relies on custom build of Storm.) > > > > Yes it works anyway but if Storm provides the feature naturally, with > > > > pluggable way like MetricsConsumer it would be great for users who > want > > > to > > > > configure dashboard regarding this. > > > > > > > > I also saw that STORM-1158 < > > > > http://issues.apache.org/jira/browse/STORM-1158> > > > > adds metrics reporter for internal actions, but it has some > limitation > > on > > > > it. > > > > > > > > - It focuses how many requests the daemon receives, not cluster > stats. > > > > - It requires reporter to use codahale metrics. Moreover it shades > > > > codahale-metrics so other implementations of reporter may not work > > > properly > > > > if it uses other codahale-metrics plugin like ganglia. > > > > > > > > So I'm planning to add metrics feature of cluster stat by not relying > > on > > > > daemon metrics reporter, but introduce new pluggable > consumer/reporter. > > > > > > > > What do you think? Do you have opinion that we need to add cluster > > stats > > > to > > > > daemon metrics reporter? > > > > > > > > Thanks, > > > > Jungtaek Lim (HeartSaVioR) > > > > > > > > > >
