+1 We can probably start by opening a JIRA for this and adding a design approach for the same? I would like to help in the coding-effort for this.
-SG On Mon, Oct 10, 2016 at 1:51 PM, P. Taylor Goetz <ptgo...@gmail.com> wrote: > I’ve been thinking about metrics lately, specifically the fact that people > tend to struggle with implementing a metrics consumer. (Like this one [1]). > > The IMetricsConsumer interface is pretty low level, and common > aggregations, calculations, etc. are left up to each individual > implementation. That seems like an area where further abstraction would > make it easier to support different back ends (Graphite, JMX, Splunk, etc.). > > My thought is to create an abstract IMetricsConsumer implementation that > does common aggregations and calculations, and then delegates to a plugable > “metrics sink” implementation (e.g. “IMetricsSink”, etc.). That would > greatly simplify the effort required to integrate with various external > metrics systems. I know of at least a few users that would be interested, > one is currently scraping the logs from LoggingMetricsConsumer and polling > the Storm REST API for their metrics. > > -Taylor > > [1] http://twocentsonsoftware.blogspot.co.il/2014/12/ > sending-out-storm-metrics.html > > > > On Oct 10, 2016, at 12:14 PM, Bobby Evans <ev...@yahoo-inc.com.INVALID> > wrote: > > > > First of all the server exposes essentially the same interface that the > IMetricsConsumer exposes. It mostly just adds a bunch of overhead in the > middle to serialize out the objects send them over http to another process > which then has to deserialize them and process them. If you really don't > need the metrics to show up on a special known box you can have that exact > same code running inside the metrics consumer without all of the overhead. > > The server/client are insecure, have to deal with thread issues that a > normal IMetricsConsumer does not, and are not written to be robust (If the > HTTP server is down the consumer crashes and continues to crash until the > server is brought back up). It was written very quickly for a test > situation and it honestly never crossed my mind that anyone would want to > use it in production. > > > > - Bobby > > > > On Monday, October 10, 2016 10:59 AM, S G <sg.online.em...@gmail.com> > wrote: > > > > > > Thanks Bobby. > > > > If we write our own metrics consumer, how do we ensure that it is better > > than HttpForwardingMetricsServer? In other words, what aspects of the > > HttpForwardingMetricsServer > > should we avoid to make our own metrics consumer better and ready for > > production? > > > > Is versign/storm-graphite <https://github.com/verisign/storm-graphite> > > production > > ready? > > > > Also, we should add a line about production-readiness of > > HttpForwardingMetricsServer > > in the documentation at http://storm.apache.org/ > releases/1.0.2/Metrics.html > > (We were just about to think seriously on using this for production as we > > thought this to be the standard solution for metrics' consumption in 1.0+ > > version). > > > > -SG > > > > On Mon, Oct 10, 2016 at 6:37 AM, Bobby Evans <ev...@yahoo-inc.com> > wrote: > > > >> First of all there really are two different sets of metrics. One set is > >> the topology metrics and the other set is the daemon metrics (metrics > for > >> things like the ui and nimbus). The JmxPreparableReporter plugin only > >> exposes daemon metrics not the topology metrics through JMX. Exposing > >> topology metrics through JMX is a non trivial task. The current metrics > >> feature was not designed for this. We are in the process of trying to > >> redesign the metrics system to allow for features like this, but it is > >> still a ways off. > >> > >> - Bobby > >> > >> > >> On Saturday, October 8, 2016 11:39 AM, S G <sg.online.em...@gmail.com> > >> wrote: > >> > >> > >> Thanks Bobby, > >> > >> We will need some kind of IMetricsConsumer to talk to telegraf. > >> Many other softwares like Solr, Elastic-Search, Cassandra etc. provide > >> metrics through a URL making it very easy to consume by tools like > telegraf. > >> How about a IMetricsConsumer that will run on storm-ui and provide the > >> metrics through a URL such as <storm-ui-host>/metrics ? > >> > >> Also, I see the following option in defaults.yaml: > >> #default storm daemon metrics reporter plugins > >> storm.daemon.metrics.reporter.plugins: > >> - "org.apache.storm.daemon.metrics.reporters. > JmxPreparableReporter" > >> > >> Is this a good option to use for converting metrics into JMX ? > >> > >> Thanks > >> SG > >> > >> > >> > >> > >> On Fri, Oct 7, 2016 at 8:11 AM, Bobby Evans <ev...@yahoo-inc.com.invalid > > > >> wrote: > >> > >> HttpForwardingMetricsServer is a real hack intended really for tests. I > >> know I wrote it :). Please don't use it in production. You can write > your > >> own IMetricesConsumer to do whatever you want to with the metrics. > >> https://github.com/apache/ storm/blob/master/storm-core/ > >> src/jvm/org/apache/storm/ metric/api/IMetricsConsumer. java > >> <https://github.com/apache/storm/blob/master/storm-core/ > src/jvm/org/apache/storm/metric/api/IMetricsConsumer.java> > >> > >> That is the correct way to get the data out. If you want to write a > >> bridge to JMX for this that might work, but going directly to telegraph > >> would probably be better. - Bobby > >> > >> On Thursday, October 6, 2016 1:43 PM, S G < > sg.online.em...@gmail.com> > >> wrote: > >> > >> > >> Hi, > >> > >> We want to use Telegraf ( > >> https://github.com/influxdata/ telegraf/tree/master/plugins > >> <https://github.com/influxdata/telegraf/tree/master/plugins>) for > getting > >> storm's metrics. > >> > >> But we do not want to add a HttpForwardingMetricsServer just to get the > >> metrics and send them to telegraf. > >> > >> Other option is to use Jolokia (https://jolokia.org/) that can read JMX > >> and > >> write into telegraf. > >> > >> Does storm report all its metrics (including those of custom > spouts/bolts) > >> into JMX? > >> Or spawning a HttpForwardingMetricsServer is the only option? > >> > >> Thanks > >> SG > >> > >> > >> > >> > >> > >> > >> > >> > > > > > >
