I agree that IMetricsConsumer is not good, but the reality is that all of the
metrics system needs to be redone. The problem is that we ship an object as a
metric. If I get an object I have no idea what it is hand hence no idea how to
report it or what to do with it. What is more the common types we use in the
metrics we provide are really not enough. For example CountMetric sends a
Long. Well when I get it in the metrics consumer I have no idea if I should
report it like a counter or if I should report it like a gauge (something that
every metrics system I have used wants to know). But then we support
pre-aggregation of the metrics with IReducer so the number I get might be an
average instead of either a gauge or a counter, which no good metrics system
will want to collect because I cannot aggregate it with anything else, the math
just does not work.
The proposal I have said before and I still believe is that we need to put in
place a parallel metrics API/system. We will deprecate all of
https://git.corp.yahoo.com/storm/storm/tree/master-security/storm-core/src/jvm/backtype/storm/metric/api
and create a new parallel one that provides an API similar to
http://metrics.dropwizard.io/3.1.0/. I would even be fine in just using their
API and exposing that to end users. Dropwizard has solved all of these
problems already and I don't see a reason to reinvent the wheel. I don't
personally see a lot of value in trying to send all of the metrics through
storm itslef. I am fine if we are able to support that, but it is far from a
requirement. - Bobby
On Monday, October 10, 2016 10:47 PM, S G <sg.online.em...@gmail.com> wrote:
+1
We can probably start by opening a JIRA for this and adding a design
approach for the same?
I would like to help in the coding-effort for this.
-SG
On Mon, Oct 10, 2016 at 1:51 PM, P. Taylor Goetz <ptgo...@gmail.com> wrote:
> I’ve been thinking about metrics lately, specifically the fact that people
> tend to struggle with implementing a metrics consumer. (Like this one [1]).
>
> The IMetricsConsumer interface is pretty low level, and common
> aggregations, calculations, etc. are left up to each individual
> implementation. That seems like an area where further abstraction would
> make it easier to support different back ends (Graphite, JMX, Splunk, etc.).
>
> My thought is to create an abstract IMetricsConsumer implementation that
> does common aggregations and calculations, and then delegates to a plugable
> “metrics sink” implementation (e.g. “IMetricsSink”, etc.). That would
> greatly simplify the effort required to integrate with various external
> metrics systems. I know of at least a few users that would be interested,
> one is currently scraping the logs from LoggingMetricsConsumer and polling
> the Storm REST API for their metrics.
>
> -Taylor
>
> [1] http://twocentsonsoftware.blogspot.co.il/2014/12/
> sending-out-storm-metrics.html
>
>
> > On Oct 10, 2016, at 12:14 PM, Bobby Evans <ev...@yahoo-inc.com.INVALID>
> wrote:
> >
> > First of all the server exposes essentially the same interface that the
> IMetricsConsumer exposes. It mostly just adds a bunch of overhead in the
> middle to serialize out the objects send them over http to another process
> which then has to deserialize them and process them. If you really don't
> need the metrics to show up on a special known box you can have that exact
> same code running inside the metrics consumer without all of the overhead.
> > The server/client are insecure, have to deal with thread issues that a
> normal IMetricsConsumer does not, and are not written to be robust (If the
> HTTP server is down the consumer crashes and continues to crash until the
> server is brought back up). It was written very quickly for a test
> situation and it honestly never crossed my mind that anyone would want to
> use it in production.
> >
> > - Bobby
> >
> > On Monday, October 10, 2016 10:59 AM, S G <sg.online.em...@gmail.com>
> wrote:
> >
> >
> > Thanks Bobby.
> >
> > If we write our own metrics consumer, how do we ensure that it is better
> > than HttpForwardingMetricsServer? In other words, what aspects of the
> > HttpForwardingMetricsServer
> > should we avoid to make our own metrics consumer better and ready for
> > production?
> >
> > Is versign/storm-graphite <https://github.com/verisign/storm-graphite>
> > production
> > ready?
> >
> > Also, we should add a line about production-readiness of
> > HttpForwardingMetricsServer
> > in the documentation at http://storm.apache.org/
> releases/1.0.2/Metrics.html
> > (We were just about to think seriously on using this for production as we
> > thought this to be the standard solution for metrics' consumption in 1.0+
> > version).
> >
> > -SG
> >
> > On Mon, Oct 10, 2016 at 6:37 AM, Bobby Evans <ev...@yahoo-inc.com>
> wrote:
> >
> >> First of all there really are two different sets of metrics. One set is
> >> the topology metrics and the other set is the daemon metrics (metrics
> for
> >> things like the ui and nimbus). The JmxPreparableReporter plugin only
> >> exposes daemon metrics not the topology metrics through JMX. Exposing
> >> topology metrics through JMX is a non trivial task. The current metrics
> >> feature was not designed for this. We are in the process of trying to
> >> redesign the metrics system to allow for features like this, but it is
> >> still a ways off.
> >>
> >> - Bobby
> >>
> >>
> >> On Saturday, October 8, 2016 11:39 AM, S G <sg.online.em...@gmail.com>
> >> wrote:
> >>
> >>
> >> Thanks Bobby,
> >>
> >> We will need some kind of IMetricsConsumer to talk to telegraf.
> >> Many other softwares like Solr, Elastic-Search, Cassandra etc. provide
> >> metrics through a URL making it very easy to consume by tools like
> telegraf.
> >> How about a IMetricsConsumer that will run on storm-ui and provide the
> >> metrics through a URL such as <storm-ui-host>/metrics ?
> >>
> >> Also, I see the following option in defaults.yaml:
> >> #default storm daemon metrics reporter plugins
> >> storm.daemon.metrics.reporter.plugins:
> >> - "org.apache.storm.daemon.metrics.reporters.
> JmxPreparableReporter"
> >>
> >> Is this a good option to use for converting metrics into JMX ?
> >>
> >> Thanks
> >> SG
> >>
> >>
> >>
> >>
> >> On Fri, Oct 7, 2016 at 8:11 AM, Bobby Evans <ev...@yahoo-inc.com.invalid
> >
> >> wrote:
> >>
> >> HttpForwardingMetricsServer is a real hack intended really for tests. I
> >> know I wrote it :). Please don't use it in production. You can write
> your
> >> own IMetricesConsumer to do whatever you want to with the metrics.
> >> https://github.com/apache/ storm/blob/master/storm-core/
> >> src/jvm/org/apache/storm/ metric/api/IMetricsConsumer. java
> >> <https://github.com/apache/storm/blob/master/storm-core/
> src/jvm/org/apache/storm/metric/api/IMetricsConsumer.java>
> >>
> >> That is the correct way to get the data out. If you want to write a
> >> bridge to JMX for this that might work, but going directly to telegraph
> >> would probably be better. - Bobby
> >>
> >> On Thursday, October 6, 2016 1:43 PM, S G <
> sg.online.em...@gmail.com>
> >> wrote:
> >>
> >>
> >> Hi,
> >>
> >> We want to use Telegraf (
> >> https://github.com/influxdata/ telegraf/tree/master/plugins
> >> <https://github.com/influxdata/telegraf/tree/master/plugins>) for
> getting
> >> storm's metrics.
> >>
> >> But we do not want to add a HttpForwardingMetricsServer just to get the
> >> metrics and send them to telegraf.
> >>
> >> Other option is to use Jolokia (https://jolokia.org/) that can read JMX
> >> and
> >> write into telegraf.
> >>
> >> Does storm report all its metrics (including those of custom
> spouts/bolts)
> >> into JMX?
> >> Or spawning a HttpForwardingMetricsServer is the only option?
> >>
> >> Thanks
> >> SG
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >
> >
>
>
