Yes. Will PR this tonight Taylor. Thanks! On Monday, December 5, 2016, 12:37:18 PM CST, P. Taylor Goetz <[email protected]> wrote:Alessandro,
Are you in a position to open a pull request against the metrics_v2 branch? I’d like to start integrating the work I’ve been doing with the reporter configuration stuff you have. If what you have is incomplete/WIP, that’s not a big deal as the metrics_v2 branch is a feature branch and we’ll have plenty of opportunities to clean things up. -Taylor > On Nov 29, 2016, at 3:27 PM, Alessandro Bellina <[email protected]> > wrote: > > Taylor, > > Ok maybe there is some effort duplication. For the config, I have the bare > minimum to get the default reporter up. I'll focus on that since you could > use it. Will update JIRA with more. > > Alessandro > > > ----- Forwarded Message ----- > From: P. Taylor Goetz <[email protected]> > To: "[email protected]" <[email protected]> > Cc: S G <[email protected]>; [email protected] > <[email protected]>; Austin Chung <[email protected]> > Sent: Tuesday, November 29, 2016, 1:27:58 PM CST > Subject: Re: Are storm metrics reported through JMX too? > > Alessandro, > > Where do you stand with the reporter configuration via the storm.yaml config > file? > > I have metrics collection for workers and disruptor queues almost ready, but > now I’m looking for flexible configuration (right now I have reporters hard > coded). > > -Taylor > > > On Nov 29, 2016, at 1:57 PM, Alessandro Bellina > > <[email protected] <mailto:[email protected]>> > > wrote: > > > > S G, > > I am slowly making progress and plan to share something this week, > > specifically for hooking up a default codahale reporter to the workers. > > Also the U of I students for the open source class have made progress on > > their rocksdb implementation for the default store (the stuff I am doing > > should feed into their store). I don't think any of this is going to be in > > 1.1.0 given that it hasn't really been looked at by others, let alone > > reviewed. > > Thanks > > Alessandro > > On Monday, November 28, 2016, 4:51:34 PM CST, S G > > <[email protected] <mailto:[email protected]>> wrote:Hi, > > > > I see https://issues.apache.org/jira/browse/STORM-2153 > > <https://issues.apache.org/jira/browse/STORM-2153>being open for this > > and a lot of good discussion happening too. > > There is also a talk for releasing 1.1.0 version soon. > > > > So I just wanted to check if any metric related improvements will be > > available in 1.1.0. > > It would be great to try out these new improvements. > > > > Thanks > > SG > > > > On Tue, Oct 18, 2016 at 8:57 PM, Abhishek Nigam <[email protected] > > <mailto:[email protected]>> wrote: > > > >> Hi all, > >> > >> I'm Abhishek Nigam, one of the students from the group at the University of > >> Illinois; I'm really looking forward to working on this issue! We'll be > >> sure to keep you all updated as to how we progress. > >> > >> Cheers, > >> -- > >> Abhishek > >> > >> On Tue, Oct 18, 2016 at 1:23 PM Alessandro Bellina <[email protected] > >> <mailto:[email protected]> > >>> > >> wrote: > >> > >>> + Naren, Austin, and Abhishek, the students from the University of > >>> Illinois Open Source class. > >>> > >>> > >>> On Tuesday, October 11, 2016 11:48 PM, S G <[email protected] > >>> <mailto:[email protected]>> > >>> wrote: > >>> > >>> > >>> Response on this important issue is pretty good. I am happily surprised > >> :) > >>> > >>> I want to mention our strategy for extracting metrics from other > >> products. > >>> We use jolokia_proxy (https://jolokia.org/features/proxy.html > >>> <https://jolokia.org/features/proxy.html>) to get > >> JMX > >>> beans from several softwares and feed them to telegraf. That way, we > >> avoid > >>> writing custom processors for all these different products. > >>> > >>> Telegraf is quickly becoming a standard for metrics data. (Just see the > >>> list of input plugins here: > >>> https://github.com/influxdata/telegraf/tree/master/plugins/inputs > >>> <https://github.com/influxdata/telegraf/tree/master/plugins/inputs>). And > >> it > >>> integrates well with several outputs too ( > >>> https://github.com/influxdata/telegraf/tree/master/plugins/outputs > >>> <https://github.com/influxdata/telegraf/tree/master/plugins/outputs>). > >>> > >>> Also since the metrics in JMX, they can be queried by jolokia-agent > >>> installed per node. This avoids the extra metrics-consumer bolt which can > >>> hit the topology throughtput too. > >>> > >>> So I cast my vote in favor of JMX-implementation of metrics. > >>> Other approaches are welcome to be discussed. > >>> > >>> On Tue, Oct 11, 2016 at 7:30 PM, Alessandro Bellina < > >>> [email protected] <mailto:[email protected]>> > >>> wrote: > >>> > >>>> blockquote, div.yahoo_quoted { margin-left: 0 !important; > >>> border-left:1px > >>>> #715FFA solid !important; padding-left:1ex !important; > >>>> background-color:white !important; } Yeap that's a requirement from our > >>>> perspective (working through this list). > >>>> Sure I think as usual we can start with master with an eye for what > >> would > >>>> need to be back ported. > >>>> Sent from Yahoo Mail for iPhone > >>>> > >>>> > >>>> On Tuesday, October 11, 2016, 8:50 PM, P. Taylor Goetz < > >>> [email protected] <mailto:[email protected]>> > >>>> wrote: > >>>> > >>>> I hope I didn't come across as overly critical. You did the best with > >>> what > >>>> you had to work with. Which isn't pretty. > >>>> > >>>> We could potentially do a parallel metrics API in 1.1, 1.2, or master > >> and > >>>> still stay close to semantic versioning...? > >>>> > >>>> -Taylor > >>>> > >>>>> On Oct 11, 2016, at 9:28 PM, Jungtaek Lim <[email protected] > >>>>> <mailto:[email protected]>> wrote: > >>>>> > >>>>> Yeah I admit that configuration flag was bad for me also, but I have > >> no > >>>>> alternatives. Only way to avoid struggling with design limitation is > >>>> revamp > >>>>> / redesign. > >>>>> Thanks S G for exposing willingness of volunteer and great news > >>>> Alessandro > >>>>> for that project. > >>>>> Alessandro, could you forward the upcoming news for the project to > >> dev@ > >>>>> list? > >>>>> > >>>>> - Jungtaek Lim (HeartSaVioR) > >>>>> > >>>>> 2016년 10월 12일 (수) 오전 10:22, P. Taylor Goetz <[email protected] > >>>>> <mailto:[email protected]>>님이 > >> 작성: > >>>>> > >>>>>> I was thinking on a smaller scale in terms of effort, but the more I > >>>> think > >>>>>> about it, the more supportive I would be of a full revamp (new API) > >>> for > >>>>>> metrics based on Coda Hale's metrics library. It's proven and > >> stable. > >>>> I've > >>>>>> used it many times. I think either approach would be roughly the > >> same > >>>>>> amount of work. > >>>>>> > >>>>>> Some of the metrics API improvements in the 1.1.x branch are nice, > >> but > >>>>>> IMHO are lipstick on a pig. > >>>>>> > >>>>>> With apologies to Jungtaek, who has done amazing work all across the > >>>>>> codebase, I'm a little squeamish about the proposed change to > >> metrics > >>>> that > >>>>>> changes the consumer API based on a configuration flag (don't know > >> the > >>>> PR > >>>>>> number offhand). > >>>>>> > >>>>>> I'm +1 for moving in this direction (revamped metrics). Let's end > >> the > >>>>>> metrics pain. > >>>>>> > >>>>>> -Taylor > >>>>>> > >>>>>>> On Oct 11, 2016, at 10:15 AM, Bobby Evans < > >>> [email protected] <mailto:[email protected]> > >>>>> > >>>>>> wrote: > >>>>>>> > >>>>>>> I agree that IMetricsConsumer is not good, but the reality is that > >>> all > >>>>>> of the metrics system needs to be redone. The problem is that we > >> ship > >>>> an > >>>>>> object as a metric. If I get an object I have no idea what it is > >> hand > >>>>>> hence no idea how to report it or what to do with it. What is more > >>> the > >>>>>> common types we use in the metrics we provide are really not enough. > >>>> For > >>>>>> example CountMetric sends a Long. Well when I get it in the metrics > >>>>>> consumer I have no idea if I should report it like a counter or if I > >>>> should > >>>>>> report it like a gauge (something that every metrics system I have > >>> used > >>>>>> wants to know). But then we support pre-aggregation of the metrics > >>> with > >>>>>> IReducer so the number I get might be an average instead of either a > >>>> gauge > >>>>>> or a counter, which no good metrics system will want to collect > >>> because > >>>> I > >>>>>> cannot aggregate it with anything else, the math just does not work. > >>>>>>> The proposal I have said before and I still believe is that we need > >>> to > >>>>>> put in place a parallel metrics API/system. We will deprecate all > >> of > >>>>>> https://git.corp.yahoo.com/storm/storm/tree/master- > >>>>>> <https://git.corp.yahoo.com/storm/storm/tree/master-> > >>>> security/storm-core/src/jvm/backtype/storm/metric/api > >>>>>> and create a new parallel one that provides an API similar to > >>>>>> http://metrics.dropwizard.io/3.1.0/. > >>> <http://metrics.dropwizard.io/3.1.0/ > >>> <http://metrics.dropwizard.io/3.1.0/>> I would even be fine in just > >>>> using > >>>>>> their API and exposing that to end users. Dropwizard has solved all > >>> of > >>>>>> these problems already and I don't see a reason to reinvent the > >> wheel. > >>>> I > >>>>>> don't personally see a lot of value in trying to send all of the > >>> metrics > >>>>>> through storm itslef. I am fine if we are able to support that, but > >>> it > >>>> is > >>>>>> far from a requirement. - Bobby > >>>>>>> > >>>>>>> On Monday, October 10, 2016 10:47 PM, S G < > >>> [email protected] <mailto:[email protected]>> > >>>>>> wrote: > >>>>>>> > >>>>>>> > >>>>>>> +1 > >>>>>>> We can probably start by opening a JIRA for this and adding a > >> design > >>>>>>> approach for the same? > >>>>>>> I would like to help in the coding-effort for this. > >>>>>>> > >>>>>>> -SG > >>>>>>> > >>>>>>>> On Mon, Oct 10, 2016 at 1:51 PM, P. Taylor Goetz < > >> [email protected] <mailto:[email protected]> > >>>> > >>>>>> wrote: > >>>>>>>> > >>>>>>>> I’ve been thinking about metrics lately, specifically the fact > >> that > >>>>>> people > >>>>>>>> tend to struggle with implementing a metrics consumer. (Like this > >>> one > >>>>>> [1]). > >>>>>>>> > >>>>>>>> The IMetricsConsumer interface is pretty low level, and common > >>>>>>>> aggregations, calculations, etc. are left up to each individual > >>>>>>>> implementation. That seems like an area where further abstraction > >>>> would > >>>>>>>> make it easier to support different back ends (Graphite, JMX, > >>> Splunk, > >>>>>> etc.). > >>>>>>>> > >>>>>>>> My thought is to create an abstract IMetricsConsumer > >> implementation > >>>> that > >>>>>>>> does common aggregations and calculations, and then delegates to a > >>>>>> plugable > >>>>>>>> “metrics sink” implementation (e.g. “IMetricsSink”, etc.). That > >>> would > >>>>>>>> greatly simplify the effort required to integrate with various > >>>> external > >>>>>>>> metrics systems. I know of at least a few users that would be > >>>>>> interested, > >>>>>>>> one is currently scraping the logs from LoggingMetricsConsumer and > >>>>>> polling > >>>>>>>> the Storm REST API for their metrics. > >>>>>>>> > >>>>>>>> -Taylor > >>>>>>>> > >>>>>>>> [1] http://twocentsonsoftware.blogspot.co.il/2014/12/ > >>>>>>>> <http://twocentsonsoftware.blogspot.co.il/2014/12/> > >>>>>>>> sending-out-storm-metrics.html > >>>>>>>> > >>>>>>>> > >>>>>>>>> On Oct 10, 2016, at 12:14 PM, Bobby Evans > >>>> <[email protected] <mailto:[email protected]> > >>>>>>> > >>>>>>>> wrote: > >>>>>>>>> > >>>>>>>>> First of all the server exposes essentially the same interface > >> that > >>>> the > >>>>>>>> IMetricsConsumer exposes. It mostly just adds a bunch of overhead > >>> in > >>>>>> the > >>>>>>>> middle to serialize out the objects send them over http to another > >>>>>> process > >>>>>>>> which then has to deserialize them and process them. If you > >> really > >>>>>> don't > >>>>>>>> need the metrics to show up on a special known box you can have > >> that > >>>>>> exact > >>>>>>>> same code running inside the metrics consumer without all of the > >>>>>> overhead. > >>>>>>>>> The server/client are insecure, have to deal with thread issues > >>> that > >>>> a > >>>>>>>> normal IMetricsConsumer does not, and are not written to be robust > >>> (If > >>>>>> the > >>>>>>>> HTTP server is down the consumer crashes and continues to crash > >>> until > >>>>>> the > >>>>>>>> server is brought back up). It was written very quickly for a > >> test > >>>>>>>> situation and it honestly never crossed my mind that anyone would > >>> want > >>>>>> to > >>>>>>>> use it in production. > >>>>>>>>> > >>>>>>>>> - Bobby > >>>>>>>>> > >>>>>>>>> On Monday, October 10, 2016 10:59 AM, S G < > >>>>>> [email protected] <mailto:[email protected]>> > >>>>>>>> wrote: > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> Thanks Bobby. > >>>>>>>>> > >>>>>>>>> If we write our own metrics consumer, how do we ensure that it is > >>>>>> better > >>>>>>>>> than HttpForwardingMetricsServer? In other words, what aspects of > >>> the > >>>>>>>>> HttpForwardingMetricsServer > >>>>>>>>> should we avoid to make our own metrics consumer better and ready > >>> for > >>>>>>>>> production? > >>>>>>>>> > >>>>>>>>> Is versign/storm-graphite <https://github.com/verisign/ > >>>>>>>>> <https://github.com/verisign/> > >>>> storm-graphite> > >>>>>>>>> production > >>>>>>>>> ready? > >>>>>>>>> > >>>>>>>>> Also, we should add a line about production-readiness of > >>>>>>>>> HttpForwardingMetricsServer > >>>>>>>>> in the documentation at http://storm.apache.org/ > >>>>>>>>> <http://storm.apache.org/> > >>>>>>>> releases/1.0.2/Metrics.html > >>>>>>>>> (We were just about to think seriously on using this for > >> production > >>>> as > >>>>>> we > >>>>>>>>> thought this to be the standard solution for metrics' consumption > >>> in > >>>>>> 1.0+ > >>>>>>>>> version). > >>>>>>>>> > >>>>>>>>> -SG > >>>>>>>>> > >>>>>>>>> On Mon, Oct 10, 2016 at 6:37 AM, Bobby Evans < > >> [email protected] <mailto:[email protected]>> > >>>>>>>> wrote: > >>>>>>>>> > >>>>>>>>>> First of all there really are two different sets of metrics. > >> One > >>>> set > >>>>>> is > >>>>>>>>>> the topology metrics and the other set is the daemon metrics > >>>> (metrics > >>>>>>>> for > >>>>>>>>>> things like the ui and nimbus). The JmxPreparableReporter > >> plugin > >>>> only > >>>>>>>>>> exposes daemon metrics not the topology metrics through JMX. > >>>> Exposing > >>>>>>>>>> topology metrics through JMX is a non trivial task. The current > >>>>>> metrics > >>>>>>>>>> feature was not designed for this. We are in the process of > >>> trying > >>>> to > >>>>>>>>>> redesign the metrics system to allow for features like this, but > >>> it > >>>> is > >>>>>>>>>> still a ways off. > >>>>>>>>>> > >>>>>>>>>> - Bobby > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> On Saturday, October 8, 2016 11:39 AM, S G < > >>>> [email protected] <mailto:[email protected]> > >>>>>>> > >>>>>>>>>> wrote: > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> Thanks Bobby, > >>>>>>>>>> > >>>>>>>>>> We will need some kind of IMetricsConsumer to talk to telegraf. > >>>>>>>>>> Many other softwares like Solr, Elastic-Search, Cassandra etc. > >>>> provide > >>>>>>>>>> metrics through a URL making it very easy to consume by tools > >> like > >>>>>>>> telegraf. > >>>>>>>>>> How about a IMetricsConsumer that will run on storm-ui and > >> provide > >>>> the > >>>>>>>>>> metrics through a URL such as <storm-ui-host>/metrics ? > >>>>>>>>>> > >>>>>>>>>> Also, I see the following option in defaults.yaml: > >>>>>>>>>> #default storm daemon metrics reporter plugins > >>>>>>>>>> storm.daemon.metrics.reporter.plugins: > >>>>>>>>>> - "org.apache.storm.daemon.metrics.reporters. > >>>>>>>> JmxPreparableReporter" > >>>>>>>>>> > >>>>>>>>>> Is this a good option to use for converting metrics into JMX ? > >>>>>>>>>> > >>>>>>>>>> Thanks > >>>>>>>>>> SG > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> On Fri, Oct 7, 2016 at 8:11 AM, Bobby Evans > >>>>>> <[email protected] <mailto:[email protected]> > >>>>>>>>> > >>>>>>>>>> wrote: > >>>>>>>>>> > >>>>>>>>>> HttpForwardingMetricsServer is a real hack intended really for > >>>>>> tests. I > >>>>>>>>>> know I wrote it :). Please don't use it in production. You can > >>>> write > >>>>>>>> your > >>>>>>>>>> own IMetricesConsumer to do whatever you want to with the > >> metrics. > >>>>>>>>>> https://github.com/apache/ > >>>>>>>>>> <https://github.com/apache/>storm/blob/master/storm-core/ > >>>>>>>>>> src/jvm/org/apache/storm/ metric/api/IMetricsConsumer. java > >>>>>>>>>> <https://github.com/apache/storm/blob/master/storm-core/ > >>>>>>>>>> <https://github.com/apache/storm/blob/master/storm-core/> > >>>>>>>> src/jvm/org/apache/storm/metric/api/IMetricsConsumer.java> > >>>>>>>>>> > >>>>>>>>>> That is the correct way to get the data out. If you want to > >>> write a > >>>>>>>>>> bridge to JMX for this that might work, but going directly to > >>>>>> telegraph > >>>>>>>>>> would probably be better. - Bobby > >>>>>>>>>> > >>>>>>>>>> On Thursday, October 6, 2016 1:43 PM, S G < > >>>>>>>> [email protected] <mailto:[email protected]>> > >>>>>>>>>> wrote: > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> Hi, > >>>>>>>>>> > >>>>>>>>>> We want to use Telegraf ( > >>>>>>>>>> https://github.com/influxdata/ > >>>>>>>>>> <https://github.com/influxdata/>telegraf/tree/master/plugins > >>>>>>>>>> <https://github.com/influxdata/telegraf/tree/master/plugins > >>>>>>>>>> <https://github.com/influxdata/telegraf/tree/master/plugins>>) > >> for > >>>>>>>> getting > >>>>>>>>>> storm's metrics. > >>>>>>>>>> > >>>>>>>>>> But we do not want to add a HttpForwardingMetricsServer just to > >>> get > >>>>>> the > >>>>>>>>>> metrics and send them to telegraf. > >>>>>>>>>> > >>>>>>>>>> Other option is to use Jolokia (https://jolokia.org/ > >>>>>>>>>> <https://jolokia.org/>) that can > >>> read > >>>>>> JMX > >>>>>>>>>> and > >>>>>>>>>> write into telegraf. > >>>>>>>>>> > >>>>>>>>>> Does storm report all its metrics (including those of custom > >>>>>>>> spouts/bolts) > >>>>>>>>>> into JMX? > >>>>>>>>>> Or spawning a HttpForwardingMetricsServer is the only option? > >>>>>>>>>> > >>>>>>>>>> Thanks > >>>>>>>>>> SG > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>> > >>>>>> > >>>> > >>>> > >>>> > >>> > >>> > >>>
