Taylor, Ok maybe there is some effort duplication. For the config, I have the bare minimum to get the default reporter up. I'll focus on that since you could use it. Will update JIRA with more. Alessandro
----- Forwarded Message -----From: P. Taylor Goetz <ptgo...@gmail.com>To: "dev@storm.apache.org" <dev@storm.apache.org>Cc: S G <sg.online.em...@gmail.com>; na...@narendasan.com <na...@narendasan.com>; Austin Chung <achun...@illinois.edu>Sent: Tuesday, November 29, 2016, 1:27:58 PM CST Subject: Re: Are storm metrics reported through JMX too? Alessandro, Where do you stand with the reporter configuration via the storm.yaml config file? I have metrics collection for workers and disruptor queues almost ready, but now I’m looking for flexible configuration (right now I have reporters hard coded). -Taylor > On Nov 29, 2016, at 1:57 PM, Alessandro Bellina > <abell...@yahoo-inc.com.INVALID> wrote: > > S G, > I am slowly making progress and plan to share something this week, > specifically for hooking up a default codahale reporter to the workers. Also > the U of I students for the open source class have made progress on their > rocksdb implementation for the default store (the stuff I am doing should > feed into their store). I don't think any of this is going to be in 1.1.0 > given that it hasn't really been looked at by others, let alone reviewed. > Thanks > Alessandro > On Monday, November 28, 2016, 4:51:34 PM CST, S G <sg.online.em...@gmail.com> > wrote:Hi, > > I see https://issues.apache.org/jira/browse/STORM-2153 being open for this > and a lot of good discussion happening too. > There is also a talk for releasing 1.1.0 version soon. > > So I just wanted to check if any metric related improvements will be > available in 1.1.0. > It would be great to try out these new improvements. > > Thanks > SG > > On Tue, Oct 18, 2016 at 8:57 PM, Abhishek Nigam <adn5...@gmail.com> wrote: > >> Hi all, >> >> I'm Abhishek Nigam, one of the students from the group at the University of >> Illinois; I'm really looking forward to working on this issue! We'll be >> sure to keep you all updated as to how we progress. >> >> Cheers, >> -- >> Abhishek >> >> On Tue, Oct 18, 2016 at 1:23 PM Alessandro Bellina <abell...@yahoo-inc.com >>> >> wrote: >> >>> + Naren, Austin, and Abhishek, the students from the University of >>> Illinois Open Source class. >>> >>> >>> On Tuesday, October 11, 2016 11:48 PM, S G <sg.online.em...@gmail.com> >>> wrote: >>> >>> >>> Response on this important issue is pretty good. I am happily surprised >> :) >>> >>> I want to mention our strategy for extracting metrics from other >> products. >>> We use jolokia_proxy (https://jolokia.org/features/proxy.html) to get >> JMX >>> beans from several softwares and feed them to telegraf. That way, we >> avoid >>> writing custom processors for all these different products. >>> >>> Telegraf is quickly becoming a standard for metrics data. (Just see the >>> list of input plugins here: >>> https://github.com/influxdata/telegraf/tree/master/plugins/inputs). And >> it >>> integrates well with several outputs too ( >>> https://github.com/influxdata/telegraf/tree/master/plugins/outputs). >>> >>> Also since the metrics in JMX, they can be queried by jolokia-agent >>> installed per node. This avoids the extra metrics-consumer bolt which can >>> hit the topology throughtput too. >>> >>> So I cast my vote in favor of JMX-implementation of metrics. >>> Other approaches are welcome to be discussed. >>> >>> On Tue, Oct 11, 2016 at 7:30 PM, Alessandro Bellina < >>> abell...@yahoo-inc.com.invalid> wrote: >>> >>>> blockquote, div.yahoo_quoted { margin-left: 0 !important; >>> border-left:1px >>>> #715FFA solid !important; padding-left:1ex !important; >>>> background-color:white !important; } Yeap that's a requirement from our >>>> perspective (working through this list). >>>> Sure I think as usual we can start with master with an eye for what >> would >>>> need to be back ported. >>>> Sent from Yahoo Mail for iPhone >>>> >>>> >>>> On Tuesday, October 11, 2016, 8:50 PM, P. Taylor Goetz < >>> ptgo...@gmail.com> >>>> wrote: >>>> >>>> I hope I didn't come across as overly critical. You did the best with >>> what >>>> you had to work with. Which isn't pretty. >>>> >>>> We could potentially do a parallel metrics API in 1.1, 1.2, or master >> and >>>> still stay close to semantic versioning...? >>>> >>>> -Taylor >>>> >>>>> On Oct 11, 2016, at 9:28 PM, Jungtaek Lim <kabh...@gmail.com> wrote: >>>>> >>>>> Yeah I admit that configuration flag was bad for me also, but I have >> no >>>>> alternatives. Only way to avoid struggling with design limitation is >>>> revamp >>>>> / redesign. >>>>> Thanks S G for exposing willingness of volunteer and great news >>>> Alessandro >>>>> for that project. >>>>> Alessandro, could you forward the upcoming news for the project to >> dev@ >>>>> list? >>>>> >>>>> - Jungtaek Lim (HeartSaVioR) >>>>> >>>>> 2016년 10월 12일 (수) 오전 10:22, P. Taylor Goetz <ptgo...@gmail.com>님이 >> 작성: >>>>> >>>>>> I was thinking on a smaller scale in terms of effort, but the more I >>>> think >>>>>> about it, the more supportive I would be of a full revamp (new API) >>> for >>>>>> metrics based on Coda Hale's metrics library. It's proven and >> stable. >>>> I've >>>>>> used it many times. I think either approach would be roughly the >> same >>>>>> amount of work. >>>>>> >>>>>> Some of the metrics API improvements in the 1.1.x branch are nice, >> but >>>>>> IMHO are lipstick on a pig. >>>>>> >>>>>> With apologies to Jungtaek, who has done amazing work all across the >>>>>> codebase, I'm a little squeamish about the proposed change to >> metrics >>>> that >>>>>> changes the consumer API based on a configuration flag (don't know >> the >>>> PR >>>>>> number offhand). >>>>>> >>>>>> I'm +1 for moving in this direction (revamped metrics). Let's end >> the >>>>>> metrics pain. >>>>>> >>>>>> -Taylor >>>>>> >>>>>>> On Oct 11, 2016, at 10:15 AM, Bobby Evans < >>> ev...@yahoo-inc.com.INVALID >>>>> >>>>>> wrote: >>>>>>> >>>>>>> I agree that IMetricsConsumer is not good, but the reality is that >>> all >>>>>> of the metrics system needs to be redone. The problem is that we >> ship >>>> an >>>>>> object as a metric. If I get an object I have no idea what it is >> hand >>>>>> hence no idea how to report it or what to do with it. What is more >>> the >>>>>> common types we use in the metrics we provide are really not enough. >>>> For >>>>>> example CountMetric sends a Long. Well when I get it in the metrics >>>>>> consumer I have no idea if I should report it like a counter or if I >>>> should >>>>>> report it like a gauge (something that every metrics system I have >>> used >>>>>> wants to know). But then we support pre-aggregation of the metrics >>> with >>>>>> IReducer so the number I get might be an average instead of either a >>>> gauge >>>>>> or a counter, which no good metrics system will want to collect >>> because >>>> I >>>>>> cannot aggregate it with anything else, the math just does not work. >>>>>>> The proposal I have said before and I still believe is that we need >>> to >>>>>> put in place a parallel metrics API/system. We will deprecate all >> of >>>>>> https://git.corp.yahoo.com/storm/storm/tree/master- >>>> security/storm-core/src/jvm/backtype/storm/metric/api >>>>>> and create a new parallel one that provides an API similar to >>>>>> http://metrics.dropwizard.io/3.1.0/. >>> <http://metrics.dropwizard.io/3.1.0/> I would even be fine in just >>>> using >>>>>> their API and exposing that to end users. Dropwizard has solved all >>> of >>>>>> these problems already and I don't see a reason to reinvent the >> wheel. >>>> I >>>>>> don't personally see a lot of value in trying to send all of the >>> metrics >>>>>> through storm itslef. I am fine if we are able to support that, but >>> it >>>> is >>>>>> far from a requirement. - Bobby >>>>>>> >>>>>>> On Monday, October 10, 2016 10:47 PM, S G < >>> sg.online.em...@gmail.com> >>>>>> wrote: >>>>>>> >>>>>>> >>>>>>> +1 >>>>>>> We can probably start by opening a JIRA for this and adding a >> design >>>>>>> approach for the same? >>>>>>> I would like to help in the coding-effort for this. >>>>>>> >>>>>>> -SG >>>>>>> >>>>>>>> On Mon, Oct 10, 2016 at 1:51 PM, P. Taylor Goetz < >> ptgo...@gmail.com >>>> >>>>>> wrote: >>>>>>>> >>>>>>>> I’ve been thinking about metrics lately, specifically the fact >> that >>>>>> people >>>>>>>> tend to struggle with implementing a metrics consumer. (Like this >>> one >>>>>> [1]). >>>>>>>> >>>>>>>> The IMetricsConsumer interface is pretty low level, and common >>>>>>>> aggregations, calculations, etc. are left up to each individual >>>>>>>> implementation. That seems like an area where further abstraction >>>> would >>>>>>>> make it easier to support different back ends (Graphite, JMX, >>> Splunk, >>>>>> etc.). >>>>>>>> >>>>>>>> My thought is to create an abstract IMetricsConsumer >> implementation >>>> that >>>>>>>> does common aggregations and calculations, and then delegates to a >>>>>> plugable >>>>>>>> “metrics sink” implementation (e.g. “IMetricsSink”, etc.). That >>> would >>>>>>>> greatly simplify the effort required to integrate with various >>>> external >>>>>>>> metrics systems. I know of at least a few users that would be >>>>>> interested, >>>>>>>> one is currently scraping the logs from LoggingMetricsConsumer and >>>>>> polling >>>>>>>> the Storm REST API for their metrics. >>>>>>>> >>>>>>>> -Taylor >>>>>>>> >>>>>>>> [1] http://twocentsonsoftware.blogspot.co.il/2014/12/ >>>>>>>> sending-out-storm-metrics.html >>>>>>>> >>>>>>>> >>>>>>>>> On Oct 10, 2016, at 12:14 PM, Bobby Evans >>>> <ev...@yahoo-inc.com.INVALID >>>>>>> >>>>>>>> wrote: >>>>>>>>> >>>>>>>>> First of all the server exposes essentially the same interface >> that >>>> the >>>>>>>> IMetricsConsumer exposes. It mostly just adds a bunch of overhead >>> in >>>>>> the >>>>>>>> middle to serialize out the objects send them over http to another >>>>>> process >>>>>>>> which then has to deserialize them and process them. If you >> really >>>>>> don't >>>>>>>> need the metrics to show up on a special known box you can have >> that >>>>>> exact >>>>>>>> same code running inside the metrics consumer without all of the >>>>>> overhead. >>>>>>>>> The server/client are insecure, have to deal with thread issues >>> that >>>> a >>>>>>>> normal IMetricsConsumer does not, and are not written to be robust >>> (If >>>>>> the >>>>>>>> HTTP server is down the consumer crashes and continues to crash >>> until >>>>>> the >>>>>>>> server is brought back up). It was written very quickly for a >> test >>>>>>>> situation and it honestly never crossed my mind that anyone would >>> want >>>>>> to >>>>>>>> use it in production. >>>>>>>>> >>>>>>>>> - Bobby >>>>>>>>> >>>>>>>>> On Monday, October 10, 2016 10:59 AM, S G < >>>>>> sg.online.em...@gmail.com> >>>>>>>> wrote: >>>>>>>>> >>>>>>>>> >>>>>>>>> Thanks Bobby. >>>>>>>>> >>>>>>>>> If we write our own metrics consumer, how do we ensure that it is >>>>>> better >>>>>>>>> than HttpForwardingMetricsServer? In other words, what aspects of >>> the >>>>>>>>> HttpForwardingMetricsServer >>>>>>>>> should we avoid to make our own metrics consumer better and ready >>> for >>>>>>>>> production? >>>>>>>>> >>>>>>>>> Is versign/storm-graphite <https://github.com/verisign/ >>>> storm-graphite> >>>>>>>>> production >>>>>>>>> ready? >>>>>>>>> >>>>>>>>> Also, we should add a line about production-readiness of >>>>>>>>> HttpForwardingMetricsServer >>>>>>>>> in the documentation at http://storm.apache.org/ >>>>>>>> releases/1.0.2/Metrics.html >>>>>>>>> (We were just about to think seriously on using this for >> production >>>> as >>>>>> we >>>>>>>>> thought this to be the standard solution for metrics' consumption >>> in >>>>>> 1.0+ >>>>>>>>> version). >>>>>>>>> >>>>>>>>> -SG >>>>>>>>> >>>>>>>>> On Mon, Oct 10, 2016 at 6:37 AM, Bobby Evans < >> ev...@yahoo-inc.com> >>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> First of all there really are two different sets of metrics. >> One >>>> set >>>>>> is >>>>>>>>>> the topology metrics and the other set is the daemon metrics >>>> (metrics >>>>>>>> for >>>>>>>>>> things like the ui and nimbus). The JmxPreparableReporter >> plugin >>>> only >>>>>>>>>> exposes daemon metrics not the topology metrics through JMX. >>>> Exposing >>>>>>>>>> topology metrics through JMX is a non trivial task. The current >>>>>> metrics >>>>>>>>>> feature was not designed for this. We are in the process of >>> trying >>>> to >>>>>>>>>> redesign the metrics system to allow for features like this, but >>> it >>>> is >>>>>>>>>> still a ways off. >>>>>>>>>> >>>>>>>>>> - Bobby >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Saturday, October 8, 2016 11:39 AM, S G < >>>> sg.online.em...@gmail.com >>>>>>> >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Thanks Bobby, >>>>>>>>>> >>>>>>>>>> We will need some kind of IMetricsConsumer to talk to telegraf. >>>>>>>>>> Many other softwares like Solr, Elastic-Search, Cassandra etc. >>>> provide >>>>>>>>>> metrics through a URL making it very easy to consume by tools >> like >>>>>>>> telegraf. >>>>>>>>>> How about a IMetricsConsumer that will run on storm-ui and >> provide >>>> the >>>>>>>>>> metrics through a URL such as <storm-ui-host>/metrics ? >>>>>>>>>> >>>>>>>>>> Also, I see the following option in defaults.yaml: >>>>>>>>>> #default storm daemon metrics reporter plugins >>>>>>>>>> storm.daemon.metrics.reporter.plugins: >>>>>>>>>> - "org.apache.storm.daemon.metrics.reporters. >>>>>>>> JmxPreparableReporter" >>>>>>>>>> >>>>>>>>>> Is this a good option to use for converting metrics into JMX ? >>>>>>>>>> >>>>>>>>>> Thanks >>>>>>>>>> SG >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Fri, Oct 7, 2016 at 8:11 AM, Bobby Evans >>>>>> <ev...@yahoo-inc.com.invalid >>>>>>>>> >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>> HttpForwardingMetricsServer is a real hack intended really for >>>>>> tests. I >>>>>>>>>> know I wrote it :). Please don't use it in production. You can >>>> write >>>>>>>> your >>>>>>>>>> own IMetricesConsumer to do whatever you want to with the >> metrics. >>>>>>>>>> https://github.com/apache/ storm/blob/master/storm-core/ >>>>>>>>>> src/jvm/org/apache/storm/ metric/api/IMetricsConsumer. java >>>>>>>>>> <https://github.com/apache/storm/blob/master/storm-core/ >>>>>>>> src/jvm/org/apache/storm/metric/api/IMetricsConsumer.java> >>>>>>>>>> >>>>>>>>>> That is the correct way to get the data out. If you want to >>> write a >>>>>>>>>> bridge to JMX for this that might work, but going directly to >>>>>> telegraph >>>>>>>>>> would probably be better. - Bobby >>>>>>>>>> >>>>>>>>>> On Thursday, October 6, 2016 1:43 PM, S G < >>>>>>>> sg.online.em...@gmail.com> >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Hi, >>>>>>>>>> >>>>>>>>>> We want to use Telegraf ( >>>>>>>>>> https://github.com/influxdata/ telegraf/tree/master/plugins >>>>>>>>>> <https://github.com/influxdata/telegraf/tree/master/plugins>) >> for >>>>>>>> getting >>>>>>>>>> storm's metrics. >>>>>>>>>> >>>>>>>>>> But we do not want to add a HttpForwardingMetricsServer just to >>> get >>>>>> the >>>>>>>>>> metrics and send them to telegraf. >>>>>>>>>> >>>>>>>>>> Other option is to use Jolokia (https://jolokia.org/) that can >>> read >>>>>> JMX >>>>>>>>>> and >>>>>>>>>> write into telegraf. >>>>>>>>>> >>>>>>>>>> Does storm report all its metrics (including those of custom >>>>>>>> spouts/bolts) >>>>>>>>>> into JMX? >>>>>>>>>> Or spawning a HttpForwardingMetricsServer is the only option? >>>>>>>>>> >>>>>>>>>> Thanks >>>>>>>>>> SG >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>> >>>> >>>> >>> >>> >>>
