Hi all, This is just a quick update on Metrics integration.
- We decided to use a separate repository for metrics [1]. The main reason is that we want to integrate Metrics to upcoming releases and it will take a longer time for products to use a new Carbon Kernel version. - Developed a new JDBC Reporter for Metrics [2] - Now I'm working on a new UI for Metrics visualization. I experimented with other tools and we decided to have a lightweight UI, which can be integrated to WSO2 Products. I'm planning to use the data from a database and that's the main reason to develop a new JDBC reporter. I'll share the details of the UI later. Since the core part for Metrics API is ready, we can start integrating to the products. Srinath, please let me know which product to start with. Then we can discuss with relevant product team. Thanks! Best Regards, [1] https://github.com/wso2/carbon-metrics [2] https://github.com/wso2/carbon-metrics/blob/master/components/org.wso2.carbon.metrics.jdbc.reporter/src/main/java/org/wso2/carbon/metrics/reporter/JDBCReporter.java On Tue, Dec 23, 2014 at 3:35 PM, Isuru Perera <[email protected]> wrote: > Hi Sameera, > > I sent a PR to carbon4-kernel repository [1]. Could you please review and > merge it? I created CARBON-15115 [2] for that. > > Thanks! > > [1] https://github.com/wso2/carbon4-kernel/pull/125 > [2] https://wso2.org/jira/browse/CARBON-15115 > > On Wed, Dec 3, 2014 at 1:53 PM, Ramith Jayasinghe <[email protected]> wrote: > >> Yep. This needs to be evaluated on loaded environments. >> >> On Wed, Dec 3, 2014 at 1:39 PM, Isuru Perera <[email protected]> wrote: >> >>> Hi all, >>> >>> I have committed what I did so far to the WSO2 SVN scratch area [1]. >>> >>> There are two components. >>> >>> 1. org.wso2.carbon.metrics.manager - Contains the WSO2 Metrics API, >>> which will be used in the WSO2 platform. >>> 2. org.wso2.carbon.metrics.impl - The implementation of Metric >>> Service, which will internally use the Metrics Java Library [2]. >>> >>> The main reason for having our own API is to introduce the support for >>> different metrics levels. This will also make sure that we are not tightly >>> coupled with Metrics Java library. >>> >>> The org.wso2.carbon.metrics.manager.MetricManager [3] class has the main >>> API. This class is similar to the Metrics' MetricRegistry class [4] and I >>> have taken the helper methods to generate names etc. >>> >>> Yesterday, I organized a code review and following are the notes: >>> >>> Participants: Azeez, Sameera, Ramith. >>> >>> Code review was based on revision 210170 [5] >>> >>> - Azeez mentioned that having a global configuration for Level [6] >>> might not be useful. Since we allow to create different Metrics with >>> different levels, we need to have a way to specify level for each Metric >>> similar to Log4J. Configurations should be decided initially rather than >>> introducing later. >>> - Ramith asked whether we can support annotations. Metrics [2] >>> support annotations for Spring with Spring's AOP support. Besides, the >>> annotation will help to measure up to method level only. It was decided >>> that we can use the APIs directly as we can use the given APIs whenever >>> we >>> need to measure. >>> - The use of ConcurrentHashMap need to be reevaluated. The >>> MetricManager also uses a ConcurrentHashMap similar to MetricRegistry to >>> avoid duplicate Metric objects. We must test the Metrics API in high >>> concurrent scenarios. >>> - We decided to integrate the metrics components to Carbon Kernel >>> first. I will start integrating the metrics to Carbon Kernel 4.4 in a >>> different branch. >>> - When logging, isErrorEnabled may not be necessary [7], but it's >>> okay to have it. >>> >>> Azeez, Sameera, Ramith, please add if I missed anything. >>> >>> In addition to the code review, I had a chat with Srinath yesterday >>> morning. >>> >>> We discussed that we must have a UI for Metrics. I will share the >>> details on that later. >>> >>> Thanks! >>> >>> Best Regards, >>> >>> [1] https://svn.wso2.org/repos/wso2/scratch/metrics >>> [2] https://dropwizard.github.io/metrics >>> [3] >>> https://svn.wso2.org/repos/wso2/scratch/metrics/components/org.wso2.carbon.metrics.manager/src/main/java/org/wso2/carbon/metrics/manager/MetricManager.java >>> [4] >>> https://github.com/dropwizard/metrics/blob/master/metrics-core/src/main/java/com/codahale/metrics/MetricRegistry.java >>> [5] http://wso2.org/svn/browse/wso2?view=revision&revision=210170 >>> [6] >>> http://wso2.org/svn/browse/wso2/scratch/metrics/components/org.wso2.carbon.metrics.impl/src/test/resources/metrics.xml?view=markup&pathrev=210170 >>> [7] >>> http://wso2.org/svn/browse/wso2/scratch/metrics/components/org.wso2.carbon.metrics.manager/src/main/java/org/wso2/carbon/metrics/manager/internal/MetricManagerComponent.java?view=markup&pathrev=210170 >>> >>> On Tue, Nov 18, 2014 at 12:11 PM, Isuru Perera <[email protected]> wrote: >>> >>>> Hi, >>>> >>>> Me & Srinath discussed a bit more on implementation. Following are the >>>> notes. >>>> >>>> - We need to have different levels for each metric similar to Log4J >>>> levels. The API for metric will have a way to specify the level. >>>> - There should be a way to view historical data. By default, we can >>>> use a log file. >>>> - There should be a configuration file for metrics. The enable >>>> option will be part of the configuration file. But it can be overridden >>>> by >>>> a System property. >>>> >>>> Srinath, >>>> >>>> Please add if I missed anything. Also regarding the reporting options, >>>> the metrics provides support for Ganglia & Graphite [1]. The in-built >>>> support for HTTP reporting might not be directly useful for us [2]. I will >>>> check on these and see how we can get the data. >>>> >>>> Thanks! >>>> >>>> Best Regards, >>>> [1] >>>> https://dropwizard.github.io/metrics/3.1.0/getting-started/#other-reporting >>>> [2] >>>> https://dropwizard.github.io/metrics/3.1.0/manual/servlets/#manual-servlets >>>> >>>> >>>> On Tue, Nov 18, 2014 at 7:53 AM, Srinath Perera <[email protected]> >>>> wrote: >>>> >>>>> Look good Isuru!! >>>>> >>>>> On Thu, Nov 13, 2014 at 6:58 PM, Isuru Perera <[email protected]> wrote: >>>>> >>>>>> Hi all, >>>>>> >>>>>> I played with following performance libraries. >>>>>> >>>>>> - Metrics >>>>>> - Parfait >>>>>> - JAMon >>>>>> - Java Simon >>>>>> - Perf4J >>>>>> >>>>>> I did a simple comparison and blogged [1]. >>>>>> >>>>>> As Srinath mentioned, the Metrics library is more suitable for this >>>>>> proposal. >>>>>> >>>>>> It supports various measuring instruments like Meters, Counters, >>>>>> Histograms, Timers etc. >>>>>> >>>>>> Metrics supports JMX reporting and we should be able to write our own >>>>>> reporting implementations easily. We can use a reporting implementation >>>>>> to >>>>>> integrate with BAM. >>>>>> >>>>>> There is no support to enable/disable metrics. Since this is an >>>>>> important feature for us, I'm thinking of introducing a custom component >>>>>> for performance monitoring. >>>>>> >>>>>> The custom component will support to enable/disable performance >>>>>> monitoring via system property and JMX. The WSO2 components will use the >>>>>> new performance monitoring component and we will be using the Metrics >>>>>> only >>>>>> inside the new component. >>>>>> >>>>>> I'm working on this and for testing we need to select a WSO2 product. >>>>>> I'm thinking of adding the performance probes to WSO2 API Manager first. >>>>>> Since WSO2 CEP is also mentioned, we can have a look at that also. >>>>>> >>>>>> Please share any suggestions or feedback. >>>>>> >>>>>> I will update the thread once I create the initial design. >>>>>> >>>>>> Thanks! >>>>>> >>>>>> Best Regards, >>>>>> >>>>>> [1] >>>>>> http://isuru-perera.blogspot.com/2014/11/java-performance-monitoring-libraries.html >>>>>> >>>>>> >>>>>> On Mon, Sep 29, 2014 at 10:00 AM, Srinath Perera <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> Thanks Thomas!! >>>>>>> >>>>>>> Paul, I looked at all 3 and really liked what Thomas sent ( >>>>>>> https://dropwizard.github.io/metrics/3.1.0/getting-started/). >>>>>>> Licence is Apache 2. >>>>>>> >>>>>>> I think I will try this out with a training project. >>>>>>> >>>>>>> --Srinath >>>>>>> >>>>>>> On Sat, Sep 27, 2014 at 11:37 AM, Thomas Wieger < >>>>>>> [email protected]> wrote: >>>>>>> >>>>>>>> please have a look on the metrics lib, which quite nicely provides >>>>>>>> support for capturing different kinds of statistics. see >>>>>>>> https://dropwizard.github.io/metrics/3.1.0/ >>>>>>>> >>>>>>>> >>>>>>>> regards, >>>>>>>> >>>>>>>> thomas >>>>>>>> >>>>>>>> Am Freitag, 26. September 2014 schrieb Sajith Ravindra : >>>>>>>> >>>>>>>> >>>>>>>>> IMHO, CEP is another product where this kind of probes can be >>>>>>>>> really useful as we have to deal with performance related >>>>>>>>> requirements. As >>>>>>>>> there's a major refactoring going on in siddhi I think we can we plant >>>>>>>>> probes like this into the Siddhi engine. It will be useful to >>>>>>>>> evaluate the >>>>>>>>> re-factored Sindhi engines performance as well as to figure out >>>>>>>>> performance >>>>>>>>> bottlenecks in the future. >>>>>>>>> >>>>>>>>> >>>>>>>>> Thanks >>>>>>>>> *,Sajith Ravindra* >>>>>>>>> Senior Software Engineer >>>>>>>>> WSO2 Inc.; http://wso2.com >>>>>>>>> lean.enterprise.middleware >>>>>>>>> >>>>>>>>> mobile: +94 77 2273550 >>>>>>>>> blog: http://sajithr.blogspot.com/ >>>>>>>>> <http://lk.linkedin.com/pub/shani-ranasinghe/34/111/ab> >>>>>>>>> >>>>>>>>> On Fri, Sep 26, 2014 at 8:38 AM, Srinath Perera <[email protected]> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> We have some pretty nice probes already in ESB, that let you look >>>>>>>>>> into what is going in the same running in production. I think we >>>>>>>>>> need to >>>>>>>>>> do this generally and build it to other products. That should reduce >>>>>>>>>> time >>>>>>>>>> we spent debugging issues significantly. >>>>>>>>>> >>>>>>>>>> My proposal is to build a probe Lib that looks like following. >>>>>>>>>> >>>>>>>>>> There are two kinds of things you need to collect. >>>>>>>>>> >>>>>>>>>> 1. Throughput at give point of code (how fast calls goes >>>>>>>>>> though) >>>>>>>>>> 2. Latency between two points. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> We will have two types of Probes. Code would look like following. >>>>>>>>>> >>>>>>>>>> Probe probe = new Probe("Name", "throughput", timeDuration); >>>>>>>>>> .. >>>>>>>>>> probe.recordThroughput(); >>>>>>>>>> >>>>>>>>>> Probe probe = new Probe("Name", "latency", timeDuration, >>>>>>>>>> logLevel); >>>>>>>>>> ... >>>>>>>>>> long id = probe.startTicking() // this so same probe can be used >>>>>>>>>> by many threads >>>>>>>>>> ... >>>>>>>>>> probe.endTicking(id); >>>>>>>>>> >>>>>>>>>> Probe will summarise data over given duration and expose. We need >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> 1. JMX bean >>>>>>>>>> 2. BAM Agent >>>>>>>>>> 3. Can turn on, off via JMX agent or via System property >>>>>>>>>> 4. Have Tool Box >>>>>>>>>> 5. Have in product UI >>>>>>>>>> 6. Can configured to write data to logs >>>>>>>>>> >>>>>>>>>> Each probe should be very small and should be able to create >>>>>>>>>> thousands without much effect. (e.g. create one for each mediator >>>>>>>>>> type) >>>>>>>>>> >>>>>>>>>> WDYT? >>>>>>>>>> >>>>>>>>>> --Srinath >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> ============================ >>>>>>>>>> Blog: http://srinathsview.blogspot.com twitter:@srinath_perera >>>>>>>>>> Site: http://people.apache.org/~hemapani/ >>>>>>>>>> Photos: http://www.flickr.com/photos/hemapani/ >>>>>>>>>> Phone: 0772360902 >>>>>>>>>> >>>>>>>>>> _______________________________________________ >>>>>>>>>> Architecture mailing list >>>>>>>>>> [email protected] >>>>>>>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> ============================ >>>>>>> Blog: http://srinathsview.blogspot.com twitter:@srinath_perera >>>>>>> Site: http://people.apache.org/~hemapani/ >>>>>>> Photos: http://www.flickr.com/photos/hemapani/ >>>>>>> Phone: 0772360902 >>>>>>> >>>>>>> _______________________________________________ >>>>>>> Architecture mailing list >>>>>>> [email protected] >>>>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Isuru Perera >>>>>> Senior Software Engineer | WSO2, Inc. | http://wso2.com/ >>>>>> Lean . Enterprise . Middleware >>>>>> >>>>>> about.me/chrishantha >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> ============================ >>>>> Blog: http://srinathsview.blogspot.com twitter:@srinath_perera >>>>> Site: http://people.apache.org/~hemapani/ >>>>> Photos: http://www.flickr.com/photos/hemapani/ >>>>> Phone: 0772360902 >>>>> >>>> >>>> >>>> >>>> -- >>>> Isuru Perera >>>> Senior Software Engineer | WSO2, Inc. | http://wso2.com/ >>>> Lean . Enterprise . Middleware >>>> >>>> about.me/chrishantha >>>> >>> >>> >>> >>> -- >>> Isuru Perera >>> Senior Software Engineer | WSO2, Inc. | http://wso2.com/ >>> Lean . Enterprise . Middleware >>> >>> about.me/chrishantha >>> >> >> >> >> -- >> Ramith Jayasinghe >> Technical Lead >> WSO2 Inc., http://wso2.com >> lean.enterprise.middleware >> >> E: [email protected] >> P: +94 777542851 >> >> > > > -- > Isuru Perera > Senior Software Engineer | WSO2, Inc. | http://wso2.com/ > Lean . Enterprise . Middleware > > about.me/chrishantha > -- Isuru Perera Senior Software Engineer | WSO2, Inc. | http://wso2.com/ Lean . Enterprise . Middleware about.me/chrishantha
_______________________________________________ Architecture mailing list [email protected] https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
