On Tue, Jun 10, 2014 at 12:45 AM, Maninda Edirisooriya <[email protected]> wrote:
> This abstraction of Data Agents will be very useful when there are many > Data Agents getting created for BAM/CEP. > +1 > Hope these agents can be used for other WSO2 products as well. > JVM Agent and Server agent can be reused. However the AS agent is specific to AS. We may need different agents for different products. > As I understand all these 3 data agents are installable features for a > Carbon server although their monitoring scope is different. Isn't it? > AS Agent is running on Carbon to grab whatever the info that can be grabbed within Carbon environment. So it can be a Carbon feature. The JVM agent jar is started by the JVM directly even before the main(String args[]) method is called. So it is not possible to run within Carbon. Please refer to java.lang.instrumentation javadoc. So it is not possible to make it as a Carbon feature. The Server agent is OS Specific. We need different agents per different OS as of my current research. This runs outside the JVM because within a JVM we cannot monitor the other processes running on the OS. So it cannot be a carbon feature. > > And these 3 agents seems to be more focused on statistics collection other > than message dumping and activity monitoring. It is possible to collect > statistics (e.g.: memory, message throughput) with periodic message > pushing. But if we need message tracing we may need to monitor each message > passing through. And may require message filtering parameters and have to > think about activity ID manipulation as well. It would be much interesting > if we can think about these requirements as well. WDYT? > The BAM receives a record per each and every HTTP call from the AS Agent (These records can be pushed periodically or as soon as the record is generated). If we look at activity monitoring, message tracing etc.. the data we sent to BAM is 99% identical. One major design goal of this system is to avoid duplicate pushes. So we will handle the Activity ID manipulation & message filtering parameters as well. > > > Thanks. > > > *Maninda Edirisooriya* > Senior Software Engineer > > *WSO2, Inc. *lean.enterprise.middleware. > > *Blog* : http://maninda.blogspot.com/ > *E-mail* : [email protected] > *Skype* : @manindae > *Twitter* : @maninda > > > On Mon, Jun 9, 2014 at 4:12 AM, Chamil Jeewantha <[email protected]> wrote: > >> I am working on AS-BAM application & server monitoring. Kalpa will work >> with me for three weeks. He will do the BAM related work for HTTP call >> monitoring. >> >> AS monitoring with BAM is the first step towards a monitoring product >> based on BAM to monitor the entire WSO2 product stack including their >> infrastructure. >> >> *Current AS monitoring with BAM:* >> >> There are some BAM toolboxes which gives some limited capabilities of >> monitoring. >> >> HTTPD Logs Analysis Toolbox (Sample) [1] >> Real-time Traffic Monitoring Toolbox (Sample) [2] >> Service Statistics Toolbox [3] >> Activity Monitoring Toolbox [4] >> >> *Limitations:* >> >> All above comes as different features >> aggregated results are not provided >> Drill down capabilities are limited. >> duplicate data pushes from AS to BAM slow down the overall system. >> The end-user should do some of the technical work to extract the result >> (writng queries etc...). >> Presentation of the result is not use friendly. >> >> *Solution:* >> >> Coming up with a new product to monitor WSO2 Application Server cluster. >> The product's monitoring capabilities are categorized in to four (4) >> categories. >> >> - HTTP call Monitoring (Http call throughput, average response time >> etc...) >> - JDBC & Third-party call monitoring (Number of queries per each Data >> source etc...) >> - JVM level monitoring (GC, Threads etc...) >> - Infrastructure monitoring (CPU utilization, Memory etc...) >> >> *Overall Architecture:* >> >> >> >> Each AS Node has three Agents running on it. These agents pushes data >> through several data streams to BAM. >> >> *The Agents:* >> >> >> - Node Agent >> >> The Hardware level & Operating System level data are gathered and pushed >> by this agent. (CPU, Memory, Network, ...) There will be an implementations >> per each Operating System. >> >> >> - Java Agent >> >> The JVM details like Garbage Collection, Threads are gathered and pushed >> by this agent. this will be implemented as a javaagent [5] >> >> >> - AS Agent >> >> The data that can be gathered as AS level are gathered and pushed by this >> Agent. Examples: Http call response time, headers, payload, session >> details, JDBC connections etc... >> >> *Streams:* >> >> There can be many streams used to push data to BAM. The only constraint >> is "Non of the data streams should publish duplicate data" >> >> *BAM:* >> >> The AS agents only pushes raw data. We develop a BAM toolbox (or a UES >> dashboard) which manipulate the data from streams and show them on a >> user-friendly dashboard with many inter-connected data visualizations. >> >> *First Step:* >> >> Implementing the HTTP call monitoring capability is the first step. It >> will be discussed in a separate thread to limit the length of the email. >> >> >> >> [1] https://docs.wso2.org/display/BAM240/Analysing+HTTPD+Logs >> [2] https://docs.wso2.org/display/BAM241/Monitoring+Realtime+Traffic >> [3] https://docs.wso2.org/display/BAM240/Service+Statistics+Toolbox >> [4] https://docs.wso2.org/display/BAM240/Activity+Monitoring+Toolbox >> [5] >> http://docs.oracle.com/javase/6/docs/api/java/lang/instrument/package-summary.html >> -- >> K.D. Chamil Jeewantha >> Associate Technical Lead >> WSO2, Inc.; http://wso2.com >> Mobile: +94716813892 >> >> >> >> _______________________________________________ >> Architecture mailing list >> [email protected] >> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >> >> > -- K.D. Chamil Jeewantha Associate Technical Lead WSO2, Inc.; http://wso2.com Mobile: +94716813892
_______________________________________________ Architecture mailing list [email protected] https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
