Hi I have done the data publishing part to bam. Now I am working on hive query to analyze data.
Thanks *G. K. S. Munasinghe * *Software Engineer,* *WSO2, Inc. http://wso2.com * *lean.enterprise.middleware.* * * email: [email protected] phone:(+94) 777911226 On Fri, May 17, 2013 at 4:44 PM, Sagara Gunathunga <[email protected]> wrote: > Hi Geeth, > > Could you please update current status of this task ? > > Thanks ! > > > On Mon, May 13, 2013 at 12:04 PM, Geeth Munasinghe <[email protected]> wrote: > >> Hi >> >> Following fields are chosen to use as payload data and meta data as >> suggestions >> >> Payload data - >> >> webappName >> webappDisplayName - The <display-name> set in web.xml >> webappOwnerTenant >> webappVersion - web-app servlet version. Ex. 2.5, 3.0 >> timestamp >> webappType - Possible values are JAX-WS/JAX-RS/Generic >> >> >> Metadata - >> >> webappContext - ex. /t/example.com/jaxwebapps/jaxrs_basic >> resourcePath - ex. For a request url of >> >> http://localhost:9763/t/example.com/jaxwebapps/jaxrs_basic/services/customers/customerservice/customers/123 >> , >> the resource path will be /customers/customerservice/customers/123 >> httpMethod - The HTTP method of the request >> contentType - Content Type of the request >> responseContentType - Content Type of the response >> responseHttpStatusCode - HTTP status code of the response >> userAgent - client user-agent >> remoteAddress - client address >> referer - HTTP referer header >> authType - Tomcat level authentication, if set. ex. >> BASIC_AUTH. >> responseTime - time taken by the applications process the request >> and send response >> sessionId - >> browser - >> browserVersion >> operatingSystem >> operatingSystemVersion >> searchEngine - applications was referred from a search engine >> country - Country which request originated. >> >> By capturing these data we can give users much detailed analysis of >> web app stats. Please give your suggestions to finalize the list. >> >> >> Thanks >> Geeth >> >> G. K. S. Munasinghe >> Software Engineer, >> WSO2, Inc. http://wso2.com >> lean.enterprise.middleware. >> >> email: [email protected] >> phone:(+94) 777911226 >> >> >> On Mon, May 13, 2013 at 10:46 AM, Amila Suriarachchi <[email protected]> >> wrote: >> > >> > >> > >> > On Sat, May 11, 2013 at 1:45 PM, Kasun Gajasinghe <[email protected]> >> wrote: >> >> >> >> >> >> >> >> >> >> On Sat, May 11, 2013 at 1:00 PM, Amila Suriarachchi <[email protected]> >> >> wrote: >> >>> >> >>> >> >>> >> >>> >> >>> On Fri, May 10, 2013 at 11:24 PM, Amila Suriarachchi <[email protected]> >> >>> wrote: >> >>>> >> >>>> >> >>>> >> >>>> >> >>>> On Fri, May 10, 2013 at 8:49 PM, Kasun Gajasinghe <[email protected]> >> >>>> wrote: >> >>>>> >> >>>>> >> >>>>> I couldn't really find the different between the two. But I have >> >>>>> categorized the http request header info as metadata. >> >>>> >> >>>> >> >>>> IIRC these concepts came from the SOAP message level. >> >>>> >> >>>> Payload data means the data it receives within the soap envelop. ie. >> >>>> things like tradeamount, tradeid etc.. Basically the business >> entities. Meta >> >>>> data was refers to the things related to generic http data. >> >>> >> >>> >> >>> From this point of view we need to send query data in pay load and >> others >> >>> in meta data. >> >>> >> >> >> >> Currently, it's possible to send only a pre-defined set of data via the >> >> data agents (which we call as stream definition). So, for the default >> >> toolboxes, there is no way to predict what info a given message >> contains, >> >> and how to extract it. That is unless the user defines a new streamdef >> which >> >> may include the said properties like tradeamount, tradeid etc. And, >> user >> >> should have separate stream definition for each service because each >> service >> >> will have a different payload. This is my understanding. >> > >> > >> > IIRC we talk about sending this undefined fields using a map or some >> thing. >> > That may not have implemented. But I think it is better to define what >> is >> > payload data and what is metadata not to make any confusions. >> > >> > thanks, >> > Amila. >> > >> >> >> >> >> >> If you take axis2 service data publishing for example, we have >> metadata, >> >> and payload data like the following. We can additionally send the whole >> >> message to BAM too (for activity service monitoring). >> >> >> >> Payload data - >> >> Service Name >> >> Op Name >> >> Timestamp >> >> Response Time >> >> Request count - will always be 1 >> >> Response count - 0/1 >> >> Fault count - 0/1 >> >> >> >> Additinally, if activity service enabled, it also send >> >> soap header >> >> soap body >> >> message direction >> >> >> >> Metadata - >> >> Request URL >> >> Remote Address >> >> ContentType >> >> User Agent >> >> Host >> >> Referer >> >> >> >> >> >> I'm not sure whether it's practical provide support for custom data. >> >> >> >> Thanks, >> >> KasunG >> >> >> >> >> >>> thanks, >> >>> Amila. >> >>>> >> >>>> >> >>>> I think we better analyses those aspects across all data agents and >> have >> >>>> a consistent convention. >> >>>> >> >>>> thanks, >> >>>> Amila. >> >>>> >> >>>>> >> >>>>> >> >>>>> >> >>>>> On Sat, May 11, 2013 at 8:59 AM, Amila Suriarachchi <[email protected] >> > >> >>>>> wrote: >> >>>>>> >> >>>>>> >> >>>>>> >> >>>>>> >> >>>>>> On Wed, May 8, 2013 at 5:23 AM, Kasun Gajasinghe <[email protected]> >> >>>>>> wrote: >> >>>>>>> >> >>>>>>> Hi, >> >>>>>>> >> >>>>>>> I'm trying to finalize the webapp request stats data that AS >> should >> >>>>>>> send to BAM. I have come with the following list of stats to be >> sent to BAM >> >>>>>>> as part of the Event Data. These are categorized in to payload >> data and >> >>>>>>> metadata. Please share your suggestions on these, and what other >> data we can >> >>>>>>> send. >> >>>>>>> >> >>>>>>> We can also make it to send the request message body, and response >> >>>>>>> message body but it would add a significant overhead. >> >>>>>>> >> >>>>>>> Currently, webapp developers need to add a a context-param called >> >>>>>>> 'enable.statistics' to the web.xml in webapps. >> >>>>>>> >> >>>>>>> <context-param> >> >>>>>>> <param-name>enable.statistics</param-name> >> >>>>>>> <param-value>true</param-value> >> >>>>>>> </context-param> >> >>>>>>> >> >>>>>>> They can configure the webapp data agent configuration and BAM >> >>>>>>> credentials via the management console. The configuration is >> saved in the >> >>>>>>> registry of the given tenant. Each tenant has to configure the >> data agent >> >>>>>>> this way. This behavior is quite like the Service Data Agent >> configuration. >> >>>>>>> >> >>>>>>> >> >>>>>>> streamId='bam_webapp_statistics_publisher:3.0.0', >> >>>>>>> name='bam_webapp_statistics_publisher', >> >>>>>>> version='1.0.0', >> >>>>>>> nickName='WebappDataAgent', >> >>>>>>> description='Publish webapp statistics events', >> >>>>>>> >> >>>>>>> Payload data - >> >>>>>>> >> >>>>>>> webappName >> >>>>>>> webappDisplayName - The <display-name> set in web.xml >> >>>>>>> webappOwnerTenant >> >>>>>>> webappVersion - web-app servlet version. Ex. 2.5, 3.0 >> >>>>>>> timestamp >> >>>>>>> webappContext - ex. /t/ >> example.com/jaxwebapps/jaxrs_basic >> >>>>>>> resourcePath - ex. For a request url of >> >>>>>>> >> http://localhost:9763/t/example.com/jaxwebapps/jaxrs_basic/services/customers/customerservice/customers/123 >> , >> >>>>>>> the resource path will be /customers/customerservice/customers/123 >> >>>>>>> webappType - Possible values are >> JAX-WS/JAX-RS/Generic >> >>>>>>> >> >>>>>>> >> >>>>>>> Metadata - >> >>>>>>> >> >>>>>>> >> >>>>>>> httpMethod - The HTTP method of the request >> >>>>>>> contentType - Content Type of the request >> >>>>>>> responseContentType - Content Type of the response >> >>>>>>> responseHttpStatusCode - HTTP status code of the response >> >>>>>>> userAgent - client user-agent >> >>>>>>> remoteAddress - client address >> >>>>>>> referer - HTTP referer header >> >>>>>>> authType - Tomcat level authentication, if set. ex. >> >>>>>>> BASIC_AUTH. >> >>>>>> >> >>>>>> >> >>>>>> What is the convention you have followed to determine what is >> payload >> >>>>>> data and what is metadata. >> >>>>>> >> >>>>>> For me most of the meta data you have given specific to that >> request. >> >>>>>> >> >>>>>> thanks, >> >>>>>> Amilla. >> >>>>>>> >> >>>>>>> >> >>>>>>> >> >>>>>>> Thanks, >> >>>>>>> KasunG >> >>>>>>> >> >>>>>>> _______________________________________________ >> >>>>>>> Architecture mailing list >> >>>>>>> [email protected] >> >>>>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >> >>>>>>> >> >>>>>> >> >>>>>> >> >>>>>> >> >>>>>> -- >> >>>>>> Amila Suriarachchi >> >>>>>> >> >>>>>> Software Architect >> >>>>>> WSO2 Inc. ; http://wso2.com >> >>>>>> lean . enterprise . middleware >> >>>>>> >> >>>>>> phone : +94 71 3082805 >> >>>>>> >> >>>>>> _______________________________________________ >> >>>>>> Architecture mailing list >> >>>>>> [email protected] >> >>>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >> >>>>>> >> >>>>> >> >>>>> >> >>>>> >> >>>>> -- >> >>>>> Kasun Gajasinghe >> >>>>> Software Engineer; >> >>>>> Development Technologies Team, WSO2 Inc.; http://wso2.com >> >>>>> >> >>>>> , >> >>>>> email: kasung AT spamfree wso2.com >> >>>>> >> >>>>> cell: +94 (77) 678-0813 >> >>>>> linked-in: http://lk.linkedin.com/in/gajasinghe >> >>>>> >> >>>>> >> >>>>> blog: http://kasunbg.org >> >>>>> >> >>>>> >> >>>>> twitter: http://twitter.com/kasunbg >> >>>>> >> >>>>> >> >>>>> >> >>>>> _______________________________________________ >> >>>>> Architecture mailing list >> >>>>> [email protected] >> >>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >> >>>>> >> >>>> >> >>>> >> >>>> >> >>>> -- >> >>>> Amila Suriarachchi >> >>>> >> >>>> Software Architect >> >>>> WSO2 Inc. ; http://wso2.com >> >>>> lean . enterprise . middleware >> >>>> >> >>>> phone : +94 71 3082805 >> >>> >> >>> >> >>> >> >>> >> >>> -- >> >>> Amila Suriarachchi >> >>> >> >>> Software Architect >> >>> WSO2 Inc. ; http://wso2.com >> >>> lean . enterprise . middleware >> >>> >> >>> phone : +94 71 3082805 >> >>> >> >>> _______________________________________________ >> >>> Architecture mailing list >> >>> [email protected] >> >>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >> >>> >> >> >> >> >> >> >> >> -- >> >> Kasun Gajasinghe >> >> Software Engineer; >> >> Development Technologies Team, WSO2 Inc.; http://wso2.com >> >> >> >> , >> >> email: kasung AT spamfree wso2.com >> >> >> >> cell: +94 (77) 678-0813 >> >> linked-in: http://lk.linkedin.com/in/gajasinghe >> >> >> >> >> >> blog: http://kasunbg.org >> >> >> >> >> >> twitter: http://twitter.com/kasunbg >> >> >> >> >> >> >> >> _______________________________________________ >> >> Architecture mailing list >> >> [email protected] >> >> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >> >> >> > >> > >> > >> > -- >> > Amila Suriarachchi >> > >> > Software Architect >> > WSO2 Inc. ; http://wso2.com >> > lean . enterprise . middleware >> > >> > phone : +94 71 3082805 >> > >> > _______________________________________________ >> > Architecture mailing list >> > [email protected] >> > https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >> > >> _______________________________________________ >> Architecture mailing list >> [email protected] >> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >> > > > > -- > Sagara Gunathunga > > Technical Lead; WSO2, Inc.; http://wso2.com > V.P Apache Web Services; http://ws.apache.org/ > Linkedin; http://www.linkedin.com/in/ssagara > Blog ; http://ssagara.blogspot.com > >
_______________________________________________ Architecture mailing list [email protected] https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
