On Sat, May 11, 2013 at 1:50 PM, Kasun Gajasinghe <[email protected]> wrote:
> > We also wanted to capture the browser and operating system the request > originated from. These can be retrieved via the user-agent header after a > little string manipulation. But as I got to know, there is no easy way to > do this on the analyzer side. ie. doing string manipulations, and > retrieving the browser, os etc. So, we have added another set of fields to > the event we send to BAM. They are browser, browserVersion, OS, OS version. > With these we can have a set of cool dashboard gadgets I believe. :) > > Like to know whether the above can be done in the analytics side rather > than processing those in data agent. > You can do string manipulation using Hive udf [1]. If those functions doesn't satisfy your requirement you can write your own custom UDF to do that job. Thanks, KasunW. [1] https://cwiki.apache.org/Hive/languagemanual-udf.html#LanguageManualUDF-StringFunctions > Thanks, > KasunG > > > > On Sat, May 11, 2013 at 1:45 PM, Kasun Gajasinghe <[email protected]> wrote: > >> >> >> >> On Sat, May 11, 2013 at 1:00 PM, Amila Suriarachchi <[email protected]>wrote: >> >>> >>> >>> >>> On Fri, May 10, 2013 at 11:24 PM, Amila Suriarachchi <[email protected]>wrote: >>> >>>> >>>> >>>> >>>> On Fri, May 10, 2013 at 8:49 PM, Kasun Gajasinghe <[email protected]>wrote: >>>> >>>>> >>>>> I couldn't really find the different between the two. But I have >>>>> categorized the http request header info as metadata. >>>>> >>>> >>>> IIRC these concepts came from the SOAP message level. >>>> >>>> Payload data means the data it receives within the soap envelop. ie. >>>> things like tradeamount, tradeid etc.. Basically the business entities. >>>> Meta data was refers to the things related to generic http data. >>>> >>> >>> From this point of view we need to send query data in pay load and >>> others in meta data. >>> >>> >> Currently, it's possible to send only a pre-defined set of data via the >> data agents (which we call as stream definition). So, for the default >> toolboxes, there is no way to predict what info a given message contains, >> and how to extract it. That is unless the user defines a new streamdef >> which may include the said properties like tradeamount, tradeid etc. And, >> user should have separate stream definition for each service because each >> service will have a different payload. This is my understanding. >> >> If you take axis2 service data publishing for example, we have metadata, >> and payload data like the following. We can additionally send the whole >> message to BAM too (for activity service monitoring). >> >> Payload data - >> Service Name >> Op Name >> Timestamp >> Response Time >> Request count - will always be 1 >> Response count - 0/1 >> Fault count - 0/1 >> >> Additinally, if activity service enabled, it also send >> soap header >> soap body >> message direction >> >> Metadata - >> Request URL >> Remote Address >> ContentType >> User Agent >> Host >> Referer >> >> >> I'm not sure whether it's practical provide support for custom data. >> >> Thanks, >> KasunG >> >> >> thanks, >>> Amila. >>> >>>> >>>> I think we better analyses those aspects across all data agents and >>>> have a consistent convention. >>>> >>>> thanks, >>>> Amila. >>>> >>>> >>>>> >>>>> >>>>> On Sat, May 11, 2013 at 8:59 AM, Amila Suriarachchi <[email protected]>wrote: >>>>> >>>>>> >>>>>> >>>>>> >>>>>> On Wed, May 8, 2013 at 5:23 AM, Kasun Gajasinghe <[email protected]>wrote: >>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> I'm trying to finalize the webapp request stats data that AS should >>>>>>> send to BAM. I have come with the following list of stats to be sent to >>>>>>> BAM >>>>>>> as part of the Event Data. These are categorized in to payload data and >>>>>>> metadata. Please share your suggestions on these, and what other data we >>>>>>> can send. >>>>>>> >>>>>>> We can also make it to send the request message body, and response >>>>>>> message body but it would add a significant overhead. >>>>>>> >>>>>>> Currently, webapp developers need to add a a context-param called >>>>>>> 'enable.statistics' to the web.xml in webapps. >>>>>>> >>>>>>> <context-param> >>>>>>> <param-name>*enable.statistics*</param-name> >>>>>>> <param-value>true</param-value> >>>>>>> </context-param> >>>>>>> >>>>>>> They can configure the webapp data agent configuration and BAM >>>>>>> credentials via the management console. The configuration is saved in >>>>>>> the >>>>>>> registry of the given tenant. Each tenant has to configure the data >>>>>>> agent >>>>>>> this way. This behavior is quite like the Service Data Agent >>>>>>> configuration. >>>>>>> >>>>>>> >>>>>>> streamId='bam_webapp_statistics_publisher:3.0.0', >>>>>>> name='bam_webapp_statistics_publisher', >>>>>>> version='1.0.0', >>>>>>> nickName='WebappDataAgent', >>>>>>> description='Publish webapp statistics events', >>>>>>> >>>>>>> Payload data - >>>>>>> >>>>>>> webappName >>>>>>> webappDisplayName - The <display-name> set in web.xml >>>>>>> webappOwnerTenant >>>>>>> webappVersion - web-app servlet version. Ex. 2.5, 3.0 >>>>>>> timestamp >>>>>>> webappContext - ex. /t/example.com/jaxwebapps/jaxrs_basic >>>>>>> resourcePath - ex. For a request url of >>>>>>> http://localhost:9763/t/example.com/jaxwebapps/jaxrs_basic/services/customers/customerservice/customers/123, >>>>>>> the resource path will be /customers/customerservice/customers/123 >>>>>>> webappType - Possible values are JAX-WS/JAX-RS/Generic >>>>>>> >>>>>>> >>>>>>> Metadata - >>>>>>> >>>>>>> >>>>>>> httpMethod - The HTTP method of the request >>>>>>> contentType - Content Type of the request >>>>>>> responseContentType - Content Type of the response >>>>>>> responseHttpStatusCode - HTTP status code of the response >>>>>>> userAgent - client user-agent >>>>>>> remoteAddress - client address >>>>>>> referer - HTTP referer header >>>>>>> authType - Tomcat level authentication, if set. ex. >>>>>>> BASIC_AUTH. >>>>>>> >>>>>>> >>>>>> What is the convention you have followed to determine what is payload >>>>>> data and what is metadata. >>>>>> >>>>>> For me most of the meta data you have given specific to that request. >>>>>> >>>>>> thanks, >>>>>> Amilla. >>>>>> >>>>>>> >>>>>>> >>>>>>> Thanks, >>>>>>> KasunG >>>>>>> >>>>>>> _______________________________________________ >>>>>>> Architecture mailing list >>>>>>> [email protected] >>>>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> *Amila Suriarachchi* >>>>>> >>>>>> Software Architect >>>>>> WSO2 Inc. ; http://wso2.com >>>>>> lean . enterprise . middleware >>>>>> >>>>>> phone : +94 71 3082805 >>>>>> >>>>>> _______________________________________________ >>>>>> Architecture mailing list >>>>>> [email protected] >>>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> *Kasun Gajasinghe* >>>>> Software Engineer; >>>>> Development Technologies Team, WSO2 Inc.; http://wso2.com >>>>> >>>>> , >>>>> *email: **kasung AT spamfree wso2.com >>>>> >>>>> ** cell: **+94 (77) 678-0813* >>>>> *linked-in: *http://lk.linkedin.com/in/gajasinghe >>>>> >>>>> * >>>>> * >>>>> *blog: **http://kasunbg.org* <http://kasunbg.org> >>>>> >>>>> * >>>>> twitter: **http://twitter.com/kasunbg* <http://twitter.com/kasunbg> >>>>> >>>>> * >>>>> * >>>>> >>>>> _______________________________________________ >>>>> Architecture mailing list >>>>> [email protected] >>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>> >>>>> >>>> >>>> >>>> -- >>>> *Amila Suriarachchi* >>>> >>>> Software Architect >>>> WSO2 Inc. ; http://wso2.com >>>> lean . enterprise . middleware >>>> >>>> phone : +94 71 3082805 >>>> >>> >>> >>> >>> -- >>> *Amila Suriarachchi* >>> >>> Software Architect >>> WSO2 Inc. ; http://wso2.com >>> lean . enterprise . middleware >>> >>> phone : +94 71 3082805 >>> >>> _______________________________________________ >>> Architecture mailing list >>> [email protected] >>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>> >>> >> >> >> -- >> *Kasun Gajasinghe* >> Software Engineer; >> Development Technologies Team, WSO2 Inc.; http://wso2.com >> >> , >> *email: **kasung AT spamfree wso2.com >> >> ** cell: **+94 (77) 678-0813* >> *linked-in: *http://lk.linkedin.com/in/gajasinghe >> >> * >> * >> *blog: **http://kasunbg.org* <http://kasunbg.org> >> >> * >> twitter: **http://twitter.com/kasunbg* <http://twitter.com/kasunbg> >> >> * >> * >> > > > > -- > *Kasun Gajasinghe* > Software Engineer; > Development Technologies Team, WSO2 Inc.; http://wso2.com > > , > *email: **kasung AT spamfree wso2.com > > ** cell: **+94 (77) 678-0813* > *linked-in: *http://lk.linkedin.com/in/gajasinghe > > * > * > *blog: **http://kasunbg.org* <http://kasunbg.org> > > * > twitter: **http://twitter.com/kasunbg* <http://twitter.com/kasunbg> > > * > * > > _______________________________________________ > Architecture mailing list > [email protected] > https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture > > -- *Kasun Weranga* ** Member, Management Committee - Data Technologies Software Engineer *WSO2, Inc. *lean.enterprise.middleware. mobile : +94 772314602 <http://sanjeewamalalgoda.blogspot.com/>blog : http://kasunweranga.blogspot.com/
_______________________________________________ Architecture mailing list [email protected] https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
