Hi Maninda, Big +1
I think we should let user nominate different attributes as "activity IDs" for different Streams. For example for "StockQuotesStream" is txID vs. PaymentsStream it is "transactionID" etc. They will be tied into one activity by BAM. Also way we do this, there are two things. 1. Processing Data at BAM - Given data, how to calculate activities. That is the above. 2. How we propagate activities IDs across WSO2 products. We have had a convention, please review it in detail before creating new. But BAM processing must be independent from this. --Srinath On Wed, Nov 26, 2014 at 6:35 PM, Maninda Edirisooriya <[email protected]> wrote: > Hi Gokul, > > +1 for introducing a convention to propagate activity id field. But still > we cannot introduce a tight convention because the propagation mechanism > depends on the use case. Data publisher and/or the message flow channel > should handle that complexity of activity ID propagation. In BAM side we > have no control in that aspect other than supporting activity-ID and > sorting-field field configurability > > Thanks. > > > *Maninda Edirisooriya* > Senior Software Engineer > > *WSO2, Inc.*lean.enterprise.middleware. > > *Blog* : http://maninda.blogspot.com/ > *E-mail* : [email protected] > *Skype* : @manindae > *Twitter* : @maninda > > On Wed, Nov 26, 2014 at 4:02 AM, Gokul Balakrishnan <[email protected]> > wrote: > >> Hi Maninda, >> >> There's another point worth considering here: how the correlation field >> is propagated across nodes. Currently we're using the transport headers >> (HTTP) to carry the activity_id field. When we're making the activity >> monitoring more generic, we should consider correlating across different >> transports such as JMS. We had a discussion on this sometime back in the >> architecture@ thread "Improving activity ID propagation in activity data >> publishing". >> >> Thanks, >> Gokul. >> >> On 25 November 2014 at 06:05, Maninda Edirisooriya <[email protected]> >> wrote: >> >>> >>> On Tue, Nov 25, 2014 at 7:03 PM, Inosh Goonewardena <[email protected]> >>> wrote: >>> >>>> Hi Maninda, >>>> >>>> On Tue, Nov 25, 2014 at 2:48 PM, Maninda Edirisooriya <[email protected] >>>> > wrote: >>>> >>>>> Current implementation of BAM Activity Monitoring is based on a the >>>>> activity ID field and sorting the timestamps. It has some limitations. >>>>> >>>>> 1. Only one activity monitoring scenario is possible per tenant >>>>> >>>> >>>> Can you elaborate more on this. Do you mean that one activity can have >>>> events from different tenants? >>>> >>> No. I mean activity monitoring should be possible with different >>> correlation fields other than hard coded "activityID" field per tenant. >>> >>>> >>>> >>>>> 2. Activity ID the the hard coded parameter used for correlating among >>>>> events >>>>> >>>> 3. Ordering mechanism is based only on sorting operation of timestamp >>>>> field >>>>> 4. Tightly bound with Cassandra custom indexes >>>>> >>>> >>>> Yes. We have above known limitations in the current architecture >>>> basically because indexing and searching is implemented based on Cassandra >>>> custom index CFs which doesn't support advance sorting/searching and >>>> grouping operations. >>>> >>>> >>>>> >>>>> With the introduction of custom data stores in BAM we should support >>>>> activity monitoring for RDBMS indexes. There we can use SQL queries as >>>>> tasks for updating indexes. This enables us to use multiple activity >>>>> monitoring scenarios per each stream. For example all session ID, >>>>> transaction ID and username can be considered as activities for different >>>>> monitoring requirements at the same time. With the toolbox deployment we >>>>> can define all the activity monitoring scenarios so that indexes are >>>>> created for each field. And also we can make it possible to configure the >>>>> sorting mechanism per each activity. The user can be able to select >>>>> whatever field like timestamp or hop_number as the sorting field. This >>>>> would enable user to introduce their own field to order the activity >>>>> results. WDYT? >>>>> >>>> >>>> Multiple activity monitoring scenarios per each stream is already >>>> supported in current implementation. Sorting and pagination are the missing >>>> pieces which we have design properly in our next major release. And we >>>> should also make the activity correlation field configurable via the >>>> toolbox. >>>> >>> We should have a way to specify different fields for monitoring >>> different activities and see them in the Activity Monitoring dashboard. For >>> example if you want to monitor the flow of session IDs among streams and >>> suppose at the same time you want to see the transaction ID flow between >>> ESB mediation flows. You need two correlation ID fields and two indexes and >>> should be able to see each scenario separately in the activity dashboard. >>> There the session ID may be sorted with the "timestamp" but transaction >>> flow should be sorted based on a constant field set by the BAM mediator, >>> "mediator_number". The later one can be used in many situations where the >>> message flow time difference between the two interception points is lesser >>> than a 1 millisecond. (for e.g. in the same sequence where no significant >>> operations are done to the message) >>> >>>> >>>> >>>>> >>>>> *Maninda Edirisooriya* >>>>> Senior Software Engineer >>>>> >>>>> *WSO2, Inc.*lean.enterprise.middleware. >>>>> >>>>> *Blog* : http://maninda.blogspot.com/ >>>>> *E-mail* : [email protected] >>>>> *Skype* : @manindae >>>>> *Twitter* : @maninda >>>>> >>>> >>>> >>>> >>>> -- >>>> Regards, >>>> >>>> Inosh Goonewardena >>>> Associate Technical Lead- WSO2 Inc. >>>> Mobile: +94779966317 >>>> >>> >>> >>> _______________________________________________ >>> Architecture mailing list >>> [email protected] >>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>> >>> >> >> >> -- >> *Balakrishnan Gokulakrishnan* >> Software Engineer, >> WSO2, Inc. http://wso2.com >> Mob: +94 77 593 5789 | +1 650 272 9927 >> > > > _______________________________________________ > Architecture mailing list > [email protected] > https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture > > -- ============================ Srinath Perera, Ph.D. http://people.apache.org/~hemapani/ http://srinathsview.blogspot.com/
_______________________________________________ Architecture mailing list [email protected] https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
