Hi Asanka, Let me clarify about current BAM indexing feature a bit more. In the streams.properties file we can define two types of indexes, i.e secondary indexes and custom indexes. However, through the message console, filtering criteria can be only added on custom index properties. The reason is, secondary indexes we refer here are default cassandra indexes[1] and there are several limitations on using those secondary indexes in advanced search queries(explained in [1] and mail thread [2]). As explained in [2], we have chosen custom indexing mechanism because of those limitations we had with secondary indexes.
As you have mentioned, currently, custom index affect only for the data that is published after the index creation. This could be a useful thing to have, but we have to think properly on the overhead it is going to add when handling the size of already existing data and the time it will take to complete the process. On the other hand, according the conclusion of mail thread [3], we are planning to remove the define stream capabilities from data agents and define all WSO2 inbuilt event definitions in corresponding toolboxes. If that's the case, it is mandatory to deploy the toolbox before we publish data, so that indexes also has to be added along with the toolbox. But anyway, there are some other limitations(pagination support, sorting) as well in this message console/activity dashboard. The reason for most of those limitations is using Cassandra based CFs for filtered searching critireas. So that, we are planning to get rid of cassandra indexing and introduce some other indexing mechanism in BAM 2.5.0(already in our roadmap). [1] - http://www.datastax.com/docs/1.1/ddl/indexes [2] - [Architecture] Indexing in Cassandra to support BAM events & activity search [3] - [Architecture] Should we have one place to define event streams? On Fri, Jul 25, 2014 at 5:15 PM, Asanka Vithanage <[email protected]> wrote: > Hi Pandula, > > I am trying on a standalone pack with built-in cassandra. Basically we can > add secondary indexs by updating the tBox streams.properties file, for Ex. > we can add below line to Mediation_Statistics_Monitoring toolBox to create > secondary indexs on direction,resource_id,stats_type fields. > > streams.definitions.defn1.secondaryindexes=direction,resource_id,stats_type > > My Point here is once we deploy the TBox with secondary indexes, It should > start indexing immediately.Reported Bug[1] for this issue. > > [1] https://wso2.org/jira/browse/BAM-1715 > > > > > On Fri, Jul 25, 2014 at 3:38 PM, Pandula Kariyawasam <[email protected]> > wrote: > >> Hi Asanka, >> >> It was a different story at that time :). Anyway, have you tried this out >> in the latest pack in standalone mode, with built in Cassandra, and >> Datastax Cassandra? Is there any special configuration to enable this? if >> secondary index should index the already existing data, and if it is not >> working in current release, then there is a bug, either on Cassandra side, >> or on BAM side. You can get this confirm by adding indexing externally >> using Cassandra CLI (As in doc you shared above) This should index the data >> which are already there in the column. If it doesn't work, then Cassandra >> side has a problem, else you have to debug with BAM team. >> >> Thanks, >> Pandula >> >> >> On Fri, Jul 25, 2014 at 2:38 PM, Asanka Vithanage <[email protected]> >> wrote: >> >>> Hi Pandula, >>> >>> As per the BAM team secondary index should index the already existing >>> data as well. When we create a secondary index on a cassandra column >>> family,It should start the indexing process immediately so there shouldn't >>> be a requirement to publish data again. >>> >>> I agree with your concern about load handling but as per Cassandra doc >>> [1], It won't be a problem.But we have to do a test run to make sure. >>> >>> [1]http://www.datastax.com/docs/1.0/ddl/indexes >>> >>> >>> On Fri, Jul 25, 2014 at 2:03 PM, Pandula Kariyawasam <[email protected]> >>> wrote: >>> >>>> Hi Asanka, >>>> >>>> As in BAM 241, Cassandra indexing will affect only for the data which >>>> receive after adding the index, and there is no mechanism to index the >>>> already existing data. If this implemented, we have to handle the size of >>>> already >>>> existing data, and the load it will create, and the time it will take >>>> to complete the process. I believe, it will be better if this can be added >>>> to BAM road map with a proper solution to all the upcoming problems. >>>> >>>> Thanks, >>>> Pandula >>>> >>>> >>>> On Fri, Jul 25, 2014 at 12:49 PM, Asanka Vithanage <[email protected]> >>>> wrote: >>>> >>>>> Hi All, >>>>> >>>>> According to the current BAM implementation, We are supporting >>>>> message and activity search based on the Cassandra indexing via message >>>>> console UI. >>>>> >>>>> Currently to make data indexing work, we have to define indexing on >>>>> tBox and deploy it.But even after tBox deployed, indexing won't get >>>>> executed till we publish new set of data. >>>>> So According to this there is no way to index existing cassandra data >>>>> set through BAM. >>>>> If its possible how can i achieve that? >>>>> >>>>> Further since indexing can be considered as separate feature directly >>>>> bind with cassandra, I believe we should provide a separate UI to execute >>>>> indexing stuffs and It will increase the usability of BAM indexing >>>>> feature. >>>>> WDYT? >>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> Asanka Vithanage >>>>> Senior Software Engineer -QA >>>>> Mobile: +94 0716286708 >>>>> Email: [email protected] >>>>> WSO2 Inc. www.wso2.com >>>>> >>>>> >>>>> _______________________________________________ >>>>> Dev mailing list >>>>> [email protected] >>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>>>> >>>>> >>>> >>>> >>>> -- >>>> >>>> *Pandula Kariyawasam K.B.* >>>> *Senior Software Engineer - QA* >>>> Mobile: +94772314510 >>>> >>>> <http://www.wso2.com> >>>> >>> >>> >>> >>> -- >>> Asanka Vithanage >>> Senior Software Engineer -QA >>> Mobile: +94 0716286708 >>> Email: [email protected] >>> WSO2 Inc. www.wso2.com >>> >>> >> >> >> -- >> >> *Pandula Kariyawasam K.B.* >> *Senior Software Engineer - QA* >> Mobile: +94772314510 >> >> <http://www.wso2.com> >> > > > > -- > Asanka Vithanage > Senior Software Engineer -QA > Mobile: +94 0716286708 > Email: [email protected] > WSO2 Inc. www.wso2.com > > -- Regards, Inosh Goonewardena Associate Technical Lead- WSO2 Inc. Mobile: +94779966317
_______________________________________________ Dev mailing list [email protected] http://wso2.org/cgi-bin/mailman/listinfo/dev
