Hi Lakshman, First you should make sure all the required statistics are published to BAM with the data publisher. In order to do it you can, 1. Add a debug log to the data publisher publishing method to log all the messages passed through it. 2. Then you can use Cassandra Explorer or another tool like cassandra cli to see whether all the events sent to BAM are stored in Cassandra. 3. ONLY AFTER the above two was confirmed working properly we have to look into next stage, summarised data. 4. Clean H2/MySQL tables and run the hive script again with drop table commands for all Hive tables at the start of the script. 5. Then if the Hive query runs without any issue look at the summarised data in H2/MySQL tables.
Therefore let the publisher publish several events to BAM and then verify the above given steps. There were several situations I have seen earlier that had problems with not publishing data due to an other publisher side error in logic. Thanks. *Maninda Edirisooriya* Senior Software Engineer *WSO2, Inc.*lean.enterprise.middleware. *Blog* : http://maninda.blogspot.com/ *E-mail* : [email protected] *Skype* : @manindae *Twitter* : @maninda On Wed, Oct 22, 2014 at 9:28 AM, Lakshman Udayakantha <[email protected]> wrote: > Hi Gihan, > > I will do that test and get back to you. > > Thanks > > On Wed, Oct 22, 2014 at 9:23 AM, Inosh Goonewardena <[email protected]> > wrote: > >> Hi, >> >> On Wednesday, October 22, 2014, Lakshman Udayakantha <[email protected]> >> wrote: >> >>> Hi Inosh, >>> >>> data in cassandra database are summized to a relational database. >>> records in cassandra database and this relational table is same. so I think >>> there is no problem with cassandra explorer. >>> >> >> Ok understood. I have misread your previous email. >> >>> >>> Thanks >>> >>> On Wed, Oct 22, 2014 at 9:14 AM, Inosh Goonewardena <[email protected]> >>> wrote: >>> >>>> Hi Lakshman, >>>> >>>> You mean data is therw in cassandra and cassandra explorer doesn't show >>>> these data? Did you compare by using cassandra CLI tool? >>>> >>>> On Wednesday, October 22, 2014, Lakshman Udayakantha < >>>> [email protected]> wrote: >>>> >>>>> HI Gihan , >>>>> >>>>> They are lost randomly. >>>>> >>>>> Thanks >>>>> >>>>> On Wed, Oct 22, 2014 at 9:00 AM, Gihan Anuruddha <[email protected]> >>>>> wrote: >>>>> >>>>>> Hi Lakshman, >>>>>> >>>>>> How many records did you send to BAM and how many events are lost? >>>>>> >>>>>> Regards, >>>>>> Gihan >>>>>> >>>>>> On Wed, Oct 22, 2014 at 8:18 AM, Lakshman Udayakantha < >>>>>> [email protected]> wrote: >>>>>> >>>>>>> Hi Gihan, >>>>>>> >>>>>>> These data in cassandra database are summarised by a hive script and >>>>>>> insert into a relational database. I checked that relational database >>>>>>> table >>>>>>> also. Number of records in cassandra database and relational database >>>>>>> table >>>>>>> are same. Then it should not be a problem with cassandra explorer. >>>>>>> >>>>>>> Thanks >>>>>>> >>>>>>> On Wed, Oct 22, 2014 at 7:53 AM, Manoj Gunawardena <[email protected]> >>>>>>> wrote: >>>>>>> >>>>>>>> Hi, >>>>>>>> >>>>>>>> This can be an issue for other products as well. ES, AS and EMM >>>>>>>> also publishing data to BAM. In ES we are using load balancing data >>>>>>>> publisher. I also notice if we call stop method, sometimes data not get >>>>>>>> publish to BAM. We changed this to initiate data publisher at the first >>>>>>>> event and call data publishers stop method in the bundle activator's >>>>>>>> stop >>>>>>>> method. >>>>>>>> Need to test with load. >>>>>>>> >>>>>>>> Thanks >>>>>>>> >>>>>>>> >>>>>>>> On Tue, Oct 21, 2014 at 10:07 PM, Gihan Anuruddha <[email protected]> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Hi Lakshman, >>>>>>>>> >>>>>>>>> It should be in Cassandra if publishEvent method executes. How do >>>>>>>>> you validate the data loss in Cassandra side? BAM embedded >>>>>>>>> Cassandra explorer is a not good tool for that. Can you please >>>>>>>>> verify that with tool like CQL or cassandra-cli? >>>>>>>>> >>>>>>>>> Regards, >>>>>>>>> Gihan >>>>>>>>> >>>>>>>>> On Tue, Oct 21, 2014 at 9:18 PM, Lakshman Udayakantha < >>>>>>>>> [email protected]> wrote: >>>>>>>>> >>>>>>>>>> Hi All, >>>>>>>>>> >>>>>>>>>> I am working on https://wso2.org/jira/browse/APIMANAGER-2064. >>>>>>>>>> >>>>>>>>>> By looking at the Cassandra database on BAM, I could figure out >>>>>>>>>> some requests publish to Cassandra database and some do not when I >>>>>>>>>> invoke >>>>>>>>>> the api. That is the reason for the missing requests in statistics >>>>>>>>>> dashboard. >>>>>>>>>> >>>>>>>>>> I tried debugging the APIMgtUsageDataBridgeDataPublisher class >>>>>>>>>> since it has publishEvent method to publish data. >>>>>>>>>> APIMgtUsageDataBridgeDataPublisher class uses >>>>>>>>>> LoadBalancingDataPublisher as >>>>>>>>>> the dataPublisher in this case. >>>>>>>>>> >>>>>>>>>> I found that every time when I invoke the api via rest client, >>>>>>>>>> this method will execute. but randomly some requests are not >>>>>>>>>> publishing in >>>>>>>>>> Cassandra database. I used the >>>>>>>>>> latest org.wso2.carbon.databridge.agent.thrift jar. but the problem >>>>>>>>>> is >>>>>>>>>> still there. >>>>>>>>>> >>>>>>>>>> if requests come to publishEvent method, should they publish in >>>>>>>>>> Cassandra database?and is there any known reason for data get not >>>>>>>>>> published >>>>>>>>>> in cassendra database. if not where I can find the data publisher to >>>>>>>>>> cassendra in BAM from API Manager. >>>>>>>>>> >>>>>>>>>> Thanks >>>>>>>>>> -- >>>>>>>>>> Lakshman Udayakantha >>>>>>>>>> WSO2 Inc. www.wso2.com >>>>>>>>>> lean.enterprise.middleware >>>>>>>>>> Mobile: *0711241005 <0711241005>* >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> _______________________________________________ >>>>>>>>>> Dev mailing list >>>>>>>>>> [email protected] >>>>>>>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> W.G. Gihan Anuruddha >>>>>>>>> Senior Software Engineer | WSO2, Inc. >>>>>>>>> M: +94772272595 >>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> Dev mailing list >>>>>>>>> [email protected] >>>>>>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Manoj Gunawardena >>>>>>>> Tech Lead >>>>>>>> WSO2, Inc.: http://wso2.com >>>>>>>> lean.enterprise.middleware >>>>>>>> Mobile : +94 77 2291643 >>>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Lakshman Udayakantha >>>>>>> WSO2 Inc. www.wso2.com >>>>>>> lean.enterprise.middleware >>>>>>> Mobile: *0711241005 <0711241005>* >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> W.G. Gihan Anuruddha >>>>>> Senior Software Engineer | WSO2, Inc. >>>>>> M: +94772272595 >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> Lakshman Udayakantha >>>>> WSO2 Inc. www.wso2.com >>>>> lean.enterprise.middleware >>>>> Mobile: *0711241005 <0711241005>* >>>>> >>>>> >>>> >>>> -- >>>> Regards, >>>> >>>> Inosh Goonewardena >>>> Associate Technical Lead- WSO2 Inc. >>>> Mobile: +94779966317 >>>> >>>> >>> >>> >>> -- >>> Lakshman Udayakantha >>> WSO2 Inc. www.wso2.com >>> lean.enterprise.middleware >>> Mobile: *0711241005 <0711241005>* >>> >>> >> >> -- >> Regards, >> >> Inosh Goonewardena >> Associate Technical Lead- WSO2 Inc. >> Mobile: +94779966317 >> >> > > > -- > Lakshman Udayakantha > WSO2 Inc. www.wso2.com > lean.enterprise.middleware > Mobile: *0711241005 <0711241005>* > > > _______________________________________________ > Dev mailing list > [email protected] > http://wso2.org/cgi-bin/mailman/listinfo/dev > >
_______________________________________________ Dev mailing list [email protected] http://wso2.org/cgi-bin/mailman/listinfo/dev
