Hi Ravindra, Thanks for your suggestion. Hi all, Seems like the following is the root cause for this issue:
When an endpoint is unavailable, we noticed that, whatever is given after the send mediator (in the insequence) doesn't get executed. As a result, the BAM mediator, which is placed after the send mediator, doesn't get executed; and it will not receive the event that the API was invoked. Hence the stats go wrong. I created an API with a load-balanced endpoint in ESB 4.8.1 and noticed the same behaviour. Refer to mail thread in dev list with subject "[ESB] Issue in Load Balanced Endpoint when one end point is unavailable". Thanks, Dilini On Tue, Oct 28, 2014 at 9:11 AM, Ravindra Ranwala <[email protected]> wrote: > Hi Lakshman, > > AFAIK this looks like a configuration issue. You have given a name to > endpoints which were defined inline. Can you please change the > configuration as given in article [1] and simplify this. Then try that out > again. > > > [1] > https://docs.wso2.com/display/ESB481/Sample+54%3A+Session+Affinity+Load+Balancing+between+Three+Endpoints > > > Thanks & Regards, > > On Mon, Oct 27, 2014 at 10:31 AM, Lakshman Udayakantha <[email protected] > > wrote: > >> Hi All, >> >> As per the offline discussion with Lakmali, The reason for this issue is >> when one end point is down, It will fail to go to the bam mediator(which >> is responsible for publishing event to destination statistics table in BAM >> cassandra database)which is added after send mediator in synapse >> configuration(that is generated when new api is created) for every 30 >> seconds. I just checked that by adding a log mediator between send mediator >> and bam mediator. Seems like no issue with bam mediator. It publishes all >> the request it receives. need to find why call will not through bam >> mediator for every 30 seconds. I added the sample inSequence generated >> here. kindly expect your feedback on this. >> <inSequence> >> <filter source="$ctx:AM_KEY_TYPE" regex="PRODUCTION"> >> <then> >> <send> >> <endpoint name="admin--helloSearvice_APIproductionEndpoint_0"> >> <loadbalance algorithm=" >> org.apache.synapse.endpoints.algorithms.RoundRobin"> >> <endpoint name="admin--helloSearvice_APIproductionEndpoint_0_0"> >> <http uri-template="http://10.100.5.161:8310/services"/> >> <property name="ENDPOINT_ADDRESS" value=" >> http://10.100.5.161:8310/services"/> >> </endpoint> >> <endpoint name="admin--helloSearvice_APIproductionEndpoint_0_1"> >> <http uri-template="http://10.100.5.161:8290/services"/> >> <property name="ENDPOINT_ADDRESS" value=" >> http://10.100.5.161:8290/services"/> >> </endpoint> >> <endpoint name="admin--helloSearvice_APIproductionEndpoint_0_2"> >> <http uri-template="http://10.100.5.161:8300/services"/> >> <property name="ENDPOINT_ADDRESS" value=" >> http://10.100.5.161:8300/services"/> >> </endpoint> >> </loadbalance> >> </endpoint> >> </send> >> <bam> >> <serverProfile name="bam-profile"> >> <streamConfig name="org_wso2_apimgt_statistics_destination" version=" >> 1.0.0"/> >> </serverProfile> >> </bam> >> </then> >> <else> >> <sequence key="_sandbox_key_error_"/> >> </else> >> </filter> >> </inSequence> >> >> Thanks >> >> On Wed, Oct 22, 2014 at 9:00 PM, Lakshman Udayakantha <[email protected] >> > wrote: >> >>> Hi Manoj, >>> >>> I tested this with single BAM instance. As Gihan mentioned ,Still need >>> to check there is a problem in APIM side. I will update this thread if >>> problem is in APIM >>> >>> Thanks >>> >>> On Wed, Oct 22, 2014 at 7:49 PM, Manoj Gunawardena <[email protected]> >>> wrote: >>> >>>> Hi Lakshman, >>>> >>>> This is very useful results, Thanks for sharing. We will do the same >>>> test for ES as well. Is this test run for single BAM instance, or multiple? >>>> (load balanced environment) >>>> >>>> Thanks >>>> >>>> On Wed, Oct 22, 2014 at 6:37 PM, Lakshman Udayakantha < >>>> [email protected]> wrote: >>>> >>>>> Hi Gihan, >>>>> >>>>> I sent 10000 requests with the preconditions you have mentioned. only >>>>> one record is missed in cassendra database for every time I sent 10000 >>>>> requests via APIM. >>>>> >>>>> Thanks >>>>> >>>>> On Wed, Oct 22, 2014 at 9:28 AM, Lakshman Udayakantha < >>>>> [email protected]> wrote: >>>>> >>>>>> Hi Gihan, >>>>>> >>>>>> I will do that test and get back to you. >>>>>> >>>>>> Thanks >>>>>> >>>>>> On Wed, Oct 22, 2014 at 9:23 AM, Inosh Goonewardena <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> On Wednesday, October 22, 2014, Lakshman Udayakantha < >>>>>>> [email protected]> wrote: >>>>>>> >>>>>>>> Hi Inosh, >>>>>>>> >>>>>>>> data in cassandra database are summized to a relational database. >>>>>>>> records in cassandra database and this relational table is same. so I >>>>>>>> think >>>>>>>> there is no problem with cassandra explorer. >>>>>>>> >>>>>>> >>>>>>> Ok understood. I have misread your previous email. >>>>>>> >>>>>>>> >>>>>>>> Thanks >>>>>>>> >>>>>>>> On Wed, Oct 22, 2014 at 9:14 AM, Inosh Goonewardena <[email protected] >>>>>>>> > wrote: >>>>>>>> >>>>>>>>> Hi Lakshman, >>>>>>>>> >>>>>>>>> You mean data is therw in cassandra and cassandra explorer doesn't >>>>>>>>> show these data? Did you compare by using cassandra CLI tool? >>>>>>>>> >>>>>>>>> On Wednesday, October 22, 2014, Lakshman Udayakantha < >>>>>>>>> [email protected]> wrote: >>>>>>>>> >>>>>>>>>> HI Gihan , >>>>>>>>>> >>>>>>>>>> They are lost randomly. >>>>>>>>>> >>>>>>>>>> Thanks >>>>>>>>>> >>>>>>>>>> On Wed, Oct 22, 2014 at 9:00 AM, Gihan Anuruddha <[email protected]> >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> Hi Lakshman, >>>>>>>>>>> >>>>>>>>>>> How many records did you send to BAM and how many events are >>>>>>>>>>> lost? >>>>>>>>>>> >>>>>>>>>>> Regards, >>>>>>>>>>> Gihan >>>>>>>>>>> >>>>>>>>>>> On Wed, Oct 22, 2014 at 8:18 AM, Lakshman Udayakantha < >>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>> >>>>>>>>>>>> Hi Gihan, >>>>>>>>>>>> >>>>>>>>>>>> These data in cassandra database are summarised by a hive >>>>>>>>>>>> script and insert into a relational database. I checked that >>>>>>>>>>>> relational >>>>>>>>>>>> database table also. Number of records in cassandra database and >>>>>>>>>>>> relational >>>>>>>>>>>> database table are same. Then it should not be a problem with >>>>>>>>>>>> cassandra >>>>>>>>>>>> explorer. >>>>>>>>>>>> >>>>>>>>>>>> Thanks >>>>>>>>>>>> >>>>>>>>>>>> On Wed, Oct 22, 2014 at 7:53 AM, Manoj Gunawardena < >>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Hi, >>>>>>>>>>>>> >>>>>>>>>>>>> This can be an issue for other products as well. ES, AS and >>>>>>>>>>>>> EMM also publishing data to BAM. In ES we are using load >>>>>>>>>>>>> balancing data >>>>>>>>>>>>> publisher. I also notice if we call stop method, sometimes data >>>>>>>>>>>>> not get >>>>>>>>>>>>> publish to BAM. We changed this to initiate data publisher at the >>>>>>>>>>>>> first >>>>>>>>>>>>> event and call data publishers stop method in the bundle >>>>>>>>>>>>> activator's stop >>>>>>>>>>>>> method. >>>>>>>>>>>>> Need to test with load. >>>>>>>>>>>>> >>>>>>>>>>>>> Thanks >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> On Tue, Oct 21, 2014 at 10:07 PM, Gihan Anuruddha < >>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> Hi Lakshman, >>>>>>>>>>>>>> >>>>>>>>>>>>>> It should be in Cassandra if publishEvent method executes. >>>>>>>>>>>>>> How do you validate the data loss in Cassandra side? BAM >>>>>>>>>>>>>> embedded Cassandra explorer is a not good tool for that. Can >>>>>>>>>>>>>> you please verify that with tool like CQL or cassandra-cli? >>>>>>>>>>>>>> >>>>>>>>>>>>>> Regards, >>>>>>>>>>>>>> Gihan >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Tue, Oct 21, 2014 at 9:18 PM, Lakshman Udayakantha < >>>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> Hi All, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I am working on https://wso2.org/jira/browse/APIMANAGER-2064 >>>>>>>>>>>>>>> . >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> By looking at the Cassandra database on BAM, I could figure >>>>>>>>>>>>>>> out some requests publish to Cassandra database and some do not >>>>>>>>>>>>>>> when I >>>>>>>>>>>>>>> invoke the api. That is the reason for the missing requests in >>>>>>>>>>>>>>> statistics >>>>>>>>>>>>>>> dashboard. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I tried debugging the APIMgtUsageDataBridgeDataPublisher >>>>>>>>>>>>>>> class since it has publishEvent method to publish data. >>>>>>>>>>>>>>> APIMgtUsageDataBridgeDataPublisher class uses >>>>>>>>>>>>>>> LoadBalancingDataPublisher as >>>>>>>>>>>>>>> the dataPublisher in this case. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I found that every time when I invoke the api via rest >>>>>>>>>>>>>>> client, this method will execute. but randomly some requests >>>>>>>>>>>>>>> are not >>>>>>>>>>>>>>> publishing in Cassandra database. I used the >>>>>>>>>>>>>>> latest org.wso2.carbon.databridge.agent.thrift jar. but the >>>>>>>>>>>>>>> problem is >>>>>>>>>>>>>>> still there. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> if requests come to publishEvent method, should they publish >>>>>>>>>>>>>>> in Cassandra database?and is there any known reason for data >>>>>>>>>>>>>>> get not >>>>>>>>>>>>>>> published in cassendra database. if not where I can find the >>>>>>>>>>>>>>> data publisher >>>>>>>>>>>>>>> to cassendra in BAM from API Manager. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Thanks >>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>> Lakshman Udayakantha >>>>>>>>>>>>>>> WSO2 Inc. www.wso2.com >>>>>>>>>>>>>>> lean.enterprise.middleware >>>>>>>>>>>>>>> Mobile: *0711241005 <0711241005>* >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>>> Dev mailing list >>>>>>>>>>>>>>> [email protected] >>>>>>>>>>>>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> -- >>>>>>>>>>>>>> W.G. Gihan Anuruddha >>>>>>>>>>>>>> Senior Software Engineer | WSO2, Inc. >>>>>>>>>>>>>> M: +94772272595 >>>>>>>>>>>>>> >>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>> Dev mailing list >>>>>>>>>>>>>> [email protected] >>>>>>>>>>>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> -- >>>>>>>>>>>>> Manoj Gunawardena >>>>>>>>>>>>> Tech Lead >>>>>>>>>>>>> WSO2, Inc.: http://wso2.com >>>>>>>>>>>>> lean.enterprise.middleware >>>>>>>>>>>>> Mobile : +94 77 2291643 >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>>>>>> Lakshman Udayakantha >>>>>>>>>>>> WSO2 Inc. www.wso2.com >>>>>>>>>>>> lean.enterprise.middleware >>>>>>>>>>>> Mobile: *0711241005 <0711241005>* >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> W.G. Gihan Anuruddha >>>>>>>>>>> Senior Software Engineer | WSO2, Inc. >>>>>>>>>>> M: +94772272595 >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> Lakshman Udayakantha >>>>>>>>>> WSO2 Inc. www.wso2.com >>>>>>>>>> lean.enterprise.middleware >>>>>>>>>> Mobile: *0711241005* >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> Regards, >>>>>>>>> >>>>>>>>> Inosh Goonewardena >>>>>>>>> Associate Technical Lead- WSO2 Inc. >>>>>>>>> Mobile: +94779966317 >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Lakshman Udayakantha >>>>>>>> WSO2 Inc. www.wso2.com >>>>>>>> lean.enterprise.middleware >>>>>>>> Mobile: *0711241005* >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Regards, >>>>>>> >>>>>>> Inosh Goonewardena >>>>>>> Associate Technical Lead- WSO2 Inc. >>>>>>> Mobile: +94779966317 >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Lakshman Udayakantha >>>>>> WSO2 Inc. www.wso2.com >>>>>> lean.enterprise.middleware >>>>>> Mobile: *0711241005* >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> Lakshman Udayakantha >>>>> WSO2 Inc. www.wso2.com >>>>> lean.enterprise.middleware >>>>> Mobile: *0711241005* >>>>> >>>>> >>>>> _______________________________________________ >>>>> Dev mailing list >>>>> [email protected] >>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>>>> >>>>> >>>> >>>> >>>> -- >>>> Manoj Gunawardena >>>> Tech Lead >>>> WSO2, Inc.: http://wso2.com >>>> lean.enterprise.middleware >>>> Mobile : +94 77 2291643 >>>> >>> >>> >>> >>> -- >>> Lakshman Udayakantha >>> WSO2 Inc. www.wso2.com >>> lean.enterprise.middleware >>> Mobile: *0711241005* >>> >>> >> >> >> -- >> Lakshman Udayakantha >> WSO2 Inc. www.wso2.com >> lean.enterprise.middleware >> Mobile: *0711241005* >> >> > > > -- > Ravindra Ranwala > Software Engineer > WSO2, Inc: http://wso2.com > <http://www.google.com/url?q=http%3A%2F%2Fwso2.com&sa=D&sntz=1&usg=AFQjCNEZvyc0uMD1HhBaEGCBxs6e9fBObg> > Mobile: +94714198770 > > > _______________________________________________ > Dev mailing list > [email protected] > http://wso2.org/cgi-bin/mailman/listinfo/dev > > -- *Dilini Muthumala* Software Engineer, WSO2 Inc. *E-mail :* [email protected] *Mobile: *+94713 400 029
_______________________________________________ Dev mailing list [email protected] http://wso2.org/cgi-bin/mailman/listinfo/dev
