Hi Sanjeewa, Yes. Earlier we had a queue filling issue in throttle publisher, which was due to the publisher. It was able overcome by increasing the threads and batch size allocated to the publisher. But beyond the 19,000 TPS, the queue filling occurs due to the limitation at the receiver side (persisting + execution plan). Checked this by removing 1 publisher (1 APIM node), while other nodes were publishing at the same rate. Then the remaining nodes managed to publish at the same rate as earlier without any issue.
regards, Supun On Mon, Aug 22, 2016 at 11:04 AM, Sanjeewa Malalgoda <[email protected]> wrote: > > > On Sun, Aug 21, 2016 at 1:49 PM, Sajith Ravindra <[email protected]> wrote: > >> Persistence could be a case here. When we send events to CEP, it only >>> processes them in-memory and forgets about it. But in the case of DAS it >>> needs to persist each and every event it receives in the EVENT tables. This >>> could be taking up cycles and hence affecting data receival. >>> >> >> >> I also agree with Nuwan. We did a similar performance test for IS >> analytics[1] and we observed that performance with persistence the is >> several factors less than the performance without persistence. Since we >> persist each incoming event into DB there's big impact on performance. >> > Yes it can be due to persistence layer. > @Supun, did we checked this Que filling issue happen due to limitation of > publisher or limitation of receiver? > > Thanks, > sanjeewa. > >> >> [1] - [Architecture] IS-Analytics performance >> >> Thanks >> *,Sajith Ravindra* >> Senior Software Engineer >> WSO2 Inc.; http://wso2.com >> lean.enterprise.middleware >> >> mobile: +94 77 2273550 >> blog: http://sajithr.blogspot.com/ >> <http://lk.linkedin.com/pub/shani-ranasinghe/34/111/ab> >> >> On Sun, Aug 21, 2016 at 11:27 AM, Imesh Gunaratne <[email protected]> wrote: >> >>> >>> >>> On Fri, Aug 19, 2016 at 2:23 PM, Supun Sethunga <[email protected]> wrote: >>> >>>> Hi all, >>>> >>>> Please find the deployment setup details at [1]. Some parts of it are >>>> yet to be completed though. >>>> >>>> @Imesh: No, we haven't created any puppet scripts for the setup. >>>> >>> >>> Thanks Supun! Few things I noticed: >>> >>> - >>> As I see we have run this on AWS. If so we can create a Cloud >>> Formation script to automate the entire deployment (may be Puppet can be >>> internally used but not mandatory). >>> - The reason why we need automation for this is, as I believe >>> performance tests should be able to run by any user for verifying the >>> results. We have done this in MSF4J. >>> >>> - Is there any reason to use a VM for MySQL without using RDS? >>> >>> T >>> hanks >>> >>> >>> >>>> >>>> @MalithJ: Input was controlled using Jmeter. Throughput is measured at >>>> the receiver side, in databridge level. (We can get it using >>>> -DprofileReceiver=true property) >>>> >>>> @Sanjeew/Harsha: This was tested with the default protocols (Binary for >>>> throttle publisher and thrift for the other publishers), but with tuned up >>>> data-agent-configs. Here I used four separate standalone APIm nodes, only >>>> to generate the events. This was not possible to simulate with a thrift >>>> client like we did for ESB Analytics, because here we have multiple >>>> receivers (unlike in ESB Analytics case), and the events are received at >>>> differents rates to those different receivers. We can publish to the >>>> streams separately, but it would not be same as a real world scenario.. >>>> Here the limitation of the throughput is caused by the complexity of >>>> the execution plans and the stream persisting, at the Analytics Server. >>>> >>>> [1] https://docs.google.com/document/d/1A1Pq-nOi9PmGdZz3wNov >>>> Pc7kG3vc74p93tOTh_WYp_o/edit# >>>> >>>> Regards, >>>> Supun >>>> >>>> On Fri, Aug 19, 2016 at 1:55 PM, Sanjeewa Malalgoda <[email protected]> >>>> wrote: >>>> >>>>> >>>>> >>>>> On Tue, Aug 16, 2016 at 10:43 PM, Harsha Kumara <[email protected]> >>>>> wrote: >>>>> >>>>>> >>>>>> >>>>>> On Tue, Aug 16, 2016 at 5:29 PM, Supun Sethunga <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> Hi all, >>>>>>> >>>>>>> We carried out some performance tests for APIM Analytics server in a >>>>>>> Minimum HA setup. Idea was to saturate the Analytics server up to the >>>>>>> maximum number of requests it can handle, without APIM (client side) >>>>>>> getting their publisher queues full, see how it will behave in the long >>>>>>> run >>>>>>> (for couple of hours). With this test, found out that the maximum >>>>>>> throughput it can handle is around 18,000 requests / sec. Please see >>>>>>> below. >>>>>>> For this test, used tomcat echo service as the backend. Used 4 >>>>>>> standalone >>>>>>> APIM servers to achieve the throughput. >>>>>>> >>>>>> This means if we scale up the API gateways, it will lead to getting >>>>>> queue fill messages frequently when request throughput to DAS node is >>>>>> larger than 19000 TPS. Since DAS node can handle more than 19000 TPS, >>>>>> this >>>>>> behavior may cause due to rate of DAS server accepting messages is less >>>>>> than the rate of message coming to gateways which leads to filling the >>>>>> queues in gateways. Is this tested with binary or thrift transport? May >>>>>> be >>>>>> better if we can have results for both binary and thrift transports. >>>>>> >>>>> When we test API Manager throttling we sent throttling events to CEP >>>>> data receiver, we were able to send much more requests than this. >>>>> I think in DAS also we are using same data receiver >>>>> implementation(like harsha mentioned is it on thrift or binary?). >>>>> If i remember correctly single data receiver handled 30000 or more >>>>> requests(we had 10 gateways or so). >>>>> I think here we need to check how data receiver and analyzer behaves >>>>> when it have high load. So we may directly test it with client tool. If >>>>> you >>>>> test with gateway then anyway it will fail at 4500TPS(even if Que filling >>>>> issue sorted) >>>>> >>>>> Thanks, >>>>> sanjeewa. >>>>> >>>>>> >>>>>>> Raw data can be found at [1]. >>>>>>> >>>>>>> [image: Inline image 1] >>>>>>> If we increase the throughput beyond 19,000 requests/sec, then the >>>>>>> queues at APIM side get filled after sometime. Please refer the 2nd, >>>>>>> 3rd, >>>>>>> and 4th sheets of [1]. The huge variance towards the end in those charts >>>>>>> were due the queues getting filled. >>>>>>> >>>>>>> I will add these to a performance results template, including the >>>>>>> advanced details about the setups and the configs. >>>>>>> >>>>>>> [1] https://docs.google.com/spreadsheets/d/1vVRPp8b31DccJ1A6 >>>>>>> p9MRGxzmLAX2yiGAebT25kpsLzg/edit#gid=0 >>>>>>> >>>>>>> Regards, >>>>>>> Supun >>>>>>> >>>>>>> -- >>>>>>> *Supun Sethunga* >>>>>>> Senior Software Engineer >>>>>>> WSO2, Inc. >>>>>>> http://wso2.com/ >>>>>>> lean | enterprise | middleware >>>>>>> Mobile : +94 716546324 >>>>>>> Blog: http://supunsetunga.blogspot.com >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Harsha Kumara >>>>>> Software Engineer, WSO2 Inc. >>>>>> Mobile: +94775505618 >>>>>> Blog:harshcreationz.blogspot.com >>>>>> >>>>>> _______________________________________________ >>>>>> Architecture mailing list >>>>>> [email protected] >>>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> >>>>> *Sanjeewa Malalgoda* >>>>> WSO2 Inc. >>>>> Mobile : +94713068779 >>>>> >>>>> <http://sanjeewamalalgoda.blogspot.com/>blog >>>>> :http://sanjeewamalalgoda.blogspot.com/ >>>>> <http://sanjeewamalalgoda.blogspot.com/> >>>>> >>>>> >>>>> >>>> >>>> >>>> -- >>>> *Supun Sethunga* >>>> Senior Software Engineer >>>> WSO2, Inc. >>>> http://wso2.com/ >>>> lean | enterprise | middleware >>>> Mobile : +94 716546324 >>>> Blog: http://supunsetunga.blogspot.com >>>> >>> >>> >>> >>> -- >>> *Imesh Gunaratne* >>> Software Architect >>> WSO2 Inc: http://wso2.com >>> T: +94 11 214 5345 M: +94 77 374 2057 >>> W: https://medium.com/@imesh TW: @imesh >>> lean. enterprise. middleware >>> >>> >>> _______________________________________________ >>> Architecture mailing list >>> [email protected] >>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>> >>> >> >> _______________________________________________ >> Architecture mailing list >> [email protected] >> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >> >> > > > -- > > *Sanjeewa Malalgoda* > WSO2 Inc. > Mobile : +94713068779 > > <http://sanjeewamalalgoda.blogspot.com/>blog :http://sanjeewamalalgoda. > blogspot.com/ <http://sanjeewamalalgoda.blogspot.com/> > > > > _______________________________________________ > Architecture mailing list > [email protected] > https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture > > -- *Supun Sethunga* Senior Software Engineer WSO2, Inc. http://wso2.com/ lean | enterprise | middleware Mobile : +94 716546324 Blog: http://supunsetunga.blogspot.com
_______________________________________________ Architecture mailing list [email protected] https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
