On Sun, Aug 21, 2016 at 1:49 PM, Sajith Ravindra <[email protected]> wrote:
> Persistence could be a case here. When we send events to CEP, it only >> processes them in-memory and forgets about it. But in the case of DAS it >> needs to persist each and every event it receives in the EVENT tables. This >> could be taking up cycles and hence affecting data receival. >> > > > I also agree with Nuwan. We did a similar performance test for IS > analytics[1] and we observed that performance with persistence the is > several factors less than the performance without persistence. Since we > persist each incoming event into DB there's big impact on performance. > Yes it can be due to persistence layer. @Supun, did we checked this Que filling issue happen due to limitation of publisher or limitation of receiver? Thanks, sanjeewa. > > [1] - [Architecture] IS-Analytics performance > > Thanks > *,Sajith Ravindra* > Senior Software Engineer > WSO2 Inc.; http://wso2.com > lean.enterprise.middleware > > mobile: +94 77 2273550 > blog: http://sajithr.blogspot.com/ > <http://lk.linkedin.com/pub/shani-ranasinghe/34/111/ab> > > On Sun, Aug 21, 2016 at 11:27 AM, Imesh Gunaratne <[email protected]> wrote: > >> >> >> On Fri, Aug 19, 2016 at 2:23 PM, Supun Sethunga <[email protected]> wrote: >> >>> Hi all, >>> >>> Please find the deployment setup details at [1]. Some parts of it are >>> yet to be completed though. >>> >>> @Imesh: No, we haven't created any puppet scripts for the setup. >>> >> >> Thanks Supun! Few things I noticed: >> >> - >> As I see we have run this on AWS. If so we can create a Cloud >> Formation script to automate the entire deployment (may be Puppet can be >> internally used but not mandatory). >> - The reason why we need automation for this is, as I believe >> performance tests should be able to run by any user for verifying the >> results. We have done this in MSF4J. >> >> - Is there any reason to use a VM for MySQL without using RDS? >> >> T >> hanks >> >> >> >>> >>> @MalithJ: Input was controlled using Jmeter. Throughput is measured at >>> the receiver side, in databridge level. (We can get it using >>> -DprofileReceiver=true property) >>> >>> @Sanjeew/Harsha: This was tested with the default protocols (Binary for >>> throttle publisher and thrift for the other publishers), but with tuned up >>> data-agent-configs. Here I used four separate standalone APIm nodes, only >>> to generate the events. This was not possible to simulate with a thrift >>> client like we did for ESB Analytics, because here we have multiple >>> receivers (unlike in ESB Analytics case), and the events are received at >>> differents rates to those different receivers. We can publish to the >>> streams separately, but it would not be same as a real world scenario.. >>> Here the limitation of the throughput is caused by the complexity of the >>> execution plans and the stream persisting, at the Analytics Server. >>> >>> [1] https://docs.google.com/document/d/1A1Pq-nOi9PmGdZz3wNov >>> Pc7kG3vc74p93tOTh_WYp_o/edit# >>> >>> Regards, >>> Supun >>> >>> On Fri, Aug 19, 2016 at 1:55 PM, Sanjeewa Malalgoda <[email protected]> >>> wrote: >>> >>>> >>>> >>>> On Tue, Aug 16, 2016 at 10:43 PM, Harsha Kumara <[email protected]> >>>> wrote: >>>> >>>>> >>>>> >>>>> On Tue, Aug 16, 2016 at 5:29 PM, Supun Sethunga <[email protected]> >>>>> wrote: >>>>> >>>>>> Hi all, >>>>>> >>>>>> We carried out some performance tests for APIM Analytics server in a >>>>>> Minimum HA setup. Idea was to saturate the Analytics server up to the >>>>>> maximum number of requests it can handle, without APIM (client side) >>>>>> getting their publisher queues full, see how it will behave in the long >>>>>> run >>>>>> (for couple of hours). With this test, found out that the maximum >>>>>> throughput it can handle is around 18,000 requests / sec. Please see >>>>>> below. >>>>>> For this test, used tomcat echo service as the backend. Used 4 standalone >>>>>> APIM servers to achieve the throughput. >>>>>> >>>>> This means if we scale up the API gateways, it will lead to getting >>>>> queue fill messages frequently when request throughput to DAS node is >>>>> larger than 19000 TPS. Since DAS node can handle more than 19000 TPS, this >>>>> behavior may cause due to rate of DAS server accepting messages is less >>>>> than the rate of message coming to gateways which leads to filling the >>>>> queues in gateways. Is this tested with binary or thrift transport? May be >>>>> better if we can have results for both binary and thrift transports. >>>>> >>>> When we test API Manager throttling we sent throttling events to CEP >>>> data receiver, we were able to send much more requests than this. >>>> I think in DAS also we are using same data receiver implementation(like >>>> harsha mentioned is it on thrift or binary?). >>>> If i remember correctly single data receiver handled 30000 or more >>>> requests(we had 10 gateways or so). >>>> I think here we need to check how data receiver and analyzer behaves >>>> when it have high load. So we may directly test it with client tool. If you >>>> test with gateway then anyway it will fail at 4500TPS(even if Que filling >>>> issue sorted) >>>> >>>> Thanks, >>>> sanjeewa. >>>> >>>>> >>>>>> Raw data can be found at [1]. >>>>>> >>>>>> [image: Inline image 1] >>>>>> If we increase the throughput beyond 19,000 requests/sec, then the >>>>>> queues at APIM side get filled after sometime. Please refer the 2nd, 3rd, >>>>>> and 4th sheets of [1]. The huge variance towards the end in those charts >>>>>> were due the queues getting filled. >>>>>> >>>>>> I will add these to a performance results template, including the >>>>>> advanced details about the setups and the configs. >>>>>> >>>>>> [1] https://docs.google.com/spreadsheets/d/1vVRPp8b31DccJ1A6 >>>>>> p9MRGxzmLAX2yiGAebT25kpsLzg/edit#gid=0 >>>>>> >>>>>> Regards, >>>>>> Supun >>>>>> >>>>>> -- >>>>>> *Supun Sethunga* >>>>>> Senior Software Engineer >>>>>> WSO2, Inc. >>>>>> http://wso2.com/ >>>>>> lean | enterprise | middleware >>>>>> Mobile : +94 716546324 >>>>>> Blog: http://supunsetunga.blogspot.com >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> Harsha Kumara >>>>> Software Engineer, WSO2 Inc. >>>>> Mobile: +94775505618 >>>>> Blog:harshcreationz.blogspot.com >>>>> >>>>> _______________________________________________ >>>>> Architecture mailing list >>>>> [email protected] >>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>> >>>>> >>>> >>>> >>>> -- >>>> >>>> *Sanjeewa Malalgoda* >>>> WSO2 Inc. >>>> Mobile : +94713068779 >>>> >>>> <http://sanjeewamalalgoda.blogspot.com/>blog >>>> :http://sanjeewamalalgoda.blogspot.com/ >>>> <http://sanjeewamalalgoda.blogspot.com/> >>>> >>>> >>>> >>> >>> >>> -- >>> *Supun Sethunga* >>> Senior Software Engineer >>> WSO2, Inc. >>> http://wso2.com/ >>> lean | enterprise | middleware >>> Mobile : +94 716546324 >>> Blog: http://supunsetunga.blogspot.com >>> >> >> >> >> -- >> *Imesh Gunaratne* >> Software Architect >> WSO2 Inc: http://wso2.com >> T: +94 11 214 5345 M: +94 77 374 2057 >> W: https://medium.com/@imesh TW: @imesh >> lean. enterprise. middleware >> >> >> _______________________________________________ >> Architecture mailing list >> [email protected] >> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >> >> > > _______________________________________________ > Architecture mailing list > [email protected] > https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture > > -- *Sanjeewa Malalgoda* WSO2 Inc. Mobile : +94713068779 <http://sanjeewamalalgoda.blogspot.com/>blog :http://sanjeewamalalgoda.blogspot.com/ <http://sanjeewamalalgoda.blogspot.com/>
_______________________________________________ Architecture mailing list [email protected] https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
