Hi Nuwan, This was indeed very good information. We were able to create a Jmeter script to simulate the scenario you have explained earlier. Also, we have configured Java Flight Recorder as advised. Please find the long running test details for API-M1.10.0 GA pack below.
*Long running test details:-* - 20 concurrent invocations continuously being run using permanent tokens (10 each http and https) - 1000 tokens were generated by using 1000 different apps > subscribed to an existing API using those apps > tokens expire after 45 minutes > expired tokens are regenerated after every 45 minutes > invoke the API using newly generated tokens concurrently with 30 threads (Altogether 50 concurrent threads being executed continuously.) We are planning to run these tests for two weeks and then get back to you with outcomes. Thanks for your feedback on this. Cheers, Pubudu. Pubudu D.P Senior Software Engineer - QA Team | WSO2 inc. Mobile : +94775464547 On Fri, Jan 8, 2016 at 12:20 PM, Nuwan Dias <[email protected]> wrote: > +1. > > Thanks, > NuwanD. > > On Fri, Jan 8, 2016 at 11:19 AM, Aparna Karunarathna <[email protected]> > wrote: > >> Hi Nuwan, >> >> Understood your concern Nuwan, I'm the one who proposed to add peak load >> for few hours( ~ 4 hours) and check the memory growth, GC , exceptions etc >> in Gateway and KM sides. In here we are not trying to hit the server with >> max load, just want to check the server behavior. :) If you are not >> agreeing to this, will do proper long running test for 2 weeks with load >> that you suggested and will share the results with you. Then 3rd week will >> do some ad-hoc test like we suggested and check server behavior. WDYT? >> >> Regards, >> Aparna. >> >> On Fri, Jan 8, 2016 at 11:01 AM, Nuwan Dias <[email protected]> wrote: >> >>> Hi Aparna, >>> >>> I'm totally +1 for the long running test. But my concern is that Pubudu >>> is planning for a strange test. It doesn't look like a typical long running >>> test. See his comment on the last reply "*generate a relatively higher >>> number of threads lets say for a few hours of the day and stop it on daily >>> basis*". The idea I get from this is that you're trying to hit the >>> server at max load and then stop... hit it at max again and stop... >>> likewise. Maybe I'm getting the wrong message but the information you've >>> requested is for the max threads count and not the moderate threads count. >>> >>> IMO to mimick a real world scenario you should be hitting the sever with >>> a moderate load (about 50 threads max) for a long time period. Not with a >>> peak load (250 threads) for several hours. And we should monitor the >>> servers during this period in order to have insight of any issues that may >>> occur. The closer we get to the real world occurrence, the more >>> valuable the outcome of our efforts would be. >>> >>> Thanks, >>> NuwanD. >>> >>> On Fri, Jan 8, 2016 at 10:47 AM, Aparna Karunarathna <[email protected]> >>> wrote: >>> >>>> Hi Nuwan, >>>> >>>> Objective of doing a long running test after the release is, in the >>>> release time we won't be able to execute a long running test for couple of >>>> weeks. Because sometime we have to stop the test to change the packs or >>>> integrate with other product with setup or to enable some configs to test a >>>> feature, etc. Most of the time we are getting some cloud issue as >>>> well(resource issues and set of nodes were getting slow). So most of time >>>> we were able to run the long running test for day or 2 days. >>>> >>>> Therefor we thought of doing a proper long running test against the GA >>>> pack for 3 weeks. Last year we did long running test after the release for >>>> couple of releases and then we shared our findings(memory growth, >>>> exceptions, warnings, etc) with dev teams. So if we can identify a bug at >>>> this stage then we can prepare a fix. >>>> >>>> Regards, >>>> Aparna. >>>> >>>> >>>> >>>> On Fri, Jan 8, 2016 at 9:22 AM, Nuwan Dias <[email protected]> wrote: >>>> >>>>> >>>>> >>>>> On Fri, Jan 8, 2016 at 7:03 AM, Pubudu Priyashan <[email protected]> >>>>> wrote: >>>>> >>>>>> Hi Nuwan, >>>>>> >>>>>> The idea is not to keep the peak level constant throughout the >>>>>> duration of the long running test but, to generate a relatively higher >>>>>> number of threads lets say for a few hours of the day and stop it on >>>>>> daily >>>>>> basis. >>>>>> >>>>> >>>>> What is the objective of this test? From what I understand, its not a >>>>> performance test and its not a typical long running test as well. So can >>>>> you explain what you're trying to achieve and how it benefits the product >>>>> or any other? When doing performance tests the peak thread count we hit >>>>> was >>>>> around 250 and you can use it if you want. But its better to do it with a >>>>> plan and clear objective. >>>>> >>>>> >>>>>> Appreciate if you could suggest a number for the peak thread count >>>>>> per single gateway node so we can monitor the behaviour. >>>>>> >>>>>> Currently we don't have scripts to support the scenario you >>>>>> suggested. We will start working on this and create relevant scripts and >>>>>> update you once we have tested the scenario you have mentioned. Thanks. >>>>>> >>>>>> Cheers, >>>>>> Pubudu. >>>>>> >>>>>> Pubudu D.P >>>>>> Senior Software Engineer - QA Team | WSO2 inc. >>>>>> Mobile : +94775464547 >>>>>> >>>>>> On Thu, Jan 7, 2016 at 4:01 PM, Nuwan Dias <[email protected]> wrote: >>>>>> >>>>>>> Hi Pubudu, >>>>>>> >>>>>>> I don't think its useful to do a long running test at peak load >>>>>>> because it doesn't happen in reality. Any system can hit peaks, but it >>>>>>> would be a momentary occurrence rather than a long running situation. >>>>>>> >>>>>>> What would be more useful is to do the long running test under >>>>>>> moderate load. This would mimick many real world scenarios. Therefore I >>>>>>> would recommend to have the concurrency at 50 threads, using about 1000 >>>>>>> access tokens and run the test in such a way that it renews its access >>>>>>> tokens from time to time (either when they expire or once every 45 >>>>>>> minutes >>>>>>> or so). >>>>>>> >>>>>>> We should also configure the Java Flight Recorder (remotely, to >>>>>>> avoid filling up the file system) so that we would have insight into >>>>>>> issues >>>>>>> if any. Also please make sure that the back-end system you front through >>>>>>> the gateway is stable and can withhold the tests. >>>>>>> >>>>>>> Thanks, >>>>>>> NuwanD. >>>>>>> >>>>>>> On Thu, Jan 7, 2016 at 3:18 PM, Pubudu Priyashan <[email protected]> >>>>>>> wrote: >>>>>>> >>>>>>>> >>>>>>>> Hi Nuwan, >>>>>>>> >>>>>>>> We are planning to start a long running test on the API-M 1.10.0 >>>>>>>> released pack. As a part of this we need to simulate a peak time in >>>>>>>> gateway >>>>>>>> worker nodes (by sending a large number of concurrent threads) could >>>>>>>> you >>>>>>>> please let us know what is the maximum expected limit of concurrent >>>>>>>> threads >>>>>>>> that we can be served in a single gateway worker node? We will then >>>>>>>> set up >>>>>>>> the long-running tests accordingly. Thanks! >>>>>>>> >>>>>>>> Cheers, >>>>>>>> Pubudu D.P >>>>>>>> Senior Software Engineer - QA Team | WSO2 inc. >>>>>>>> Mobile : +94775464547 >>>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Nuwan Dias >>>>>>> >>>>>>> Technical Lead - WSO2, Inc. http://wso2.com >>>>>>> email : [email protected] >>>>>>> Phone : +94 777 775 729 >>>>>>> >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> Nuwan Dias >>>>> >>>>> Technical Lead - WSO2, Inc. http://wso2.com >>>>> email : [email protected] >>>>> Phone : +94 777 775 729 >>>>> >>>> >>>> >>>> >>>> -- >>>> *Regards,* >>>> >>>> *Aparna Karunarathna.* >>>> >>>> >>>> *Associate Technical Lead - QAWSO2 Inc.Mobile: 0714002533 <0714002533>* >>>> >>> >>> >>> >>> -- >>> Nuwan Dias >>> >>> Technical Lead - WSO2, Inc. http://wso2.com >>> email : [email protected] >>> Phone : +94 777 775 729 >>> >> >> >> >> -- >> *Regards,* >> >> *Aparna Karunarathna.* >> >> >> *Associate Technical Lead - QAWSO2 Inc.Mobile: 0714002533* >> > > > > -- > Nuwan Dias > > Technical Lead - WSO2, Inc. http://wso2.com > email : [email protected] > Phone : +94 777 775 729 >
_______________________________________________ Dev mailing list [email protected] http://wso2.org/cgi-bin/mailman/listinfo/dev
