On 15 August 2012 19:40, Milamber <[email protected]> wrote: > > > Le 15/08/2012 09:34, sebb a ecrit : > >> On 15 August 2012 08:05, Milamber<[email protected]> wrote: >>> >>> >>> Le 14/08/2012 21:10, sebb a ecrit : >>> >>>> On 14 August 2012 11:32, sebb<[email protected]> wrote: >>>>> >>>>> On 14 August 2012 10:51, Milamber<[email protected]> wrote: >>>>>> >>>>>> >>>>>> Le 13/08/2012 18:58, sebb a ecrit : >>>>>> >>>>>>> On 13 August 2012 17:07, sebb<[email protected]> wrote: >>>>>>>> >>>>>>>> I've made the changes to merge the OnDemand code back into the >>>>>>>> standard ThreadGroup. >>>>>>>> >>>>>>>> This means it's now trivial for users to switch between the >>>>>>>> different >>>>>>>> startup types. >>>>>>>> >>>>>>>> I hope I've not broken anything ... >>>>>>> >>>>>>> Seems to work OK with a simple test plan of a single Java sampler, 10 >>>>>>> loops. >>>>>>> Setting the ramp-up to the same as the thread count gives me a max of >>>>>>> 3 threads concurrently. >>>>>>> >>>>>>> This runs out of memory when run without delayed start and 10000 >>>>>>> threads. >>>>>>> The same test starts fine and runs for a while with delayed start. >>>>>> >>>>>> >>>>>> I've run a simple test of 2 Java sampler (with 1s sleep time), 10 >>>>>> loops, >>>>>> with 120000 threads and ramp-up 6000s with jmeter-trunk (on Linux 64 >>>>>> bits >>>>>> server with 48 cores, sun jdk 6 64bits) >>>>>> >>>>>> === >>>>>> With delayed start, test works fine. >>>>>> 2400004 of sampler results >>>>>> >>>>>> Max heap size during tests: ~2.6 GB >>>>>> (jmeter java opts : HEAP="-Xms8g -Xmx8g -XX:+UseConcMarkSweepGC >>>>>> -Xloggc:/tmp/gclog_`date '+20%y%m%d%H%M%S'`.txt") >>>>>> >>>>>> Average number of LWP (Light-weight process): ~500 (~threads inside >>>>>> the >>>>>> java process via the linux command 'ps') >>>>>> >>>>>> === >>>>>> Without delayed start: OutOfMemoryError: unable to create new native >>>>>> thread. >>>>>> 45176 of sampler results (test not ended, kill by me after the OOME >>>>>> notification) >>>>>> >>>>>> Max heap size during tests: ~2.3 GB >>>>>> (jmeter java opts : HEAP="-Xms8g -Xmx8g -XX:+UseConcMarkSweepGC >>>>>> -Xloggc:/tmp/gclog_`date '+20%y%m%d%H%M%S'`.txt") >>>>>> >>>>>> Max number of LWP (Light-weight process): 30396 (~threads inside the >>>>>> java >>>>>> process) >>>>>> >>>>>> === >>>>>> >>>>>> Conclusion IMHO: A great behavior for JMeter and tests with a huge >>>>>> number of >>>>>> threads (users) with small number of individual loops per user, like >>>>>> webservice's test >>>>> >>>>> Thanks! >>>>> >>>>> I'm currently working on delaying the creation of JMeterThread >>>>> instances. >>>>> These take quite a bit of memory, and they aren't needed until just >>>>> before the thread starts. >>>>> >>>>> This means moving code from StandardJMeterEngine to ThreadGroup. >>>>> Quite a lot of changes, but it's looking very promising - many more >>>>> threads can be started, as memory is then only taken for active >>>>> threads. >>>>> >>>>> I should be able to commit the code later today or tomorrow. >>>> >>>> Committed. Seems to be working OK, though I've not tested multiple >>>> thread groups yet. >>>> >>>> When using delayed start, the thread count should now be limited only >>>> by the number of concurrent threads. >>>> I was able to start a test with 10 million threads (though I did not >>>> let it run for very long). >>>> >>>> Assuming that the thread-related data is properly released when the >>>> thread finishes, there's no reason why such a test should not >>>> complete. >>> >>> >>> >>> I've run again the simple test of 2 Java sampler (with 1s sleep time), 10 >>> loops, with 120000 threads and ramp-up 6000s with jmeter-trunk (on Linux >>> 64 >>> bits server with 48 cores, sun jdk 6 64bits) >>> >>> >>> === >>> With delayed start, test works fine. >>> 2400004 of sampler results >>> >>> Max heap size during tests: end with 2.3 GB (without any Full GC) >>> >>> (jmeter java opts : HEAP="-Xms8g -Xmx8g -XX:+UseConcMarkSweepGC >>> -Xloggc:/tmp/gclog_`date '+20%y%m%d%H%M%S'`.txt") >>> >>> Average number of LWP (Light-weight process): ~500 (~threads inside the >>> java process via the linux command 'ps') >>> >>> === >>> Without delayed start: OutOfMemoryError: unable to create new native >>> thread. >> >> That's to be expected. >> >>> 105704 of sampler results (test not ended, kill by me after the OOME >>> notification) >>> >>> Max heap size during tests: ~7.8 GB >>> >>> (jmeter java opts : HEAP="-Xms8g -Xmx8g -XX:+UseConcMarkSweepGC >>> -Xloggc:/tmp/gclog_`date '+20%y%m%d%H%M%S'`.txt") >>> >>> Max number of LWP (Light-weight process): 31083 (~threads inside the >>> java >>> process) >>> >>> === >>> >>> A new test: >>> same scenario but sleep_time = 100 ms, 2 loops, with 9,600,000 threads >>> and >>> ramp-up 7200s, delayed start (80000 new threads by minute) >>> >>> Test have OOME after 50 min >>> ~7,715,164 of sampler results >>> >>> Max heap size during tests: end with 99 GB (after GC) (big OOME ;-)) >>> (jmeter java opts : HEAP="-Xms100g -Xmx100g -XX:NewSize=50g >>> -XX:MaxNewSize=50g -XX:PermSize=1g -XX:MaxPermSize=20g >>> -XX:+UseConcMarkSweepGC -Xloggc:/tmp/gclog_`date '+20%y%m%d%H%M%S'`.txt") >>> >>> Average number of LWP (Light-weight process): up to 769 (before the first >>> Full OOME) (~threads inside the java process via the linux command 'ps') >> >> This is a higher concurrent number than before. It would be >> interesting to know if the test would succeed with a similar count to >> before. >> >> i.e. keep the ratio of thread-count / ramp-up the same. > > > If we keep the ratio (120,000 thread/6000 sec ramp-up), for 9,600,000 > threads, we need a ramp-up of 480,000 seconds (~5,5 days)
How about running the shorter test with the same ratio instead? > I can make a test with this ratio only for 800 minute (~13,3 hours) and > 960,000 threads. I think I must keep the first JVM config (heap at 8GB). > I will send results asap. > > > >> >> That should show if there are thread resources that are not being >> cleaned up when the thread finishes. > > > If the OOME occurs, I will try to get a heap dump. > > > > >> >>> Max TG number before the 1st OOME : Thread started: Thread Group >>> 1-1875217 >>> >>> === >>> Milamber >>> >>> >>> >>> >>> >>>>>> Milamber >>>>>> >>>>>> >>>>>>> It's much quicker to stop the test as well. >>>>>>> >>>>>>> I was able to start a test with 50000 threads, but after that memory >>>>>>> was a problem. >>>>>>> Also, the test takes quite a long time to start up, presumably >>>>>>> because >>>>>>> of the array of JMeterThread instances. >>>>>>> >>>>>>>> There's probably still some tidying up that could be done. >>>>>>>> >>>>>>>> For example, StandardJMeterEngine still creates the JMeterThread >>>>>>>> instances at the start. It might be better to make the thread group >>>>>>>> class responsible for handling these, so the additional objects are >>>>>>>> not created until needed. >>>>>>> >>>>>>> I think that's what broke my tests with more than about 60000 >>>>>>> threads. >>>>>>> >>>>>>> As well as deferring creation of these objects,they need to be >>>>>>> released once the thread has completed. >>>>>>> >>>>>>>> BTW, my French is not up to translating the delayed_start property >>>>>>>> entry - please can someone else fix that so the tests no longer >>>>>>>> fail? >>>>>>>> Thanks! >>>>>> >>>>>> >
