Great !!!! Thanks sebb for your last days work and for finding this memory issue, JMeter is on a highway to heaven :-)
I really think we should in next release notes emphasize the following improvements: - Delayed Thread creation - New Response Time Graph - Full IpV6 support thanks to new Cookie Manager - And all the technical memory and performance imrpovements (enhance the existing wiki page ?) Regards Philippe On Fri, Aug 31, 2012 at 10:46 PM, Milamber <[email protected]> wrote: > > > Le 18/08/2012 15:12, sebb a ecrit : > > On 18 August 2012 13:02, Milamber<[email protected]> wrote: >> >>> >>> Le 15/08/2012 19:54, sebb a ecrit : >>> >>>> [snip] >>>> >>>> 105704 of sampler results (test not ended, kill by me after the OOME >>>>>>> notification) >>>>>>> >>>>>>> Max heap size during tests: ~7.8 GB >>>>>>> >>>>>>> (jmeter java opts : HEAP="-Xms8g -Xmx8g -XX:+UseConcMarkSweepGC >>>>>>> -Xloggc:/tmp/gclog_`date '+20%y%m%d%H%M%S'`.txt") >>>>>>> >>>>>>> Max number of LWP (Light-weight process): 31083 (~threads inside the >>>>>>> java >>>>>>> process) >>>>>>> >>>>>>> === >>>>>>> >>>>>>> A new test: >>>>>>> same scenario but sleep_time = 100 ms, 2 loops, with 9,600,000 >>>>>>> threads >>>>>>> and >>>>>>> ramp-up 7200s, delayed start (80000 new threads by minute) >>>>>>> >>>>>>> Test have OOME after 50 min >>>>>>> ~7,715,164 of sampler results >>>>>>> >>>>>>> Max heap size during tests: end with 99 GB (after GC) (big OOME ;-)) >>>>>>> (jmeter java opts : HEAP="-Xms100g -Xmx100g -XX:NewSize=50g >>>>>>> -XX:MaxNewSize=50g -XX:PermSize=1g -XX:MaxPermSize=20g >>>>>>> -XX:+UseConcMarkSweepGC -Xloggc:/tmp/gclog_`date >>>>>>> '+20%y%m%d%H%M%S'`.txt") >>>>>>> >>>>>>> Average number of LWP (Light-weight process): up to 769 (before the >>>>>>> first >>>>>>> Full OOME) (~threads inside the java process via the linux command >>>>>>> 'ps') >>>>>>> >>>>>> This is a higher concurrent number than before. It would be >>>>>> interesting to know if the test would succeed with a similar count to >>>>>> before. >>>>>> >>>>>> i.e. keep the ratio of thread-count / ramp-up the same. >>>>>> >>>>> >>>>> If we keep the ratio (120,000 thread/6000 sec ramp-up), for 9,600,000 >>>>> threads, we need a ramp-up of 480,000 seconds (~5,5 days) >>>>> >>>> How about running the shorter test with the same ratio instead? >>>> >>> >>> Same thing. a OutOfMemroyError. >>> I've run several tests (Xmx = 100G or 8G or 512 M). Always the OOME. >>> >>> Heap Dump show a huge retained memory in o.a.j.threads.TestCompiler in >>> the >>> private static final Set<ObjectPair> pairing = new >>> HashSet<ObjectPair>(); >>> >>> Memory Analyzer : >>> The class*"org.apache.jmeter.**threads.TestCompiler"*, loaded >>> by*"org.apache.jmeter.**DynamicClassLoader @ 0xe00272a8"*, >>> occupies*516,943,728 (99.53%)*bytes. The memory is accumulated in one >>> instance of*"java.util.HashMap$Entry[]"***loaded by*"<system class >>> loader>"*. >>> >>> *Keywords* >>> org.apache.jmeter.threads.**TestCompiler >>> java.util.HashMap$Entry[] >>> org.apache.jmeter.**DynamicClassLoader @ 0xe00272a8 >>> >>> ==== >>> Inside TestCompiler object (in memory analyzer), we found : >>> >>> Class Name | Shallow Heap | Retained Heap >>> ------------------------------**------------------------------** >>> --------- >>> pairing java.util.HashSet @ 0xe013da30| 16 | 516,943,720 >>> ------------------------------**------------------------------** >>> --------- >>> >>> The pairing hashset size is 221236 in my test (each element count 56 >>> bytes >>> to 112 bytes) >>> >> Thanks! >> >> I was thinking the pairing Set could be made an instance variable so >> it would disappear at the end of a thread. >> >> However, that might perhaps cause problems for test classes that are >> marked NoThreadClone. >> >> The ObjectPair class equals uses ==, so for per-thread child and >> parent entries, it would make no difference if the Set were an >> instance variable. >> >> However, if both the child and parent classes are shared between >> threads, then it would be possible for one thread to match another's >> ObjectPair. >> It would be necessary to use a static Set to catch this case; however >> I don't know if this case can ever occur. >> >> It might be worth trying the test with a temporary version of >> TestCompiler: >> - just remove the "static" qualifier, and comment out the initialise >> body. >> >> If that makes a big difference it would be worth trying to move most >> of the data to an instance variable. >> >> Meanwhile 'll try and do some more investigation to see if we can drop >> the static qualifier. >> > > Good news: > > > Since your commit r1378780 (bug 53796), I've can run a very large test > with a lot of threads successfully! > > JMeter trunk, without JVM parameters changes (i.e. Xmx=512MB) except a log > gc option > > Number of threads : 960,000 > Ramp up : 48,000 > Loop : 10 > Delay thread creation enable > > Scenario: 2 Java requests with 1 sec sleep time > > Results : 19,200,004 lines in csv files (size 2,3 GB) > > GC never reach 135MB (used) (arround 28 MB after a GC). > > > > Possible to display a the new response time graph, with a Xmx=2G (need 1,4 > GB) > http://www.milamberspace.net/**jmeter/resptime1.png<http://www.milamberspace.net/jmeter/resptime1.png> > (great stability) > > > Milamber > > > >> ==== >>> >>> To reproduce this OOME : >>> >>> A simple test : >>> >>> Thread Group with Startup delay, 960000 Threads, 48000 ramp-up, 10 loops >>> |-- 1 Java request (or HTTP request) >>> |-- 2 Java request (or HTTP request) >>> -- Simple Writer >>> >>> >>> JVM option : default Xms and Xmx (512M), and modify variable SERVER >>> (optionnal): >>> SERVER="-server -XX:+UseConcMarkSweepGC -Xloggc:/tmp/gclog_`date >>> '+20%y%m%d%H%M%S'`.txt" >>> >>> make a heap dump (just before or) when the first Full GC come. >>> jmap -F -dump:format=b,file=Jmeter.**hprof<PID> >>> (or wait the heap dump from the option "-XX:+** >>> HeapDumpOnOutOfMemoryError") >>> >>> Milamber. >>> >>> >>> >>> >>> >>> > -- Cordialement. Philippe Mouawad.
