On 18 August 2012 13:02, Milamber <[email protected]> wrote: > > > Le 15/08/2012 19:54, sebb a ecrit : >> >> [snip] >> >>>>> 105704 of sampler results (test not ended, kill by me after the OOME >>>>> notification) >>>>> >>>>> Max heap size during tests: ~7.8 GB >>>>> >>>>> (jmeter java opts : HEAP="-Xms8g -Xmx8g -XX:+UseConcMarkSweepGC >>>>> -Xloggc:/tmp/gclog_`date '+20%y%m%d%H%M%S'`.txt") >>>>> >>>>> Max number of LWP (Light-weight process): 31083 (~threads inside the >>>>> java >>>>> process) >>>>> >>>>> === >>>>> >>>>> A new test: >>>>> same scenario but sleep_time = 100 ms, 2 loops, with 9,600,000 threads >>>>> and >>>>> ramp-up 7200s, delayed start (80000 new threads by minute) >>>>> >>>>> Test have OOME after 50 min >>>>> ~7,715,164 of sampler results >>>>> >>>>> Max heap size during tests: end with 99 GB (after GC) (big OOME ;-)) >>>>> (jmeter java opts : HEAP="-Xms100g -Xmx100g -XX:NewSize=50g >>>>> -XX:MaxNewSize=50g -XX:PermSize=1g -XX:MaxPermSize=20g >>>>> -XX:+UseConcMarkSweepGC -Xloggc:/tmp/gclog_`date >>>>> '+20%y%m%d%H%M%S'`.txt") >>>>> >>>>> Average number of LWP (Light-weight process): up to 769 (before the >>>>> first >>>>> Full OOME) (~threads inside the java process via the linux command >>>>> 'ps') >>>> >>>> This is a higher concurrent number than before. It would be >>>> interesting to know if the test would succeed with a similar count to >>>> before. >>>> >>>> i.e. keep the ratio of thread-count / ramp-up the same. >>> >>> >>> If we keep the ratio (120,000 thread/6000 sec ramp-up), for 9,600,000 >>> threads, we need a ramp-up of 480,000 seconds (~5,5 days) >> >> How about running the shorter test with the same ratio instead? > > > Same thing. a OutOfMemroyError. > I've run several tests (Xmx = 100G or 8G or 512 M). Always the OOME. > > Heap Dump show a huge retained memory in o.a.j.threads.TestCompiler in the > private static final Set<ObjectPair> pairing = new HashSet<ObjectPair>(); > > Memory Analyzer : > The class*"org.apache.jmeter.threads.TestCompiler"*, loaded > by*"org.apache.jmeter.DynamicClassLoader @ 0xe00272a8"*, > occupies*516,943,728 (99.53%)*bytes. The memory is accumulated in one > instance of*"java.util.HashMap$Entry[]"*loaded by*"<system class loader>"*. > > *Keywords* > org.apache.jmeter.threads.TestCompiler > java.util.HashMap$Entry[] > org.apache.jmeter.DynamicClassLoader @ 0xe00272a8 > > ==== > Inside TestCompiler object (in memory analyzer), we found : > > Class Name | Shallow Heap | Retained Heap > --------------------------------------------------------------------- > pairing java.util.HashSet @ 0xe013da30| 16 | 516,943,720 > --------------------------------------------------------------------- > > The pairing hashset size is 221236 in my test (each element count 56 bytes > to 112 bytes)
Thanks! I was thinking the pairing Set could be made an instance variable so it would disappear at the end of a thread. However, that might perhaps cause problems for test classes that are marked NoThreadClone. The ObjectPair class equals uses ==, so for per-thread child and parent entries, it would make no difference if the Set were an instance variable. However, if both the child and parent classes are shared between threads, then it would be possible for one thread to match another's ObjectPair. It would be necessary to use a static Set to catch this case; however I don't know if this case can ever occur. It might be worth trying the test with a temporary version of TestCompiler: - just remove the "static" qualifier, and comment out the initialise body. If that makes a big difference it would be worth trying to move most of the data to an instance variable. Meanwhile 'll try and do some more investigation to see if we can drop the static qualifier. > ==== > > To reproduce this OOME : > > A simple test : > > Thread Group with Startup delay, 960000 Threads, 48000 ramp-up, 10 loops > |-- 1 Java request (or HTTP request) > |-- 2 Java request (or HTTP request) > -- Simple Writer > > > JVM option : default Xms and Xmx (512M), and modify variable SERVER > (optionnal): > SERVER="-server -XX:+UseConcMarkSweepGC -Xloggc:/tmp/gclog_`date > '+20%y%m%d%H%M%S'`.txt" > > make a heap dump (just before or) when the first Full GC come. > jmap -F -dump:format=b,file=Jmeter.hprof <PID> > (or wait the heap dump from the option "-XX:+HeapDumpOnOutOfMemoryError") > > Milamber. > > > > >
