Hello Sebb, Milamber, Just to be sure I understand did memory leak exist before the optimisation on JMeterThread memory (ie creation when needed) (also to have a stable revision id) Reading the full discussion, i understand tests where running fine after sebb reintegrated in ThreadGroup the OnDemandThreaGroup under the delayed start option , did I understand well ? Milamber could you open a bug on nightly bug with your attached test plan ? I am not sure to understand its content ? Thanks
Sebb, in a later mail you talk about Object pair. Last time i looked at that part I was wondering what it was used for ? Do you know ? Thanks Regards Philippe On Saturday, August 18, 2012, sebb wrote: > On 18 August 2012 13:02, Milamber <[email protected]> wrote: > > > > > > Le 15/08/2012 19:54, sebb a ecrit : > >> > >> [snip] > >> > >>>>> 105704 of sampler results (test not ended, kill by me after the OOME > >>>>> notification) > >>>>> > >>>>> Max heap size during tests: ~7.8 GB > >>>>> > >>>>> (jmeter java opts : HEAP="-Xms8g -Xmx8g -XX:+UseConcMarkSweepGC > >>>>> -Xloggc:/tmp/gclog_`date '+20%y%m%d%H%M%S'`.txt") > >>>>> > >>>>> Max number of LWP (Light-weight process): 31083 (~threads inside the > >>>>> java > >>>>> process) > >>>>> > >>>>> === > >>>>> > >>>>> A new test: > >>>>> same scenario but sleep_time = 100 ms, 2 loops, with 9,600,000 > threads > >>>>> and > >>>>> ramp-up 7200s, delayed start (80000 new threads by minute) > >>>>> > >>>>> Test have OOME after 50 min > >>>>> ~7,715,164 of sampler results > >>>>> > >>>>> Max heap size during tests: end with 99 GB (after GC) (big OOME ;-)) > >>>>> (jmeter java opts : HEAP="-Xms100g -Xmx100g -XX:NewSize=50g > >>>>> -XX:MaxNewSize=50g -XX:PermSize=1g -XX:MaxPermSize=20g > >>>>> -XX:+UseConcMarkSweepGC -Xloggc:/tmp/gclog_`date > >>>>> '+20%y%m%d%H%M%S'`.txt") > >>>>> > >>>>> Average number of LWP (Light-weight process): up to 769 (before the > >>>>> first > >>>>> Full OOME) (~threads inside the java process via the linux command > >>>>> 'ps') > >>>> > >>>> This is a higher concurrent number than before. It would be > >>>> interesting to know if the test would succeed with a similar count to > >>>> before. > >>>> > >>>> i.e. keep the ratio of thread-count / ramp-up the same. > >>> > >>> > >>> If we keep the ratio (120,000 thread/6000 sec ramp-up), for 9,600,000 > >>> threads, we need a ramp-up of 480,000 seconds (~5,5 days) > >> > >> How about running the shorter test with the same ratio instead? > > > > > > Same thing. a OutOfMemroyError. > > I've run several tests (Xmx = 100G or 8G or 512 M). Always the OOME. > > > > Heap Dump show a huge retained memory in o.a.j.threads.TestCompiler in > the > > private static final Set<ObjectPair> pairing = new HashSet<ObjectPair>(); > > > > Memory Analyzer : > > The class*"org.apache.jmeter.threads.TestCompiler"*, loaded > > by*"org.apache.jmeter.DynamicClassLoader @ 0xe00272a8"*, > > occupies*516,943,728 (99.53%)*bytes. The memory is accumulated in one > > instance of*"java.util.HashMap$Entry[]"*loaded by*"<system class > loader>"*. > > > > *Keywords* > > org.apache.jmeter.threads.TestCompiler > > java.util.HashMap$Entry[] > > org.apache.jmeter.DynamicClassLoader @ 0xe00272a8 > > > > ==== > > Inside TestCompiler object (in memory analyzer), we found : > > > > Class Name | Shallow Heap | Retained Heap > > --------------------------------------------------------------------- > > pairing java.util.HashSet @ 0xe013da30| 16 | 516,943,720 > > --------------------------------------------------------------------- > > > > The pairing hashset size is 221236 in my test (each element count 56 > bytes > > to 112 bytes) > > Thanks! > > I was thinking the pairing Set could be made an instance variable so > it would disappear at the end of a thread. > > However, that might perhaps cause problems for test classes that are > marked NoThreadClone. > > The ObjectPair class equals uses ==, so for per-thread child and > parent entries, it would make no difference if the Set were an > instance variable. > > However, if both the child and parent classes are shared between > threads, then it would be possible for one thread to match another's > ObjectPair. > It would be necessary to use a static Set to catch this case; however > I don't know if this case can ever occur. > > It might be worth trying the test with a temporary version of TestCompiler: > - just remove the "static" qualifier, and comment out the initialise body. > > If that makes a big difference it would be worth trying to move most > of the data to an instance variable. > > Meanwhile 'll try and do some more investigation to see if we can drop > the static qualifier. > > > ==== > > > > To reproduce this OOME : > > > > A simple test : > > > > Thread Group with Startup delay, 960000 Threads, 48000 ramp-up, 10 loops > > |-- 1 Java request (or HTTP request) > > |-- 2 Java request (or HTTP request) > > -- Simple Writer > > > > > > JVM option : default Xms and Xmx (512M), and modify variable SERVER > > (optionnal): > > SERVER="-server -XX:+UseConcMarkSweepGC -Xloggc:/tmp/gclog_`date > > '+20%y%m%d%H%M%S'`.txt" > > > > make a heap dump (just before or) when the first Full GC come. > > jmap -F -dump:format=b,file=Jmeter.hprof <PID> > > (or wait the heap dump from the option "-XX:+HeapDumpOnOutOfMemoryError") > > > > Milamber. > > > > > > > > > > > -- Cordialement. Philippe Mouawad.
