On 18 August 2012 20:32, Philippe Mouawad <[email protected]> wrote: > Hello Sebb, Milamber, > > Just to be sure I understand did memory leak exist before the optimisation > on JMeterThread memory (ie creation when needed) (also to have a stable > revision id)
I don't think it's a memory leak - it's just that the algorithm perhaps uses more memory than it needs. > Reading the full discussion, i understand tests where running fine after > sebb reintegrated in ThreadGroup the OnDemandThreaGroup under the delayed > start option , did I understand well ? I think it's only recently that it has been possible to run with much larger thread count. That was not possible until the JMeterThread creation was delayed to just before Java thread creation. Before, when the JMeterThread array was created at startup, the memory would be used up before test start. > Milamber could you open a bug on nightly bug with your attached test plan ? > I am not sure to understand its content ? Thanks > > Sebb, in a later mail you talk about Object pair. > Last time i looked at that part I was wondering what it was used for ? Do > you know ? It seems to be used to prevent adding a child to a parent if the child has already been added. If the ObjectPair (child, parent) is present, then the child has been added. Otherwise, the child is added to the parent and the pair is added to the Set. The problem is, this can take a lot of memory if there are a lot of threads. Because the Set is shared (static), it cannot be cleared until all the threads end. Though I must admit I'm not entirely sure that the Set does need to be shared. It seems a bit odd that the pairings are shared between all ThreadGroups, regardless of whether they run in parallel or not. Indeed why should individual threads need to check if other threads have already seen the same child/parent pair? It seems wrong that they should be dependent on each other in this way. I looked at the SVN history and it has always been a static field. But maybe there were other features of the original design that made this necessary then. > Thanks > Regards > Philippe > On Saturday, August 18, 2012, sebb wrote: > >> On 18 August 2012 13:02, Milamber <[email protected]> wrote: >> > >> > >> > Le 15/08/2012 19:54, sebb a ecrit : >> >> >> >> [snip] >> >> >> >>>>> 105704 of sampler results (test not ended, kill by me after the OOME >> >>>>> notification) >> >>>>> >> >>>>> Max heap size during tests: ~7.8 GB >> >>>>> >> >>>>> (jmeter java opts : HEAP="-Xms8g -Xmx8g -XX:+UseConcMarkSweepGC >> >>>>> -Xloggc:/tmp/gclog_`date '+20%y%m%d%H%M%S'`.txt") >> >>>>> >> >>>>> Max number of LWP (Light-weight process): 31083 (~threads inside the >> >>>>> java >> >>>>> process) >> >>>>> >> >>>>> === >> >>>>> >> >>>>> A new test: >> >>>>> same scenario but sleep_time = 100 ms, 2 loops, with 9,600,000 >> threads >> >>>>> and >> >>>>> ramp-up 7200s, delayed start (80000 new threads by minute) >> >>>>> >> >>>>> Test have OOME after 50 min >> >>>>> ~7,715,164 of sampler results >> >>>>> >> >>>>> Max heap size during tests: end with 99 GB (after GC) (big OOME ;-)) >> >>>>> (jmeter java opts : HEAP="-Xms100g -Xmx100g -XX:NewSize=50g >> >>>>> -XX:MaxNewSize=50g -XX:PermSize=1g -XX:MaxPermSize=20g >> >>>>> -XX:+UseConcMarkSweepGC -Xloggc:/tmp/gclog_`date >> >>>>> '+20%y%m%d%H%M%S'`.txt") >> >>>>> >> >>>>> Average number of LWP (Light-weight process): up to 769 (before the >> >>>>> first >> >>>>> Full OOME) (~threads inside the java process via the linux command >> >>>>> 'ps') >> >>>> >> >>>> This is a higher concurrent number than before. It would be >> >>>> interesting to know if the test would succeed with a similar count to >> >>>> before. >> >>>> >> >>>> i.e. keep the ratio of thread-count / ramp-up the same. >> >>> >> >>> >> >>> If we keep the ratio (120,000 thread/6000 sec ramp-up), for 9,600,000 >> >>> threads, we need a ramp-up of 480,000 seconds (~5,5 days) >> >> >> >> How about running the shorter test with the same ratio instead? >> > >> > >> > Same thing. a OutOfMemroyError. >> > I've run several tests (Xmx = 100G or 8G or 512 M). Always the OOME. >> > >> > Heap Dump show a huge retained memory in o.a.j.threads.TestCompiler in >> the >> > private static final Set<ObjectPair> pairing = new HashSet<ObjectPair>(); >> > >> > Memory Analyzer : >> > The class*"org.apache.jmeter.threads.TestCompiler"*, loaded >> > by*"org.apache.jmeter.DynamicClassLoader @ 0xe00272a8"*, >> > occupies*516,943,728 (99.53%)*bytes. The memory is accumulated in one >> > instance of*"java.util.HashMap$Entry[]"*loaded by*"<system class >> loader>"*. >> > >> > *Keywords* >> > org.apache.jmeter.threads.TestCompiler >> > java.util.HashMap$Entry[] >> > org.apache.jmeter.DynamicClassLoader @ 0xe00272a8 >> > >> > ==== >> > Inside TestCompiler object (in memory analyzer), we found : >> > >> > Class Name | Shallow Heap | Retained Heap >> > --------------------------------------------------------------------- >> > pairing java.util.HashSet @ 0xe013da30| 16 | 516,943,720 >> > --------------------------------------------------------------------- >> > >> > The pairing hashset size is 221236 in my test (each element count 56 >> bytes >> > to 112 bytes) >> >> Thanks! >> >> I was thinking the pairing Set could be made an instance variable so >> it would disappear at the end of a thread. >> >> However, that might perhaps cause problems for test classes that are >> marked NoThreadClone. >> >> The ObjectPair class equals uses ==, so for per-thread child and >> parent entries, it would make no difference if the Set were an >> instance variable. >> >> However, if both the child and parent classes are shared between >> threads, then it would be possible for one thread to match another's >> ObjectPair. >> It would be necessary to use a static Set to catch this case; however >> I don't know if this case can ever occur. >> >> It might be worth trying the test with a temporary version of TestCompiler: >> - just remove the "static" qualifier, and comment out the initialise body. >> >> If that makes a big difference it would be worth trying to move most >> of the data to an instance variable. >> >> Meanwhile 'll try and do some more investigation to see if we can drop >> the static qualifier. >> >> > ==== >> > >> > To reproduce this OOME : >> > >> > A simple test : >> > >> > Thread Group with Startup delay, 960000 Threads, 48000 ramp-up, 10 loops >> > |-- 1 Java request (or HTTP request) >> > |-- 2 Java request (or HTTP request) >> > -- Simple Writer >> > >> > >> > JVM option : default Xms and Xmx (512M), and modify variable SERVER >> > (optionnal): >> > SERVER="-server -XX:+UseConcMarkSweepGC -Xloggc:/tmp/gclog_`date >> > '+20%y%m%d%H%M%S'`.txt" >> > >> > make a heap dump (just before or) when the first Full GC come. >> > jmap -F -dump:format=b,file=Jmeter.hprof <PID> >> > (or wait the heap dump from the option "-XX:+HeapDumpOnOutOfMemoryError") >> > >> > Milamber. >> > >> > >> > >> > >> > >> > > > -- > Cordialement. > Philippe Mouawad.
