Le 15/08/2012 19:54, sebb a ecrit :
[snip]
105704 of sampler results (test not ended, kill by me after the OOME
notification)
Max heap size during tests: ~7.8 GB
(jmeter java opts : HEAP="-Xms8g -Xmx8g -XX:+UseConcMarkSweepGC
-Xloggc:/tmp/gclog_`date '+20%y%m%d%H%M%S'`.txt")
Max number of LWP (Light-weight process): 31083 (~threads inside the
java
process)
===
A new test:
same scenario but sleep_time = 100 ms, 2 loops, with 9,600,000 threads
and
ramp-up 7200s, delayed start (80000 new threads by minute)
Test have OOME after 50 min
~7,715,164 of sampler results
Max heap size during tests: end with 99 GB (after GC) (big OOME ;-))
(jmeter java opts : HEAP="-Xms100g -Xmx100g -XX:NewSize=50g
-XX:MaxNewSize=50g -XX:PermSize=1g -XX:MaxPermSize=20g
-XX:+UseConcMarkSweepGC -Xloggc:/tmp/gclog_`date '+20%y%m%d%H%M%S'`.txt")
Average number of LWP (Light-weight process): up to 769 (before the first
Full OOME) (~threads inside the java process via the linux command 'ps')
This is a higher concurrent number than before. It would be
interesting to know if the test would succeed with a similar count to
before.
i.e. keep the ratio of thread-count / ramp-up the same.
If we keep the ratio (120,000 thread/6000 sec ramp-up), for 9,600,000
threads, we need a ramp-up of 480,000 seconds (~5,5 days)
How about running the shorter test with the same ratio instead?
Same thing. a OutOfMemroyError.
I've run several tests (Xmx = 100G or 8G or 512 M). Always the OOME.
Heap Dump show a huge retained memory in o.a.j.threads.TestCompiler in the
private static final Set<ObjectPair> pairing = new HashSet<ObjectPair>();
Memory Analyzer :
The class*"org.apache.jmeter.threads.TestCompiler"*, loaded
by*"org.apache.jmeter.DynamicClassLoader @ 0xe00272a8"*,
occupies*516,943,728 (99.53%)*bytes. The memory is accumulated in one
instance of*"java.util.HashMap$Entry[]"*loaded by*"<system class loader>"*.
*Keywords*
org.apache.jmeter.threads.TestCompiler
java.util.HashMap$Entry[]
org.apache.jmeter.DynamicClassLoader @ 0xe00272a8
====
Inside TestCompiler object (in memory analyzer), we found :
Class Name | Shallow Heap | Retained Heap
---------------------------------------------------------------------
pairing java.util.HashSet @ 0xe013da30| 16 | 516,943,720
---------------------------------------------------------------------
The pairing hashset size is 221236 in my test (each element count 56
bytes to 112 bytes)
====
To reproduce this OOME :
A simple test :
Thread Group with Startup delay, 960000 Threads, 48000 ramp-up, 10 loops
|-- 1 Java request (or HTTP request)
|-- 2 Java request (or HTTP request)
-- Simple Writer
JVM option : default Xms and Xmx (512M), and modify variable SERVER
(optionnal):
SERVER="-server -XX:+UseConcMarkSweepGC -Xloggc:/tmp/gclog_`date
'+20%y%m%d%H%M%S'`.txt"
make a heap dump (just before or) when the first Full GC come.
jmap -F -dump:format=b,file=Jmeter.hprof <PID>
(or wait the heap dump from the option "-XX:+HeapDumpOnOutOfMemoryError")
Milamber.
