On 02/13/2014 04:31 PM, jlindwall wrote:
I am stress testing activemq 5.9.0 by sending a flood of non-persistent
messages to a topic using jmeter.
My producers are 50 threads attempting to deliver 1000 msgs/sec; each of
size 1K. There is merely a 1 millisecond delay in between message sends.
There are 100 consumers on the topic. It runs fine for 10-20 seconds or so,
then I see the consumer stop working and "GC overhead limit exceeded" errors
appear in the consumer's console (Weirdly, monitoring the gc via visualvm
it does not seem like a massive gc event...). Looking at the heap dump from
the consumer client I see a large number of ActiveMQObjectMessage instances,
owned by the GC root ActiveMDSessionExecutor.messageQueue().
Is the consumer message dispatching simply falling badly behind incoming
message flow? Can I tune the consumer somehow to compensate? Or is this
just the expected failure mode for this scenario?
Thanks for any insights.
Sample output from the consumers:
Created the tree successfully using JMS_Subscribers_Forever.jmx
Starting the test @ Thu Feb 13 11:33:51 PST 2014 (1392320031599)
Waiting for possible shutdown message on port 4445
summary + 0 in 5s = 0.0/s Avg: 0 Min: 5001 Max: 5001 Err:
1 (0.00%) Active: 100 Started: 100 Finished: 0
summary + 0 in 35s = 0.0/s Avg: 0 Min: 4999 Max: 5006 Err:
600 (0.00%) Active: 100 Started: 100 Finished: 0
java.lang.OutOfMemoryError: GC overhead limit exceeded
Dumping heap to java_pid3941.hprof ...
Heap dump file created [654947324 bytes in 8.677 secs]
Exception: java.lang.OutOfMemoryError thrown from the
UncaughtExceptionHandler in thread "ActiveMQ Session Task-1"
Uncaught Exception java.lang.OutOfMemoryError: GC overhead limit exceeded.
See log file for details.
Logging Error: Unknown error writing event.
Uncaught Exception java.lang.OutOfMemoryError: GC overhead limit exceeded.
See log file for details.
java.lang.OutOfMemoryError: GC overhead limit exceeded
Uncaught Exception java.lang.OutOfMemoryError: GC overhead limit exceeded.
See log file for details.
Uncaught Exception java.lang.OutOfMemoryError: GC overhead limit exceeded.
See log file for details.
Logging Error: Unknown error writing event.
java.lang.OutOfMemoryError: GC overhead limit exceeded
Logging Error: Unknown error writing event.
java.lang.OutOfMemoryError: GC overhead limit exceeded
Uncaught Exception java.lang.OutOfMemoryError: GC overhead limit exceeded.
See log file for details.
Exception: java.lang.OutOfMemoryError thrown from the
UncaughtExceptionHandler in thread "ActiveMQ Transport:
tcp://server/172.55.33.120:61616@41330"
Uncaught Exception java.lang.OutOfMemoryError: GC overhead limit exceeded.
See log file for details.
Uncaught Exception java.lang.OutOfMemoryError: GC overhead limit exceeded.
See log file for details.
Uncaught Exception java.lang.OutOfMemoryError: GC overhead limit exceeded.
See log file for details.
Uncaught Exception java.lang.OutOfMemoryError: GC overhead limit exceeded.
See log file for details.
Uncaught Exception java.lang.OutOfMemoryError: GC overhead limit exceeded.
See log file for details.
Uncaught Exception java.lang.OutOfMemoryError: GC overhead limit exceeded.
See log file for details.
Uncaught Exception java.lang.OutOfMemoryError: GC overhead limit exceeded.
See log file for details.
--
View this message in context:
http://activemq.2283324.n4.nabble.com/Stress-testing-OOME-tp4677959.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
The topic prefetch limit by default is quite large, you could set this
to a lower value to control how many messages pile in there.
--
Tim Bish
Sr Software Engineer | RedHat Inc.
[email protected] | www.fusesource.com | www.redhat.com
skype: tabish121 | twitter: @tabish121
blog: http://timbish.blogspot.com/
