[
https://issues.apache.org/jira/browse/APLO-160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13209625#comment-13209625
]
Hiram Chirino edited comment on APLO-160 at 2/16/12 7:15 PM:
-------------------------------------------------------------
Hi Lionel, thanks for sending me that heap. I just loaded it up and it looks
like it had 48,317 connection objects in memory which were retaining 6,044 M of
memory. Which comes out to about 128k per connection. Which seems right since
we use a 64k send and receive buffer per connection[1].
That 6,044 M is quickly going to make it into the tenured GC Generation since
are associated with connection buffer which remain in memory as long as the
connection is in memory. Since your -Xmx is set to 8192 M and assuming [2]
your JVM is using 1:2 young to tenured GC ratio, then that means. 2,730 M is
dedicated to the young generation and 5,462 is dedicated to the tenured
generation. So to me it's not surprising that your having GC issues.
Things that you can try to help reduce your problems, use smaller values for
the receive_buffer_size and send_buffer_size transport settings. Using a 4k
buffer should let you support 16x more connections. I'm not sure what the
bottom limit is on this configuration value.
Increase your -Xmx setting. Increase the new generation ratio. For example
-XX:NewRatio=3 (If you increase this too much transient data will go into the
old gen and you will end up doing more old gen collections which are more
expensive).
[1]:
http://activemq.apache.org/apollo/documentation/user-manual.html#TCP_Transports
[2]:
http://www.oracle.com/technetwork/java/gc-tuning-5-138395.html#0.0.0.%20The%20Young%20Generation%7coutline
was (Author: chirino):
Hi Lionel, thanks for sending me that heap. I just loaded it up and it
looks like it had 48,317 connection objects in memory which were retaining
6,044 M of memory. Which comes out to about 128k per connection. Which seems
right since we use a 64k send and receive buffer per connection[1].
That 6,044 M is quickly going to make it into the tenured GC Generation since
are associated with connection buffer which remain in memory as long as the
connection is in memory. Since your -Xmx is set to 8192 M and assuming [2]
your JVM is using 1:2 young to tenured GC ratio, then that means. 2,730 M is
dedicated to the young generation and 5,462 is dedicated to the tenured
generation. So to me it's not surprising that your having GC issues.
Things that you can try to help reduce your problems, use smaller values for
the receive_buffer_size and send_buffer_size transport settings. Using a 4k
buffer should let you support 16x more connections. I'm not sure what the
bottom limit is on this configuration value.
Increase your -Xmx setting. Increase the new generation ratio. For example
-XX:NewRatio=3 (If you increase this too much transient data will go into the
old gen and you will end up doing more old gen collections which are more
expensive).
[1]:
http://activemq.apache.org/apollo/documentation/user-manual.html#TCP_Transports
[2]:
http://www.oracle.com/technetwork/java/gc-tuning-5-138395.html#0.0.0.%20The%20Young%20Generation|outline
> Apollo becoming unresponsive under stress test
> ----------------------------------------------
>
> Key: APLO-160
> URL: https://issues.apache.org/jira/browse/APLO-160
> Project: ActiveMQ Apollo
> Issue Type: Bug
> Environment: apollo-1.1-20120209.032648-24
> Reporter: Lionel Cons
> Attachments: apollo.dump
>
>
> While running a stress test against apollo-1.1-20120209.032648-24 (many
> concurrent TCP connections), the broker became unresponsive.
> It logged several times: java.lang.OutOfMemoryError: GC overhead limit
> exceeded
> It also logged other warnings, probably related:
> 2012-02-14 14:14:49,273 | WARN | handle failed | org.eclipse.jetty.io.nio |
> Apollo Task
> 2012-02-14 14:18:39,073 | WARN | Problem scavenging sessions |
> org.eclipse.jetty.server.session | HashSessionScavenger-0
> It could not be stopped either, I had to kill -9 it.
> What can be done to avoid these problems?
> FWIW, java has been started with -server -Xmx8192m -Xms4096m
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
