Markus Leichter created AMQ-6976:
------------------------------------
Summary: GC goes crazy under load. Heap is filled up with
TransportConnectionState objects
Key: AMQ-6976
URL: https://issues.apache.org/jira/browse/AMQ-6976
Project: ActiveMQ
Issue Type: Bug
Components: Broker
Affects Versions: 5.15.3
Environment: wrapper.conf
{{# Java Additional Parameters}}
{{# note that n is the parameter number starting from 1.}}
{{wrapper.java.additional.1=-Dactivemq.home="%ACTIVEMQ_HOME%"}}
{{wrapper.java.additional.2=-Dactivemq.base="%ACTIVEMQ_BASE%"}}
{{wrapper.java.additional.3=-Djavax.net.ssl.keyStorePassword=password}}
{{wrapper.java.additional.4=-Djavax.net.ssl.trustStorePassword=password}}
{{wrapper.java.additional.5=-Djavax.net.ssl.keyStore="%ACTIVEMQ_CONF%/broker.ks"}}
{{wrapper.java.additional.6=-Djavax.net.ssl.trustStore="%ACTIVEMQ_CONF%/broker.ts"}}
{{wrapper.java.additional.7=-Dcom.sun.management.jmxremote}}
{{wrapper.java.additional.8=-Dorg.apache.activemq.UseDedicatedTaskRunner=false}}
{{wrapper.java.additional.9=-Djava.util.logging.config.file=logging.properties}}
{{wrapper.java.additional.10=-Dactivemq.conf="%ACTIVEMQ_CONF%"}}
{{wrapper.java.additional.11=-Dactivemq.data="%ACTIVEMQ_DATA%"}}
{{wrapper.java.additional.12=-Djava.security.auth.login.config="%ACTIVEMQ_CONF%/login.config"}}
{{wrapper.java.additional.13=-Dorg.apache.activemq.SERIALIZABLE_PACKAGES="*"}}
{{wrapper.java.additional.14=-XX:+CMSParallelRemarkEnabled}}
{{wrapper.java.additional.15=-XX:+UseConcMarkSweepGC}}
{{wrapper.java.additional.16=-XX:+CMSPermGenSweepingEnabled}}
{{wrapper.java.additional.17=-XX:+CMSClassUnloadingEnabled}}
{{wrapper.java.additional.18=-XX:+DisableExplicitGC}}
{{wrapper.java.additional.19=-XX:+UseParNewGC}}
{{wrapper.java.additional.20=-XX:+PrintGC}}
{{wrapper.java.additional.21=-XX:+PrintGCDetails}}
{{wrapper.java.additional.22=-XX:+PrintGCDateStamps}}
{{wrapper.java.additional.23=-XX:+PrintGCCause}}
{{wrapper.java.additional.24=-XX:+PrintPromotionFailure}}
{{wrapper.java.additional.25=-XX:PrintFLSStatistics=1}}
{{wrapper.java.additional.26=-XX:+UseGCLogFileRotation}}
{{wrapper.java.additional.27=-Xloggc:F:/Logs/gc_AMQ.log}}
{{wrapper.java.additional.28=-XX:NumberOfGCLogFiles=20}}
{{wrapper.java.additional.29=-XX:GCLogFileSize=10M}}
{{wrapper.java.additional.30=-XX:+PrintTenuringDistribution}}
{{wrapper.java.additional.31=-XX:NewRatio=2}}
{{wrapper.java.additional.32=-server}}
{{wrapper.java.additional.33=-XX:+HeapDumpOnOutOfMemoryError}}
{{wrapper.java.additional.34=-XX:HeapDumpPath=F:/Logs/activemq_heapdumps}}
{{wrapper.java.additional.35=-XX:CMSInitiatingOccupancyFraction=60 }}
{{wrapper.java.additional.36=-XX:+UseCMSInitiatingOccupancyOnly}}
{{wrapper.java.additional.37=-Xms8192m -Xmx8192m}}
Reporter: Markus Leichter
Attachments: image-2018-05-28-13-54-21-048.png
We are running a so called "Shared Nothing" setup with several Wildfly 8.2.1
instances (10) on several host machines (4)- all connected to one standalone
Activemq installation - Normally we have 600-700 connections in total to our
broker.
The Activemq installation is configured to use 8GB RAM.
{code:java}
<broker xmlns="http://activemq.apache.org/schema/core"; brokerName="dcdng"
id="dcdng" useJmx="true" useShutdownHook="false"
schedulerSupport="false" dataDirectory="C:/activemq/data"
advisorySupport="false">
<managementContext>
<managementContext createConnector="true" connectorPort="61610"
suppressMBean="endpoint=dynamicProducer,endpoint=Consumer,connectionName=*,destinationName=ActiveMQ.Advisory.*"/>
</managementContext>
<!-- ... -->
</broker>{code}
The memory usage is set to 500mb which is sufficient since our messages are
relative small. (MaxMessageSize: 41442)
The system performs great until we put it under high load. Then the heap usage
grows and it seems the memory cannot get cleaned up properly.
!image-2018-05-28-07-19-03-374.png|width=1048,height=554!
At the end I took a heap dump and the MAT prints the following:
!image-2018-05-28-13-54-21-048.png!
So why are there so many TransportConnectionState objects left an not cleaned
up properly. When the system performs under "normal" load, everything seems to
be fine.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
