[
https://issues.apache.org/jira/browse/AMQ-5773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dmitry updated AMQ-5773:
------------------------
Environment:
$ java -version
java version "1.8.0_20"
Java(TM) SE Runtime Environment (build 1.8.0_40-b26)
Java HotSpot(TM) 64-Bit Server VM (build 25.20-b23, mixed mode)
$ uname -a
Linux mint 3.13.0-24-generic #46-Ubuntu SMP Thu Apr 10 19:11:08 UTC 2014 x86_64
x86_64 x86_64 GNU/Linux
was:
$ java -version
java version "1.8.0_20"
Java(TM) SE Runtime Environment (build 1.8.0_20-b26)
Java HotSpot(TM) 64-Bit Server VM (build 25.20-b23, mixed mode)
$ uname -a
Linux mint 3.13.0-24-generic #46-Ubuntu SMP Thu Apr 10 19:11:08 UTC 2014 x86_64
x86_64 x86_64 GNU/Linux
> Memory leak in JDBC Message Store
> ---------------------------------
>
> Key: AMQ-5773
> URL: https://issues.apache.org/jira/browse/AMQ-5773
> Project: ActiveMQ
> Issue Type: Bug
> Components: JDBC
> Affects Versions: 5.11.1
> Environment: $ java -version
> java version "1.8.0_20"
> Java(TM) SE Runtime Environment (build 1.8.0_40-b26)
> Java HotSpot(TM) 64-Bit Server VM (build 25.20-b23, mixed mode)
> $ uname -a
> Linux mint 3.13.0-24-generic #46-Ubuntu SMP Thu Apr 10 19:11:08 UTC 2014
> x86_64 x86_64 x86_64 GNU/Linux
> Reporter: Dmitry
>
> Hello. Recently I've faced with weird problem, actually I'm not sure if it's
> a bug or misconfiguration or something else. But since my broker has "almost"
> default configuration I think this is bug.
> Brief configuration description(full configuration xml file at the end of
> issue):
> - Persistent storage: MySQL Db
> - Message type: only persistent Text Messages
> - Subscribers: only durable topic subscribers
> So, when I started broker I monitored its cpu/memory usage and here is what I
> noticed, memory chart below:(for servers monitoring I'm using New Relic):
> [Heap Memory Chart |
> http://s4.postimg.org/fjiwg1v0d/amq_memory_leak_chart.png]
> As you can see usage of memory slowly but constantly growing.
> I decided to find out what exactly stored in activemq's heap, so I dumped
> heap and analyzed it with Eclipse MAT. Screenshots below:
> [Leak Analysis | http://s4.postimg.org/nqaw7ml31/amq_leak_suspect.png]
> As we can see almost all of the memory occupied in JournalTopicMessageStore.
> Going deeper:
> [First heapdump |
> http://s4.postimg.org/p6r03iddp/amq_problem_suspect_00_00.png]
> We can see *completions* list in TransactionContext, which has size of 38590
> objects. This heap dump I made at 00:00.
> Below is a heap dump that I made 4 hours later:
> [Second heapdump |
> http://s4.postimg.org/e8fqlbosd/amq_problem_suspect_04_00.png]
> The picture is the same, but now this list contains 63770 objecs.
> Analyzed code a little bit, I noticed that this list is never cleared and
> hence its size always growth. Runnables that are stored in this list have
> link to Message object and hence this message is never will be GC'ed, even
> after ActiveMQ puts it to db. Of course I don't know probably this is a
> regular behavior that objects from this list are never removed, but it's
> looks really strange that object that contains this list is never recreated
> for 10 hours.
> Below I paste configuration of my broker:
> {code:title="broker configuration"|collapse}
> <beans
> xmlns="http://www.springframework.org/schema/beans";
> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
> xsi:schemaLocation="http://www.springframework.org/schema/beans
> http://www.springframework.org/schema/beans/spring-beans.xsd
> http://activemq.apache.org/schema/core
> http://activemq.apache.org/schema/core/activemq-core.xsd";>
> <!-- Allows us to use system properties as variables in this
> configuration file -->
> <bean
> class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
> <property name="locations">
> <value>file:${activemq.conf}/credentials.properties</value>
> </property>
> </bean>
> <!-- Allows accessing the server log -->
> <bean id="logQuery" class="io.fabric8.insight.log.log4j.Log4jLogQuery"
> lazy-init="false" scope="singleton"
> init-method="start" destroy-method="stop">
> </bean>
> <!--
> The <broker> element is used to configure the ActiveMQ broker.
> -->
> <broker xmlns="http://activemq.apache.org/schema/core";
> brokerName="broker"
> persistent="true"
> schedulerSupport="true"
> advisorySupport="false"
> dataDirectory="${activemq.data}">
> <destinationPolicy>
> <policyMap>
> <policyEntries>
> <policyEntry topic=">" producerFlowControl="false"
> expireMessagesPeriod="0">
> <deadLetterStrategy>
> <sharedDeadLetterStrategy processExpired="false"
> processNonPersistent="false"/>
> </deadLetterStrategy>
> </policyEntry>
> <policyEntry queue=">" optimizedDispatch="true"
> producerFlowControl="false">
> <deadLetterStrategy>
> <sharedDeadLetterStrategy processExpired="false"
> processNonPersistent="false"/>
> </deadLetterStrategy>
> </policyEntry>
> </policyEntries>
> </policyMap>
> </destinationPolicy>
> <!--
> The managementContext is used to configure how ActiveMQ is
> exposed in
> JMX. By default, ActiveMQ uses the MBean server that is started by
> the JVM. For more information, see:
> http://activemq.apache.org/jmx.html
> -->
> <managementContext>
> <managementContext createConnector="false"/>
> </managementContext>
> <!--
> Configure message persistence for the broker. The default
> persistence
> mechanism is MySQL.
> For more information, see:
> http://activemq.apache.org/persistence.html
> -->
> <persistenceFactory>
> <journalPersistenceAdapterFactory
> journalLogFiles="8"
> journalLogFileSize="268435456"
> useJournal="true"
> useQuickJournal="false"
> dataSource="#mysql-ds"
> dataDirectory="${activemq.data}"/>
> </persistenceFactory>
> <!--
> The systemUsage controls the maximum amount of space the broker will
> use before disabling caching and/or slowing down producers. For
> more information, see:
> http://activemq.apache.org/producer-flow-control.html
> -->
> <systemUsage>
> <systemUsage>
> <memoryUsage>
> <memoryUsage limit="64 mb"/>
> </memoryUsage>
> <storeUsage>
> <storeUsage limit="100 gb"/>
> </storeUsage>
> <tempUsage>
> <tempUsage limit="50 gb"/>
> </tempUsage>
> </systemUsage>
> </systemUsage>
> <!--
> The transport connectors expose ActiveMQ over a given protocol to
> clients and other brokers. For more information, see:
> http://activemq.apache.org/configuring-transports.html
> -->
> <transportConnectors>
> <!-- DOS protection, limit concurrent connections to 1000 and
> frame size to 100MB -->
> <transportConnector name="default"
> uri="nio://activemq.lc:61616?maximumConnections=1000&connectionTimeout=600000&wireFormat.maxFrameSize=104857600"/>
> </transportConnectors>
> <!-- redelivery configuration-->
> <plugins>
> <loggingBrokerPlugin logAll="true"/>
> <redeliveryPlugin sendToDlqIfMaxRetriesExceeded="true">
> <redeliveryPolicyMap>
> <redeliveryPolicyMap>
> <defaultEntry>
> <redeliveryPolicy maximumRedeliveries="4"
> initialRedeliveryDelay="5000" redeliveryDelay="10000"
> useExponentialBackOff="true" />
> </defaultEntry>
> </redeliveryPolicyMap>
> </redeliveryPolicyMap>
> </redeliveryPlugin>
> </plugins>
> <!-- destroy the spring context on shutdown to stop jetty -->
> <shutdownHooks>
> <bean xmlns="http://www.springframework.org/schema/beans";
> class="org.apache.activemq.hooks.SpringContextHook"/>
> </shutdownHooks>
> </broker>
> <bean id="mysql-ds"
> class="org.apache.commons.dbcp.BasicDataSource"
> destroy-method="close">
> <property name="driverClassName" value="com.mysql.jdbc.Driver"/>
> <property name="url"
> value="jdbc:mysql://localhost:3306/activemq?relaxAutoCommit=true&autoReconnect=true"/>
> <property name="username" value="root"/>
> <property name="password" value=""/>
> <property name="poolPreparedStatements" value="true"/>
> <property name="maxActive" value="40"/>
> </bean>
> <!--
> Enable web consoles, REST and Ajax APIs and demos
> The web consoles requires by default login, you can disable this in
> the jetty.xml file
> Take a look at ${ACTIVEMQ_HOME}/conf/jetty.xml for more details
> -->
> <import resource="jetty.xml"/>
> </beans>
> {code}
> Please help me found out is it configuration problem of it is really a bug,
> because I suffer with this problem for about 2 weeks.
> Thanks!
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
