did you try running without the journal? ie: replace
<persistenceFactory>
<journalPersistenceAdapterFactory> ...
with:
<persistenceAdapter>
<jdbcPersistenceAdapter> ....
On 15 May 2015 at 09:17, Denary <[email protected]> wrote:
> Hello. Recently I've faced with weird problem, actually I'm not sure if it's
> a bug or misconfiguration or something else. But since my broker has
> "almost" default configuration I think this is bug.
> Brief configuration description(full configuration xml file at the end of
> issue):
> - Persistent storage: MySQL Db
> - Message type: only persistent Text Messages
> - Subscribers: only durable topic subscribers
>
> So, when I started broker I monitored its cpu/memory usage and here is what
> I noticed, memory chart below:(for servers monitoring I'm using New Relic):
> <http://activemq.2283324.n4.nabble.com/file/n4696558/amq_memory_leak_chart.png>
> As you can see usage of memory slowly but constantly growing.
> I decided to find out what exactly stored in activemq's heap, so I dumped
> heap and analyzed it with Eclipse MAT. Screenshots below:
> http://s4.postimg.org/nqaw7ml31/amq_leak_suspect.png
> <http://s4.postimg.org/nqaw7ml31/amq_leak_suspect.png>
> As we can see almost all of the memory occupied in JournalTopicMessageStore.
> Going deeper:
> http://s4.postimg.org/p6r03iddp/amq_problem_suspect_00_00.png
> <http://s4.postimg.org/p6r03iddp/amq_problem_suspect_00_00.png>
> We can see *completions* list in TransactionContext, which has size of 38590
> objects. This heap dump I made at 00:00.
> Below is a heap dump that I made 4 hours later:
> http://s4.postimg.org/e8fqlbosd/amq_problem_suspect_04_00.png
> <http://s4.postimg.org/e8fqlbosd/amq_problem_suspect_04_00.png>
>
> The picture is the same, but now this list contains 63770 objecs.
> Analyzed code a little bit, I noticed that this list is never cleared and
> hence its size always growth. Runnables that are stored in this list have
> link to Message object and hence this message is never will be GC'ed, even
> after ActiveMQ puts it to db. Of course I don't know probably this is a
> regular behavior that objects from this list are never removed, but it's
> looks really strange that object that contains this list is never recreated
> for 10 hours.
>
> Below I paste configuration of my broker:
> {code:title="broker configuration"|collapse}
> <beans
> xmlns="http://www.springframework.org/schema/beans";
> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
> xsi:schemaLocation="http://www.springframework.org/schema/beans
> http://www.springframework.org/schema/beans/spring-beans.xsd
> http://activemq.apache.org/schema/core
> http://activemq.apache.org/schema/core/activemq-core.xsd";>
>
>
> <bean
> class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
> <property name="locations">
> <value>file:${activemq.conf}/credentials.properties</value>
> </property>
> </bean>
>
>
> <bean id="logQuery" class="io.fabric8.insight.log.log4j.Log4jLogQuery"
> lazy-init="false" scope="singleton"
> init-method="start" destroy-method="stop">
> </bean>
>
>
> <broker xmlns="http://activemq.apache.org/schema/core";
> brokerName="broker"
> persistent="true"
> schedulerSupport="true"
> advisorySupport="false"
> dataDirectory="${activemq.data}">
>
> <destinationPolicy>
> <policyMap>
> <policyEntries>
> <policyEntry topic=">" producerFlowControl="false"
> expireMessagesPeriod="0">
> <deadLetterStrategy>
> <sharedDeadLetterStrategy processExpired="false"
> processNonPersistent="false"/>
> </deadLetterStrategy>
> </policyEntry>
> <policyEntry queue=">" optimizedDispatch="true"
> producerFlowControl="false">
> <deadLetterStrategy>
> <sharedDeadLetterStrategy processExpired="false"
> processNonPersistent="false"/>
> </deadLetterStrategy>
> </policyEntry>
> </policyEntries>
> </policyMap>
> </destinationPolicy>
>
>
>
> <managementContext>
> <managementContext createConnector="false"/>
> </managementContext>
>
>
> <persistenceFactory>
> <journalPersistenceAdapterFactory
> journalLogFiles="8"
> journalLogFileSize="268435456"
> useJournal="true"
> useQuickJournal="false"
> dataSource="#mysql-ds"
> dataDirectory="${activemq.data}"/>
> </persistenceFactory>
>
>
> <systemUsage>
> <systemUsage>
> <memoryUsage>
> <memoryUsage limit="64 mb"/>
> </memoryUsage>
> <storeUsage>
> <storeUsage limit="100 gb"/>
> </storeUsage>
> <tempUsage>
> <tempUsage limit="50 gb"/>
> </tempUsage>
> </systemUsage>
> </systemUsage>
>
>
> <transportConnectors>
>
> <transportConnector name="default"
> uri="nio://activemq.lc:61616?maximumConnections=1000&connectionTimeout=600000&wireFormat.maxFrameSize=104857600"/>
> </transportConnectors>
>
>
> <plugins>
> <loggingBrokerPlugin logAll="true"/>
> <redeliveryPlugin sendToDlqIfMaxRetriesExceeded="true">
> <redeliveryPolicyMap>
> <redeliveryPolicyMap>
> <defaultEntry>
> <redeliveryPolicy maximumRedeliveries="4"
> initialRedeliveryDelay="5000" redeliveryDelay="10000"
> useExponentialBackOff="true" />
> </defaultEntry>
> </redeliveryPolicyMap>
> </redeliveryPolicyMap>
> </redeliveryPlugin>
> </plugins>
>
>
> <shutdownHooks>
> <bean xmlns="http://www.springframework.org/schema/beans";
> class="org.apache.activemq.hooks.SpringContextHook"/>
> </shutdownHooks>
>
> </broker>
>
> <bean id="mysql-ds"
> class="org.apache.commons.dbcp.BasicDataSource"
> destroy-method="close">
> <property name="driverClassName" value="com.mysql.jdbc.Driver"/>
> <property name="url"
> value="jdbc:mysql://localhost:3306/activemq?relaxAutoCommit=true&autoReconnect=true"/>
> <property name="username" value="root"/>
> <property name="password" value=""/>
> <property name="poolPreparedStatements" value="true"/>
> <property name="maxActive" value="40"/>
> </bean>
>
>
> <import resource="jetty.xml"/>
>
> </beans>
> {code}
>
> Please help me fiund out is it configuration problem or it is really a bug,
> because I suffer with this problem for about 2 weeks.
> Thanks!
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/Memory-leak-in-JDBC-Message-Store-tp4696558.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
