[
https://issues.apache.org/activemq/browse/AMQ-2563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jim Harter updated AMQ-2563:
----------------------------
Attachment: heap.zip
AMQTest.war
jdbc-test.xml
Attaching sample broker configuration file, test web application to produce and
consume messages, and a zip of a heap dump showing the memory leak
> Memory Leak in DefaultJDBCAdapter
> ---------------------------------
>
> Key: AMQ-2563
> URL: https://issues.apache.org/activemq/browse/AMQ-2563
> Project: ActiveMQ
> Issue Type: Bug
> Components: Message Store
> Affects Versions: 5.3.0
> Environment: Java 1.6.0_07-b06 on Solaris (5.10) sparcv9
> Reporter: Jim Harter
> Attachments: AMQTest.war, heap.zip, jdbc-test.xml
>
>
> I was performing an endurance test on ActiveMQ 5.3.0. My test consisted of a
> single queue with a number of producers and consumers. The producers send
> very simple text messages and the consumers simply receive the message and do
> no other processing. Using the OOTB configuration (which uses -Xmx512m) with
> JDBC persistence to an Oracle database, I found the system came to a halt at
> around 10.3 million messages. Looking at the VM revealed a very full heap
> and tiny gains from garbage collection. Restarting the broker allows the
> system to run again.
> To determine the cause of the exhausted heap, I took a series of heap dumps
> over time. My examination of the heaps showed that the number of live
> instances of TreeMap$Entry and Long were increasing linearly with the number
> of messages. The Longs were owned by the TreeMap$Entry objects. The
> TreeMap$Entry objects could be tracked back to the TreeSet<Long> instance
> from the lastRecoveredMessageIds field in DefaultJDBCAdapter.
> The only method that uses lastRecoveredMessageIds is:
> public void doRecoverNextMessages(TransactionContext c, ActiveMQDestination
> destination, long nextSeq,
> int maxReturned, JDBCMessageRecoveryListener listener) throws
> Exception
> As the listener is called to recover a message, the id is added to this set.
> The id is only removed from this set if it is encountered on future run of
> doRecoverNextMessages when it is added to the cleanupIds list. The SQL that
> is executed at the beginning of the method filters messages based on having
> an id greater than nextSeq. If nextSeq is always large enough, an id is
> never added to cleanupIds and consequently never removed from
> lastRecoveredMessageIds.
> I saw that the use of lastRecoveredMessageIds was introduced with AMQ-1918.
> Also, AMQ-2436 synchronizes the TreeSet, but that should have no effect on
> this issue.
> Dejan mentioned some work done on JDBC persistence and a memory leak fix in
> association with AMQ-2519. I have performed the same test on the latest
> snapshots of 5.3.1 and 5.4. The same results were observed on those as well.
> Also, work done for AMQ-2519 modified JDBCMessageStore,
> JDBCTopicMessageStore, and JDBCPersistenceAdapter. I believe the issue is in
> DefaultJDBCAdapter.
> To reproduce:
> - Start a broker using JDBC persistence (sample configuration file will be
> attached)
> - Start a number of producers and consumers using the same queue on that
> broker (sample WAR file will be attached. Modify
> WEB-INF/applicationContext.xml to set the queue name and broker URL. The war
> file currently uses TestQueue and tcp://localhost:61616, respectively.)
> - Wait for about 20,000 messages to be processed (you can use less; it just
> gets easier in the heap dumps to see after about 20,000 messages. 29 bytes
> are retained per message that is processed)
> - Look at the old generation size after a full garbage collection over time.
> It grows slowly.
> - Obtain a heap dump. The heap dump will show a number of retained instances
> of TreeMap$Entry and Long. (sample heap dump will be attached)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
