Memory Leak in DefaultJDBCAdapter
---------------------------------
Key: AMQ-2563
URL: https://issues.apache.org/activemq/browse/AMQ-2563
Project: ActiveMQ
Issue Type: Bug
Components: Message Store
Affects Versions: 5.3.0
Environment: Java 1.6.0_07-b06 on Solaris (5.10) sparcv9
Reporter: Jim Harter
I was performing an endurance test on ActiveMQ 5.3.0. My test consisted of a
single queue with a number of producers and consumers. The producers send very
simple text messages and the consumers simply receive the message and do no
other processing. Using the OOTB configuration (which uses -Xmx512m) with JDBC
persistence to an Oracle database, I found the system came to a halt at around
10.3 million messages. Looking at the VM revealed a very full heap and tiny
gains from garbage collection. Restarting the broker allows the system to run
again.
To determine the cause of the exhausted heap, I took a series of heap dumps
over time. My examination of the heaps showed that the number of live
instances of TreeMap$Entry and Long were increasing linearly with the number of
messages. The Longs were owned by the TreeMap$Entry objects. The
TreeMap$Entry objects could be tracked back to the TreeSet<Long> instance from
the lastRecoveredMessageIds field in DefaultJDBCAdapter.
The only method that uses lastRecoveredMessageIds is:
public void doRecoverNextMessages(TransactionContext c, ActiveMQDestination
destination, long nextSeq,
int maxReturned, JDBCMessageRecoveryListener listener) throws
Exception
As the listener is called to recover a message, the id is added to this set.
The id is only removed from this set if it is encountered on future run of
doRecoverNextMessages when it is added to the cleanupIds list. The SQL that is
executed at the beginning of the method filters messages based on having an id
greater than nextSeq. If nextSeq is always large enough, an id is never added
to cleanupIds and consequently never removed from lastRecoveredMessageIds.
I saw that the use of lastRecoveredMessageIds was introduced with AMQ-1918.
Also, AMQ-2436 synchronizes the TreeSet, but that should have no effect on this
issue.
Dejan mentioned some work done on JDBC persistence and a memory leak fix in
association with AMQ-2519. I have performed the same test on the latest
snapshots of 5.3.1 and 5.4. The same results were observed on those as well.
Also, work done for AMQ-2519 modified JDBCMessageStore, JDBCTopicMessageStore,
and JDBCPersistenceAdapter. I believe the issue is in DefaultJDBCAdapter.
To reproduce:
- Start a broker using JDBC persistence (sample configuration file will be
attached)
- Start a number of producers and consumers using the same queue on that broker
(sample WAR file will be attached. Modify WEB-INF/applicationContext.xml to
set the queue name and broker URL. The war file currently uses TestQueue and
tcp://localhost:61616, respectively.)
- Wait for about 20,000 messages to be processed (you can use less; it just
gets easier in the heap dumps to see after about 20,000 messages. 29 bytes are
retained per message that is processed)
- Look at the old generation size after a full garbage collection over time.
It grows slowly.
- Obtain a heap dump. The heap dump will show a number of retained instances
of TreeMap$Entry and Long. (sample heap dump will be attached)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
