[
https://issues.apache.org/jira/browse/ARTEMIS-1996?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16561662#comment-16561662
]
ASF GitHub Bot commented on ARTEMIS-1996:
-----------------------------------------
GitHub user morefuntang opened a pull request:
https://github.com/apache/activemq-artemis/pull/2199
ARTEMIS-1996 MappedSequentialFileFactory may cause DirectByteBuffer
off-heap memory leaks
Because of some customization requirements, the readJournalFile method of
JournalImpl needs to be calledmultiple times. During the stress test, it was
found that almost every 5 hours, the Broker appeared a Full GC.
This is the information about the Full GC in the GC log:
**2018-07-25T12:14:07.323+0800: 10089.523: [Full GC (System.gc())
6767M->253M(16G), 8.7138691 secs]
[Eden: 632.0M(712.0M)->0.0B(816.0M) Survivors: 104.0M->0.0B Heap:
6767.6M(16.0G)->253.9M(16.0G)], [Metaspace: 36323K->35961K(1083392K)]
[Times: user=2.56 sys=0.42, real=8.71 secs]**
When the Full GC appears, the thread stack is as follows:
**java.lang.System.gc(System.java:993)
java.nio.Bits.reserveMemory(Bits.java:666)
java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123)
java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:311)
org.apache.activemq.artemis.core.io.mapped.MappedSequentialFileFactory.newBuffer(MappedSequentialFileFactory.java:109)
org.apache.activemq.artemis.core.journal.impl.JournalImpl.readJournalFile(JournalImpl.java:463)**
By analyzing the stack, it should be that the JVM's off-heap memory cannot
be allocated, causing the JVM to call the System.gc() method.
In the Broker, MappedSequentialFileFactory caches off-heap memory through
ThreadLocal. Once the thread is evicted by the CompactExecutor(keepalive is
60s) in the Journal, the off-heap memory is "**leaked**".
NIOSequentialFileFactory also has the same problem.
My advice is to use netty.ThreadDeathWatcher to monitor the threads of the
CompactorExcutor. Once the CompactorExcutor thread finishes, can release the
heap memory by executing the callback method in ThreadDeathWatcher.
jira link: _https://issues.apache.org/jira/browse/ARTEMIS-1996_
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/morefuntang/activemq-artemis Branch20180730
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/activemq-artemis/pull/2199.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2199
----
commit 45ad52fd63be35b7aee783c0d4ba94964df0a167
Author: tangpu <morefun.tang@...>
Date: 2018-07-30T08:41:45Z
ARTEMIS-1996 MappedSequentialFileFactory may cause DirectByteBuffer memory
leaks
----
> MappedSequentialFileFactory may cause DirectByteBuffer memory leaks
> -------------------------------------------------------------------
>
> Key: ARTEMIS-1996
> URL: https://issues.apache.org/jira/browse/ARTEMIS-1996
> Project: ActiveMQ Artemis
> Issue Type: Bug
> Components: Broker
> Reporter: tang pu
> Priority: Minor
>
> Because of some customization requirements, the readJournalFile method of
> JournalImpl needs to be calledmultiple times.
> During the stress test, it was found that almost every 5 hours, the Broker
> appeared a Full GC.
> This is the information about the Full GC in the GC log.
> {color:#FF0000}2018-07-25T12:14:07.323+0800: 10089.523: [Full GC
> (System.gc()) 6767M->253M(16G), 8.7138691 secs]{color}
> {color:#FF0000} [Eden: 632.0M(712.0M)->0.0B(816.0M) Survivors: 104.0M->0.0B
> Heap: 6767.6M(16.0G)->253.9M(16.0G)], [Metaspace:
> 36323K->35961K(1083392K)]{color}
> {color:#FF0000} [Times: user=2.56 sys=0.42, real=8.71 secs] {color}
> When the Full GC appears, the thread stack is as follows:
> {color:#FF0000}java.lang.System.gc(System.java:993){color}
> {color:#FF0000}java.nio.Bits.reserveMemory(Bits.java:666){color}
> {color:#FF0000}java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123){color}
> {color:#FF0000}java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:311){color}
> {color:#FF0000}org.apache.activemq.artemis.core.io.mapped.MappedSequentialFileFactory.newBuffer(MappedSequentialFileFactory.java:109){color}
> {color:#FF0000}org.apache.activemq.artemis.core.journal.impl.JournalImpl.readJournalFile(JournalImpl.java:463){color}
> By analyzing the stack, it should be that the JVM's heap memory cannot be
> allocated, causing the JVM to call the System.gc() method.
> In the Broker, MappedSequentialFileFactory caches off-heap memory through
> ThreadLocal. Once the thread is evicted by the CompactExecutor(keepalive is
> 60s) in the Journal, the heap memory is "leaked".
> {color:#FF0000}NIOSequentialFileFactory{color} also has the same problem
>
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
