[
https://issues.apache.org/jira/browse/ARTEMIS-1996?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16561658#comment-16561658
]
Francesco Nigro commented on ARTEMIS-1996:
------------------------------------------
> Because of some customization requirements
Which kind of customization? This will help to understand the issue...
> Once the thread is evicted by the CompactExecutor(keepalive is 60s) in the
>Journal, the heap memory is "leaked".
AFAIK ThreadLocal entries are being collected if the Thread owner got collected
(with a minor GC as well) and each direct ByteBuffer is being collected like
any other
on-heap instance, but freed through a Cleaner that will be triggered when the
buffer owner instance got collected.
If this isn't happening that means that there are many owner Threads that are
spawned (holding each one a direct ByteBuffer instance) exceeding the
configured -XX:MaxDirectMemorySize configuration.
-XX:MaxDirectMemorySize is being used by java.nio.Bits.reserveMemory to choose
if run a Full GC to force cleaning up of the current allocated native memory.
Quoting some docs: "By default, the JVM does not set a limit on how much memory
is reserved for Direct Byte Buffers. A soft limit of 64 MB is set, which the
JVM automatically expands in 32 MB chunks, as required."
There are other docs instead stating that if not configured it will be
defaulted to the -Xmx value (that means that a 5GB heap will allow additional
5GB of off heap memory): TBH I'm not feeling confident about it.
If you are experiencing FullGC that just means that you have too many live
threads concurrently pooling and no minor GCs that free up memory in time.
If that's the case I see 2 solutions:
* changing the pooling mechanism using a AtomicReferenceArray with size ==
available processors or a concurrent queue/stack (ie with fixed size and not
depending by Threads or minorGCs)
* not pooling when such huge direct ByteBuffers are needed (for NIO and
ASYNCIO, while MAPPED could directly use its internal MappedByteBuffer to read
the content)
> MappedSequentialFileFactory may cause DirectByteBuffer memory leaks
> -------------------------------------------------------------------
>
> Key: ARTEMIS-1996
> URL: https://issues.apache.org/jira/browse/ARTEMIS-1996
> Project: ActiveMQ Artemis
> Issue Type: Bug
> Components: Broker
> Reporter: tang pu
> Priority: Minor
>
> Because of some customization requirements, the readJournalFile method of
> JournalImpl needs to be calledmultiple times.
> During the stress test, it was found that almost every 5 hours, the Broker
> appeared a Full GC.
> This is the information about the Full GC in the GC log.
> {color:#FF0000}2018-07-25T12:14:07.323+0800: 10089.523: [Full GC
> (System.gc()) 6767M->253M(16G), 8.7138691 secs]{color}
> {color:#FF0000} [Eden: 632.0M(712.0M)->0.0B(816.0M) Survivors: 104.0M->0.0B
> Heap: 6767.6M(16.0G)->253.9M(16.0G)], [Metaspace:
> 36323K->35961K(1083392K)]{color}
> {color:#FF0000} [Times: user=2.56 sys=0.42, real=8.71 secs] {color}
> When the Full GC appears, the thread stack is as follows:
> {color:#FF0000}java.lang.System.gc(System.java:993){color}
> {color:#FF0000}java.nio.Bits.reserveMemory(Bits.java:666){color}
> {color:#FF0000}java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123){color}
> {color:#FF0000}java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:311){color}
> {color:#FF0000}org.apache.activemq.artemis.core.io.mapped.MappedSequentialFileFactory.newBuffer(MappedSequentialFileFactory.java:109){color}
> {color:#FF0000}org.apache.activemq.artemis.core.journal.impl.JournalImpl.readJournalFile(JournalImpl.java:463){color}
> By analyzing the stack, it should be that the JVM's heap memory cannot be
> allocated, causing the JVM to call the System.gc() method.
> In the Broker, MappedSequentialFileFactory caches off-heap memory through
> ThreadLocal. Once the thread is evicted by the CompactExecutor(keepalive is
> 60s) in the Journal, the heap memory is "leaked".
> {color:#FF0000}NIOSequentialFileFactory{color} also has the same problem
>
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
