[
https://issues.apache.org/jira/browse/ARTEMIS-2482?focusedWorklogId=310470&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-310470
]
ASF GitHub Bot logged work on ARTEMIS-2482:
-------------------------------------------
Author: ASF GitHub Bot
Created on: 11/Sep/19 10:29
Start Date: 11/Sep/19 10:29
Worklog Time Spent: 10m
Work Description: franz1981 commented on issue #2832: ARTEMIS-2482 Large
messages could leak native ByteBuffers
URL: https://github.com/apache/activemq-artemis/pull/2832#issuecomment-530319911
@wy96f I've taken a second look to the way Netty handle thread local
`ByteBuf`pools and I can confirm that
`-Dio.netty.allocator.useCacheForAllThreads=false` should save non-netty
threads from creating such thread local regions for tiny/small/medium sized
buffers.
While related to
> but they just contain wrappers to direct memory that cannot be released
(is part of the pool and they have no cleaner). The impact should be way less
then IOUtil that pool native ByteBuffers holding exclusively native memory that
won't be reused anymore...
I have to add more detail and I was wrong: the tiny/small/medium thread
local caches are actually leaking memory regions (with default
`io.netty.allocator.useCacheForAllThread`) that need the holding thread to die
in order to make them available to other threads again, but their size is quite
small by default and the size I've chosen as `LARGE_MESSAGE_CHUNK_SIZE` (ie 100
* 1024) is `medium` but doesn't fall in any of those caches, so no real leak
will happen until we allocate something smaller (could happen but is rare).
IMO we should consider 2 separate improvements for a separate PR:
- allow threads to stay alive forever and in a fixed number, but just idle
(and maybe using the FJ thread pool executor instead of the AMQ thread pool
executor in that case): in that case we can consider such caches to be ok to be
used by *all* threads (including non-netty ones), because the leak is just to
cope with future load
- for the AMQ thread pool executor as it is, make the presence of such
caches configurable and off by default
wdyt?
Thanks for the comment, good catch!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
-------------------
Worklog Id: (was: 310470)
Time Spent: 5.5h (was: 5h 20m)
> Large messages could leak native ByteBuffers
> --------------------------------------------
>
> Key: ARTEMIS-2482
> URL: https://issues.apache.org/jira/browse/ARTEMIS-2482
> Project: ActiveMQ Artemis
> Issue Type: Bug
> Components: AMQP, Broker, OpenWire
> Affects Versions: 2.10.0
> Reporter: Francesco Nigro
> Priority: Major
> Time Spent: 5.5h
> Remaining Estimate: 0h
>
> JournalStorageManager::addBytesToLargeMessage and
> LargeServerMessageImpl::DecodingContext::encode are relying on the pooling of
> direct ByteBuffers performed internally by NIO.
> Those buffers are pooled until certain size limit (ie
> jdk.nio.maxCachedBufferSize, as shown on
> https://bugs.openjdk.java.net/browse/JDK-8147468) otherwise are freed right
> after the write succeed.
> If the property jdk.nio.maxCachedBufferSize isn't set, the direct buffers are
> always pooled regardless of the size, leading to OOM issues on high load of
> variable sized writes due to the amount of direct memory allocated and not
> released/late released.
> This should be an alternative fix for
> https://issues.apache.org/jira/browse/ARTEMIS-1811 and it check if such
> pooling is happening, making large messages to be read/written in chunks by
> using the Netty ByteBuf pool to handle any intermediate buffer.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
