franz1981 commented on issue #2479: ARTEMIS-2211 Avoid duplicate code for ByteBuffer pooling and alignment URL: https://github.com/apache/activemq-artemis/pull/2479#issuecomment-457500842 @michaelandrepearce @clebertsuconic We can talk about this one on the issue/forum too: the thing is that currently we have yet several parts on artemis where we rely on thread pool storage for direct `ByteBuffer`s: 1. write side of paging 2. write side of journal with NIO/MAPPED journal types are used Considering that the context of execution in artemis is agnostic of the executing `Thread` and that the same task could move between executing `Thread`s, it could happen that after 1 minute of inactivity threads could die, they'll leak the thread pool storage related to them (until the first minor GC). If a new burst of activity will create new threads and no minor GC has happened there risk is to have too many direct buffers alive that could lead the OOM killer of the OS to kill the first process that allocate native memory. For compaction, this has been addressed (by me) by allocating upfront the compaction task a single direct buffer and reusing it for any compaction sub-task, deallocating manually (like a C `free`) at the end: although it could seems a bad solution, it avoid leaking several MBs of direct memory if the compaction operations are handled by different threads each time. It would slow down compaction, but given that compaction is not meant to be an hot path like journal writes is best to have predictable memory footprint. This same approach can't be used for the other 2 cases and I was thinking to provide a lock-free generic implementation of direct buffer pools to address it (as ASYNCIO does), but maybe there is a simpler and more effective solution: feel free to give your opinion ;)
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
