Re: Netty SSL memory leak

[Tom van der Woerdt]

Hi John,

That's the bug I filed the ticket for, yup. I recommend updating to a newer
Cassandra version (3.0.11 or newer), which fixes this issue (and many
others).

Tom


On Wed, May 31, 2017 at 12:39 AM, John Sanda  wrote:

> I have Cassandra 3.0.9 cluster that is hitting OutOfMemoryErrors with byte
> buffer allocation. The stack trace looks like:
>
> java.lang.OutOfMemoryError: Direct buffer memory
> at java.nio.Bits.reserveMemory(Bits.java:694) ~[na:1.8.0_131]
> at java.nio.DirectByteBuffer.(DirectByteBuffer.java:123)
> ~[na:1.8.0_131]
> at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:311)
> ~[na:1.8.0_131]
> at io.netty.buffer.PoolArena$DirectArena.newChunk(PoolArena.java:434)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at io.netty.buffer.PoolArena.allocateNormal(PoolArena.java:179)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at io.netty.buffer.PoolArena.allocate(PoolArena.java:168)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at io.netty.buffer.PoolArena.allocate(PoolArena.java:98)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(
> PooledByteBufAllocator.java:250) ~[netty-all-4.0.23.Final.jar:
> 4.0.23.Final]
> at io.netty.buffer.AbstractByteBufAllocator.directBuffer(
> AbstractByteBufAllocator.java:155) ~[netty-all-4.0.23.Final.jar:
> 4.0.23.Final]
> at io.netty.buffer.AbstractByteBufAllocator.directBuffer(
> AbstractByteBufAllocator.java:146) ~[netty-all-4.0.23.Final.jar:
> 4.0.23.Final]
> at io.netty.buffer.AbstractByteBufAllocator.buffer(
> AbstractByteBufAllocator.java:83) ~[netty-all-4.0.23.Final.jar:
> 4.0.23.Final]
> at io.netty.handler.ssl.SslHandler.allocate(SslHandler.java:1265)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at io.netty.handler.ssl.SslHandler.allocateOutNetBuf(SslHandler.java:1275)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at io.netty.handler.ssl.SslHandler.wrap(SslHandler.java:453)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at io.netty.handler.ssl.SslHandler.flush(SslHandler.java:432)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at io.netty.channel.AbstractChannelHandlerContext.invokeFlush(
> AbstractChannelHandlerContext.java:688) ~[netty-all-4.0.23.Final.jar:
> 4.0.23.Final]
>
> I do not yet have a heap dump. The two relevant tickets are
> CASSANDRA-13114 
>  and CASSANDRA-13126
> . The upstream
> Netty ticket is 3057 .
> Cassandra 3.0.11 upgraded Netty to the version with the fix. Is there
> anything I can check to confirm that this is in fact the issue I am hitting?
>
> Secondly, is there a way to monitor for this? The OOME does not cause the
> JVM to exit. Instead, the logs are getting filled up with OutOfMemoryErrors.
> nodetool status reports UN, and nodetool statusbinary reports running.
>
> --
>
> - John
>

Netty SSL memory leak

[John Sanda]

I have Cassandra 3.0.9 cluster that is hitting OutOfMemoryErrors with byte
buffer allocation. The stack trace looks like:

java.lang.OutOfMemoryError: Direct buffer memory
at java.nio.Bits.reserveMemory(Bits.java:694) ~[na:1.8.0_131]
at java.nio.DirectByteBuffer.(DirectByteBuffer.java:123)
~[na:1.8.0_131]
at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:311)
~[na:1.8.0_131]
at io.netty.buffer.PoolArena$DirectArena.newChunk(PoolArena.java:434)
~[netty-all-4.0.23.Final.jar:4.0.23.Final]
at io.netty.buffer.PoolArena.allocateNormal(PoolArena.java:179)
~[netty-all-4.0.23.Final.jar:4.0.23.Final]
at io.netty.buffer.PoolArena.allocate(PoolArena.java:168)
~[netty-all-4.0.23.Final.jar:4.0.23.Final]
at io.netty.buffer.PoolArena.allocate(PoolArena.java:98)
~[netty-all-4.0.23.Final.jar:4.0.23.Final]
at
io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:250)
~[netty-all-4.0.23.Final.jar:4.0.23.Final]
at
io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:155)
~[netty-all-4.0.23.Final.jar:4.0.23.Final]
at
io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:146)
~[netty-all-4.0.23.Final.jar:4.0.23.Final]
at
io.netty.buffer.AbstractByteBufAllocator.buffer(AbstractByteBufAllocator.java:83)
~[netty-all-4.0.23.Final.jar:4.0.23.Final]
at io.netty.handler.ssl.SslHandler.allocate(SslHandler.java:1265)
~[netty-all-4.0.23.Final.jar:4.0.23.Final]
at
io.netty.handler.ssl.SslHandler.allocateOutNetBuf(SslHandler.java:1275)
~[netty-all-4.0.23.Final.jar:4.0.23.Final]
at io.netty.handler.ssl.SslHandler.wrap(SslHandler.java:453)
~[netty-all-4.0.23.Final.jar:4.0.23.Final]
at io.netty.handler.ssl.SslHandler.flush(SslHandler.java:432)
~[netty-all-4.0.23.Final.jar:4.0.23.Final]
at
io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:688)
~[netty-all-4.0.23.Final.jar:4.0.23.Final]

I do not yet have a heap dump. The two relevant tickets are CASSANDRA-13114
 and CASSANDRA-13126
. The upstream Netty
ticket is 3057 . Cassandra
3.0.11 upgraded Netty to the version with the fix. Is there anything I can
check to confirm that this is in fact the issue I am hitting?

Secondly, is there a way to monitor for this? The OOME does not cause the
JVM to exit. Instead, the logs are getting filled up with OutOfMemoryErrors.
nodetool status reports UN, and nodetool statusbinary reports running.

-- 

- John