netty OOM problem

[yuneng xie]

you may suffer from OOM when we are using

public static final ByteBufAllocator byteBufAllocator =
UnpooledByteBufAllocator.DEFAULT;

and configure your Encoder as “preferDirect = false”, so you are using
HeapBuf for encoding.

public class NettyProtocolEncoder extends MessageToByteEncoder<Object> {

    public NettyProtocolEncoder() {
        super(false);
    }

the problem is caused by the code below

protected final Object filterOutboundMessage(Object msg) {
        if (msg instanceof ByteBuf) {
            ByteBuf buf = (ByteBuf) msg;
            if (buf.isDirect()) {
                return msg;
            }

            return newDirectBuffer(buf);
        }

        if (msg instanceof FileRegion) {
            return msg;
        }

        throw new UnsupportedOperationException(
                "unsupported message type: " +
StringUtil.simpleClassName(msg) + EXPECTED_TYPES);
    }

 static final class ThreadLocalDirectByteBuf extends UnpooledDirectByteBuf {

        private static final Recycler<ThreadLocalDirectByteBuf>
RECYCLER = new Recycler<ThreadLocalDirectByteBuf>() {
            @Override
            protected ThreadLocalDirectByteBuf newObject(Handle handle) {
                return new ThreadLocalDirectByteBuf(handle);
            }
        };

        static ThreadLocalDirectByteBuf newInstance() {
            ThreadLocalDirectByteBuf buf = RECYCLER.get();
            buf.setRefCnt(1);
            return buf;
        }

        private final Handle handle;

        private ThreadLocalDirectByteBuf(Handle handle) {
            super(UnpooledByteBufAllocator.DEFAULT, 256, Integer.MAX_VALUE);
            this.handle = handle;
        }

        @Override
        protected void deallocate() {
            if (capacity() > THREAD_LOCAL_BUFFER_SIZE) {
                super.deallocate();
            } else {
                clear();
                RECYCLER.recycle(this, handle);
            }
        }
    }


   - netty tried to convert the HeapBuf into a DirectBuf
   (filterOutboundMessage method)。The DirectBuf is retrieved from RECYCLER
   - RECYCLER may hold a large amount of ThreadLocalDirectByteBuf.
   - RECYCLER is not used to retrieve buffer when reading data. Attempts to
   allocate DirectBuffer may trigger FullGC
   - your peer could be in the same situation. so both side get blocked.

i’m wondering why we are not using RECYCLER when reading data  ;)

-- 
You received this message because you are subscribed to the Google Groups 
"Netty discussions" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to netty+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/netty/CAAkXsNTKe_DmjSosBfymXds-XaSEu6Gfh%3DYn%2BgYZKU%3DR7fCG-Q%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.