Another point to note: the allocator thinks it has used up 4G of memory yet
this happens on an otherwise mostly idle server after transferring <10MB of
the response.
It does appear that the arena for the thread handling the response has
filled while the others are empty, which makes sense. I've attached the
output of dumpStats at the time the exception was thrown. Perhaps this is a
case of needing to first check for channel writability? It looks like with
16 arenas and 4G total available, a single thread would exhaust its output
buffers with 256MB of pending data.
Still, I'm quite surprised to see the pooled allocator in play at all here.
On Thursday, June 30, 2016 at 5:15:09 PM UTC-4, Chris Conroy wrote:
>
> I have encountered a strange issue in some server code on top of Netty 4.1
> Final using the Java SSL server engine.
>
> When I try to send a very large file through the pipeline, I get an
> OutOfDirectMemoryError from an Sslhandler#flush.
>
> Of course, this could happen if I failed to call release somewhere in the
> pipeline, but we’re actually setting .option(ChannelOption.ALLOCATOR, new
> UnpooledByteBufAllocator(false)) in the bootstrap for this server in
> order to get Netty 3-like memory semantics. (We plan to shift to the pooled
> allocator later after shaking out any other Netty 4 issues).
>
> The fact that the SslHandler is still using a direct buffer pool despite
> the option above seems like a bug. Or, am I misunderstanding this option?
>
> If things were generally leaking, I would expect to see this eventually
> even without large requests, but it seems to only impact channels that are
> servicing a large transfer. I’m still gathering more data on the issue, but
> FWIW the leak detector in advanced mode has not reported any leaks, and
> interestingly I cannot reproduce the issue when running under paranoid
> mode.
>
> io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 16777216
> byte(s) of direct memory (used: 4110417927, max: 4127195136)
> at
> io.netty.util.internal.PlatformDependent.incrementMemoryCounter(PlatformDependent.java:592)
> at
> io.netty.util.internal.PlatformDependent.allocateDirectNoCleaner(PlatformDependent.java:546)
> at
> io.netty.buffer.PoolArena$DirectArena.allocateDirect(PoolArena.java:699)
> at io.netty.buffer.PoolArena$DirectArena.newChunk(PoolArena.java:688)
> at io.netty.buffer.PoolArena.allocateNormal(PoolArena.java:237)
> at io.netty.buffer.PoolArena.allocate(PoolArena.java:221)
> at io.netty.buffer.PoolArena.allocate(PoolArena.java:141)
> at
> io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:262)
> at
> io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:179)
> at
> io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:170)
> at
> io.netty.buffer.AbstractByteBufAllocator.buffer(AbstractByteBufAllocator.java:107)
> at io.netty.handler.ssl.SslHandler.allocate(SslHandler.java:1461)
> at
> io.netty.handler.ssl.SslHandler.allocateOutNetBuf(SslHandler.java:1471)
> at io.netty.handler.ssl.SslHandler.wrap(SslHandler.java:523)
> at io.netty.handler.ssl.SslHandler.flush(SslHandler.java:501)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:762)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:754)
> at
> io.netty.channel.AbstractChannelHandlerContext.flush(AbstractChannelHandlerContext.java:735)
> at
> io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.flush(CombinedChannelDuplexHandler.java:523)
> at
> io.netty.channel.ChannelOutboundHandlerAdapter.flush(ChannelOutboundHandlerAdapter.java:115)
> at
> io.netty.channel.CombinedChannelDuplexHandler.flush(CombinedChannelDuplexHandler.java:348)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:762)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:754)
> at
> io.netty.channel.AbstractChannelHandlerContext.flush(AbstractChannelHandlerContext.java:735)
> at
> io.netty.channel.ChannelOutboundHandlerAdapter.flush(ChannelOutboundHandlerAdapter.java:115)
> ....
>
>
>
--
You received this message because you are subscribed to the Google Groups
"Netty discussions" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/netty/39fae52d-a9ad-4e73-93a3-ac5272918cc3%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Chunk(s) at 0~25%:
none
Chunk(s) at 0~50%:
none
Chunk(s) at 25~75%:
none
Chunk(s) at 50~100%:
none
Chunk(s) at 75~100%:
none
Chunk(s) at 100%:
none
tiny subpages:
small subpages:
Chunk(s) at 0~25%:
none
Chunk(s) at 0~50%:
none
Chunk(s) at 25~75%:
none
Chunk(s) at 50~100%:
none
Chunk(s) at 75~100%:
none
Chunk(s) at 100%:
none
tiny subpages:
small subpages:
Chunk(s) at 0~25%:
none
Chunk(s) at 0~50%:
none
Chunk(s) at 25~75%:
none
Chunk(s) at 50~100%:
none
Chunk(s) at 75~100%:
none
Chunk(s) at 100%:
none
tiny subpages:
small subpages:
Chunk(s) at 0~25%:
none
Chunk(s) at 0~50%:
none
Chunk(s) at 25~75%:
none
Chunk(s) at 50~100%:
none
Chunk(s) at 75~100%:
none
Chunk(s) at 100%:
none
tiny subpages:
small subpages:
Chunk(s) at 0~25%:
none
Chunk(s) at 0~50%:
none
Chunk(s) at 25~75%:
none
Chunk(s) at 50~100%:
none
Chunk(s) at 75~100%:
none
Chunk(s) at 100%:
none
tiny subpages:
small subpages:
Chunk(s) at 0~25%:
none
Chunk(s) at 0~50%:
none
Chunk(s) at 25~75%:
none
Chunk(s) at 50~100%:
none
Chunk(s) at 75~100%:
none
Chunk(s) at 100%:
none
tiny subpages:
small subpages:
Chunk(s) at 0~25%:
none
Chunk(s) at 0~50%:
none
Chunk(s) at 25~75%:
none
Chunk(s) at 50~100%:
none
Chunk(s) at 75~100%:
none
Chunk(s) at 100%:
none
tiny subpages:
small subpages:
Chunk(s) at 0~25%:
none
Chunk(s) at 0~50%:
none
Chunk(s) at 25~75%:
none
Chunk(s) at 50~100%:
none
Chunk(s) at 75~100%:
none
Chunk(s) at 100%:
none
tiny subpages:
small subpages:
Chunk(s) at 0~25%:
none
Chunk(s) at 0~50%:
none
Chunk(s) at 25~75%:
none
Chunk(s) at 50~100%:
none
Chunk(s) at 75~100%:
none
Chunk(s) at 100%:
none
tiny subpages:
small subpages:
Chunk(s) at 0~25%:
none
Chunk(s) at 0~50%:
none
Chunk(s) at 25~75%:
none
Chunk(s) at 50~100%:
none
Chunk(s) at 75~100%:
none
Chunk(s) at 100%:
none
tiny subpages:
small subpages:
Chunk(s) at 0~25%:
none
Chunk(s) at 0~50%:
none
Chunk(s) at 25~75%:
none
Chunk(s) at 50~100%:
none
Chunk(s) at 75~100%:
none
Chunk(s) at 100%:
none
tiny subpages:
small subpages:
Chunk(s) at 0~25%:
none
Chunk(s) at 0~50%:
none
Chunk(s) at 25~75%:
none
Chunk(s) at 50~100%:
none
Chunk(s) at 75~100%:
none
Chunk(s) at 100%:
none
tiny subpages:
small subpages:
Chunk(s) at 0~25%:
none
Chunk(s) at 0~50%:
none
Chunk(s) at 25~75%:
none
Chunk(s) at 50~100%:
none
Chunk(s) at 75~100%:
none
Chunk(s) at 100%:
none
tiny subpages:
small subpages:
Chunk(s) at 0~25%:
none
Chunk(s) at 0~50%:
none
Chunk(s) at 25~75%:
none
Chunk(s) at 50~100%:
none
Chunk(s) at 75~100%:
none
Chunk(s) at 100%:
none
tiny subpages:
small subpages:
Chunk(s) at 0~25%:
none
Chunk(s) at 0~50%:
none
Chunk(s) at 25~75%:
none
Chunk(s) at 50~100%:
none
Chunk(s) at 75~100%:
none
Chunk(s) at 100%:
none
tiny subpages:
small subpages:
Chunk(s) at 0~25%:
none
Chunk(s) at 0~50%:
none
Chunk(s) at 25~75%:
none
Chunk(s) at 50~100%:
none
Chunk(s) at 75~100%:
none
Chunk(s) at 100%:
none
tiny subpages:
small subpages:
Chunk(s) at 0~25%:
none
Chunk(s) at 0~50%:
none
Chunk(s) at 25~75%:
none
Chunk(s) at 50~100%:
none
Chunk(s) at 75~100%:
none
Chunk(s) at 100%:
none
tiny subpages:
small subpages:
Chunk(s) at 0~25%:
none
Chunk(s) at 0~50%:
none
Chunk(s) at 25~75%:
none
Chunk(s) at 50~100%:
none
Chunk(s) at 75~100%:
none
Chunk(s) at 100%:
none
tiny subpages:
small subpages:
Chunk(s) at 0~25%:
none
Chunk(s) at 0~50%:
none
Chunk(s) at 25~75%:
none
Chunk(s) at 50~100%:
none
Chunk(s) at 75~100%:
none
Chunk(s) at 100%:
none
tiny subpages:
small subpages:
Chunk(s) at 0~25%:
none
Chunk(s) at 0~50%:
none
Chunk(s) at 25~75%:
none
Chunk(s) at 50~100%:
none
Chunk(s) at 75~100%:
none
Chunk(s) at 100%:
none
tiny subpages:
small subpages:
Chunk(s) at 0~25%:
none
Chunk(s) at 0~50%:
none
Chunk(s) at 25~75%:
none
Chunk(s) at 50~100%:
none
Chunk(s) at 75~100%:
none
Chunk(s) at 100%:
none
tiny subpages:
small subpages:
Chunk(s) at 0~25%:
none
Chunk(s) at 0~50%:
none
Chunk(s) at 25~75%:
none
Chunk(s) at 50~100%:
none
Chunk(s) at 75~100%:
none
Chunk(s) at 100%:
none
tiny subpages:
small subpages:
Chunk(s) at 0~25%:
none
Chunk(s) at 0~50%:
none
Chunk(s) at 25~75%:
none
Chunk(s) at 50~100%:
none
Chunk(s) at 75~100%:
none
Chunk(s) at 100%:
none
tiny subpages:
small subpages:
Chunk(s) at 0~25%:
none
Chunk(s) at 0~50%:
none
Chunk(s) at 25~75%:
none
Chunk(s) at 50~100%:
none
Chunk(s) at 75~100%:
none
Chunk(s) at 100%:
none
tiny subpages:
small subpages:
Chunk(s) at 0~25%:
none
Chunk(s) at 0~50%:
none
Chunk(s) at 25~75%:
none
Chunk(s) at 50~100%:
none
Chunk(s) at 75~100%:
none
Chunk(s) at 100%:
none
tiny subpages:
small subpages:
Chunk(s) at 0~25%:
none
Chunk(s) at 0~50%:
none
Chunk(s) at 25~75%:
none
Chunk(s) at 50~100%:
none
Chunk(s) at 75~100%:
none
Chunk(s) at 100%:
none
tiny subpages:
small subpages:
Chunk(s) at 0~25%:
none
Chunk(s) at 0~50%:
none
Chunk(s) at 25~75%:
none
Chunk(s) at 50~100%:
none
Chunk(s) at 75~100%:
none
Chunk(s) at 100%:
none
tiny subpages:
small subpages:
Chunk(s) at 0~25%:
none
Chunk(s) at 0~50%:
none
Chunk(s) at 25~75%:
none
Chunk(s) at 50~100%:
none
Chunk(s) at 75~100%:
none
Chunk(s) at 100%:
none
tiny subpages:
small subpages:
Chunk(s) at 0~25%:
none
Chunk(s) at 0~50%:
none
Chunk(s) at 25~75%:
none
Chunk(s) at 50~100%:
none
Chunk(s) at 75~100%:
none
Chunk(s) at 100%:
none
tiny subpages:
small subpages:
Chunk(s) at 0~25%:
none
Chunk(s) at 0~50%:
none
Chunk(s) at 25~75%:
none
Chunk(s) at 50~100%:
none
Chunk(s) at 75~100%:
none
Chunk(s) at 100%:
none
tiny subpages:
small subpages:
Chunk(s) at 0~25%:
none
Chunk(s) at 0~50%:
none
Chunk(s) at 25~75%:
none
Chunk(s) at 50~100%:
none
Chunk(s) at 75~100%:
none
Chunk(s) at 100%:
none
tiny subpages:
small subpages:
Chunk(s) at 0~25%:
none
Chunk(s) at 0~50%:
none
Chunk(s) at 25~75%:
none
Chunk(s) at 50~100%:
none
Chunk(s) at 75~100%:
none
Chunk(s) at 100%:
none
tiny subpages:
small subpages:
32 direct arena(s):
Chunk(s) at 0~25%:
Chunk(a851d04: 2%, 172032/16777216)
Chunk(s) at 0~50%:
none
Chunk(s) at 25~75%:
none
Chunk(s) at 50~100%:
none
Chunk(s) at 75~100%:
none
Chunk(s) at 100%:
none
tiny subpages:
5: (2068: 1/102, offset: 163840, length: 8192, elemSize: 80)
13: (2049: 1/39, offset: 8192, length: 8192, elemSize: 208)
16: (2050: 1/32, offset: 16384, length: 8192, elemSize: 256)
small subpages:
1: (2048: 2/8, offset: 0, length: 8192, elemSize: 1024)
Chunk(s) at 0~25%:
Chunk(6b024dfc: 2%, 180224/16777216)
Chunk(s) at 0~50%:
none
Chunk(s) at 25~75%:
none
Chunk(s) at 50~100%:
none
Chunk(s) at 75~100%:
none
Chunk(s) at 100%:
none
tiny subpages:
5: (2065: 1/102, offset: 139264, length: 8192, elemSize: 80)
13: (2050: 1/39, offset: 16384, length: 8192, elemSize: 208)
14: (2049: 1/36, offset: 8192, length: 8192, elemSize: 224)
16: (2051: 1/32, offset: 24576, length: 8192, elemSize: 256)
small subpages:
1: (2048: 2/8, offset: 0, length: 8192, elemSize: 1024)
Chunk(s) at 0~25%:
Chunk(3dce1775: 2%, 188416/16777216)
Chunk(s) at 0~50%:
none
Chunk(s) at 25~75%:
none
Chunk(s) at 50~100%:
none
Chunk(s) at 75~100%:
none
Chunk(s) at 100%:
none
tiny subpages:
5: (2065: 1/102, offset: 139264, length: 8192, elemSize: 80)
8: (2050: 1/64, offset: 16384, length: 8192, elemSize: 128)
13: (2066: 1/39, offset: 147456, length: 8192, elemSize: 208)
16: (2051: 1/32, offset: 24576, length: 8192, elemSize: 256)
23: (2049: 1/22, offset: 8192, length: 8192, elemSize: 368)
small subpages:
1: (2048: 2/8, offset: 0, length: 8192, elemSize: 1024)
Chunk(s) at 0~25%:
Chunk(1800969c: 2%, 188416/16777216)
Chunk(s) at 0~50%:
none
Chunk(s) at 25~75%:
none
Chunk(s) at 50~100%:
none
Chunk(s) at 75~100%:
none
Chunk(s) at 100%:
none
tiny subpages:
5: (2065: 1/102, offset: 139264, length: 8192, elemSize: 80)
8: (2050: 1/64, offset: 16384, length: 8192, elemSize: 128)
13: (2066: 1/39, offset: 147456, length: 8192, elemSize: 208)
16: (2051: 1/32, offset: 24576, length: 8192, elemSize: 256)
23: (2049: 1/22, offset: 8192, length: 8192, elemSize: 368)
small subpages:
1: (2048: 2/8, offset: 0, length: 8192, elemSize: 1024)
Chunk(s) at 0~25%:
none
Chunk(s) at 0~50%:
none
Chunk(s) at 25~75%:
none
Chunk(s) at 50~100%:
none
Chunk(s) at 75~100%:
none
Chunk(s) at 100%:
Chunk(3acfc5a5: 100%, 16777216/16777216)
Chunk(435b12ad: 100%, 16777216/16777216)
Chunk(ee3d00c: 100%, 16777216/16777216)
Chunk(124b033a: 100%, 16777216/16777216)
Chunk(566d1e3b: 100%, 16777216/16777216)
Chunk(4d38a021: 100%, 16777216/16777216)
Chunk(5dcb6a3a: 100%, 16777216/16777216)
Chunk(1753e06a: 100%, 16777216/16777216)
Chunk(29f1a87d: 100%, 16777216/16777216)
Chunk(21c02cca: 100%, 16777216/16777216)
Chunk(275c1872: 100%, 16777216/16777216)
Chunk(2f04ec43: 100%, 16777216/16777216)
Chunk(b920b9e: 100%, 16777216/16777216)
Chunk(2b94c05a: 100%, 16777216/16777216)
Chunk(6c4cd2d2: 100%, 16777216/16777216)
Chunk(64a24dec: 100%, 16777216/16777216)
Chunk(7f77641d: 100%, 16777216/16777216)
Chunk(726cfdb9: 100%, 16777216/16777216)
Chunk(78bc6b64: 100%, 16777216/16777216)
Chunk(e8784b2: 100%, 16777216/16777216)
Chunk(4a944ef3: 100%, 16777216/16777216)
Chunk(5f00e32d: 100%, 16777216/16777216)
Chunk(447c54e2: 100%, 16777216/16777216)
Chunk(766ceacb: 100%, 16777216/16777216)
Chunk(1ee7c123: 100%, 16777216/16777216)
Chunk(46264059: 100%, 16777216/16777216)
Chunk(605bd73b: 100%, 16777216/16777216)
Chunk(71f04349: 100%, 16777216/16777216)
Chunk(51be7a8d: 100%, 16777216/16777216)
Chunk(25eecc6b: 100%, 16777216/16777216)
Chunk(1b3235e2: 100%, 16777216/16777216)
Chunk(682bc7c6: 100%, 16777216/16777216)
Chunk(4a63b25c: 100%, 16777216/16777216)
Chunk(193da2ef: 100%, 16777216/16777216)
Chunk(13aa090f: 100%, 16777216/16777216)
Chunk(2589f94b: 100%, 16777216/16777216)
Chunk(722f4ef4: 100%, 16777216/16777216)
Chunk(671a084: 100%, 16777216/16777216)
Chunk(45502334: 100%, 16777216/16777216)
Chunk(2ff9c974: 100%, 16777216/16777216)
Chunk(27296a91: 100%, 16777216/16777216)
Chunk(2c579b70: 100%, 16777216/16777216)
Chunk(68da544d: 100%, 16777216/16777216)
Chunk(909924b: 100%, 16777216/16777216)
Chunk(65181c19: 100%, 16777216/16777216)
Chunk(750497d1: 100%, 16777216/16777216)
Chunk(4f46dd00: 100%, 16777216/16777216)
Chunk(a188569: 100%, 16777216/16777216)
Chunk(2fc5b8e7: 100%, 16777216/16777216)
Chunk(7e2e08d1: 100%, 16777216/16777216)
Chunk(7f964c9: 100%, 16777216/16777216)
Chunk(49848a92: 100%, 16777216/16777216)
Chunk(4b0411c5: 100%, 16777216/16777216)
Chunk(45e2f0f4: 100%, 16777216/16777216)
Chunk(9711bad: 100%, 16777216/16777216)
Chunk(644cc00b: 100%, 16777216/16777216)
Chunk(fad7f7e: 100%, 16777216/16777216)
Chunk(5d1a4b2f: 100%, 16777216/16777216)
Chunk(5cb78f52: 100%, 16777216/16777216)
Chunk(5ef94057: 100%, 16777216/16777216)
Chunk(3acb8279: 100%, 16777216/16777216)
Chunk(3587e53c: 100%, 16777216/16777216)
Chunk(552a21c1: 100%, 16777216/16777216)
Chunk(41c538cb: 100%, 16777216/16777216)
Chunk(27fbb2c0: 100%, 16777216/16777216)
Chunk(2ddf1d06: 100%, 16777216/16777216)
Chunk(297f0555: 100%, 16777216/16777216)
Chunk(419463d1: 100%, 16777216/16777216)
Chunk(3b03a491: 100%, 16777216/16777216)
Chunk(6e61b9da: 100%, 16777216/16777216)
Chunk(3fe54cee: 100%, 16777216/16777216)
Chunk(5d0d3a28: 100%, 16777216/16777216)
Chunk(7b0c2a34: 100%, 16777216/16777216)
Chunk(18c0f3e6: 100%, 16777216/16777216)
Chunk(dd75e2c: 100%, 16777216/16777216)
Chunk(39bff9c4: 100%, 16777216/16777216)
Chunk(2378394d: 100%, 16777216/16777216)
Chunk(3cb8a107: 100%, 16777216/16777216)
Chunk(b29ae82: 100%, 16777216/16777216)
Chunk(4d1e07fa: 100%, 16777216/16777216)
Chunk(2f4eed45: 100%, 16777216/16777216)
Chunk(5687d132: 100%, 16777216/16777216)
Chunk(109c28de: 100%, 16777216/16777216)
Chunk(2d00cce0: 100%, 16777216/16777216)
Chunk(75fc28c0: 100%, 16777216/16777216)
Chunk(1d1a790c: 100%, 16777216/16777216)
Chunk(6cb178a9: 100%, 16777216/16777216)
Chunk(47fdde13: 100%, 16777216/16777216)
Chunk(53534404: 100%, 16777216/16777216)
Chunk(1d4f857b: 100%, 16777216/16777216)
Chunk(7adcd24b: 100%, 16777216/16777216)
Chunk(53789645: 100%, 16777216/16777216)
Chunk(1ac29b4a: 100%, 16777216/16777216)
Chunk(7b47a134: 100%, 16777216/16777216)
Chunk(67957a9d: 100%, 16777216/16777216)
Chunk(70480294: 100%, 16777216/16777216)
Chunk(7e5edf1c: 100%, 16777216/16777216)
Chunk(38d69fe2: 100%, 16777216/16777216)
Chunk(745b5bf7: 100%, 16777216/16777216)
Chunk(4437ab4a: 100%, 16777216/16777216)
Chunk(4c193aef: 100%, 16777216/16777216)
Chunk(403c8501: 100%, 16777216/16777216)
Chunk(6e16fa12: 100%, 16777216/16777216)
Chunk(17826161: 100%, 16777216/16777216)
Chunk(1249ff3c: 100%, 16777216/16777216)
Chunk(367974f9: 100%, 16777216/16777216)
Chunk(6f66ce4e: 100%, 16777216/16777216)
Chunk(6b6b23aa: 100%, 16777216/16777216)
Chunk(360691e4: 100%, 16777216/16777216)
Chunk(4ba5f561: 100%, 16777216/16777216)
Chunk(12e856e5: 100%, 16777216/16777216)
Chunk(202c5161: 100%, 16777216/16777216)
Chunk(22a77d31: 100%, 16777216/16777216)
Chunk(46cd9af9: 100%, 16777216/16777216)
Chunk(16c2a3bb: 100%, 16777216/16777216)
Chunk(54a74f5b: 100%, 16777216/16777216)
Chunk(4d70fe0b: 100%, 16777216/16777216)
Chunk(67c0af0e: 100%, 16777216/16777216)
Chunk(649c0736: 100%, 16777216/16777216)
Chunk(a2f5169: 100%, 16777216/16777216)
Chunk(40e56681: 100%, 16777216/16777216)
Chunk(223f194c: 100%, 16777216/16777216)
Chunk(269e1f8a: 100%, 16777216/16777216)
Chunk(56cab629: 100%, 16777216/16777216)
Chunk(3df0031f: 100%, 16777216/16777216)
Chunk(675f9894: 100%, 16777216/16777216)
Chunk(316ba9ba: 100%, 16777216/16777216)
Chunk(32133a40: 100%, 16777216/16777216)
Chunk(fc63fe6: 100%, 16777216/16777216)
Chunk(14468db2: 100%, 16777216/16777216)
Chunk(788cc0f9: 100%, 16777216/16777216)
Chunk(53663492: 100%, 16777216/16777216)
Chunk(6de10817: 100%, 16777216/16777216)
Chunk(4d6a2304: 100%, 16777216/16777216)
Chunk(53ffa958: 100%, 16777216/16777216)
Chunk(315fc511: 100%, 16777216/16777216)
Chunk(541bd29d: 100%, 16777216/16777216)
Chunk(4875b919: 100%, 16777216/16777216)
Chunk(666e7c26: 100%, 16777216/16777216)
Chunk(29d64ab7: 100%, 16777216/16777216)
Chunk(2353ea2a: 100%, 16777216/16777216)
Chunk(468b3228: 100%, 16777216/16777216)
Chunk(5311e993: 100%, 16777216/16777216)
Chunk(488982b4: 100%, 16777216/16777216)
Chunk(74b72335: 100%, 16777216/16777216)
Chunk(6b3255f9: 100%, 16777216/16777216)
Chunk(3733747d: 100%, 16777216/16777216)
Chunk(332bcc0d: 100%, 16777216/16777216)
Chunk(7ec8ffd6: 100%, 16777216/16777216)
Chunk(7ace06b: 100%, 16777216/16777216)
Chunk(2b90ec7f: 100%, 16777216/16777216)
Chunk(463616a4: 100%, 16777216/16777216)
Chunk(7fb967fb: 100%, 16777216/16777216)
Chunk(1f766fdf: 100%, 16777216/16777216)
Chunk(33a99b5a: 100%, 16777216/16777216)
Chunk(45dc38bd: 100%, 16777216/16777216)
Chunk(716e0942: 100%, 16777216/16777216)
Chunk(5d4a9827: 100%, 16777216/16777216)
Chunk(22c6ab9f: 100%, 16777216/16777216)
Chunk(67b76aa: 100%, 16777216/16777216)
Chunk(75e4b3e: 100%, 16777216/16777216)
Chunk(edc6052: 100%, 16777216/16777216)
Chunk(19d1a845: 100%, 16777216/16777216)
Chunk(44c253f3: 100%, 16777216/16777216)
Chunk(3130e7ee: 100%, 16777216/16777216)
Chunk(5c03df6a: 100%, 16777216/16777216)
Chunk(4884c72f: 100%, 16777216/16777216)
Chunk(1e8f5807: 100%, 16777216/16777216)
Chunk(28b6c3af: 100%, 16777216/16777216)
Chunk(6a0cb15c: 100%, 16777216/16777216)
Chunk(4dfbe8d: 100%, 16777216/16777216)
Chunk(60f49171: 100%, 16777216/16777216)
Chunk(7ec18d7b: 100%, 16777216/16777216)
Chunk(714845ae: 100%, 16777216/16777216)
Chunk(89aa6dc: 100%, 16777216/16777216)
Chunk(4c21c1a7: 100%, 16777216/16777216)
Chunk(3e79270d: 100%, 16777216/16777216)
Chunk(daf7059: 100%, 16777216/16777216)
Chunk(50de8b76: 100%, 16777216/16777216)
Chunk(12305433: 100%, 16777216/16777216)
Chunk(65864837: 100%, 16777216/16777216)
Chunk(135dd166: 100%, 16777216/16777216)
Chunk(377c7d40: 100%, 16777216/16777216)
Chunk(27fd2650: 100%, 16777216/16777216)
Chunk(706d1568: 100%, 16777216/16777216)
Chunk(dc62ce2: 100%, 16777216/16777216)
Chunk(5984c667: 100%, 16777216/16777216)
Chunk(1785b4a1: 100%, 16777216/16777216)
Chunk(f5b256c: 100%, 16777216/16777216)
Chunk(33c6b959: 100%, 16777216/16777216)
Chunk(4c8ef6b8: 100%, 16777216/16777216)
Chunk(76947334: 100%, 16777216/16777216)
Chunk(203258cf: 100%, 16777216/16777216)
Chunk(263821e1: 100%, 16777216/16777216)
Chunk(1db828a8: 100%, 16777216/16777216)
Chunk(48e0e5f8: 100%, 16777216/16777216)
Chunk(7aa6280b: 100%, 16777216/16777216)
Chunk(1d763997: 100%, 16777216/16777216)
Chunk(41d7bff9: 100%, 16777216/16777216)
Chunk(ed74d1e: 100%, 16777216/16777216)
Chunk(457adabf: 100%, 16777216/16777216)
Chunk(69ece906: 100%, 16777216/16777216)
Chunk(15b8bbb5: 100%, 16777216/16777216)
Chunk(21b3bba1: 100%, 16777216/16777216)
Chunk(328f220a: 100%, 16777216/16777216)
Chunk(3c255984: 100%, 16777216/16777216)
Chunk(6c9028a6: 100%, 16777216/16777216)
Chunk(50c2e826: 100%, 16777216/16777216)
Chunk(1b161f46: 100%, 16777216/16777216)
Chunk(de99cd9: 100%, 16777216/16777216)
Chunk(605140da: 100%, 16777216/16777216)
Chunk(2715072e: 100%, 16777216/16777216)
Chunk(cd2c0a5: 100%, 16777216/16777216)
Chunk(4d1c9ab2: 100%, 16777216/16777216)
Chunk(2721445b: 100%, 16777216/16777216)
Chunk(28820be6: 100%, 16777216/16777216)
Chunk(32cf31bb: 100%, 16777216/16777216)
Chunk(1b2fe45a: 100%, 16777216/16777216)
Chunk(368168f8: 100%, 16777216/16777216)
Chunk(e6488f5: 100%, 16777216/16777216)
Chunk(45ed226b: 100%, 16777216/16777216)
Chunk(210403b8: 100%, 16777216/16777216)
Chunk(1cff69c8: 100%, 16777216/16777216)
Chunk(36f66982: 100%, 16777216/16777216)
Chunk(1aa4eb4d: 100%, 16777216/16777216)
Chunk(1bbc2262: 100%, 16777216/16777216)
Chunk(7ceaa9f4: 100%, 16777216/16777216)
Chunk(79d56592: 100%, 16777216/16777216)
Chunk(44960766: 100%, 16777216/16777216)
Chunk(3e4392a4: 100%, 16777216/16777216)
tiny subpages:
1: (2071: 96/512, offset: 188416, length: 8192, elemSize: 16)
5: (2068: 0/102, offset: 163840, length: 8192, elemSize: 80)
9: (2069: 0/56, offset: 172032, length: 8192, elemSize: 144)
13: (2049: 0/39, offset: 8192, length: 8192, elemSize: 208)
16: (2050: 0/32, offset: 16384, length: 8192, elemSize: 256)
18: (2070: 0/28, offset: 180224, length: 8192, elemSize: 288)
small subpages:
1: (2048: 0/8, offset: 0, length: 8192, elemSize: 1024)
Chunk(s) at 0~25%:
Chunk(17cf3991: 2%, 172032/16777216)
Chunk(s) at 0~50%:
none
Chunk(s) at 25~75%:
none
Chunk(s) at 50~100%:
none
Chunk(s) at 75~100%:
none
Chunk(s) at 100%:
none
tiny subpages:
5: (2068: 1/102, offset: 163840, length: 8192, elemSize: 80)
13: (2049: 1/39, offset: 8192, length: 8192, elemSize: 208)
16: (2050: 1/32, offset: 16384, length: 8192, elemSize: 256)
small subpages:
1: (2048: 2/8, offset: 0, length: 8192, elemSize: 1024)
Chunk(s) at 0~25%:
Chunk(4564370f: 2%, 180224/16777216)
Chunk(s) at 0~50%:
none
Chunk(s) at 25~75%:
none
Chunk(s) at 50~100%:
none
Chunk(s) at 75~100%:
none
Chunk(s) at 100%:
none
tiny subpages:
5: (2065: 1/102, offset: 139264, length: 8192, elemSize: 80)
13: (2050: 1/39, offset: 16384, length: 8192, elemSize: 208)
14: (2049: 1/36, offset: 8192, length: 8192, elemSize: 224)
16: (2051: 1/32, offset: 24576, length: 8192, elemSize: 256)
small subpages:
1: (2048: 2/8, offset: 0, length: 8192, elemSize: 1024)
Chunk(s) at 0~25%:
Chunk(313da5f2: 2%, 172032/16777216)
Chunk(s) at 0~50%:
none
Chunk(s) at 25~75%:
none
Chunk(s) at 50~100%:
none
Chunk(s) at 75~100%:
none
Chunk(s) at 100%:
none
tiny subpages:
5: (2068: 1/102, offset: 163840, length: 8192, elemSize: 80)
13: (2049: 1/39, offset: 8192, length: 8192, elemSize: 208)
16: (2050: 1/32, offset: 16384, length: 8192, elemSize: 256)
small subpages:
1: (2048: 2/8, offset: 0, length: 8192, elemSize: 1024)
Chunk(s) at 0~25%:
Chunk(13c9d7c3: 2%, 188416/16777216)
Chunk(s) at 0~50%:
none
Chunk(s) at 25~75%:
none
Chunk(s) at 50~100%:
none
Chunk(s) at 75~100%:
none
Chunk(s) at 100%:
none
tiny subpages:
5: (2068: 1/102, offset: 163840, length: 8192, elemSize: 80)
6: (2069: 1/85, offset: 172032, length: 8192, elemSize: 96)
8: (2070: 1/64, offset: 180224, length: 8192, elemSize: 128)
13: (2049: 1/39, offset: 8192, length: 8192, elemSize: 208)
16: (2050: 1/32, offset: 16384, length: 8192, elemSize: 256)
small subpages:
1: (2048: 2/8, offset: 0, length: 8192, elemSize: 1024)
Chunk(s) at 0~25%:
Chunk(7da4779a: 1%, 139264/16777216)
Chunk(s) at 0~50%:
none
Chunk(s) at 25~75%:
none
Chunk(s) at 50~100%:
none
Chunk(s) at 75~100%:
none
Chunk(s) at 100%:
none
tiny subpages:
13: (2050: 1/39, offset: 16384, length: 8192, elemSize: 208)
14: (2049: 1/36, offset: 8192, length: 8192, elemSize: 224)
16: (2051: 1/32, offset: 24576, length: 8192, elemSize: 256)
small subpages:
1: (2048: 2/8, offset: 0, length: 8192, elemSize: 1024)
Chunk(s) at 0~25%:
Chunk(191cfb6e: 2%, 172032/16777216)
Chunk(s) at 0~50%:
none
Chunk(s) at 25~75%:
none
Chunk(s) at 50~100%:
none
Chunk(s) at 75~100%:
none
Chunk(s) at 100%:
none
tiny subpages:
5: (2068: 1/102, offset: 163840, length: 8192, elemSize: 80)
13: (2049: 1/39, offset: 8192, length: 8192, elemSize: 208)
16: (2050: 1/32, offset: 16384, length: 8192, elemSize: 256)
small subpages:
1: (2048: 2/8, offset: 0, length: 8192, elemSize: 1024)
Chunk(s) at 0~25%:
Chunk(4552cf8a: 2%, 172032/16777216)
Chunk(s) at 0~50%:
none
Chunk(s) at 25~75%:
none
Chunk(s) at 50~100%:
none
Chunk(s) at 75~100%:
none
Chunk(s) at 100%:
none
tiny subpages:
5: (2068: 1/102, offset: 163840, length: 8192, elemSize: 80)
13: (2049: 1/39, offset: 8192, length: 8192, elemSize: 208)
16: (2050: 1/32, offset: 16384, length: 8192, elemSize: 256)
small subpages:
1: (2048: 2/8, offset: 0, length: 8192, elemSize: 1024)
Chunk(s) at 0~25%:
Chunk(18f86f2c: 2%, 172032/16777216)
Chunk(s) at 0~50%:
none
Chunk(s) at 25~75%:
none
Chunk(s) at 50~100%:
none
Chunk(s) at 75~100%:
none
Chunk(s) at 100%:
none
tiny subpages:
5: (2068: 1/102, offset: 163840, length: 8192, elemSize: 80)
13: (2049: 1/39, offset: 8192, length: 8192, elemSize: 208)
16: (2050: 1/32, offset: 16384, length: 8192, elemSize: 256)
small subpages:
1: (2048: 2/8, offset: 0, length: 8192, elemSize: 1024)
Chunk(s) at 0~25%:
Chunk(463ed30e: 2%, 180224/16777216)
Chunk(s) at 0~50%:
none
Chunk(s) at 25~75%:
none
Chunk(s) at 50~100%:
none
Chunk(s) at 75~100%:
none
Chunk(s) at 100%:
none
tiny subpages:
5: (2069: 1/102, offset: 172032, length: 8192, elemSize: 80)
8: (2068: 1/64, offset: 163840, length: 8192, elemSize: 128)
13: (2049: 1/39, offset: 8192, length: 8192, elemSize: 208)
16: (2050: 1/32, offset: 16384, length: 8192, elemSize: 256)
small subpages:
1: (2048: 2/8, offset: 0, length: 8192, elemSize: 1024)
Chunk(s) at 0~25%:
Chunk(3887872c: 1%, 147456/16777216)
Chunk(s) at 0~50%:
none
Chunk(s) at 25~75%:
none
Chunk(s) at 50~100%:
none
Chunk(s) at 75~100%:
none
Chunk(s) at 100%:
none
tiny subpages:
13: (2050: 1/39, offset: 16384, length: 8192, elemSize: 208)
14: (2049: 1/36, offset: 8192, length: 8192, elemSize: 224)
16: (2051: 1/32, offset: 24576, length: 8192, elemSize: 256)
31: (2065: 1/16, offset: 139264, length: 8192, elemSize: 496)
small subpages:
1: (2048: 2/8, offset: 0, length: 8192, elemSize: 1024)
Chunk(s) at 0~25%:
Chunk(6bfa2129: 2%, 172032/16777216)
Chunk(s) at 0~50%:
none
Chunk(s) at 25~75%:
none
Chunk(s) at 50~100%:
none
Chunk(s) at 75~100%:
none
Chunk(s) at 100%:
none
tiny subpages:
5: (2068: 1/102, offset: 163840, length: 8192, elemSize: 80)
13: (2049: 1/39, offset: 8192, length: 8192, elemSize: 208)
16: (2050: 1/32, offset: 16384, length: 8192, elemSize: 256)
small subpages:
1: (2048: 2/8, offset: 0, length: 8192, elemSize: 1024)
Chunk(s) at 0~25%:
none
Chunk(s) at 0~50%:
none
Chunk(s) at 25~75%:
none
Chunk(s) at 50~100%:
none
Chunk(s) at 75~100%:
none
Chunk(s) at 100%:
none
tiny subpages:
small subpages:
Chunk(s) at 0~25%:
none
Chunk(s) at 0~50%:
none
Chunk(s) at 25~75%:
none
Chunk(s) at 50~100%:
none
Chunk(s) at 75~100%:
none
Chunk(s) at 100%:
none
tiny subpages:
small subpages:
Chunk(s) at 0~25%:
none
Chunk(s) at 0~50%:
none
Chunk(s) at 25~75%:
none
Chunk(s) at 50~100%:
none
Chunk(s) at 75~100%:
none
Chunk(s) at 100%:
none
tiny subpages:
small subpages:
Chunk(s) at 0~25%:
none
Chunk(s) at 0~50%:
none
Chunk(s) at 25~75%:
none
Chunk(s) at 50~100%:
none
Chunk(s) at 75~100%:
none
Chunk(s) at 100%:
none
tiny subpages:
small subpages:
Chunk(s) at 0~25%:
none
Chunk(s) at 0~50%:
none
Chunk(s) at 25~75%:
none
Chunk(s) at 50~100%:
none
Chunk(s) at 75~100%:
none
Chunk(s) at 100%:
none
tiny subpages:
small subpages:
Chunk(s) at 0~25%:
none
Chunk(s) at 0~50%:
none
Chunk(s) at 25~75%:
none
Chunk(s) at 50~100%:
none
Chunk(s) at 75~100%:
none
Chunk(s) at 100%:
none
tiny subpages:
small subpages:
Chunk(s) at 0~25%:
none
Chunk(s) at 0~50%:
none
Chunk(s) at 25~75%:
none
Chunk(s) at 50~100%:
none
Chunk(s) at 75~100%:
none
Chunk(s) at 100%:
none
tiny subpages:
small subpages:
Chunk(s) at 0~25%:
none
Chunk(s) at 0~50%:
none
Chunk(s) at 25~75%:
none
Chunk(s) at 50~100%:
none
Chunk(s) at 75~100%:
none
Chunk(s) at 100%:
none
tiny subpages:
small subpages:
Chunk(s) at 0~25%:
none
Chunk(s) at 0~50%:
none
Chunk(s) at 25~75%:
none
Chunk(s) at 50~100%:
none
Chunk(s) at 75~100%:
none
Chunk(s) at 100%:
none
tiny subpages:
small subpages:
Chunk(s) at 0~25%:
none
Chunk(s) at 0~50%:
none
Chunk(s) at 25~75%:
none
Chunk(s) at 50~100%:
none
Chunk(s) at 75~100%:
none
Chunk(s) at 100%:
none
tiny subpages:
small subpages:
Chunk(s) at 0~25%:
none
Chunk(s) at 0~50%:
none
Chunk(s) at 25~75%:
none
Chunk(s) at 50~100%:
none
Chunk(s) at 75~100%:
none
Chunk(s) at 100%:
none
tiny subpages:
small subpages:
Chunk(s) at 0~25%:
none
Chunk(s) at 0~50%:
none
Chunk(s) at 25~75%:
none
Chunk(s) at 50~100%:
none
Chunk(s) at 75~100%:
none
Chunk(s) at 100%:
none
tiny subpages:
small subpages:
Chunk(s) at 0~25%:
none
Chunk(s) at 0~50%:
none
Chunk(s) at 25~75%:
none
Chunk(s) at 50~100%:
none
Chunk(s) at 75~100%:
none
Chunk(s) at 100%:
none
tiny subpages:
small subpages:
Chunk(s) at 0~25%:
none
Chunk(s) at 0~50%:
none
Chunk(s) at 25~75%:
none
Chunk(s) at 50~100%:
none
Chunk(s) at 75~100%:
none
Chunk(s) at 100%:
none
tiny subpages:
small subpages:
Chunk(s) at 0~25%:
none
Chunk(s) at 0~50%:
none
Chunk(s) at 25~75%:
none
Chunk(s) at 50~100%:
none
Chunk(s) at 75~100%:
none
Chunk(s) at 100%:
none
tiny subpages:
small subpages:
Chunk(s) at 0~25%:
none
Chunk(s) at 0~50%:
none
Chunk(s) at 25~75%:
none
Chunk(s) at 50~100%:
none
Chunk(s) at 75~100%:
none
Chunk(s) at 100%:
none
tiny subpages:
small subpages:
io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 16777216
byte(s) of direct memory (used: 4112515079, max: 4127195136)
at
io.netty.util.internal.PlatformDependent.incrementMemoryCounter(PlatformDependent.java:592)
at
io.netty.util.internal.PlatformDependent.allocateDirectNoCleaner(PlatformDependent.java:546)
at
io.netty.buffer.PoolArena$DirectArena.allocateDirect(PoolArena.java:699)
at io.netty.buffer.PoolArena$DirectArena.newChunk(PoolArena.java:688)
at io.netty.buffer.PoolArena.allocateNormal(PoolArena.java:237)
at io.netty.buffer.PoolArena.allocate(PoolArena.java:221)
at io.netty.buffer.PoolArena.allocate(PoolArena.java:141)
at
io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:262)
at
io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:179)
at
io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:170)
at
io.netty.buffer.AbstractByteBufAllocator.buffer(AbstractByteBufAllocator.java:107)
at io.netty.handler.ssl.SslHandler.allocate(SslHandler.java:1461)
at
io.netty.handler.ssl.SslHandler.allocateOutNetBuf(SslHandler.java:1471)
at io.netty.handler.ssl.SslHandler.wrap(SslHandler.java:523)
at io.netty.handler.ssl.SslHandler.flush(SslHandler.java:501)
at
io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:762)
at
io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:754)
at
io.netty.channel.AbstractChannelHandlerContext.flush(AbstractChannelHandlerContext.java:735)
at
io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.flush(CombinedChannelDuplexHandler.java:523)
at
io.netty.channel.ChannelOutboundHandlerAdapter.flush(ChannelOutboundHandlerAdapter.java:115)
at
io.netty.channel.CombinedChannelDuplexHandler.flush(CombinedChannelDuplexHandler.java:348)
at
io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:762)
at
io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:754)
at
io.netty.channel.AbstractChannelHandlerContext.flush(AbstractChannelHandlerContext.java:735)
at
io.netty.channel.ChannelOutboundHandlerAdapter.flush(ChannelOutboundHandlerAdapter.java:115)
at
io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:762)
at
io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:754)
at
io.netty.channel.AbstractChannelHandlerContext.flush(AbstractChannelHandlerContext.java:735)
at
io.netty.channel.ChannelDuplexHandler.flush(ChannelDuplexHandler.java:117)
at
io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:762)
at
io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:754)
at
io.netty.channel.AbstractChannelHandlerContext.flush(AbstractChannelHandlerContext.java:735)
at
io.netty.channel.ChannelDuplexHandler.flush(ChannelDuplexHandler.java:117)
at
io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:762)
at
io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:754)
at
io.netty.channel.AbstractChannelHandlerContext.flush(AbstractChannelHandlerContext.java:735)
at
io.netty.channel.ChannelOutboundHandlerAdapter.flush(ChannelOutboundHandlerAdapter.java:115)
