Ralf Steppacher created CAMEL-9040: -------------------------------------- Summary: netty4-http - LEAK: ByteBuf.release() was not called before it's garbage-collected Key: CAMEL-9040 URL: https://issues.apache.org/jira/browse/CAMEL-9040 Project: Camel Issue Type: Bug Components: camel-netty4-http Affects Versions: 2.15.2 Reporter: Ralf Steppacher
In a reverse-proxy with multiple routes that utilize the netty4-http component both as the consumer and producer I am receiving the following error: {noformat} 2015-07-30 11:52:34,416 | ERROR | yServerTCPWorker | ResourceLeakDetector | 97 - io.netty.common - 4.0.27.Final | | LEAK: ByteBuf.release() was not called before it's garbage-collected. See http://netty.io/wiki/reference-counted-objects.html for more information. {noformat} Setting the Netty leak reporting to paranoid ({{-Dio.netty.leakDetectionLevel=paranoid}}) yields an error for almost every request (it probably is 1:1). The stacktraces come in two variants. Variant 1 is far less frequent than variant 2. Variant 1: {noformat} Recent access records: 0 Created at: io.netty.buffer.CompositeByteBuf.<init>(CompositeByteBuf.java:60) io.netty.buffer.Unpooled.compositeBuffer(Unpooled.java:353) io.netty.handler.codec.http.HttpObjectAggregator.decode(HttpObjectAggregator.java:153) io.netty.handler.codec.http.HttpObjectAggregator.decode(HttpObjectAggregator.java:54) io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:89) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339) io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324) io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:242) io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:147) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339) io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324) io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:847) io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131) io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511) io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468) io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354) io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111) java.lang.Thread.run(Thread.java:745) {noformat} Variant 2: {noformat} Recent access records: 5 #5: io.netty.buffer.AdvancedLeakAwareByteBuf.getBytes(AdvancedLeakAwareByteBuf.java:223) io.netty.buffer.CompositeByteBuf.getBytes(CompositeByteBuf.java:684) io.netty.buffer.CompositeByteBuf.getBytes(CompositeByteBuf.java:40) io.netty.buffer.AbstractByteBuf.readBytes(AbstractByteBuf.java:677) io.netty.buffer.CompositeByteBuf.readBytes(CompositeByteBuf.java:1493) io.netty.buffer.CompositeByteBuf.readBytes(CompositeByteBuf.java:40) io.netty.buffer.ByteBufInputStream.read(ByteBufInputStream.java:120) java.io.BufferedInputStream.fill(BufferedInputStream.java:235) java.io.BufferedInputStream.read1(BufferedInputStream.java:275) java.io.BufferedInputStream.read(BufferedInputStream.java:334) org.bouncycastle.util.io.TeeInputStream.read(Unknown Source) com.ctc.wstx.io.BaseReader.readBytes(BaseReader.java:155) com.ctc.wstx.io.UTF8Reader.loadMore(UTF8Reader.java:368) com.ctc.wstx.io.UTF8Reader.read(UTF8Reader.java:111) com.ctc.wstx.io.ReaderBootstrapper.initialLoad(ReaderBootstrapper.java:250) com.ctc.wstx.io.ReaderBootstrapper.bootstrapInput(ReaderBootstrapper.java:133) com.ctc.wstx.stax.WstxInputFactory.doCreateSR(WstxInputFactory.java:545) com.ctc.wstx.stax.WstxInputFactory.createSR(WstxInputFactory.java:605) com.ctc.wstx.stax.WstxInputFactory.createSR(WstxInputFactory.java:628) com.ctc.wstx.stax.WstxInputFactory.createXMLStreamReader(WstxInputFactory.java:331) ch.vivates.pep.stream.ResponseStatusFilter.filter(ResponseStatusFilter.java:41) ch.vivates.pep.stream.BaseStreamFilter.run(BaseStreamFilter.java:141) java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) java.util.concurrent.FutureTask.run(FutureTask.java:262) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) #4: io.netty.buffer.AdvancedLeakAwareByteBuf.release(AdvancedLeakAwareByteBuf.java:45) io.netty.handler.codec.http.DefaultHttpContent.release(DefaultHttpContent.java:72) io.netty.util.ReferenceCountUtil.release(ReferenceCountUtil.java:59) io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:91) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339) io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324) io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:242) io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:147) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339) io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324) io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:847) io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131) io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511) io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468) io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354) io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111) java.lang.Thread.run(Thread.java:745) #3: io.netty.buffer.AdvancedLeakAwareByteBuf.slice(AdvancedLeakAwareByteBuf.java:73) io.netty.buffer.CompositeByteBuf.addComponent0(CompositeByteBuf.java:170) io.netty.buffer.CompositeByteBuf.addComponent(CompositeByteBuf.java:112) io.netty.handler.codec.http.HttpObjectAggregator.decode(HttpObjectAggregator.java:191) io.netty.handler.codec.http.HttpObjectAggregator.decode(HttpObjectAggregator.java:54) io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:89) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339) io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324) io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:242) io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:147) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339) io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324) io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:847) io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131) io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511) io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468) io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354) io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111) java.lang.Thread.run(Thread.java:745) #2: io.netty.buffer.AdvancedLeakAwareByteBuf.order(AdvancedLeakAwareByteBuf.java:63) io.netty.buffer.CompositeByteBuf.addComponent0(CompositeByteBuf.java:170) io.netty.buffer.CompositeByteBuf.addComponent(CompositeByteBuf.java:112) io.netty.handler.codec.http.HttpObjectAggregator.decode(HttpObjectAggregator.java:191) io.netty.handler.codec.http.HttpObjectAggregator.decode(HttpObjectAggregator.java:54) io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:89) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339) io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324) io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:242) io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:147) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339) io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324) io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:847) io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131) io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511) io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468) io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354) io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111) java.lang.Thread.run(Thread.java:745) #1: io.netty.buffer.AdvancedLeakAwareByteBuf.retain(AdvancedLeakAwareByteBuf.java:709) io.netty.handler.codec.http.DefaultHttpContent.retain(DefaultHttpContent.java:60) io.netty.handler.codec.http.HttpObjectAggregator.decode(HttpObjectAggregator.java:190) io.netty.handler.codec.http.HttpObjectAggregator.decode(HttpObjectAggregator.java:54) io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:89) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339) io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324) io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:242) io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:147) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339) io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324) io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:847) io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131) io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511) io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468) io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354) io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111) java.lang.Thread.run(Thread.java:745) Created at: io.netty.buffer.UnpooledByteBufAllocator.newDirectBuffer(UnpooledByteBufAllocator.java:55) io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:155) io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:146) io.netty.buffer.AbstractByteBufAllocator.ioBuffer(AbstractByteBufAllocator.java:107) io.netty.channel.AdaptiveRecvByteBufAllocator$HandleImpl.allocate(AdaptiveRecvByteBufAllocator.java:104) io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:117) io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511) io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468) io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354) io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111) java.lang.Thread.run(Thread.java:745) {noformat} Variant 2 creates the buffer in {{UnpooledByteBufAllocator}}. Which probably means that there is no real resource leak. Rather the coding pattern required for objects that might be pooled is not adhered to. This is a sample route that triggers the error: {code:xml} <route id="pep-xds_b-registry"> <from uri="netty4-http:http://{{pep.host}}:{{pep.xds.b.registry.port}}/XDS3/registry?matchOnUriPrefix=true&disableStreamCache=true" /> <process ref="registryInboundProcessor" /> <to uri="netty4-http:http://{{xds.b.registry.host}}:{{xds.b.registry.port}}/XDS3/registry?bridgeEndpoint=true&throwExceptionOnFailure=false&chunkedMaxContentLength={{pep.xds.b.registry.max.message.size}}&disableStreamCache=false" /> <!-- Unfortunately disabling the stream cache only works for requests, but not responses: https://issues.apache.org/jira/browse/CAMEL-7638 --> <bean ref="mdcUpdateProcessor" method="restoreBreadcrumbFromProperty" /> <process ref="registryOutboundProcessor" /> </route> {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)