[ https://issues.apache.org/jira/browse/CAMEL-9040?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ralf Steppacher updated CAMEL-9040: ----------------------------------- Description: In a reverse-proxy with multiple routes that utilize the netty4-http component both as the consumer and producer I am receiving the following error: {noformat} 2015-07-30 11:52:34,416 | ERROR | yServerTCPWorker | ResourceLeakDetector | 97 - io.netty.common - 4.0.27.Final | | LEAK: ByteBuf.release() was not called before it's garbage-collected. See http://netty.io/wiki/reference-counted-objects.html for more information. {noformat} Setting the Netty leak reporting to paranoid ({{-Dio.netty.leakDetectionLevel=paranoid}}) yields an error for almost every request (it probably is 1:1). The stacktraces come in two variants. Variant 1 is far less frequent than variant 2. Variant 1: {noformat} Recent access records: 0 Created at: io.netty.buffer.CompositeByteBuf.<init>(CompositeByteBuf.java:60) io.netty.buffer.Unpooled.compositeBuffer(Unpooled.java:353) io.netty.handler.codec.http.HttpObjectAggregator.decode(HttpObjectAggregator.java:153) io.netty.handler.codec.http.HttpObjectAggregator.decode(HttpObjectAggregator.java:54) io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:89) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339) io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324) io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:242) io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:147) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339) io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324) io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:847) io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131) io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511) io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468) io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354) io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111) java.lang.Thread.run(Thread.java:745) {noformat} Variant 2: {noformat} Recent access records: 5 #5: io.netty.buffer.AdvancedLeakAwareByteBuf.getBytes(AdvancedLeakAwareByteBuf.java:223) io.netty.buffer.CompositeByteBuf.getBytes(CompositeByteBuf.java:684) io.netty.buffer.CompositeByteBuf.getBytes(CompositeByteBuf.java:40) io.netty.buffer.AbstractByteBuf.readBytes(AbstractByteBuf.java:677) io.netty.buffer.CompositeByteBuf.readBytes(CompositeByteBuf.java:1493) io.netty.buffer.CompositeByteBuf.readBytes(CompositeByteBuf.java:40) io.netty.buffer.ByteBufInputStream.read(ByteBufInputStream.java:120) java.io.BufferedInputStream.fill(BufferedInputStream.java:235) java.io.BufferedInputStream.read1(BufferedInputStream.java:275) java.io.BufferedInputStream.read(BufferedInputStream.java:334) org.bouncycastle.util.io.TeeInputStream.read(Unknown Source) com.ctc.wstx.io.BaseReader.readBytes(BaseReader.java:155) com.ctc.wstx.io.UTF8Reader.loadMore(UTF8Reader.java:368) com.ctc.wstx.io.UTF8Reader.read(UTF8Reader.java:111) com.ctc.wstx.io.ReaderBootstrapper.initialLoad(ReaderBootstrapper.java:250) com.ctc.wstx.io.ReaderBootstrapper.bootstrapInput(ReaderBootstrapper.java:133) com.ctc.wstx.stax.WstxInputFactory.doCreateSR(WstxInputFactory.java:545) com.ctc.wstx.stax.WstxInputFactory.createSR(WstxInputFactory.java:605) com.ctc.wstx.stax.WstxInputFactory.createSR(WstxInputFactory.java:628) com.ctc.wstx.stax.WstxInputFactory.createXMLStreamReader(WstxInputFactory.java:331) ch.vivates.pep.stream.ResponseStatusFilter.filter(ResponseStatusFilter.java:41) ch.vivates.pep.stream.BaseStreamFilter.run(BaseStreamFilter.java:141) java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) java.util.concurrent.FutureTask.run(FutureTask.java:262) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) #4: io.netty.buffer.AdvancedLeakAwareByteBuf.release(AdvancedLeakAwareByteBuf.java:45) io.netty.handler.codec.http.DefaultHttpContent.release(DefaultHttpContent.java:72) io.netty.util.ReferenceCountUtil.release(ReferenceCountUtil.java:59) io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:91) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339) io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324) io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:242) io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:147) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339) io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324) io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:847) io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131) io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511) io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468) io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354) io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111) java.lang.Thread.run(Thread.java:745) #3: io.netty.buffer.AdvancedLeakAwareByteBuf.slice(AdvancedLeakAwareByteBuf.java:73) io.netty.buffer.CompositeByteBuf.addComponent0(CompositeByteBuf.java:170) io.netty.buffer.CompositeByteBuf.addComponent(CompositeByteBuf.java:112) io.netty.handler.codec.http.HttpObjectAggregator.decode(HttpObjectAggregator.java:191) io.netty.handler.codec.http.HttpObjectAggregator.decode(HttpObjectAggregator.java:54) io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:89) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339) io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324) io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:242) io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:147) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339) io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324) io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:847) io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131) io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511) io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468) io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354) io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111) java.lang.Thread.run(Thread.java:745) #2: io.netty.buffer.AdvancedLeakAwareByteBuf.order(AdvancedLeakAwareByteBuf.java:63) io.netty.buffer.CompositeByteBuf.addComponent0(CompositeByteBuf.java:170) io.netty.buffer.CompositeByteBuf.addComponent(CompositeByteBuf.java:112) io.netty.handler.codec.http.HttpObjectAggregator.decode(HttpObjectAggregator.java:191) io.netty.handler.codec.http.HttpObjectAggregator.decode(HttpObjectAggregator.java:54) io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:89) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339) io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324) io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:242) io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:147) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339) io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324) io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:847) io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131) io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511) io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468) io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354) io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111) java.lang.Thread.run(Thread.java:745) #1: io.netty.buffer.AdvancedLeakAwareByteBuf.retain(AdvancedLeakAwareByteBuf.java:709) io.netty.handler.codec.http.DefaultHttpContent.retain(DefaultHttpContent.java:60) io.netty.handler.codec.http.HttpObjectAggregator.decode(HttpObjectAggregator.java:190) io.netty.handler.codec.http.HttpObjectAggregator.decode(HttpObjectAggregator.java:54) io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:89) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339) io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324) io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:242) io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:147) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339) io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324) io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:847) io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131) io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511) io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468) io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354) io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111) java.lang.Thread.run(Thread.java:745) Created at: io.netty.buffer.UnpooledByteBufAllocator.newDirectBuffer(UnpooledByteBufAllocator.java:55) io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:155) io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:146) io.netty.buffer.AbstractByteBufAllocator.ioBuffer(AbstractByteBufAllocator.java:107) io.netty.channel.AdaptiveRecvByteBufAllocator$HandleImpl.allocate(AdaptiveRecvByteBufAllocator.java:104) io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:117) io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511) io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468) io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354) io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111) java.lang.Thread.run(Thread.java:745) {noformat} Variant 2 creates the buffer in {{UnpooledByteBufAllocator}}. Which probably means that there is no real resource leak. Rather the coding pattern required for objects that might be pooled is not adhered to. This is a sample route that triggers the error: {code:xml} <route id="pep-xds_b-registry"> <from uri="netty4-http:http://{{pep.host}}:{{pep.xds.b.registry.port}}/XDS3/registry?matchOnUriPrefix=true&disableStreamCache=true" /> <process ref="registryInboundProcessor" /> <to uri="netty4-http:http://{{xds.b.registry.host}}:{{xds.b.registry.port}}/XDS3/registry?bridgeEndpoint=true&throwExceptionOnFailure=false&chunkedMaxContentLength={{pep.xds.b.registry.max.message.size}}&disableStreamCache=false" /> <!-- Unfortunately disabling the stream cache only works for requests, but not responses: https://issues.apache.org/jira/browse/CAMEL-7638 --> <process ref="registryOutboundProcessor" /> </route> {code} was: In a reverse-proxy with multiple routes that utilize the netty4-http component both as the consumer and producer I am receiving the following error: {noformat} 2015-07-30 11:52:34,416 | ERROR | yServerTCPWorker | ResourceLeakDetector | 97 - io.netty.common - 4.0.27.Final | | LEAK: ByteBuf.release() was not called before it's garbage-collected. See http://netty.io/wiki/reference-counted-objects.html for more information. {noformat} Setting the Netty leak reporting to paranoid ({{-Dio.netty.leakDetectionLevel=paranoid}}) yields an error for almost every request (it probably is 1:1). The stacktraces come in two variants. Variant 1 is far less frequent than variant 2. Variant 1: {noformat} Recent access records: 0 Created at: io.netty.buffer.CompositeByteBuf.<init>(CompositeByteBuf.java:60) io.netty.buffer.Unpooled.compositeBuffer(Unpooled.java:353) io.netty.handler.codec.http.HttpObjectAggregator.decode(HttpObjectAggregator.java:153) io.netty.handler.codec.http.HttpObjectAggregator.decode(HttpObjectAggregator.java:54) io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:89) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339) io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324) io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:242) io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:147) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339) io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324) io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:847) io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131) io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511) io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468) io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354) io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111) java.lang.Thread.run(Thread.java:745) {noformat} Variant 2: {noformat} Recent access records: 5 #5: io.netty.buffer.AdvancedLeakAwareByteBuf.getBytes(AdvancedLeakAwareByteBuf.java:223) io.netty.buffer.CompositeByteBuf.getBytes(CompositeByteBuf.java:684) io.netty.buffer.CompositeByteBuf.getBytes(CompositeByteBuf.java:40) io.netty.buffer.AbstractByteBuf.readBytes(AbstractByteBuf.java:677) io.netty.buffer.CompositeByteBuf.readBytes(CompositeByteBuf.java:1493) io.netty.buffer.CompositeByteBuf.readBytes(CompositeByteBuf.java:40) io.netty.buffer.ByteBufInputStream.read(ByteBufInputStream.java:120) java.io.BufferedInputStream.fill(BufferedInputStream.java:235) java.io.BufferedInputStream.read1(BufferedInputStream.java:275) java.io.BufferedInputStream.read(BufferedInputStream.java:334) org.bouncycastle.util.io.TeeInputStream.read(Unknown Source) com.ctc.wstx.io.BaseReader.readBytes(BaseReader.java:155) com.ctc.wstx.io.UTF8Reader.loadMore(UTF8Reader.java:368) com.ctc.wstx.io.UTF8Reader.read(UTF8Reader.java:111) com.ctc.wstx.io.ReaderBootstrapper.initialLoad(ReaderBootstrapper.java:250) com.ctc.wstx.io.ReaderBootstrapper.bootstrapInput(ReaderBootstrapper.java:133) com.ctc.wstx.stax.WstxInputFactory.doCreateSR(WstxInputFactory.java:545) com.ctc.wstx.stax.WstxInputFactory.createSR(WstxInputFactory.java:605) com.ctc.wstx.stax.WstxInputFactory.createSR(WstxInputFactory.java:628) com.ctc.wstx.stax.WstxInputFactory.createXMLStreamReader(WstxInputFactory.java:331) ch.vivates.pep.stream.ResponseStatusFilter.filter(ResponseStatusFilter.java:41) ch.vivates.pep.stream.BaseStreamFilter.run(BaseStreamFilter.java:141) java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) java.util.concurrent.FutureTask.run(FutureTask.java:262) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) #4: io.netty.buffer.AdvancedLeakAwareByteBuf.release(AdvancedLeakAwareByteBuf.java:45) io.netty.handler.codec.http.DefaultHttpContent.release(DefaultHttpContent.java:72) io.netty.util.ReferenceCountUtil.release(ReferenceCountUtil.java:59) io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:91) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339) io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324) io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:242) io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:147) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339) io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324) io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:847) io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131) io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511) io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468) io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354) io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111) java.lang.Thread.run(Thread.java:745) #3: io.netty.buffer.AdvancedLeakAwareByteBuf.slice(AdvancedLeakAwareByteBuf.java:73) io.netty.buffer.CompositeByteBuf.addComponent0(CompositeByteBuf.java:170) io.netty.buffer.CompositeByteBuf.addComponent(CompositeByteBuf.java:112) io.netty.handler.codec.http.HttpObjectAggregator.decode(HttpObjectAggregator.java:191) io.netty.handler.codec.http.HttpObjectAggregator.decode(HttpObjectAggregator.java:54) io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:89) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339) io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324) io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:242) io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:147) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339) io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324) io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:847) io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131) io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511) io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468) io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354) io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111) java.lang.Thread.run(Thread.java:745) #2: io.netty.buffer.AdvancedLeakAwareByteBuf.order(AdvancedLeakAwareByteBuf.java:63) io.netty.buffer.CompositeByteBuf.addComponent0(CompositeByteBuf.java:170) io.netty.buffer.CompositeByteBuf.addComponent(CompositeByteBuf.java:112) io.netty.handler.codec.http.HttpObjectAggregator.decode(HttpObjectAggregator.java:191) io.netty.handler.codec.http.HttpObjectAggregator.decode(HttpObjectAggregator.java:54) io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:89) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339) io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324) io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:242) io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:147) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339) io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324) io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:847) io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131) io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511) io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468) io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354) io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111) java.lang.Thread.run(Thread.java:745) #1: io.netty.buffer.AdvancedLeakAwareByteBuf.retain(AdvancedLeakAwareByteBuf.java:709) io.netty.handler.codec.http.DefaultHttpContent.retain(DefaultHttpContent.java:60) io.netty.handler.codec.http.HttpObjectAggregator.decode(HttpObjectAggregator.java:190) io.netty.handler.codec.http.HttpObjectAggregator.decode(HttpObjectAggregator.java:54) io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:89) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339) io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324) io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:242) io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:147) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339) io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324) io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:847) io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131) io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511) io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468) io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354) io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111) java.lang.Thread.run(Thread.java:745) Created at: io.netty.buffer.UnpooledByteBufAllocator.newDirectBuffer(UnpooledByteBufAllocator.java:55) io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:155) io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:146) io.netty.buffer.AbstractByteBufAllocator.ioBuffer(AbstractByteBufAllocator.java:107) io.netty.channel.AdaptiveRecvByteBufAllocator$HandleImpl.allocate(AdaptiveRecvByteBufAllocator.java:104) io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:117) io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511) io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468) io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354) io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111) java.lang.Thread.run(Thread.java:745) {noformat} Variant 2 creates the buffer in {{UnpooledByteBufAllocator}}. Which probably means that there is no real resource leak. Rather the coding pattern required for objects that might be pooled is not adhered to. This is a sample route that triggers the error: {code:xml} <route id="pep-xds_b-registry"> <from uri="netty4-http:http://{{pep.host}}:{{pep.xds.b.registry.port}}/XDS3/registry?matchOnUriPrefix=true&disableStreamCache=true" /> <process ref="registryInboundProcessor" /> <to uri="netty4-http:http://{{xds.b.registry.host}}:{{xds.b.registry.port}}/XDS3/registry?bridgeEndpoint=true&throwExceptionOnFailure=false&chunkedMaxContentLength={{pep.xds.b.registry.max.message.size}}&disableStreamCache=false" /> <!-- Unfortunately disabling the stream cache only works for requests, but not responses: https://issues.apache.org/jira/browse/CAMEL-7638 --> <bean ref="mdcUpdateProcessor" method="restoreBreadcrumbFromProperty" /> <process ref="registryOutboundProcessor" /> </route> {code} > netty4-http - LEAK: ByteBuf.release() was not called before it's > garbage-collected > ---------------------------------------------------------------------------------- > > Key: CAMEL-9040 > URL: https://issues.apache.org/jira/browse/CAMEL-9040 > Project: Camel > Issue Type: Bug > Components: camel-netty4-http > Affects Versions: 2.15.2 > Reporter: Ralf Steppacher > > In a reverse-proxy with multiple routes that utilize the netty4-http > component both as the consumer and producer I am receiving the following > error: > {noformat} > 2015-07-30 11:52:34,416 | ERROR | yServerTCPWorker | ResourceLeakDetector > | 97 - io.netty.common - 4.0.27.Final | | LEAK: ByteBuf.release() > was not called before it's garbage-collected. See > http://netty.io/wiki/reference-counted-objects.html for more information. > {noformat} > Setting the Netty leak reporting to paranoid > ({{-Dio.netty.leakDetectionLevel=paranoid}}) yields an error for almost every > request (it probably is 1:1). > The stacktraces come in two variants. Variant 1 is far less frequent than > variant 2. > Variant 1: > {noformat} > Recent access records: 0 > Created at: > io.netty.buffer.CompositeByteBuf.<init>(CompositeByteBuf.java:60) > io.netty.buffer.Unpooled.compositeBuffer(Unpooled.java:353) > io.netty.handler.codec.http.HttpObjectAggregator.decode(HttpObjectAggregator.java:153) > io.netty.handler.codec.http.HttpObjectAggregator.decode(HttpObjectAggregator.java:54) > io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:89) > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339) > io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324) > io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:242) > io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:147) > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339) > io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324) > io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:847) > io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131) > io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511) > io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468) > io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382) > io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354) > io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111) > java.lang.Thread.run(Thread.java:745) > {noformat} > Variant 2: > {noformat} > Recent access records: 5 > #5: > io.netty.buffer.AdvancedLeakAwareByteBuf.getBytes(AdvancedLeakAwareByteBuf.java:223) > io.netty.buffer.CompositeByteBuf.getBytes(CompositeByteBuf.java:684) > io.netty.buffer.CompositeByteBuf.getBytes(CompositeByteBuf.java:40) > io.netty.buffer.AbstractByteBuf.readBytes(AbstractByteBuf.java:677) > io.netty.buffer.CompositeByteBuf.readBytes(CompositeByteBuf.java:1493) > io.netty.buffer.CompositeByteBuf.readBytes(CompositeByteBuf.java:40) > io.netty.buffer.ByteBufInputStream.read(ByteBufInputStream.java:120) > java.io.BufferedInputStream.fill(BufferedInputStream.java:235) > java.io.BufferedInputStream.read1(BufferedInputStream.java:275) > java.io.BufferedInputStream.read(BufferedInputStream.java:334) > org.bouncycastle.util.io.TeeInputStream.read(Unknown Source) > com.ctc.wstx.io.BaseReader.readBytes(BaseReader.java:155) > com.ctc.wstx.io.UTF8Reader.loadMore(UTF8Reader.java:368) > com.ctc.wstx.io.UTF8Reader.read(UTF8Reader.java:111) > com.ctc.wstx.io.ReaderBootstrapper.initialLoad(ReaderBootstrapper.java:250) > com.ctc.wstx.io.ReaderBootstrapper.bootstrapInput(ReaderBootstrapper.java:133) > com.ctc.wstx.stax.WstxInputFactory.doCreateSR(WstxInputFactory.java:545) > com.ctc.wstx.stax.WstxInputFactory.createSR(WstxInputFactory.java:605) > com.ctc.wstx.stax.WstxInputFactory.createSR(WstxInputFactory.java:628) > com.ctc.wstx.stax.WstxInputFactory.createXMLStreamReader(WstxInputFactory.java:331) > ch.vivates.pep.stream.ResponseStatusFilter.filter(ResponseStatusFilter.java:41) > ch.vivates.pep.stream.BaseStreamFilter.run(BaseStreamFilter.java:141) > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > java.util.concurrent.FutureTask.run(FutureTask.java:262) > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > java.lang.Thread.run(Thread.java:745) > #4: > io.netty.buffer.AdvancedLeakAwareByteBuf.release(AdvancedLeakAwareByteBuf.java:45) > io.netty.handler.codec.http.DefaultHttpContent.release(DefaultHttpContent.java:72) > io.netty.util.ReferenceCountUtil.release(ReferenceCountUtil.java:59) > io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:91) > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339) > io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324) > io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:242) > io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:147) > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339) > io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324) > io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:847) > io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131) > io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511) > io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468) > io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382) > io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354) > io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111) > java.lang.Thread.run(Thread.java:745) > #3: > io.netty.buffer.AdvancedLeakAwareByteBuf.slice(AdvancedLeakAwareByteBuf.java:73) > io.netty.buffer.CompositeByteBuf.addComponent0(CompositeByteBuf.java:170) > io.netty.buffer.CompositeByteBuf.addComponent(CompositeByteBuf.java:112) > io.netty.handler.codec.http.HttpObjectAggregator.decode(HttpObjectAggregator.java:191) > io.netty.handler.codec.http.HttpObjectAggregator.decode(HttpObjectAggregator.java:54) > io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:89) > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339) > io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324) > io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:242) > io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:147) > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339) > io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324) > io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:847) > io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131) > io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511) > io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468) > io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382) > io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354) > io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111) > java.lang.Thread.run(Thread.java:745) > #2: > io.netty.buffer.AdvancedLeakAwareByteBuf.order(AdvancedLeakAwareByteBuf.java:63) > io.netty.buffer.CompositeByteBuf.addComponent0(CompositeByteBuf.java:170) > io.netty.buffer.CompositeByteBuf.addComponent(CompositeByteBuf.java:112) > io.netty.handler.codec.http.HttpObjectAggregator.decode(HttpObjectAggregator.java:191) > io.netty.handler.codec.http.HttpObjectAggregator.decode(HttpObjectAggregator.java:54) > io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:89) > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339) > io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324) > io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:242) > io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:147) > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339) > io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324) > io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:847) > io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131) > io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511) > io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468) > io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382) > io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354) > io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111) > java.lang.Thread.run(Thread.java:745) > #1: > io.netty.buffer.AdvancedLeakAwareByteBuf.retain(AdvancedLeakAwareByteBuf.java:709) > io.netty.handler.codec.http.DefaultHttpContent.retain(DefaultHttpContent.java:60) > io.netty.handler.codec.http.HttpObjectAggregator.decode(HttpObjectAggregator.java:190) > io.netty.handler.codec.http.HttpObjectAggregator.decode(HttpObjectAggregator.java:54) > io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:89) > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339) > io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324) > io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:242) > io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:147) > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339) > io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324) > io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:847) > io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131) > io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511) > io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468) > io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382) > io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354) > io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111) > java.lang.Thread.run(Thread.java:745) > Created at: > io.netty.buffer.UnpooledByteBufAllocator.newDirectBuffer(UnpooledByteBufAllocator.java:55) > io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:155) > io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:146) > io.netty.buffer.AbstractByteBufAllocator.ioBuffer(AbstractByteBufAllocator.java:107) > io.netty.channel.AdaptiveRecvByteBufAllocator$HandleImpl.allocate(AdaptiveRecvByteBufAllocator.java:104) > io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:117) > io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511) > io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468) > io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382) > io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354) > io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111) > java.lang.Thread.run(Thread.java:745) > {noformat} > Variant 2 creates the buffer in {{UnpooledByteBufAllocator}}. Which probably > means that there is no real resource leak. Rather the coding pattern required > for objects that might be pooled is not adhered to. > This is a sample route that triggers the error: > {code:xml} > <route id="pep-xds_b-registry"> > <from > uri="netty4-http:http://{{pep.host}}:{{pep.xds.b.registry.port}}/XDS3/registry?matchOnUriPrefix=true&disableStreamCache=true" > /> > <process ref="registryInboundProcessor" /> > <to > uri="netty4-http:http://{{xds.b.registry.host}}:{{xds.b.registry.port}}/XDS3/registry?bridgeEndpoint=true&throwExceptionOnFailure=false&chunkedMaxContentLength={{pep.xds.b.registry.max.message.size}}&disableStreamCache=false" > /> > <!-- Unfortunately disabling the stream cache only works for requests, > but not responses: https://issues.apache.org/jira/browse/CAMEL-7638 --> > <process ref="registryOutboundProcessor" /> > </route> > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)