[
https://issues.apache.org/jira/browse/ARTEMIS-1940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robbie Gemmell updated ARTEMIS-1940:
------------------------------------
Affects Version/s: 2.6.0
> premature release of pooled buffers during send can cause AMQP connection
> failures
> ----------------------------------------------------------------------------------
>
> Key: ARTEMIS-1940
> URL: https://issues.apache.org/jira/browse/ARTEMIS-1940
> Project: ActiveMQ Artemis
> Issue Type: Bug
> Components: AMQP
> Affects Versions: 2.6.0, 2.6.1
> Reporter: Robbie Gemmell
> Assignee: Robbie Gemmell
> Priority: Critical
> Fix For: 2.6.2
>
>
> When sending messages over AMQP connections to consumers, in certain cases
> the broker can prematurely release a pooled buffer before the message send
> process has completed using it. When attempt is made to send more data for
> the message later on the process fails since the buffer is no longer valid,
> per stacktrace below.
> The issue is that the broker assumes the send is entirely complete after
> pumping the proton transport, but there are cases where that may not yet be
> the case, perhaps leading to this issue. Most sends are not affected by the
> issue at all, but some such as for redelivered messages can be due to their
> use of pooled buffers.
> To address the issue for now, cases where the pooled buffers are being used
> should revert to the older copying send methods of proton, while the
> remainder continue to use the newer send methods that directly use the
> provided buffer (which is often just going to be the one created when the
> message arrived at the broker).
> {noformat}
> Thread-1 (activemq-netty-threads)] 18:49:34,857 WARN
> [org.apache.activemq.artemis.core.server] AMQ222218: Server disconnecting:
> Error decoding buffer: io.netty.util.IllegalReferenceCountException: refCnt: 0
> at
> io.netty.buffer.AbstractByteBuf.ensureAccessible(AbstractByteBuf.java:1417)
> [netty-buffer-4.1.24.Final.jar:4.1.24.Final]
> at io.netty.buffer.PooledHeapByteBuf.array(PooledHeapByteBuf.java:318)
> [netty-buffer-4.1.24.Final.jar:4.1.24.Final]
> at
> org.apache.activemq.artemis.protocol.amqp.util.NettyReadable.get(NettyReadable.java:211)
> [artemis-amqp-protocol-2.7.0-SNAPSHOT.jar:]
> at
> org.apache.qpid.proton.codec.WritableBuffer$ByteBufferWrapper.put(WritableBuffer.java:140)
> [proton-j-0.27.1.jar:]
> at
> org.apache.qpid.proton.engine.impl.FrameWriter.writeFrame(FrameWriter.java:180)
> [proton-j-0.27.1.jar:]
> at
> org.apache.qpid.proton.engine.impl.TransportImpl.writeFrame(TransportImpl.java:1075)
> [proton-j-0.27.1.jar:]
> at
> org.apache.qpid.proton.engine.impl.TransportImpl.processTransportWorkSender(TransportImpl.java:599)
> [proton-j-0.27.1.jar:]
> at
> org.apache.qpid.proton.engine.impl.TransportImpl.processTransportWork(TransportImpl.java:518)
> [proton-j-0.27.1.jar:]
> at
> org.apache.qpid.proton.engine.impl.TransportImpl.writeInto(TransportImpl.java:347)
> [proton-j-0.27.1.jar:]
> at
> org.apache.qpid.proton.engine.impl.TransportOutputAdaptor.pending(TransportOutputAdaptor.java:59)
> [proton-j-0.27.1.jar:]
> at
> org.apache.qpid.proton.engine.impl.TransportOutputAdaptor.head(TransportOutputAdaptor.java:80)
> [proton-j-0.27.1.jar:]
> at
> org.apache.qpid.proton.engine.impl.SaslImpl$SwitchingSaslTransportWrapper.head(SaslImpl.java:820)
> [proton-j-0.27.1.jar:]
> at
> org.apache.qpid.proton.engine.impl.HandshakeSniffingTransportWrapper.head(HandshakeSniffingTransportWrapper.java:151)
> [proton-j-0.27.1.jar:]
> at
> org.apache.qpid.proton.engine.impl.TransportImpl.head(TransportImpl.java:1533)
> [proton-j-0.27.1.jar:]
> at
> org.apache.activemq.artemis.protocol.amqp.proton.handler.ProtonHandler.flushBytes(ProtonHandler.java:212)
> [artemis-amqp-protocol-2.7.0-SNAPSHOT.jar:]
> at
> org.apache.activemq.artemis.protocol.amqp.proton.handler.ProtonHandler.dispatch(ProtonHandler.java:516)
> [artemis-amqp-protocol-2.7.0-SNAPSHOT.jar:]
> at
> org.apache.activemq.artemis.protocol.amqp.proton.handler.ProtonHandler.flush(ProtonHandler.java:307)
> [artemis-amqp-protocol-2.7.0-SNAPSHOT.jar:]
> at
> org.apache.activemq.artemis.protocol.amqp.proton.handler.ProtonHandler.inputBuffer(ProtonHandler.java:272)
> [artemis-amqp-protocol-2.7.0-SNAPSHOT.jar:]
> at
> org.apache.activemq.artemis.protocol.amqp.proton.AMQPConnectionContext.inputBuffer(AMQPConnectionContext.java:158)
> [artemis-amqp-protocol-2.7.0-SNAPSHOT.jar:]
> at
> org.apache.activemq.artemis.protocol.amqp.broker.ActiveMQProtonRemotingConnection.bufferReceived(ActiveMQProtonRemotingConnection.java:147)
> [artemis-amqp-protocol-2.7.0-SNAPSHOT.jar:]
> at
> org.apache.activemq.artemis.core.remoting.server.impl.RemotingServiceImpl$DelegatingBufferHandler.bufferReceived(RemotingServiceImpl.java:643)
> [artemis-server-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
> at
> org.apache.activemq.artemis.core.remoting.impl.netty.ActiveMQChannelHandler.channelRead(ActiveMQChannelHandler.java:73)
> [artemis-core-client-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
> [netty-transport-4.1.24.Final.jar:4.1.24.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
> [netty-transport-4.1.24.Final.jar:4.1.24.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
> [netty-transport-4.1.24.Final.jar:4.1.24.Final]
> at
> io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434)
> [netty-transport-4.1.24.Final.jar:4.1.24.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
> [netty-transport-4.1.24.Final.jar:4.1.24.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
> [netty-transport-4.1.24.Final.jar:4.1.24.Final]
> at
> io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965)
> [netty-transport-4.1.24.Final.jar:4.1.24.Final]
> at
> io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:808)
> [netty-transport-native-epoll-4.1.24.Final-linux-x86_64.jar:4.1.24.Final]
> at
> io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:404)
> [netty-transport-native-epoll-4.1.24.Final-linux-x86_64.jar:4.1.24.Final]
> at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:304)
> [netty-transport-native-epoll-4.1.24.Final-linux-x86_64.jar:4.1.24.Final]
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)
> [netty-common-4.1.24.Final.jar:4.1.24.Final]
> at
> org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
> [artemis-commons-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
> [Thread-1 (activemq-netty-threads)] 18:49:34,863 WARN
> [org.apache.activemq.artemis.core.client] AMQ212037: Connection failure has
> been detected: refCnt: 0 [code=GENERIC_EXCEPTION]
> [Thread-1 (activemq-netty-threads)] 18:49:34,865 WARN
> [org.apache.activemq.artemis.core.server] AMQ222061: Client connection
> failed, clearing up resources for session f5f36741-731f-11e8-8513-000c29e9f622
> [Thread-1 (activemq-netty-threads)] 18:49:34,871 WARN
> [org.apache.activemq.artemis.core.server] AMQ222107: Cleared up resources for
> session f5f36741-731f-11e8-8513-000c29e9f622
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
