Robbie Gemmell created ARTEMIS-1940:
---------------------------------------
Summary: premature release of pooled buffers during send can cause
AMQP connection failures
Key: ARTEMIS-1940
URL: https://issues.apache.org/jira/browse/ARTEMIS-1940
Project: ActiveMQ Artemis
Issue Type: Bug
Components: AMQP
Affects Versions: 2.6.1
Reporter: Robbie Gemmell
Assignee: Robbie Gemmell
Fix For: 2.6.2
When sending messages over AMQP connections to consumers, in certain cases the
broker can prematurely release a pooled buffer before the message send process
has completed using it. When attempt is made to send more data for the message
later on the process fails since the buffer is no longer valid, per stacktrace
below.
The issue is that the broker assumes the send is entirely complete after
pumping the proton transport, but there are cases where that may not yet be the
case, perhaps leading to this issue. Most sends are not affected by the issue
at all, but some such as for redelivered messages can be due to their use of
pooled buffers.
To address the issue for now, cases where the pooled buffers are being used
should revert to the older copying send methods of proton, while the remainder
continue to use the newer send methods that directly use the provided buffer
(which is often just going to be the one created when the message arrived at
the broker).
{noformat}
Thread-1 (activemq-netty-threads)] 18:49:34,857 WARN
[org.apache.activemq.artemis.core.server] AMQ222218: Server disconnecting:
Error decoding buffer: io.netty.util.IllegalReferenceCountException: refCnt: 0
at
io.netty.buffer.AbstractByteBuf.ensureAccessible(AbstractByteBuf.java:1417)
[netty-buffer-4.1.24.Final.jar:4.1.24.Final]
at io.netty.buffer.PooledHeapByteBuf.array(PooledHeapByteBuf.java:318)
[netty-buffer-4.1.24.Final.jar:4.1.24.Final]
at
org.apache.activemq.artemis.protocol.amqp.util.NettyReadable.get(NettyReadable.java:211)
[artemis-amqp-protocol-2.7.0-SNAPSHOT.jar:]
at
org.apache.qpid.proton.codec.WritableBuffer$ByteBufferWrapper.put(WritableBuffer.java:140)
[proton-j-0.27.1.jar:]
at
org.apache.qpid.proton.engine.impl.FrameWriter.writeFrame(FrameWriter.java:180)
[proton-j-0.27.1.jar:]
at
org.apache.qpid.proton.engine.impl.TransportImpl.writeFrame(TransportImpl.java:1075)
[proton-j-0.27.1.jar:]
at
org.apache.qpid.proton.engine.impl.TransportImpl.processTransportWorkSender(TransportImpl.java:599)
[proton-j-0.27.1.jar:]
at
org.apache.qpid.proton.engine.impl.TransportImpl.processTransportWork(TransportImpl.java:518)
[proton-j-0.27.1.jar:]
at
org.apache.qpid.proton.engine.impl.TransportImpl.writeInto(TransportImpl.java:347)
[proton-j-0.27.1.jar:]
at
org.apache.qpid.proton.engine.impl.TransportOutputAdaptor.pending(TransportOutputAdaptor.java:59)
[proton-j-0.27.1.jar:]
at
org.apache.qpid.proton.engine.impl.TransportOutputAdaptor.head(TransportOutputAdaptor.java:80)
[proton-j-0.27.1.jar:]
at
org.apache.qpid.proton.engine.impl.SaslImpl$SwitchingSaslTransportWrapper.head(SaslImpl.java:820)
[proton-j-0.27.1.jar:]
at
org.apache.qpid.proton.engine.impl.HandshakeSniffingTransportWrapper.head(HandshakeSniffingTransportWrapper.java:151)
[proton-j-0.27.1.jar:]
at
org.apache.qpid.proton.engine.impl.TransportImpl.head(TransportImpl.java:1533)
[proton-j-0.27.1.jar:]
at
org.apache.activemq.artemis.protocol.amqp.proton.handler.ProtonHandler.flushBytes(ProtonHandler.java:212)
[artemis-amqp-protocol-2.7.0-SNAPSHOT.jar:]
at
org.apache.activemq.artemis.protocol.amqp.proton.handler.ProtonHandler.dispatch(ProtonHandler.java:516)
[artemis-amqp-protocol-2.7.0-SNAPSHOT.jar:]
at
org.apache.activemq.artemis.protocol.amqp.proton.handler.ProtonHandler.flush(ProtonHandler.java:307)
[artemis-amqp-protocol-2.7.0-SNAPSHOT.jar:]
at
org.apache.activemq.artemis.protocol.amqp.proton.handler.ProtonHandler.inputBuffer(ProtonHandler.java:272)
[artemis-amqp-protocol-2.7.0-SNAPSHOT.jar:]
at
org.apache.activemq.artemis.protocol.amqp.proton.AMQPConnectionContext.inputBuffer(AMQPConnectionContext.java:158)
[artemis-amqp-protocol-2.7.0-SNAPSHOT.jar:]
at
org.apache.activemq.artemis.protocol.amqp.broker.ActiveMQProtonRemotingConnection.bufferReceived(ActiveMQProtonRemotingConnection.java:147)
[artemis-amqp-protocol-2.7.0-SNAPSHOT.jar:]
at
org.apache.activemq.artemis.core.remoting.server.impl.RemotingServiceImpl$DelegatingBufferHandler.bufferReceived(RemotingServiceImpl.java:643)
[artemis-server-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
at
org.apache.activemq.artemis.core.remoting.impl.netty.ActiveMQChannelHandler.channelRead(ActiveMQChannelHandler.java:73)
[artemis-core-client-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
[netty-transport-4.1.24.Final.jar:4.1.24.Final]
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
[netty-transport-4.1.24.Final.jar:4.1.24.Final]
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
[netty-transport-4.1.24.Final.jar:4.1.24.Final]
at
io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434)
[netty-transport-4.1.24.Final.jar:4.1.24.Final]
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
[netty-transport-4.1.24.Final.jar:4.1.24.Final]
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
[netty-transport-4.1.24.Final.jar:4.1.24.Final]
at
io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965)
[netty-transport-4.1.24.Final.jar:4.1.24.Final]
at
io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:808)
[netty-transport-native-epoll-4.1.24.Final-linux-x86_64.jar:4.1.24.Final]
at
io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:404)
[netty-transport-native-epoll-4.1.24.Final-linux-x86_64.jar:4.1.24.Final]
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:304)
[netty-transport-native-epoll-4.1.24.Final-linux-x86_64.jar:4.1.24.Final]
at
io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)
[netty-common-4.1.24.Final.jar:4.1.24.Final]
at
org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
[artemis-commons-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
[Thread-1 (activemq-netty-threads)] 18:49:34,863 WARN
[org.apache.activemq.artemis.core.client] AMQ212037: Connection failure has
been detected: refCnt: 0 [code=GENERIC_EXCEPTION]
[Thread-1 (activemq-netty-threads)] 18:49:34,865 WARN
[org.apache.activemq.artemis.core.server] AMQ222061: Client connection failed,
clearing up resources for session f5f36741-731f-11e8-8513-000c29e9f622
[Thread-1 (activemq-netty-threads)] 18:49:34,871 WARN
[org.apache.activemq.artemis.core.server] AMQ222107: Cleared up resources for
session f5f36741-731f-11e8-8513-000c29e9f622
{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
