[
https://issues.apache.org/jira/browse/ARTEMIS-4970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Justin Bertram updated ARTEMIS-4970:
------------------------------------
Description:
The IndexOutOfBoundsException error occurs randomly when messages are being
replicated via async mirroring.
Several thousands of messages can be replicated successfully before it happens.
I have no reproduction scenario yet, as it is random but it happens several
times per day.
If needed, specific logging level can be enabled if that helps with the
investigation.
*Artemis setup:*
The Artemis topology is composed by two Artemis clusters (of 3 groups) with ZK
quorum (primary/backup).
Dual async mirroring is enabled on queues on both clusters.
*IndexOutOfBound error details*
Most messages going through the replication link are of standard size and
originated by Openwire or Core protocol. Large messages, averaging 150KB can be
replicated too but are less frequent.
Please note that the message is altered by an interceptor to add property
"_BT_MAX_DELIVERY" when it reaches the broker.
The message embedded in the stack trace below appears to have been
redistributed within the cluster before being replicated, as user is
ACTIVEMQ.CLUSTER.ADMIN.USER. I have seen it failing in non-redistributed
scenario too.
{noformat}
2024-08-06 03:44:00,596 WARN
[org.apache.activemq.artemis.protocol.amqp.proton.ProtonServerSenderContext]
readerIndex: 0, writerIndex: 11960 (expected: 0 <= readerIndex <= writerIndex
<= capacity(11715))
java.lang.IndexOutOfBoundsException: readerIndex: 0, writerIndex: 11960
(expected: 0 <= readerIndex <= writerIndex <= capacity(11715))
at
io.netty.buffer.AbstractByteBuf.checkIndexBounds(AbstractByteBuf.java:112)
~[netty-buffer-4.1.112.Final.jar:4.1.112.Final]
at io.netty.buffer.AbstractByteBuf.writerIndex(AbstractByteBuf.java:135)
~[netty-buffer-4.1.112.Final.jar:4.1.112.Final]
at
org.apache.activemq.artemis.protocol.amqp.proton.AMQPTunneledCoreMessageWriter.writeBytes(AMQPTunneledCoreMessageWriter.java:107)
[artemis-amqp-protocol-2.36.0.jar:2.36.0]
at
org.apache.activemq.artemis.protocol.amqp.proton.MessageWriter.accept(MessageWriter.java:41)
[artemis-amqp-protocol-2.36.0.jar:2.36.0]
at
org.apache.activemq.artemis.protocol.amqp.proton.MessageWriter.accept(MessageWriter.java:28)
[artemis-amqp-protocol-2.36.0.jar:2.36.0]
at
org.apache.activemq.artemis.core.server.impl.MessageReferenceImpl.run(MessageReferenceImpl.java:136)
[artemis-server-2.36.0.jar:2.36.0]
at
io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173)
[netty-common-4.1.112.Final.jar:4.1.112.Final]
at
io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166)
[netty-common-4.1.112.Final.jar:4.1.112.Final]
at
io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:469)
[netty-common-4.1.112.Final.jar:4.1.112.Final]
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:405)
[netty-transport-classes-epoll-4.1.112.Final.jar:4.1.112.Final]
at
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:994)
[netty-common-4.1.112.Final.jar:4.1.112.Final]
at
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
[netty-common-4.1.112.Final.jar:4.1.112.Final]
at
org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
[artemis-commons-2.36.0.jar:2.36.0]
2024-08-06 03:44:00,596 WARN [org.apache.activemq.artemis.core.server]
AMQ222151: removing consumer which did not handle a message,
consumer=ServerConsumerImpl [id=0, filter=null, binding=LocalQueueBinding
[address=$ACTIVEMQ_ARTEMIS_MIRROR_dc2-group-1,
queue=QueueImpl[name=$ACTIVEMQ_ARTEMIS_MIRROR_dc2-group-1,
postOffice=PostOfficeImpl
[server=ActiveMQServerImpl::name=artemis-dc1-primary-1], temp=false]@290df877,
filter=null, name=$ACTIVEMQ_ARTEMIS_MIRROR_dc2-group-1,
clusterName=$ACTIVEMQ_ARTEMIS_MIRROR_dc2-group-154a6ae45-26e5-11ee-837c-506b8d97040b],
closed=false],
message=Reference[282007612256]:RELIABLE:CoreMessage[messageID=282007612256,
durable=true, userID=a51c7f00-53c7-11ef-bb35-506b8db21b19, priority=4,
timestamp=Tue Aug 06 03:44:00 EDT 2024, expiration=0, durable=true,
address=queue.ua.release.shared.internal-message.message-manager.to.message-services,
size=11703,
properties=TypedProperties[traceparent=00-f369c10867855a63b6d9c86cfd612249-e57085be9b3c4fc5-01,
__AMQ_CID=ID:us01vwfmmm02-51592-638581215922967464-0:0, _AMQ_GROUP_SEQUENCE=0,
__HDR_BROKER_IN_TIME=1722930240548, _AMQ_ROUTING_TYPE=1, __HDR_ARRIVAL=0,
_BT_MAX_DELIVERY=30, __HDR_COMMAND_ID=7438781,
JMSCorrelationID=O202408060944000003Q,
_AMQ_VALIDATED_USER=ACTIVEMQ.CLUSTER.ADMIN.USER,
_AMQ_DUPL_ID=ID:us01vwfmmm02-51592-638581215922967464-1:0:11:153438:1,
__HDR_PRODUCER_ID=ID:us01vwfmmm02-51592-638581215922967464-1:0:11:153438,
__HDR_MESSAGE_ID=ID:us01vwfmmm02-51592-638581215922967464-1:0:11:153438:1,
__HDR_DROPPABLE=false]]@266147141
java.lang.IndexOutOfBoundsException: readerIndex: 0, writerIndex: 11960
(expected: 0 <= readerIndex <= writerIndex <= capacity(11715))
at
io.netty.buffer.AbstractByteBuf.checkIndexBounds(AbstractByteBuf.java:112)
~[netty-buffer-4.1.112.Final.jar:4.1.112.Final]
at io.netty.buffer.AbstractByteBuf.writerIndex(AbstractByteBuf.java:135)
~[netty-buffer-4.1.112.Final.jar:4.1.112.Final]
at
org.apache.activemq.artemis.protocol.amqp.proton.AMQPTunneledCoreMessageWriter.writeBytes(AMQPTunneledCoreMessageWriter.java:107)
[artemis-amqp-protocol-2.36.0.jar:2.36.0]
at
org.apache.activemq.artemis.protocol.amqp.proton.MessageWriter.accept(MessageWriter.java:41)
[artemis-amqp-protocol-2.36.0.jar:2.36.0]
at
org.apache.activemq.artemis.protocol.amqp.proton.MessageWriter.accept(MessageWriter.java:28)
[artemis-amqp-protocol-2.36.0.jar:2.36.0]
at
org.apache.activemq.artemis.core.server.impl.MessageReferenceImpl.run(MessageReferenceImpl.java:136)
[artemis-server-2.36.0.jar:2.36.0]
at
io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173)
[netty-common-4.1.112.Final.jar:4.1.112.Final]
at
io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166)
[netty-common-4.1.112.Final.jar:4.1.112.Final]
at
io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:469)
[netty-common-4.1.112.Final.jar:4.1.112.Final]
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:405)
[netty-transport-classes-epoll-4.1.112.Final.jar:4.1.112.Final]
at
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:994)
[netty-common-4.1.112.Final.jar:4.1.112.Final]
at
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
[netty-common-4.1.112.Final.jar:4.1.112.Final]
at
org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
[artemis-commons-2.36.0.jar:2.36.0]{noformat}
*Mirroring consumer stopped*
The error triggers another problem where message replication is immediately
stopped because the mirroring consumer is destroyed and is never recreated
automatically (null pointer problem?).
A workaround found is to close AMQP broker connections, which will trigger
broker connection to restart and automatically restart mirroring consumer.
Stacktrace of consumer failing to reconnect after the IndexOutOfBound Exception:
{noformat}
2024-08-06 03:44:00,597 WARN [io.netty.util.concurrent.AbstractEventExecutor]
A task raised an exception. Task:
org.apache.activemq.artemis.protocol.amqp.broker.AMQPSessionCallback$$Lambda$1173/0x00007ff7dc82c228@41bb7410
java.lang.NullPointerException: Cannot invoke
"org.apache.activemq.artemis.protocol.amqp.proton.ProtonServerSenderContext.close(org.apache.qpid.proton.amqp.transport.ErrorCondition)"
because the return value of
"org.apache.activemq.artemis.core.server.ServerConsumer.getProtocolContext()"
is null
at
org.apache.activemq.artemis.protocol.amqp.broker.AMQPSessionCallback.lambda$disconnect$5(AMQPSessionCallback.java:747)
~[artemis-amqp-protocol-2.36.0.jar:2.36.0]
at
io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173)
~[netty-common-4.1.112.Final.jar:4.1.112.Final]
at
io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166)
[netty-common-4.1.112.Final.jar:4.1.112.Final]
at
io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:469)
[netty-common-4.1.112.Final.jar:4.1.112.Final]
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:405)
[netty-transport-classes-epoll-4.1.112.Final.jar:4.1.112.Final]
at
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:994)
[netty-common-4.1.112.Final.jar:4.1.112.Final]
at
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
[netty-common-4.1.112.Final.jar:4.1.112.Final]
at
org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
[artemis-commons-2.36.0.jar:2.36.0]{noformat}
*Error on replica-side*
{noformat}
2024-08-06 03:44:00,598 WARN
[org.apache.activemq.artemis.protocol.amqp.proton.ProtonAbstractReceiver]
Received an AmqpValue payload in core tunneled AMQP message
java.lang.IllegalArgumentException: Received an AmqpValue payload in core
tunneled AMQP message
at
org.apache.activemq.artemis.protocol.amqp.proton.AMQPTunneledCoreMessageReader.readBytes(AMQPTunneledCoreMessageReader.java:122)
~[artemis-amqp-protocol-2.36.0.jar:2.36.0]
at
org.apache.activemq.artemis.protocol.amqp.proton.ProtonAbstractReceiver.onMessage(ProtonAbstractReceiver.java:329)
[artemis-amqp-protocol-2.36.0.jar:2.36.0]
at
org.apache.activemq.artemis.protocol.amqp.proton.AMQPConnectionContext.onDelivery(AMQPConnectionContext.java:890)
[artemis-amqp-protocol-2.36.0.jar:2.36.0]
at
org.apache.activemq.artemis.protocol.amqp.proton.handler.Events.dispatch(Events.java:95)
[artemis-amqp-protocol-2.36.0.jar:2.36.0]
at
org.apache.activemq.artemis.protocol.amqp.proton.handler.ProtonHandler.dispatch(ProtonHandler.java:585)
[artemis-amqp-protocol-2.36.0.jar:2.36.0]
at
org.apache.activemq.artemis.protocol.amqp.proton.handler.ProtonHandler.handleFlush(ProtonHandler.java:391)
[artemis-amqp-protocol-2.36.0.jar:2.36.0]
at
org.apache.activemq.artemis.protocol.amqp.proton.handler.ProtonHandler.flush(ProtonHandler.java:376)
[artemis-amqp-protocol-2.36.0.jar:2.36.0]
at
org.apache.activemq.artemis.protocol.amqp.proton.handler.ProtonHandler.inputBuffer(ProtonHandler.java:334)
[artemis-amqp-protocol-2.36.0.jar:2.36.0]
at
org.apache.activemq.artemis.protocol.amqp.proton.AMQPConnectionContext.inputBuffer(AMQPConnectionContext.java:293)
[artemis-amqp-protocol-2.36.0.jar:2.36.0]
at
org.apache.activemq.artemis.protocol.amqp.broker.ActiveMQProtonRemotingConnection.bufferReceived(ActiveMQProtonRemotingConnection.java:174)
[artemis-amqp-protocol-2.36.0.jar:2.36.0]
at
org.apache.activemq.artemis.core.remoting.server.impl.RemotingServiceImpl$DelegatingBufferHandler.bufferReceived(RemotingServiceImpl.java:723)
[artemis-server-2.36.0.jar:2.36.0]
at
org.apache.activemq.artemis.core.remoting.impl.netty.ActiveMQChannelHandler.channelRead(ActiveMQChannelHandler.java:73)
[artemis-core-client-2.36.0.jar:2.36.0]
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
[netty-transport-4.1.112.Final.jar:4.1.112.Final]
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
[netty-transport-4.1.112.Final.jar:4.1.112.Final]
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
[netty-transport-4.1.112.Final.jar:4.1.112.Final]
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1473)
[netty-handler-4.1.112.Final.jar:4.1.112.Final]
at
io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1336)
[netty-handler-4.1.112.Final.jar:4.1.112.Final]
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1385)
[netty-handler-4.1.112.Final.jar:4.1.112.Final]
at
io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:530)
[netty-codec-4.1.112.Final.jar:4.1.112.Final]
at
io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:469)
[netty-codec-4.1.112.Final.jar:4.1.112.Final]
at
io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290)
[netty-codec-4.1.112.Final.jar:4.1.112.Final]
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
[netty-transport-4.1.112.Final.jar:4.1.112.Final]
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
[netty-transport-4.1.112.Final.jar:4.1.112.Final]
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
[netty-transport-4.1.112.Final.jar:4.1.112.Final]
at
io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1407)
[netty-transport-4.1.112.Final.jar:4.1.112.Final]
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
[netty-transport-4.1.112.Final.jar:4.1.112.Final]
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
[netty-transport-4.1.112.Final.jar:4.1.112.Final]
at
io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:918)
[netty-transport-4.1.112.Final.jar:4.1.112.Final]
at
io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:799)
[netty-transport-classes-epoll-4.1.112.Final.jar:4.1.112.Final]
at
io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:501)
[netty-transport-classes-epoll-4.1.112.Final.jar:4.1.112.Final]
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:399)
[netty-transport-classes-epoll-4.1.112.Final.jar:4.1.112.Final]
at
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:994)
[netty-common-4.1.112.Final.jar:4.1.112.Final]
at
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
[netty-common-4.1.112.Final.jar:4.1.112.Final]
at
org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
[artemis-commons-2.36.0.jar:2.36.0]
2024-08-06 03:44:00,603 WARN
[org.apache.activemq.artemis.protocol.amqp.proton.handler.ProtonHandler]
Received an AmqpValue payload in core tunneled AMQP message
org.apache.activemq.artemis.protocol.amqp.exceptions.ActiveMQAMQPInternalErrorException:
Received an AmqpValue payload in core tunneled AMQP message
at
org.apache.activemq.artemis.protocol.amqp.proton.ProtonAbstractReceiver.onMessage(ProtonAbstractReceiver.java:337)
~[artemis-amqp-protocol-2.36.0.jar:2.36.0]
at
org.apache.activemq.artemis.protocol.amqp.proton.AMQPConnectionContext.onDelivery(AMQPConnectionContext.java:890)
~[artemis-amqp-protocol-2.36.0.jar:2.36.0]
at
org.apache.activemq.artemis.protocol.amqp.proton.handler.Events.dispatch(Events.java:95)
~[artemis-amqp-protocol-2.36.0.jar:2.36.0]
at
org.apache.activemq.artemis.protocol.amqp.proton.handler.ProtonHandler.dispatch(ProtonHandler.java:585)
[artemis-amqp-protocol-2.36.0.jar:2.36.0]
at
org.apache.activemq.artemis.protocol.amqp.proton.handler.ProtonHandler.handleFlush(ProtonHandler.java:391)
[artemis-amqp-protocol-2.36.0.jar:2.36.0]
at
org.apache.activemq.artemis.protocol.amqp.proton.handler.ProtonHandler.flush(ProtonHandler.java:376)
[artemis-amqp-protocol-2.36.0.jar:2.36.0]
at
org.apache.activemq.artemis.protocol.amqp.proton.handler.ProtonHandler.inputBuffer(ProtonHandler.java:334)
[artemis-amqp-protocol-2.36.0.jar:2.36.0]
at
org.apache.activemq.artemis.protocol.amqp.proton.AMQPConnectionContext.inputBuffer(AMQPConnectionContext.java:293)
[artemis-amqp-protocol-2.36.0.jar:2.36.0]
at
org.apache.activemq.artemis.protocol.amqp.broker.ActiveMQProtonRemotingConnection.bufferReceived(ActiveMQProtonRemotingConnection.java:174)
[artemis-amqp-protocol-2.36.0.jar:2.36.0]
at
org.apache.activemq.artemis.core.remoting.server.impl.RemotingServiceImpl$DelegatingBufferHandler.bufferReceived(RemotingServiceImpl.java:723)
[artemis-server-2.36.0.jar:2.36.0]
at
org.apache.activemq.artemis.core.remoting.impl.netty.ActiveMQChannelHandler.channelRead(ActiveMQChannelHandler.java:73)
[artemis-core-client-2.36.0.jar:2.36.0]
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
[netty-transport-4.1.112.Final.jar:4.1.112.Final]
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
[netty-transport-4.1.112.Final.jar:4.1.112.Final]
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
[netty-transport-4.1.112.Final.jar:4.1.112.Final]
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1473)
[netty-handler-4.1.112.Final.jar:4.1.112.Final]
at
io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1336)
[netty-handler-4.1.112.Final.jar:4.1.112.Final]
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1385)
[netty-handler-4.1.112.Final.jar:4.1.112.Final]
at
io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:530)
[netty-codec-4.1.112.Final.jar:4.1.112.Final]
at
io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:469)
[netty-codec-4.1.112.Final.jar:4.1.112.Final]
at
io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290)
[netty-codec-4.1.112.Final.jar:4.1.112.Final]
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
[netty-transport-4.1.112.Final.jar:4.1.112.Final]
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
[netty-transport-4.1.112.Final.jar:4.1.112.Final]
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
[netty-transport-4.1.112.Final.jar:4.1.112.Final]
at
io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1407)
[netty-transport-4.1.112.Final.jar:4.1.112.Final]
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
[netty-transport-4.1.112.Final.jar:4.1.112.Final]
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
[netty-transport-4.1.112.Final.jar:4.1.112.Final]
at
io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:918)
[netty-transport-4.1.112.Final.jar:4.1.112.Final]
at
io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:799)
[netty-transport-classes-epoll-4.1.112.Final.jar:4.1.112.Final]
at
io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:501)
[netty-transport-classes-epoll-4.1.112.Final.jar:4.1.112.Final]
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:399)
[netty-transport-classes-epoll-4.1.112.Final.jar:4.1.112.Final]
at
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:994)
[netty-common-4.1.112.Final.jar:4.1.112.Final]
at
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
[netty-common-4.1.112.Final.jar:4.1.112.Final]
at
org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
[artemis-commons-2.36.0.jar:2.36.0]
Caused by: java.lang.IllegalArgumentException: Received an AmqpValue payload in
core tunneled AMQP message
at
org.apache.activemq.artemis.protocol.amqp.proton.AMQPTunneledCoreMessageReader.readBytes(AMQPTunneledCoreMessageReader.java:122)
~[artemis-amqp-protocol-2.36.0.jar:2.36.0]
at
org.apache.activemq.artemis.protocol.amqp.proton.ProtonAbstractReceiver.onMessage(ProtonAbstractReceiver.java:329)
~[artemis-amqp-protocol-2.36.0.jar:2.36.0]
... 32 more{noformat}
was:
The IndexOutOfBoundsException error occurs randomly when messages are being
replicated via async mirroring.
Several thousands of messages can be replicated successfully before it happens.
I have no reproduction scenario yet, as it is random but it happens several
times per day.
If needed, specific logging level can be enabled if that helps with the
investigation.
*Artemis setup:*
The Artemis topology is composed by two Artemis clusters (of 3 groups) with ZK
quorum (primary/backup).
Dual async mirroring is enabled on queues on both clusters.
*IndexOutOfBound error details*
Most messages going through the replication link are of standard size and
originated by Openwire or Core protocol. Large messages, averaging 150KB can be
replicated too but are less frequent.
Please note that the message is altered by an interceptor to add property
"_BT_MAX_DELIVERY" when it reaches the broker.
The message embedded in the stack trace below appears to have been
redistributed within the cluster before being replicated, as user is
ACTIVEMQ.CLUSTER.ADMIN.USER. I have seen it failing in non-redistributed
scenario too.
{code:java}
2024-08-06 03:44:00,596 WARN
[org.apache.activemq.artemis.protocol.amqp.proton.ProtonServerSenderContext]
readerIndex: 0, writerIndex: 11960 (expected: 0 <= readerIndex <= writerIndex
<= capacity(11715))
java.lang.IndexOutOfBoundsException: readerIndex: 0, writerIndex: 11960
(expected: 0 <= readerIndex <= writerIndex <= capacity(11715))
at
io.netty.buffer.AbstractByteBuf.checkIndexBounds(AbstractByteBuf.java:112)
~[netty-buffer-4.1.112.Final.jar:4.1.112.Final]
at io.netty.buffer.AbstractByteBuf.writerIndex(AbstractByteBuf.java:135)
~[netty-buffer-4.1.112.Final.jar:4.1.112.Final]
at
org.apache.activemq.artemis.protocol.amqp.proton.AMQPTunneledCoreMessageWriter.writeBytes(AMQPTunneledCoreMessageWriter.java:107)
[artemis-amqp-protocol-2.36.0.jar:2.36.0]
at
org.apache.activemq.artemis.protocol.amqp.proton.MessageWriter.accept(MessageWriter.java:41)
[artemis-amqp-protocol-2.36.0.jar:2.36.0]
at
org.apache.activemq.artemis.protocol.amqp.proton.MessageWriter.accept(MessageWriter.java:28)
[artemis-amqp-protocol-2.36.0.jar:2.36.0]
at
org.apache.activemq.artemis.core.server.impl.MessageReferenceImpl.run(MessageReferenceImpl.java:136)
[artemis-server-2.36.0.jar:2.36.0]
at
io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173)
[netty-common-4.1.112.Final.jar:4.1.112.Final]
at
io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166)
[netty-common-4.1.112.Final.jar:4.1.112.Final]
at
io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:469)
[netty-common-4.1.112.Final.jar:4.1.112.Final]
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:405)
[netty-transport-classes-epoll-4.1.112.Final.jar:4.1.112.Final]
at
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:994)
[netty-common-4.1.112.Final.jar:4.1.112.Final]
at
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
[netty-common-4.1.112.Final.jar:4.1.112.Final]
at
org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
[artemis-commons-2.36.0.jar:2.36.0]
2024-08-06 03:44:00,596 WARN [org.apache.activemq.artemis.core.server]
AMQ222151: removing consumer which did not handle a message,
consumer=ServerConsumerImpl [id=0, filter=null, binding=LocalQueueBinding
[address=$ACTIVEMQ_ARTEMIS_MIRROR_dc2-group-1,
queue=QueueImpl[name=$ACTIVEMQ_ARTEMIS_MIRROR_dc2-group-1,
postOffice=PostOfficeImpl
[server=ActiveMQServerImpl::name=artemis-dc1-primary-1], temp=false]@290df877,
filter=null, name=$ACTIVEMQ_ARTEMIS_MIRROR_dc2-group-1,
clusterName=$ACTIVEMQ_ARTEMIS_MIRROR_dc2-group-154a6ae45-26e5-11ee-837c-506b8d97040b],
closed=false],
message=Reference[282007612256]:RELIABLE:CoreMessage[messageID=282007612256,
durable=true, userID=a51c7f00-53c7-11ef-bb35-506b8db21b19, priority=4,
timestamp=Tue Aug 06 03:44:00 EDT 2024, expiration=0, durable=true,
address=queue.ua.release.shared.internal-message.message-manager.to.message-services,
size=11703,
properties=TypedProperties[traceparent=00-f369c10867855a63b6d9c86cfd612249-e57085be9b3c4fc5-01,
__AMQ_CID=ID:us01vwfmmm02-51592-638581215922967464-0:0, _AMQ_GROUP_SEQUENCE=0,
__HDR_BROKER_IN_TIME=1722930240548, _AMQ_ROUTING_TYPE=1, __HDR_ARRIVAL=0,
_BT_MAX_DELIVERY=30, __HDR_COMMAND_ID=7438781,
JMSCorrelationID=O202408060944000003Q,
_AMQ_VALIDATED_USER=ACTIVEMQ.CLUSTER.ADMIN.USER,
_AMQ_DUPL_ID=ID:us01vwfmmm02-51592-638581215922967464-1:0:11:153438:1,
__HDR_PRODUCER_ID=ID:us01vwfmmm02-51592-638581215922967464-1:0:11:153438,
__HDR_MESSAGE_ID=ID:us01vwfmmm02-51592-638581215922967464-1:0:11:153438:1,
__HDR_DROPPABLE=false]]@266147141
java.lang.IndexOutOfBoundsException: readerIndex: 0, writerIndex: 11960
(expected: 0 <= readerIndex <= writerIndex <= capacity(11715))
at
io.netty.buffer.AbstractByteBuf.checkIndexBounds(AbstractByteBuf.java:112)
~[netty-buffer-4.1.112.Final.jar:4.1.112.Final]
at io.netty.buffer.AbstractByteBuf.writerIndex(AbstractByteBuf.java:135)
~[netty-buffer-4.1.112.Final.jar:4.1.112.Final]
at
org.apache.activemq.artemis.protocol.amqp.proton.AMQPTunneledCoreMessageWriter.writeBytes(AMQPTunneledCoreMessageWriter.java:107)
[artemis-amqp-protocol-2.36.0.jar:2.36.0]
at
org.apache.activemq.artemis.protocol.amqp.proton.MessageWriter.accept(MessageWriter.java:41)
[artemis-amqp-protocol-2.36.0.jar:2.36.0]
at
org.apache.activemq.artemis.protocol.amqp.proton.MessageWriter.accept(MessageWriter.java:28)
[artemis-amqp-protocol-2.36.0.jar:2.36.0]
at
org.apache.activemq.artemis.core.server.impl.MessageReferenceImpl.run(MessageReferenceImpl.java:136)
[artemis-server-2.36.0.jar:2.36.0]
at
io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173)
[netty-common-4.1.112.Final.jar:4.1.112.Final]
at
io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166)
[netty-common-4.1.112.Final.jar:4.1.112.Final]
at
io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:469)
[netty-common-4.1.112.Final.jar:4.1.112.Final]
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:405)
[netty-transport-classes-epoll-4.1.112.Final.jar:4.1.112.Final]
at
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:994)
[netty-common-4.1.112.Final.jar:4.1.112.Final]
at
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
[netty-common-4.1.112.Final.jar:4.1.112.Final]
at
org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
[artemis-commons-2.36.0.jar:2.36.0]{code}
*Mirroring consumer stopped*
The error triggers another problem where message replication is immediately
stopped because the mirroring consumer is destroyed and is never recreated
automatically (null pointer problem?).
A workaround found is to close AMQP broker connections, which will trigger
broker connection to restart and automatically restart mirroring consumer.
Stacktrace of consumer failing to reconnect after the IndexOutOfBound Exception:
{code:java}
2024-08-06 03:44:00,597 WARN [io.netty.util.concurrent.AbstractEventExecutor]
A task raised an exception. Task:
org.apache.activemq.artemis.protocol.amqp.broker.AMQPSessionCallback$$Lambda$1173/0x00007ff7dc82c228@41bb7410
java.lang.NullPointerException: Cannot invoke
"org.apache.activemq.artemis.protocol.amqp.proton.ProtonServerSenderContext.close(org.apache.qpid.proton.amqp.transport.ErrorCondition)"
because the return value of
"org.apache.activemq.artemis.core.server.ServerConsumer.getProtocolContext()"
is null
at
org.apache.activemq.artemis.protocol.amqp.broker.AMQPSessionCallback.lambda$disconnect$5(AMQPSessionCallback.java:747)
~[artemis-amqp-protocol-2.36.0.jar:2.36.0]
at
io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173)
~[netty-common-4.1.112.Final.jar:4.1.112.Final]
at
io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166)
[netty-common-4.1.112.Final.jar:4.1.112.Final]
at
io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:469)
[netty-common-4.1.112.Final.jar:4.1.112.Final]
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:405)
[netty-transport-classes-epoll-4.1.112.Final.jar:4.1.112.Final]
at
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:994)
[netty-common-4.1.112.Final.jar:4.1.112.Final]
at
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
[netty-common-4.1.112.Final.jar:4.1.112.Final]
at
org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
[artemis-commons-2.36.0.jar:2.36.0]{code}
*Error on replica-side*
{code:java}
2024-08-06 03:44:00,598 WARN
[org.apache.activemq.artemis.protocol.amqp.proton.ProtonAbstractReceiver]
Received an AmqpValue payload in core tunneled AMQP message
java.lang.IllegalArgumentException: Received an AmqpValue payload in core
tunneled AMQP message
at
org.apache.activemq.artemis.protocol.amqp.proton.AMQPTunneledCoreMessageReader.readBytes(AMQPTunneledCoreMessageReader.java:122)
~[artemis-amqp-protocol-2.36.0.jar:2.36.0]
at
org.apache.activemq.artemis.protocol.amqp.proton.ProtonAbstractReceiver.onMessage(ProtonAbstractReceiver.java:329)
[artemis-amqp-protocol-2.36.0.jar:2.36.0]
at
org.apache.activemq.artemis.protocol.amqp.proton.AMQPConnectionContext.onDelivery(AMQPConnectionContext.java:890)
[artemis-amqp-protocol-2.36.0.jar:2.36.0]
at
org.apache.activemq.artemis.protocol.amqp.proton.handler.Events.dispatch(Events.java:95)
[artemis-amqp-protocol-2.36.0.jar:2.36.0]
at
org.apache.activemq.artemis.protocol.amqp.proton.handler.ProtonHandler.dispatch(ProtonHandler.java:585)
[artemis-amqp-protocol-2.36.0.jar:2.36.0]
at
org.apache.activemq.artemis.protocol.amqp.proton.handler.ProtonHandler.handleFlush(ProtonHandler.java:391)
[artemis-amqp-protocol-2.36.0.jar:2.36.0]
at
org.apache.activemq.artemis.protocol.amqp.proton.handler.ProtonHandler.flush(ProtonHandler.java:376)
[artemis-amqp-protocol-2.36.0.jar:2.36.0]
at
org.apache.activemq.artemis.protocol.amqp.proton.handler.ProtonHandler.inputBuffer(ProtonHandler.java:334)
[artemis-amqp-protocol-2.36.0.jar:2.36.0]
at
org.apache.activemq.artemis.protocol.amqp.proton.AMQPConnectionContext.inputBuffer(AMQPConnectionContext.java:293)
[artemis-amqp-protocol-2.36.0.jar:2.36.0]
at
org.apache.activemq.artemis.protocol.amqp.broker.ActiveMQProtonRemotingConnection.bufferReceived(ActiveMQProtonRemotingConnection.java:174)
[artemis-amqp-protocol-2.36.0.jar:2.36.0]
at
org.apache.activemq.artemis.core.remoting.server.impl.RemotingServiceImpl$DelegatingBufferHandler.bufferReceived(RemotingServiceImpl.java:723)
[artemis-server-2.36.0.jar:2.36.0]
at
org.apache.activemq.artemis.core.remoting.impl.netty.ActiveMQChannelHandler.channelRead(ActiveMQChannelHandler.java:73)
[artemis-core-client-2.36.0.jar:2.36.0]
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
[netty-transport-4.1.112.Final.jar:4.1.112.Final]
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
[netty-transport-4.1.112.Final.jar:4.1.112.Final]
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
[netty-transport-4.1.112.Final.jar:4.1.112.Final]
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1473)
[netty-handler-4.1.112.Final.jar:4.1.112.Final]
at
io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1336)
[netty-handler-4.1.112.Final.jar:4.1.112.Final]
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1385)
[netty-handler-4.1.112.Final.jar:4.1.112.Final]
at
io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:530)
[netty-codec-4.1.112.Final.jar:4.1.112.Final]
at
io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:469)
[netty-codec-4.1.112.Final.jar:4.1.112.Final]
at
io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290)
[netty-codec-4.1.112.Final.jar:4.1.112.Final]
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
[netty-transport-4.1.112.Final.jar:4.1.112.Final]
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
[netty-transport-4.1.112.Final.jar:4.1.112.Final]
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
[netty-transport-4.1.112.Final.jar:4.1.112.Final]
at
io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1407)
[netty-transport-4.1.112.Final.jar:4.1.112.Final]
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
[netty-transport-4.1.112.Final.jar:4.1.112.Final]
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
[netty-transport-4.1.112.Final.jar:4.1.112.Final]
at
io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:918)
[netty-transport-4.1.112.Final.jar:4.1.112.Final]
at
io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:799)
[netty-transport-classes-epoll-4.1.112.Final.jar:4.1.112.Final]
at
io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:501)
[netty-transport-classes-epoll-4.1.112.Final.jar:4.1.112.Final]
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:399)
[netty-transport-classes-epoll-4.1.112.Final.jar:4.1.112.Final]
at
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:994)
[netty-common-4.1.112.Final.jar:4.1.112.Final]
at
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
[netty-common-4.1.112.Final.jar:4.1.112.Final]
at
org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
[artemis-commons-2.36.0.jar:2.36.0]
2024-08-06 03:44:00,603 WARN
[org.apache.activemq.artemis.protocol.amqp.proton.handler.ProtonHandler]
Received an AmqpValue payload in core tunneled AMQP message
org.apache.activemq.artemis.protocol.amqp.exceptions.ActiveMQAMQPInternalErrorException:
Received an AmqpValue payload in core tunneled AMQP message
at
org.apache.activemq.artemis.protocol.amqp.proton.ProtonAbstractReceiver.onMessage(ProtonAbstractReceiver.java:337)
~[artemis-amqp-protocol-2.36.0.jar:2.36.0]
at
org.apache.activemq.artemis.protocol.amqp.proton.AMQPConnectionContext.onDelivery(AMQPConnectionContext.java:890)
~[artemis-amqp-protocol-2.36.0.jar:2.36.0]
at
org.apache.activemq.artemis.protocol.amqp.proton.handler.Events.dispatch(Events.java:95)
~[artemis-amqp-protocol-2.36.0.jar:2.36.0]
at
org.apache.activemq.artemis.protocol.amqp.proton.handler.ProtonHandler.dispatch(ProtonHandler.java:585)
[artemis-amqp-protocol-2.36.0.jar:2.36.0]
at
org.apache.activemq.artemis.protocol.amqp.proton.handler.ProtonHandler.handleFlush(ProtonHandler.java:391)
[artemis-amqp-protocol-2.36.0.jar:2.36.0]
at
org.apache.activemq.artemis.protocol.amqp.proton.handler.ProtonHandler.flush(ProtonHandler.java:376)
[artemis-amqp-protocol-2.36.0.jar:2.36.0]
at
org.apache.activemq.artemis.protocol.amqp.proton.handler.ProtonHandler.inputBuffer(ProtonHandler.java:334)
[artemis-amqp-protocol-2.36.0.jar:2.36.0]
at
org.apache.activemq.artemis.protocol.amqp.proton.AMQPConnectionContext.inputBuffer(AMQPConnectionContext.java:293)
[artemis-amqp-protocol-2.36.0.jar:2.36.0]
at
org.apache.activemq.artemis.protocol.amqp.broker.ActiveMQProtonRemotingConnection.bufferReceived(ActiveMQProtonRemotingConnection.java:174)
[artemis-amqp-protocol-2.36.0.jar:2.36.0]
at
org.apache.activemq.artemis.core.remoting.server.impl.RemotingServiceImpl$DelegatingBufferHandler.bufferReceived(RemotingServiceImpl.java:723)
[artemis-server-2.36.0.jar:2.36.0]
at
org.apache.activemq.artemis.core.remoting.impl.netty.ActiveMQChannelHandler.channelRead(ActiveMQChannelHandler.java:73)
[artemis-core-client-2.36.0.jar:2.36.0]
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
[netty-transport-4.1.112.Final.jar:4.1.112.Final]
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
[netty-transport-4.1.112.Final.jar:4.1.112.Final]
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
[netty-transport-4.1.112.Final.jar:4.1.112.Final]
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1473)
[netty-handler-4.1.112.Final.jar:4.1.112.Final]
at
io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1336)
[netty-handler-4.1.112.Final.jar:4.1.112.Final]
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1385)
[netty-handler-4.1.112.Final.jar:4.1.112.Final]
at
io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:530)
[netty-codec-4.1.112.Final.jar:4.1.112.Final]
at
io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:469)
[netty-codec-4.1.112.Final.jar:4.1.112.Final]
at
io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290)
[netty-codec-4.1.112.Final.jar:4.1.112.Final]
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
[netty-transport-4.1.112.Final.jar:4.1.112.Final]
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
[netty-transport-4.1.112.Final.jar:4.1.112.Final]
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
[netty-transport-4.1.112.Final.jar:4.1.112.Final]
at
io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1407)
[netty-transport-4.1.112.Final.jar:4.1.112.Final]
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
[netty-transport-4.1.112.Final.jar:4.1.112.Final]
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
[netty-transport-4.1.112.Final.jar:4.1.112.Final]
at
io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:918)
[netty-transport-4.1.112.Final.jar:4.1.112.Final]
at
io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:799)
[netty-transport-classes-epoll-4.1.112.Final.jar:4.1.112.Final]
at
io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:501)
[netty-transport-classes-epoll-4.1.112.Final.jar:4.1.112.Final]
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:399)
[netty-transport-classes-epoll-4.1.112.Final.jar:4.1.112.Final]
at
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:994)
[netty-common-4.1.112.Final.jar:4.1.112.Final]
at
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
[netty-common-4.1.112.Final.jar:4.1.112.Final]
at
org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
[artemis-commons-2.36.0.jar:2.36.0]
Caused by: java.lang.IllegalArgumentException: Received an AmqpValue payload in
core tunneled AMQP message
at
org.apache.activemq.artemis.protocol.amqp.proton.AMQPTunneledCoreMessageReader.readBytes(AMQPTunneledCoreMessageReader.java:122)
~[artemis-amqp-protocol-2.36.0.jar:2.36.0]
at
org.apache.activemq.artemis.protocol.amqp.proton.ProtonAbstractReceiver.onMessage(ProtonAbstractReceiver.java:329)
~[artemis-amqp-protocol-2.36.0.jar:2.36.0]
... 32 more {code}
> IndexOutOfBoundsException in AMQP tunnelling of Core Messages and permanent
> stop of message replication via mirroring
> ---------------------------------------------------------------------------------------------------------------------
>
> Key: ARTEMIS-4970
> URL: https://issues.apache.org/jira/browse/ARTEMIS-4970
> Project: ActiveMQ Artemis
> Issue Type: Bug
> Components: AMQP, Broker
> Affects Versions: 2.35.0, 2.36.0
> Reporter: Jean-Pascal Briquet
> Priority: Major
>
> The IndexOutOfBoundsException error occurs randomly when messages are being
> replicated via async mirroring.
> Several thousands of messages can be replicated successfully before it
> happens.
> I have no reproduction scenario yet, as it is random but it happens several
> times per day.
> If needed, specific logging level can be enabled if that helps with the
> investigation.
> *Artemis setup:*
> The Artemis topology is composed by two Artemis clusters (of 3 groups) with
> ZK quorum (primary/backup).
> Dual async mirroring is enabled on queues on both clusters.
> *IndexOutOfBound error details*
> Most messages going through the replication link are of standard size and
> originated by Openwire or Core protocol. Large messages, averaging 150KB can
> be replicated too but are less frequent.
> Please note that the message is altered by an interceptor to add property
> "_BT_MAX_DELIVERY" when it reaches the broker.
> The message embedded in the stack trace below appears to have been
> redistributed within the cluster before being replicated, as user is
> ACTIVEMQ.CLUSTER.ADMIN.USER. I have seen it failing in non-redistributed
> scenario too.
> {noformat}
> 2024-08-06 03:44:00,596 WARN
> [org.apache.activemq.artemis.protocol.amqp.proton.ProtonServerSenderContext]
> readerIndex: 0, writerIndex: 11960 (expected: 0 <= readerIndex <= writerIndex
> <= capacity(11715))
> java.lang.IndexOutOfBoundsException: readerIndex: 0, writerIndex: 11960
> (expected: 0 <= readerIndex <= writerIndex <= capacity(11715))
> at
> io.netty.buffer.AbstractByteBuf.checkIndexBounds(AbstractByteBuf.java:112)
> ~[netty-buffer-4.1.112.Final.jar:4.1.112.Final]
> at io.netty.buffer.AbstractByteBuf.writerIndex(AbstractByteBuf.java:135)
> ~[netty-buffer-4.1.112.Final.jar:4.1.112.Final]
> at
> org.apache.activemq.artemis.protocol.amqp.proton.AMQPTunneledCoreMessageWriter.writeBytes(AMQPTunneledCoreMessageWriter.java:107)
> [artemis-amqp-protocol-2.36.0.jar:2.36.0]
> at
> org.apache.activemq.artemis.protocol.amqp.proton.MessageWriter.accept(MessageWriter.java:41)
> [artemis-amqp-protocol-2.36.0.jar:2.36.0]
> at
> org.apache.activemq.artemis.protocol.amqp.proton.MessageWriter.accept(MessageWriter.java:28)
> [artemis-amqp-protocol-2.36.0.jar:2.36.0]
> at
> org.apache.activemq.artemis.core.server.impl.MessageReferenceImpl.run(MessageReferenceImpl.java:136)
> [artemis-server-2.36.0.jar:2.36.0]
> at
> io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173)
> [netty-common-4.1.112.Final.jar:4.1.112.Final]
> at
> io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166)
> [netty-common-4.1.112.Final.jar:4.1.112.Final]
> at
> io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:469)
> [netty-common-4.1.112.Final.jar:4.1.112.Final]
> at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:405)
> [netty-transport-classes-epoll-4.1.112.Final.jar:4.1.112.Final]
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:994)
> [netty-common-4.1.112.Final.jar:4.1.112.Final]
> at
> io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
> [netty-common-4.1.112.Final.jar:4.1.112.Final]
> at
> org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
> [artemis-commons-2.36.0.jar:2.36.0]
> 2024-08-06 03:44:00,596 WARN [org.apache.activemq.artemis.core.server]
> AMQ222151: removing consumer which did not handle a message,
> consumer=ServerConsumerImpl [id=0, filter=null, binding=LocalQueueBinding
> [address=$ACTIVEMQ_ARTEMIS_MIRROR_dc2-group-1,
> queue=QueueImpl[name=$ACTIVEMQ_ARTEMIS_MIRROR_dc2-group-1,
> postOffice=PostOfficeImpl
> [server=ActiveMQServerImpl::name=artemis-dc1-primary-1],
> temp=false]@290df877, filter=null, name=$ACTIVEMQ_ARTEMIS_MIRROR_dc2-group-1,
> clusterName=$ACTIVEMQ_ARTEMIS_MIRROR_dc2-group-154a6ae45-26e5-11ee-837c-506b8d97040b],
> closed=false],
> message=Reference[282007612256]:RELIABLE:CoreMessage[messageID=282007612256,
> durable=true, userID=a51c7f00-53c7-11ef-bb35-506b8db21b19, priority=4,
> timestamp=Tue Aug 06 03:44:00 EDT 2024, expiration=0, durable=true,
> address=queue.ua.release.shared.internal-message.message-manager.to.message-services,
> size=11703,
> properties=TypedProperties[traceparent=00-f369c10867855a63b6d9c86cfd612249-e57085be9b3c4fc5-01,
> __AMQ_CID=ID:us01vwfmmm02-51592-638581215922967464-0:0,
> _AMQ_GROUP_SEQUENCE=0, __HDR_BROKER_IN_TIME=1722930240548,
> _AMQ_ROUTING_TYPE=1, __HDR_ARRIVAL=0, _BT_MAX_DELIVERY=30,
> __HDR_COMMAND_ID=7438781, JMSCorrelationID=O202408060944000003Q,
> _AMQ_VALIDATED_USER=ACTIVEMQ.CLUSTER.ADMIN.USER,
> _AMQ_DUPL_ID=ID:us01vwfmmm02-51592-638581215922967464-1:0:11:153438:1,
> __HDR_PRODUCER_ID=ID:us01vwfmmm02-51592-638581215922967464-1:0:11:153438,
> __HDR_MESSAGE_ID=ID:us01vwfmmm02-51592-638581215922967464-1:0:11:153438:1,
> __HDR_DROPPABLE=false]]@266147141
> java.lang.IndexOutOfBoundsException: readerIndex: 0, writerIndex: 11960
> (expected: 0 <= readerIndex <= writerIndex <= capacity(11715))
> at
> io.netty.buffer.AbstractByteBuf.checkIndexBounds(AbstractByteBuf.java:112)
> ~[netty-buffer-4.1.112.Final.jar:4.1.112.Final]
> at io.netty.buffer.AbstractByteBuf.writerIndex(AbstractByteBuf.java:135)
> ~[netty-buffer-4.1.112.Final.jar:4.1.112.Final]
> at
> org.apache.activemq.artemis.protocol.amqp.proton.AMQPTunneledCoreMessageWriter.writeBytes(AMQPTunneledCoreMessageWriter.java:107)
> [artemis-amqp-protocol-2.36.0.jar:2.36.0]
> at
> org.apache.activemq.artemis.protocol.amqp.proton.MessageWriter.accept(MessageWriter.java:41)
> [artemis-amqp-protocol-2.36.0.jar:2.36.0]
> at
> org.apache.activemq.artemis.protocol.amqp.proton.MessageWriter.accept(MessageWriter.java:28)
> [artemis-amqp-protocol-2.36.0.jar:2.36.0]
> at
> org.apache.activemq.artemis.core.server.impl.MessageReferenceImpl.run(MessageReferenceImpl.java:136)
> [artemis-server-2.36.0.jar:2.36.0]
> at
> io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173)
> [netty-common-4.1.112.Final.jar:4.1.112.Final]
> at
> io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166)
> [netty-common-4.1.112.Final.jar:4.1.112.Final]
> at
> io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:469)
> [netty-common-4.1.112.Final.jar:4.1.112.Final]
> at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:405)
> [netty-transport-classes-epoll-4.1.112.Final.jar:4.1.112.Final]
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:994)
> [netty-common-4.1.112.Final.jar:4.1.112.Final]
> at
> io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
> [netty-common-4.1.112.Final.jar:4.1.112.Final]
> at
> org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
> [artemis-commons-2.36.0.jar:2.36.0]{noformat}
> *Mirroring consumer stopped*
> The error triggers another problem where message replication is immediately
> stopped because the mirroring consumer is destroyed and is never recreated
> automatically (null pointer problem?).
> A workaround found is to close AMQP broker connections, which will trigger
> broker connection to restart and automatically restart mirroring consumer.
> Stacktrace of consumer failing to reconnect after the IndexOutOfBound
> Exception:
> {noformat}
> 2024-08-06 03:44:00,597 WARN
> [io.netty.util.concurrent.AbstractEventExecutor] A task raised an exception.
> Task:
> org.apache.activemq.artemis.protocol.amqp.broker.AMQPSessionCallback$$Lambda$1173/0x00007ff7dc82c228@41bb7410
> java.lang.NullPointerException: Cannot invoke
> "org.apache.activemq.artemis.protocol.amqp.proton.ProtonServerSenderContext.close(org.apache.qpid.proton.amqp.transport.ErrorCondition)"
> because the return value of
> "org.apache.activemq.artemis.core.server.ServerConsumer.getProtocolContext()"
> is null
> at
> org.apache.activemq.artemis.protocol.amqp.broker.AMQPSessionCallback.lambda$disconnect$5(AMQPSessionCallback.java:747)
> ~[artemis-amqp-protocol-2.36.0.jar:2.36.0]
> at
> io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173)
> ~[netty-common-4.1.112.Final.jar:4.1.112.Final]
> at
> io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166)
> [netty-common-4.1.112.Final.jar:4.1.112.Final]
> at
> io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:469)
> [netty-common-4.1.112.Final.jar:4.1.112.Final]
> at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:405)
> [netty-transport-classes-epoll-4.1.112.Final.jar:4.1.112.Final]
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:994)
> [netty-common-4.1.112.Final.jar:4.1.112.Final]
> at
> io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
> [netty-common-4.1.112.Final.jar:4.1.112.Final]
> at
> org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
> [artemis-commons-2.36.0.jar:2.36.0]{noformat}
> *Error on replica-side*
> {noformat}
> 2024-08-06 03:44:00,598 WARN
> [org.apache.activemq.artemis.protocol.amqp.proton.ProtonAbstractReceiver]
> Received an AmqpValue payload in core tunneled AMQP message
> java.lang.IllegalArgumentException: Received an AmqpValue payload in core
> tunneled AMQP message
> at
> org.apache.activemq.artemis.protocol.amqp.proton.AMQPTunneledCoreMessageReader.readBytes(AMQPTunneledCoreMessageReader.java:122)
> ~[artemis-amqp-protocol-2.36.0.jar:2.36.0]
> at
> org.apache.activemq.artemis.protocol.amqp.proton.ProtonAbstractReceiver.onMessage(ProtonAbstractReceiver.java:329)
> [artemis-amqp-protocol-2.36.0.jar:2.36.0]
> at
> org.apache.activemq.artemis.protocol.amqp.proton.AMQPConnectionContext.onDelivery(AMQPConnectionContext.java:890)
> [artemis-amqp-protocol-2.36.0.jar:2.36.0]
> at
> org.apache.activemq.artemis.protocol.amqp.proton.handler.Events.dispatch(Events.java:95)
> [artemis-amqp-protocol-2.36.0.jar:2.36.0]
> at
> org.apache.activemq.artemis.protocol.amqp.proton.handler.ProtonHandler.dispatch(ProtonHandler.java:585)
> [artemis-amqp-protocol-2.36.0.jar:2.36.0]
> at
> org.apache.activemq.artemis.protocol.amqp.proton.handler.ProtonHandler.handleFlush(ProtonHandler.java:391)
> [artemis-amqp-protocol-2.36.0.jar:2.36.0]
> at
> org.apache.activemq.artemis.protocol.amqp.proton.handler.ProtonHandler.flush(ProtonHandler.java:376)
> [artemis-amqp-protocol-2.36.0.jar:2.36.0]
> at
> org.apache.activemq.artemis.protocol.amqp.proton.handler.ProtonHandler.inputBuffer(ProtonHandler.java:334)
> [artemis-amqp-protocol-2.36.0.jar:2.36.0]
> at
> org.apache.activemq.artemis.protocol.amqp.proton.AMQPConnectionContext.inputBuffer(AMQPConnectionContext.java:293)
> [artemis-amqp-protocol-2.36.0.jar:2.36.0]
> at
> org.apache.activemq.artemis.protocol.amqp.broker.ActiveMQProtonRemotingConnection.bufferReceived(ActiveMQProtonRemotingConnection.java:174)
> [artemis-amqp-protocol-2.36.0.jar:2.36.0]
> at
> org.apache.activemq.artemis.core.remoting.server.impl.RemotingServiceImpl$DelegatingBufferHandler.bufferReceived(RemotingServiceImpl.java:723)
> [artemis-server-2.36.0.jar:2.36.0]
> at
> org.apache.activemq.artemis.core.remoting.impl.netty.ActiveMQChannelHandler.channelRead(ActiveMQChannelHandler.java:73)
> [artemis-core-client-2.36.0.jar:2.36.0]
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
> [netty-transport-4.1.112.Final.jar:4.1.112.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
> [netty-transport-4.1.112.Final.jar:4.1.112.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
> [netty-transport-4.1.112.Final.jar:4.1.112.Final]
> at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1473)
> [netty-handler-4.1.112.Final.jar:4.1.112.Final]
> at
> io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1336)
> [netty-handler-4.1.112.Final.jar:4.1.112.Final]
> at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1385)
> [netty-handler-4.1.112.Final.jar:4.1.112.Final]
> at
> io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:530)
> [netty-codec-4.1.112.Final.jar:4.1.112.Final]
> at
> io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:469)
> [netty-codec-4.1.112.Final.jar:4.1.112.Final]
> at
> io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290)
> [netty-codec-4.1.112.Final.jar:4.1.112.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
> [netty-transport-4.1.112.Final.jar:4.1.112.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
> [netty-transport-4.1.112.Final.jar:4.1.112.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
> [netty-transport-4.1.112.Final.jar:4.1.112.Final]
> at
> io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1407)
> [netty-transport-4.1.112.Final.jar:4.1.112.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
> [netty-transport-4.1.112.Final.jar:4.1.112.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
> [netty-transport-4.1.112.Final.jar:4.1.112.Final]
> at
> io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:918)
> [netty-transport-4.1.112.Final.jar:4.1.112.Final]
> at
> io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:799)
> [netty-transport-classes-epoll-4.1.112.Final.jar:4.1.112.Final]
> at
> io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:501)
> [netty-transport-classes-epoll-4.1.112.Final.jar:4.1.112.Final]
> at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:399)
> [netty-transport-classes-epoll-4.1.112.Final.jar:4.1.112.Final]
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:994)
> [netty-common-4.1.112.Final.jar:4.1.112.Final]
> at
> io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
> [netty-common-4.1.112.Final.jar:4.1.112.Final]
> at
> org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
> [artemis-commons-2.36.0.jar:2.36.0]
> 2024-08-06 03:44:00,603 WARN
> [org.apache.activemq.artemis.protocol.amqp.proton.handler.ProtonHandler]
> Received an AmqpValue payload in core tunneled AMQP message
> org.apache.activemq.artemis.protocol.amqp.exceptions.ActiveMQAMQPInternalErrorException:
> Received an AmqpValue payload in core tunneled AMQP message
> at
> org.apache.activemq.artemis.protocol.amqp.proton.ProtonAbstractReceiver.onMessage(ProtonAbstractReceiver.java:337)
> ~[artemis-amqp-protocol-2.36.0.jar:2.36.0]
> at
> org.apache.activemq.artemis.protocol.amqp.proton.AMQPConnectionContext.onDelivery(AMQPConnectionContext.java:890)
> ~[artemis-amqp-protocol-2.36.0.jar:2.36.0]
> at
> org.apache.activemq.artemis.protocol.amqp.proton.handler.Events.dispatch(Events.java:95)
> ~[artemis-amqp-protocol-2.36.0.jar:2.36.0]
> at
> org.apache.activemq.artemis.protocol.amqp.proton.handler.ProtonHandler.dispatch(ProtonHandler.java:585)
> [artemis-amqp-protocol-2.36.0.jar:2.36.0]
> at
> org.apache.activemq.artemis.protocol.amqp.proton.handler.ProtonHandler.handleFlush(ProtonHandler.java:391)
> [artemis-amqp-protocol-2.36.0.jar:2.36.0]
> at
> org.apache.activemq.artemis.protocol.amqp.proton.handler.ProtonHandler.flush(ProtonHandler.java:376)
> [artemis-amqp-protocol-2.36.0.jar:2.36.0]
> at
> org.apache.activemq.artemis.protocol.amqp.proton.handler.ProtonHandler.inputBuffer(ProtonHandler.java:334)
> [artemis-amqp-protocol-2.36.0.jar:2.36.0]
> at
> org.apache.activemq.artemis.protocol.amqp.proton.AMQPConnectionContext.inputBuffer(AMQPConnectionContext.java:293)
> [artemis-amqp-protocol-2.36.0.jar:2.36.0]
> at
> org.apache.activemq.artemis.protocol.amqp.broker.ActiveMQProtonRemotingConnection.bufferReceived(ActiveMQProtonRemotingConnection.java:174)
> [artemis-amqp-protocol-2.36.0.jar:2.36.0]
> at
> org.apache.activemq.artemis.core.remoting.server.impl.RemotingServiceImpl$DelegatingBufferHandler.bufferReceived(RemotingServiceImpl.java:723)
> [artemis-server-2.36.0.jar:2.36.0]
> at
> org.apache.activemq.artemis.core.remoting.impl.netty.ActiveMQChannelHandler.channelRead(ActiveMQChannelHandler.java:73)
> [artemis-core-client-2.36.0.jar:2.36.0]
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
> [netty-transport-4.1.112.Final.jar:4.1.112.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
> [netty-transport-4.1.112.Final.jar:4.1.112.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
> [netty-transport-4.1.112.Final.jar:4.1.112.Final]
> at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1473)
> [netty-handler-4.1.112.Final.jar:4.1.112.Final]
> at
> io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1336)
> [netty-handler-4.1.112.Final.jar:4.1.112.Final]
> at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1385)
> [netty-handler-4.1.112.Final.jar:4.1.112.Final]
> at
> io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:530)
> [netty-codec-4.1.112.Final.jar:4.1.112.Final]
> at
> io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:469)
> [netty-codec-4.1.112.Final.jar:4.1.112.Final]
> at
> io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290)
> [netty-codec-4.1.112.Final.jar:4.1.112.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
> [netty-transport-4.1.112.Final.jar:4.1.112.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
> [netty-transport-4.1.112.Final.jar:4.1.112.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
> [netty-transport-4.1.112.Final.jar:4.1.112.Final]
> at
> io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1407)
> [netty-transport-4.1.112.Final.jar:4.1.112.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
> [netty-transport-4.1.112.Final.jar:4.1.112.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
> [netty-transport-4.1.112.Final.jar:4.1.112.Final]
> at
> io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:918)
> [netty-transport-4.1.112.Final.jar:4.1.112.Final]
> at
> io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:799)
> [netty-transport-classes-epoll-4.1.112.Final.jar:4.1.112.Final]
> at
> io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:501)
> [netty-transport-classes-epoll-4.1.112.Final.jar:4.1.112.Final]
> at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:399)
> [netty-transport-classes-epoll-4.1.112.Final.jar:4.1.112.Final]
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:994)
> [netty-common-4.1.112.Final.jar:4.1.112.Final]
> at
> io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
> [netty-common-4.1.112.Final.jar:4.1.112.Final]
> at
> org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
> [artemis-commons-2.36.0.jar:2.36.0]
> Caused by: java.lang.IllegalArgumentException: Received an AmqpValue payload
> in core tunneled AMQP message
> at
> org.apache.activemq.artemis.protocol.amqp.proton.AMQPTunneledCoreMessageReader.readBytes(AMQPTunneledCoreMessageReader.java:122)
> ~[artemis-amqp-protocol-2.36.0.jar:2.36.0]
> at
> org.apache.activemq.artemis.protocol.amqp.proton.ProtonAbstractReceiver.onMessage(ProtonAbstractReceiver.java:329)
> ~[artemis-amqp-protocol-2.36.0.jar:2.36.0]
> ... 32 more{noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
For further information, visit: https://activemq.apache.org/contact
