Anton Roskvist created ARTEMIS-3973:
---------------------------------------
Summary: High pressure on paging can leave address stuck in paging
Key: ARTEMIS-3973
URL: https://issues.apache.org/jira/browse/ARTEMIS-3973
Project: ActiveMQ Artemis
Issue Type: Bug
Reporter: Anton Roskvist
When sending a significant load through the paging store, there are occasional
issues that causes the address to get stuck in paging and may end up in a
generally inconsistent state (incorrect metrics, and unreadable messages for
instance).
I have seen these stack traces associated with this issue:
{code:java}
WARN [org.apache.activemq.artemis.core.server] AMQ222225: Sending unexpected
exception to the client: java.lang.IllegalStateException: Unable to insert
at
io.netty.util.collection.IntObjectHashMap.put(IntObjectHashMap.java:141)
[netty-common-4.1.79.Final.jar:4.1.79.Final]
at
org.apache.activemq.artemis.core.paging.cursor.impl.PageSubscriptionImpl$PageCursorInfo.internalAddACK(PageSubscriptionImpl.java:1192)
[artemis-server-2.25.0.jar:2.25.0]
at
org.apache.activemq.artemis.core.paging.cursor.impl.PageSubscriptionImpl$PageCursorInfo.addACK(PageSubscriptionImpl.java:1174)
[artemis-server-2.25.0.jar:2.25.0]
at
org.apache.activemq.artemis.core.paging.cursor.impl.PageSubscriptionImpl.processACK(PageSubscriptionImpl.java:961)
[artemis-server-2.25.0.jar:2.25.0]
at
org.apache.activemq.artemis.core.paging.cursor.impl.PageSubscriptionImpl$PageCursorTX.afterCommit(PageSubscriptionImpl.java:1260)
[artemis-server-2.25.0.jar:2.25.0]
at
org.apache.activemq.artemis.core.transaction.impl.TransactionImpl.afterCommit(TransactionImpl.java:564)
[artemis-server-2.25.0.jar:2.25.0]
at
org.apache.activemq.artemis.core.transaction.impl.TransactionImpl$2.done(TransactionImpl.java:305)
[artemis-server-2.25.0.jar:2.25.0]
at
org.apache.activemq.artemis.core.persistence.impl.journal.OperationContextImpl.executeOnCompletion(OperationContextImpl.java:195)
[artemis-server-2.25.0.jar:2.25.0]
at
org.apache.activemq.artemis.core.persistence.impl.journal.OperationContextImpl.executeOnCompletion(OperationContextImpl.java:141)
[artemis-server-2.25.0.jar:2.25.0]
at
org.apache.activemq.artemis.core.persistence.impl.journal.AbstractJournalStorageManager.afterCompleteOperations(AbstractJournalStorageManager.java:338)
[artemis-server-2.25.0.jar:2.25.0]
at
org.apache.activemq.artemis.core.transaction.impl.TransactionImpl.commit(TransactionImpl.java:296)
[artemis-server-2.25.0.jar:2.25.0]
at
org.apache.activemq.artemis.core.transaction.impl.TransactionImpl.commit(TransactionImpl.java:247)
[artemis-server-2.25.0.jar:2.25.0]
at
org.apache.activemq.artemis.core.server.impl.ServerSessionImpl.commit(ServerSessionImpl.java:1320)
[artemis-server-2.25.0.jar:2.25.0]
at
org.apache.activemq.artemis.core.protocol.core.ServerSessionPacketHandler.slowPacketHandler(ServerSessionPacketHandler.java:474)
[artemis-server-2.25.0.jar:2.25.0]
at
org.apache.activemq.artemis.core.protocol.core.ServerSessionPacketHandler.onMessagePacket(ServerSessionPacketHandler.java:298)
[artemis-server-2.25.0.jar:2.25.0]
at org.apache.activemq.artemis.utils.actors.Actor.doTask(Actor.java:33)
[artemis-commons-2.25.0.jar:]
at
org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:67)
[artemis-commons-2.25.0.jar:]
at
org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:56)
[artemis-commons-2.25.0.jar:]
at
org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:31)
[artemis-commons-2.25.0.jar:]
at
org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:67)
[artemis-commons-2.25.0.jar:]
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
[java.base:]
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
[java.base:]
at
org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
[artemis-commons-2.25.0.jar:] {code}
and
{code:java}
WARN
[org.apache.activemq.artemis.core.paging.cursor.impl.PageSubscriptionImpl]
Index 16384 out of bounds for length 16384:
java.lang.ArrayIndexOutOfBoundsException: Index 16384 out of bounds for length
16384
at
io.netty.util.collection.IntObjectHashMap.indexOf(IntObjectHashMap.java:345)
[netty-common-4.1.79.Final.jar:4.1.79.Final]
at
io.netty.util.collection.IntObjectHashMap.get(IntObjectHashMap.java:114)
[netty-common-4.1.79.Final.jar:4.1.79.Final]
at
org.apache.activemq.artemis.core.paging.cursor.impl.PageSubscriptionImpl$PageCursorInfo.isAck(PageSubscriptionImpl.java:1061)
[artemis-server-2.25.0.jar:2.25.0]
at
org.apache.activemq.artemis.core.paging.cursor.impl.PageSubscriptionImpl$CursorIterator.moveNext(PageSubscriptionImpl.java:1407)
[artemis-server-2.25.0.jar:2.25.0]
at
org.apache.activemq.artemis.core.paging.cursor.impl.PageSubscriptionImpl$CursorIterator.next(PageSubscriptionImpl.java:1348)
[artemis-server-2.25.0.jar:2.25.0]
at
org.apache.activemq.artemis.core.paging.cursor.impl.PageSubscriptionImpl$CursorIterator.tryNext(PageSubscriptionImpl.java:1492)
[artemis-server-2.25.0.jar:2.25.0]
at
org.apache.activemq.artemis.core.server.impl.QueueImpl.depage(QueueImpl.java:3294)
[artemis-server-2.25.0.jar:2.25.0]
at
org.apache.activemq.artemis.core.server.impl.QueueImpl.lambda$scheduleDepage$7(QueueImpl.java:3266)
[artemis-server-2.25.0.jar:2.25.0]
at
org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:56)
[artemis-commons-2.25.0.jar:]
at
org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:31)
[artemis-commons-2.25.0.jar:]
at
org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:67)
[artemis-commons-2.25.0.jar:]
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
[java.base:]
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
[java.base:]
at
org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
[artemis-commons-2.25.0.jar:] {code}
I have yet to reproduce this in a test but it happens quite regularly as the
broker approaches it's throughput limit based on CPU-resources, in my case
that's somewhere in the neighborhood of 250k messages/s.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
