Ilkka Virolainen created ARTEMIS-1889:
-----------------------------------------
Summary: Replication Failure During Compacting
Key: ARTEMIS-1889
URL: https://issues.apache.org/jira/browse/ARTEMIS-1889
Project: ActiveMQ Artemis
Issue Type: Bug
Components: Broker
Affects Versions: 2.6.0, 2.5.0
Environment: RHEL 6.2, Windows 7
Reporter: Ilkka Virolainen
It is possible for the broker to corrupt its journal during operations and
subsequently fail replication when configured as a replicating master/slave
pair. Exceptions on master node:
{noformat}
15:00:18,074 ERROR [org.apache.activemq.artemis.core.server] AMQ224016: Caught
exception:
ActiveMQIllegalStateException[errorType=ILLEGAL_STATE message=AMQ119028:
Consumer 0 doesn't
exist on the server]
at
org.apache.activemq.artemis.core.server.impl.ServerSessionImpl.findConsumer(ServerSessionImpl.java:889)
[artemis-server-2.5.0.jar:2.5.0]
at
org.apache.activemq.artemis.core.server.impl.ServerSessionImpl.acknowledge(ServerSessionImpl.java:858)
[artemis-server-2.5.0.jar:2.5.0]
at
org.apache.activemq.artemis.core.protocol.core.ServerSessionPacketHandler.onSessionAcknowledge(ServerSessionPacketHandler.java:632)
[artemis-server-2.5.0.jar:2.5.0]
at
org.apache.activemq.artemis.core.protocol.core.ServerSessionPacketHandler.onMessagePacket(ServerSessionPacketHandler.java:268)
[artemis-server-2.5.0.jar:2.5.0]
at org.apache.activemq.artemis.utils.actors.Actor.doTask(Actor.java:33)
[artemis-commons-2.5.0.jar:2.5.0]
at
org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:66)
[artemis-commons-2.5.0.jar:2.5.0]
at
org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:42)
[artemis-commons-2.5.0.jar:2.5.0]
at
org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:31)
[artemis-commons-2.5.0.jar:2.5.0]
at
org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:66)
[artemis-commons-2.5.0.jar:2.5.0]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[rt.jar:1.8.0_162]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[rt.jar:1.8.0_162]
at
org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
[artemis-commons-2.5.0.jar:2.5.0]
{noformat}
After this both master and slave log these errors:
{noformat}
20:54:15,063 WARN [org.apache.activemq.artemis.journal] AMQ142032: Error
reading journal
file: java.lang.IllegalStateException: Inconsistency during compacting:
RollbackRecord ID
= 4452580183 for an already rolled back transaction during compacting
at
org.apache.activemq.artemis.core.journal.impl.JournalCompactor.onReadRollbackRecord(JournalCompactor.java:349)
[artemis-journal-2.5.0.jar:2.5.0]
at
org.apache.activemq.artemis.core.journal.impl.JournalImpl.readJournalFile(JournalImpl.java:733)
[artemis-journal-2.5.0.jar:2.5.0]
at
org.apache.activemq.artemis.core.journal.impl.JournalImpl.compact(JournalImpl.java:1566)
[artemis-journal-2.5.0.jar:2.5.0]
at
org.apache.activemq.artemis.core.journal.impl.JournalImpl$14.run(JournalImpl.java:2111)
[artemis-journal-2.5.0.jar:2.5.0]
at
org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:42)
[artemis-commons-2.5.0.jar:2.5.0]
at
org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:31)
[artemis-commons-2.5.0.jar:2.5.0]
at
org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:66)
[artemis-commons-2.5.0.jar:2.5.0]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[rt.jar:1.8.0_162]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[rt.jar:1.8.0_162]
at
org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
[artemis-commons-2.5.0.jar:2.5.0]
20:54:15,067 WARN [org.apache.activemq.artemis.journal] AMQ142011: Error on
reading compacting
for JournalFileImpl: (activemq-data-259.amq id = 355, recordID = 355)
20:54:15,104 ERROR [org.apache.activemq.artemis.journal] AMQ144003: Error
compacting: java.lang.Exception:
Error on reading compacting for JournalFileImpl: (activemq-data-259.amq id =
355, recordID
= 355)
at
org.apache.activemq.artemis.core.journal.impl.JournalImpl.compact(JournalImpl.java:1569)
[artemis-journal-2.5.0.jar:2.5.0]
at
org.apache.activemq.artemis.core.journal.impl.JournalImpl$14.run(JournalImpl.java:2111)
[artemis-journal-2.5.0.jar:2.5.0]
at
org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:42)
[artemis-commons-2.5.0.jar:2.5.0]
at
org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:31)
[artemis-commons-2.5.0.jar:2.5.0]
at
org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:66)
[artemis-commons-2.5.0.jar:2.5.0]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[rt.jar:1.8.0_162]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[rt.jar:1.8.0_162]
at
org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
[artemis-commons-2.5.0.jar:2.5.0]
Caused by: java.lang.Exception: Inconsistency during compacting: RollbackRecord
ID = 4452580183
for an already rolled back transaction during compacting
at
org.apache.activemq.artemis.core.journal.impl.JournalImpl.readJournalFile(JournalImpl.java:761)
[artemis-journal-2.5.0.jar:2.5.0]
at
org.apache.activemq.artemis.core.journal.impl.JournalImpl.compact(JournalImpl.java:1566)
[artemis-journal-2.5.0.jar:2.5.0]
... 7 more
Caused by: java.lang.IllegalStateException: Inconsistency during compacting:
RollbackRecord
ID = 4452580183 for an already rolled back transaction during compacting
at
org.apache.activemq.artemis.core.journal.impl.JournalCompactor.onReadRollbackRecord(JournalCompactor.java:349)
[artemis-journal-2.5.0.jar:2.5.0]
at
org.apache.activemq.artemis.core.journal.impl.JournalImpl.readJournalFile(JournalImpl.java:733)
[artemis-journal-2.5.0.jar:2.5.0]
... 8 more{noformat}
This results in HA not working. Restarting the broker logs:
{noformat}
08:58:28,608 WARN [org.apache.activemq.artemis.core.server] AMQ222013: Error
when trying
to start replication: java.lang.RuntimeException: Error during compact, look at
the logs
at
org.apache.activemq.artemis.core.journal.impl.JournalImpl.scheduleCompactAndBlock(JournalImpl.java:1481)
[artemis-journal-2.5.0.jar:2.5.0]
at
org.apache.activemq.artemis.core.persistence.impl.journal.JournalStorageManager.startReplication(JournalStorageManager.java:553)
[artemis-server-2.5.0.jar:2.5.0]
at
org.apache.activemq.artemis.core.server.impl.SharedNothingLiveActivation$2.run(SharedNothingLiveActivation.java:178)
[artemis-server-2.5.0.jar:2.5.0]
at java.lang.Thread.run(Thread.java:748)
[rt.jar:1.8.0_162]{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
