[
https://issues.apache.org/jira/browse/AMQ-6831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16200130#comment-16200130
]
Gary Tully commented on AMQ-6831:
---------------------------------
There is a workaround but it is temporary. Here is the background, there is a
problem with AMQ-6771 that speeds up the checkForCorruptJournalFiles. On a
restart, it is possible that the check will report corruption when there is
none. It is independent of the preallocation strategy or preallocation scope.
If ignoreMissingJournalFiles=true is configured, this can lead to unnecessary
loss of data, otherwise the broker will fail to start and report the corruption
and there will be no loss.
If there is no reason to suspect corruption it may be this defect. The defect
relates to the parsing of a batch of journal data. If the journal record header
crosses a batch boundary at a particular point, the defect can happen.
Disabling the checkForCorruptJournalFiles will avoid the problem but also
changing or varying the read batch size may avoid the problem.
The read batch size is controlled by journalMaxWriteBatchSize which default to
4mb. Setting that to 10mb or some larger number may help. It is a little random
but the window for this to occur is quite small. If the
journalMaxWriteBatchSize is half of of the journalSize (and there is enough
memory) then two reads will parse a journal file and the chance of a batch
being on a boundary and hitting this defect is very small.
> Broker fails to start complaining about missing/corrupt journal files after
> upgrading to 5.15.1
> -----------------------------------------------------------------------------------------------
>
> Key: AMQ-6831
> URL: https://issues.apache.org/jira/browse/AMQ-6831
> Project: ActiveMQ
> Issue Type: Bug
> Affects Versions: 5.15.1
> Reporter: masc
> Assignee: Gary Tully
> Priority: Critical
> Fix For: 5.16.0
>
> Attachments: kahadb.zip
>
>
> {code}
> 2017-10-09 12:29:36.802 ERROR 4197 --- [pool-3-thread-1]
> org.deku.leoz.node.Application : java.io.IOException: Detected
> missing/corrupt journal files referenced by:[0:ActiveMQ.DLQ] 3 messages
> affected.
> java.lang.Error: java.io.IOException: Detected missing/corrupt journal files
> referenced by:[0:ActiveMQ.DLQ] 3 messages affected.
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1148)
> ~[na:1.8.0_131]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> ~[na:1.8.0_131]
> at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_131]
> Caused by: java.io.IOException: Detected missing/corrupt journal files
> referenced by:[0:ActiveMQ.DLQ] 3 messages affected.
> at
> org.apache.activemq.store.kahadb.MessageDatabase.recoverIndex(MessageDatabase.java:965)
> ~[activemq-kahadb-store-5.15.1.jar:5.15.1]
> at
> org.apache.activemq.store.kahadb.MessageDatabase$5.execute(MessageDatabase.java:717)
> ~[activemq-kahadb-store-5.15.1.jar:5.15.1]
> at
> org.apache.activemq.store.kahadb.disk.page.Transaction.execute(Transaction.java:779)
> ~[activemq-kahadb-store-5.15.1.jar:5.15.1]
> at
> org.apache.activemq.store.kahadb.MessageDatabase.recover(MessageDatabase.java:714)
> ~[activemq-kahadb-store-5.15.1.jar:5.15.1]
> at
> org.apache.activemq.store.kahadb.MessageDatabase.open(MessageDatabase.java:473)
> ~[activemq-kahadb-store-5.15.1.jar:5.15.1]
> at
> org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:493)
> ~[activemq-kahadb-store-5.15.1.jar:5.15.1]
> at
> org.apache.activemq.store.kahadb.MessageDatabase.doStart(MessageDatabase.java:297)
> ~[activemq-kahadb-store-5.15.1.jar:5.15.1]
> at
> org.apache.activemq.store.kahadb.KahaDBStore.doStart(KahaDBStore.java:219)
> ~[activemq-kahadb-store-5.15.1.jar:5.15.1]
> at
> org.apache.activemq.util.ServiceSupport.start(ServiceSupport.java:55)
> ~[activemq-client-5.15.1.jar:5.15.1]
> at
> org.apache.activemq.store.kahadb.KahaDBPersistenceAdapter.doStart(KahaDBPersistenceAdapter.java:232)
> ~[activemq-kahadb-store-5.15.1.jar:5.15.1]
> {code}
> Downgrading to 5.15.0 resolves the problem.
> This is an embedded broker setup, using
> {code:none}
> val pa = brokerService.persistenceAdapter as KahaDBPersistenceAdapter
> pa.isCheckForCorruptJournalFiles = true
> pa.isIgnoreMissingJournalfiles = false
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
