Sorry, I forgot the stack-trace: 2019-07-01 01:32:37,574 ERROR [org.apache.activemq.artemis.core.server] AMQ224000: Failure in initialisation: java.lang.IllegalStateException: Invalid record type -93
at org.apache.activemq.artemis.core.persistence.impl.journal.AbstractJournalStorageManager.loadMessageJournal(AbstractJournalStorageManager.java:1137) [artemis-server-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.loadJournals(ActiveMQServerImpl.java:2575) [artemis-server-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.initialisePart2(ActiveMQServerImpl.java:2336) [artemis-server-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.core.server.impl.LiveOnlyActivation.run(LiveOnlyActivation.java:71) [artemis-server-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.internalStart(ActiveMQServerImpl.java:543) [artemis-server-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.start(ActiveMQServerImpl.java:482) [artemis-server-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.integration.FileBroker.start(FileBroker.java:111) [artemis-cli-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.cli.commands.Run.execute(Run.java:82) [artemis-cli-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.cli.Artemis.internalExecute(Artemis.java:149) [artemis-cli-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.cli.Artemis.execute(Artemis.java:96) [artemis-cli-2.4.0.jar:2.4.0] at org.apache.activemq.artemis.cli.Artemis.execute(Artemis.java:123) [artemis-cli-2.4.0.jar:2.4.0] Unluckily the operation team deleted data journals and restarted broker. So we have nothing to do but to read code to find the root cause :( Justin Bertram <[email protected]> 于2019年7月1日周一 下午10:37写道： > Do you have the full stack-trace of the exception? Also, do you still have > the damaged journal? It would be good to have it to test with. > > > Justin > > On Mon, Jul 1, 2019 at 4:57 AM yw yw <[email protected]> wrote: > > > Hi, All > > > > > > Yesterday our cluster experienced a sudden loss of power. When we started > > broker after power brought back, exception occurred: > > > > > > The exception showed the userRecordType loaded was illegal. The operation > > team deleted data journals and broker started successfully. > > > > > > It was a pity we didn't backup the problematic journal files. We checked > > dmesg command output, no disk errors. SMART tests on disk also showed > disk > > not broken. Then we digged into code(JournalImpl::readJournalFile) and > > tried to find something. We have two doubts with the code. > > > > > > First doubt: > > > > The comment says "I - We scan for any valid record on the file. If a hole > > happened on the middle of the file we keep looking until all the > > possibilities are gone". > > > > Considering we're appending journal file and fileId is strictly > increasing, > > so we can just skip the whole file if the fileId of record is not equal > to > > file id. IMO the rest records in the file are the same, no need to read > > them. Should we keep looking all the possibilities, is there a > > possibility(very low one) that we just assemble a record of which fileId, > > recordType, checkSize all qualifies but actually does not exist? > > > > Our second one: > > > > In the case of power outage where part of record is written into disk, > e.g. > > recordyType,fileId is successfully written, we may read the old record > > though fileId is latest? > > > > Can anyone shed some lights on this please? Thanks. > > >
