henrypoter opened a new issue #1641: INDEX_OUT_OF_RANGE caused broker & 
nameserver shutdown
URL: https://github.com/apache/rocketmq/issues/1641
 
 
   1. Please describe the issue you observed:
   
   - What did you do (The steps to reproduce)?
   rocketmq版本 4.6.0。使用dledger模式部署了一个broker分片,1主2从。
   其中有一个从节点在压测过程中broker和namserver进程终止了
   错误信息
   `[rocketmq@y2_web212 rocketmqlogs]$ vi broker_default.log 
   
   2019-12-05 03:38:01 ERROR EntryDispatcher-n0-n1 - [Push-n1]Error in 
EntryDispatcher-n0-n1 writeIndex=29222072 compareIndex=-1
   io.openmessaging.storage.dledger.exception.DLedgerException: 
[code=410,name=INDEX_OUT_OF_RANGE,desc=] 29221072 should between 
300627707-313007211
           at 
io.openmessaging.storage.dledger.utils.PreConditions.check(PreConditions.java:41)
 ~[dledger-0.1.jar:na]
           at 
io.openmessaging.storage.dledger.store.file.DLedgerMmapFileStore.get(DLedgerMmapFileStore.java:479)
 ~[dledger-0.1.jar:na]
           at 
io.openmessaging.storage.dledger.DLedgerEntryPusher$EntryDispatcher.doAppendInner(DLedgerEntryPusher.java:389)
 ~[dledger-0.1.jar:na]
           at 
io.openmessaging.storage.dledger.DLedgerEntryPusher$EntryDispatcher.doCheckAppendResponse(DLedgerEntryPusher.java:434)
 ~[dledger-0.1.jar:na]
           at 
io.openmessaging.storage.dledger.DLedgerEntryPusher$EntryDispatcher.doAppend(DLedgerEntryPusher.java:461)
 ~[dledger-0.1.jar:na]
           at 
io.openmessaging.storage.dledger.DLedgerEntryPusher$EntryDispatcher.doWork(DLedgerEntryPusher.java:602)
 ~[dledger-0.1.jar:na]
           at 
io.openmessaging.storage.dledger.ShutdownAbleThread.run(ShutdownAbleThread.java:87)
 [dledger-0.1.jar:na]
   2019-12-05 03:38:02 INFO DLedgerFlushDataService - Flush data cost=691 ms
   2019-12-05 03:38:02 ERROR StateMaintainer - Send heartBeat request failed 
info[group=RaftNode00,term=1,code=200,local=n0,remote=n1,leader=n0]
   org.apache.rocketmq.remoting.exception.RemotingConnectException: connect to 
192.168.111.213:40911 failed
           at 
org.apache.rocketmq.remoting.netty.NettyRemotingClient.invokeAsync(NettyRemotingClient.java:531)
 ~[rocketmq-remoting-4.6.0.jar:4.6.0]
           at 
io.openmessaging.storage.dledger.DLedgerRpcNettyService.heartBeat(DLedgerRpcNettyService.java:118)
 ~[dledger-0.1.jar:na]
           at 
io.openmessaging.storage.dledger.DLedgerLeaderElector.sendHeartbeats(DLedgerLeaderElector.java:252)
 [dledger-0.1.jar:na]
           at 
io.openmessaging.storage.dledger.DLedgerLeaderElector.maintainAsLeader(DLedgerLeaderElector.java:320)
 [dledger-0.1.jar:na]
           at 
io.openmessaging.storage.dledger.DLedgerLeaderElector.maintainState(DLedgerLeaderElector.java:506)
 [dledger-0.1.jar:na]
           at 
io.openmessaging.storage.dledger.DLedgerLeaderElector.access$200(DLedgerLeaderElector.java:40)
 [dledger-0.1.jar:na]
           at 
io.openmessaging.storage.dledger.DLedgerLeaderElector$StateMaintainer.doWork(DLedgerLeaderElector.java:548)
 [dledger-0.1.jar:na]
           at 
io.openmessaging.storage.dledger.ShutdownAbleThread.run(ShutdownAbleThread.java:87)
 [dledger-0.1.jar:na]
   2019-12-05 03:38:03 WARN EntryDispatcher-n0-n1 - [Push-n1]Retry to push 
entry at 29221072
   2019-12-05 03:38:03 ERROR EntryDispatcher-n0-n1 - [Push-n1]Error in 
EntryDispatcher-n0-n1 writeIndex=29222072 compareIndex=-1
   io.openmessaging.storage.dledger.exception.DLedgerException: 
[code=410,name=INDEX_OUT_OF_RANGE,desc=] 29221072 should between 
300627707-313007409
           at 
io.openmessaging.storage.dledger.utils.PreConditions.check(PreConditions.java:41)
 ~[dledger-0.1.jar:na]
           at 
io.openmessaging.storage.dledger.store.file.DLedgerMmapFileStore.get(DLedgerMmapFileStore.java:479)
 ~[dledger-0.1.jar:na]
           at 
io.openmessaging.storage.dledger.DLedgerEntryPusher$EntryDispatcher.doAppendInner(DLedgerEntryPusher.java:389)
 ~[dledger-0.1.jar:na]
           at 
io.openmessaging.storage.dledger.DLedgerEntryPusher$EntryDispatcher.doCheckAppendResponse(DLedgerEntryPusher.java:434)
 ~[dledger-0.1.jar:na]
           at 
io.openmessaging.storage.dledger.DLedgerEntryPusher$EntryDispatcher.doAppend(DLedgerEntryPusher.java:461)
 ~[dledger-0.1.jar:na]
           at 
io.openmessaging.storage.dledger.DLedgerEntryPusher$EntryDispatcher.doWork(DLedgerEntryPusher.java:602)
 ~[dledger-0.1.jar:na]
           at 
io.openmessaging.storage.dledger.ShutdownAbleThread.run(ShutdownAbleThread.java:87)
 [dledger-0.1.jar:na]
   2019-12-05 03:38:03 INFO QuorumAckChecker - [n0][LEADER] term=1 
ledgerBegin=300627707 ledgerEnd=313007409 committed=313007409 
watermarks={1:{"n0":313007409,"n1":29221071,"n2":313007409}}
   2019-12-05 03:38:03 WARN EntryDispatcher-n0-n1 - [Push-n1]Retry to push 
entry at 29221072
   2019-12-05 03:38:03 ERROR EntryDispatcher-n0-n1 - [Push-n1]Error in 
EntryDispatcher-n0-n1 writeIndex=29222072 compareIndex=-1
   io.openmessaging.storage.dledger.exception.DLedgerException: 
[code=410,name=INDEX_OUT_OF_RANGE,desc=] 29221072 should between 
300627707-313007431
           at 
io.openmessaging.storage.dledger.utils.PreConditions.check(PreConditions.java:41)
 ~[dledger-0.1.jar:na]
           at 
io.openmessaging.storage.dledger.store.file.DLedgerMmapFileStore.get(DLedgerMmapFileStore.java:479)
 ~[dledger-0.1.jar:na]
           at 
io.openmessaging.storage.dledger.DLedgerEntryPusher$EntryDispatcher.doAppendInner(DLedgerEntryPusher.java:389)
 ~[dledger-0.1.jar:na]
           at 
io.openmessaging.storage.dledger.DLedgerEntryPusher$EntryDispatcher.doCheckAppendResponse(DLedgerEntryPusher.java:434)
 ~[dledger-0.1.jar:na]
           at 
io.openmessaging.storage.dledger.DLedgerEntryPusher$EntryDispatcher.doAppend(DLedgerEntryPusher.java:461)
 ~[dledger-0.1.jar:na]
           at 
io.openmessaging.storage.dledger.DLedgerEntryPusher$EntryDispatcher.doWork(DLedgerEntryPusher.java:602)
 ~[dledger-0.1.jar:na]
           at 
io.openmessaging.storage.dledger.ShutdownAbleThread.run(ShutdownAbleThread.java:87)
 [dledger-0.1.jar:na]
   2019-12-05 03:38:04 WARN EntryDispatcher-n0-n1 - [Push-n1]Retry to push 
entry at 29221072
   2019-12-05 03:38:04 ERROR EntryDispatcher-n0-n1 - [Push-n1]Error in 
EntryDispatcher-n0-n1 writeIndex=29222072 compareIndex=-1
   io.openmessaging.storage.dledger.exception.DLedgerException: 
[code=410,name=INDEX_OUT_OF_RANGE,desc=] 29221072 should between 
300627707-313007473
           at 
io.openmessaging.storage.dledger.utils.PreConditions.check(PreConditions.java:41)
 ~[dledger-0.1.jar:na]
           at 
io.openmessaging.storage.dledger.store.file.DLedgerMmapFileStore.get(DLedgerMmapFileStore.java:479)
 ~[dledger-0.1.jar:na]
           at 
io.openmessaging.storage.dledger.DLedgerEntryPusher$EntryDispatcher.doAppendInner(DLedgerEntryPusher.java:389)
 ~[dledger-0.1.jar:na]
           at 
io.openmessaging.storage.dledger.DLedgerEntryPusher$EntryDispatcher.doCheckAppendResponse(DLedgerEntryPusher.java:434)
 ~[dledger-0.1.jar:na]
           at 
io.openmessaging.storage.dledger.DLedgerEntryPusher$EntryDispatcher.doAppend(DLedgerEntryPusher.java:461)
 ~[dledger-0.1.jar:na]
           at 
io.openmessaging.storage.dledger.DLedgerEntryPusher$EntryDispatcher.doWork(DLedgerEntryPusher.java:602)
 ~[dledger-0.1.jar:na]
           at 
io.openmessaging.storage.dledger.ShutdownAbleThread.run(ShutdownAbleThread.java:87)
 [dledger-0.1.jar:na]
   2019-12-05 03:38:04 WARN EntryDispatcher-n0-n1 - [Push-n1]Retry to push 
entry at 29221072
   2019-12-05 03:38:04 ERROR EntryDispatcher-n0-n1 - [Push-n1]Error in 
EntryDispatcher-n0-n1 writeIndex=29222072 compareIndex=-1
   ?ERROR      `
   
   重启broker和NameServer后恢复log
   
   `
   [rocketmq@y2zomq213 rocketmqlogs]$ tail -fn 111111111 *.log | grep recover
   2019-12-05 14:20:24 INFO main - Begin to recover data from 
entryIndex=629360492 fileIndex=17 fileSize=20 
fileName=/home/rocketmq/store/dledger-n1/data/00000000426275504128 
   2019-12-05 14:20:25 INFO main - Trying to recover data file 
/home/rocketmq/store/dledger-n1/data/00000000427349245952
   2019-12-05 14:20:26 INFO main - Trying to recover data file 
/home/rocketmq/store/dledger-n1/data/00000000428422987776
           at 
io.openmessaging.storage.dledger.store.file.DLedgerMmapFileStore.recover(DLedgerMmapFileStore.java:216)
 ~[dledger-0.1.jar:na]
           at 
org.apache.rocketmq.store.dledger.DLedgerCommitLog.recover(DLedgerCommitLog.java:253)
 [rocketmq-store-4.6.0.jar:4.6.0]
           at 
org.apache.rocketmq.store.dledger.DLedgerCommitLog.recoverNormally(DLedgerCommitLog.java:305)
 [rocketmq-store-4.6.0.jar:4.6.0]
           at 
org.apache.rocketmq.store.DefaultMessageStore.recover(DefaultMessageStore.java:1334)
 [rocketmq-store-4.6.0.jar:4.6.0]
   2019-12-05 22:28:18 INFO main - Begin to recover data from 
entryIndex=629360492 fileIndex=17 fileSize=20 
fileName=/home/rocketmq/store/dledger-n1/data/00000000426275504128 
   2019-12-05 22:28:19 INFO main - Trying to recover data file 
/home/rocketmq/store/dledger-n1/data/00000000427349245952
   2019-12-05 22:28:20 INFO main - Trying to recover data file 
/home/rocketmq/store/dledger-n1/data/00000000428422987776
           at 
io.openmessaging.storage.dledger.store.file.DLedgerMmapFileStore.recover(DLedgerMmapFileStore.java:216)
 ~[dledger-0.1.jar:na]
           at 
org.apache.rocketmq.store.dledger.DLedgerCommitLog.recover(DLedgerCommitLog.java:253)
 [rocketmq-store-4.6.0.jar:4.6.0]
           at 
org.apache.rocketmq.store.dledger.DLedgerCommitLog.recoverNormally(DLedgerCommitLog.java:305)
 [rocketmq-store-4.6.0.jar:4.6.0]
           at 
org.apache.rocketmq.store.DefaultMessageStore.recover(DefaultMessageStore.java:1334)
 [rocketmq-store-4.6.0.jar:4.6.0]
   2019-12-03 10:47:56 INFO main - recover next consume queue file, 
/home/rocketmq/store/consumequeue/uat-topic/0/00000000000012000000
   2019-12-03 10:47:56 INFO main - recover next consume queue file, 
/home/rocketmq/store/consumequeue/uat-topic/0/00000000000018000000
   2019-12-03 10:47:56 INFO main - recover current consume queue file over,  
/home/rocketmq/store/consumequeue/uat-topic/0/00000000000018000000 0 0 0
   2019-12-03 10:47:56 INFO main - recover current consume queue queue over 
/home/rocketmq/store/consumequeue/uat-topic/0/00000000000018000000 22122820
   2019-12-03 10:47:56 INFO main - recover next consume queue file, 
/home/rocketmq/store/consumequeue/uat-topic/1/00000000000012000000
   2019-12-03 10:47:56 INFO main - recover next consume queue file, 
/home/rocketmq/store/consumequeue/uat-topic/1/00000000000018000000
   2019-12-03 10:47:56 INFO main - recover current consume queue file over,  
/home/rocketmq/store/consumequeue/uat-topic/1/00000000000018000000 0 0 0
   2019-12-03 10:47:56 INFO main - recover current consume queue queue over 
/home/rocketmq/store/consumequeue/uat-topic/1/00000000000018000000 22123000
   2019-12-03 10:47:56 INFO main - recover next consume queue file, 
/home/rocketmq/store/consumequeue/uat-topic/2/00000000000012000000
   2019-12-03 10:47:56 INFO main - recover next consume queue f`
   - What did you expect to see?
   
   
   2. Please tell us about your environment:
   
   3. Other information (e.g. detailed explanation, logs, related issues, 
suggestions how to fix, etc):
    

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[email protected]


With regards,
Apache Git Services

Reply via email to