[jira] [Commented] (ROCKETMQ-279) add commit log data and consume queue data logic consistent check when start mq

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/ROCKETMQ-279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16150277#comment-16150277
 ] 

ASF GitHub Bot commented on ROCKETMQ-279:
-

Github user fuyou001 closed the pull request at:

https://github.com/apache/incubator-rocketmq/pull/159


> add commit log data and consume queue data logic consistent check when start 
> mq
> ---
>
> Key: ROCKETMQ-279
> URL: https://issues.apache.org/jira/browse/ROCKETMQ-279
> Project: Apache RocketMQ
>  Issue Type: Bug
>  Components: rocketmq-store
>Affects Versions: 4.0.0-incubating, 4.1.0-incubating
>Reporter: yubaofu
>Assignee: yukon
> Fix For: 4.2.0-incubating
>
>
> commit log and consume data may logic inconsistent ,
> because  [ROCKETMQ-265|https://issues.apache.org/jira/browse/ROCKETMQ-265],
> so we add  data consistent check,when mq start.
> if check failed, start failed.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (ROCKETMQ-279) add commit log data and consume queue data logic consistent check when start mq

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/ROCKETMQ-279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16150020#comment-16150020
 ] 

ASF GitHub Bot commented on ROCKETMQ-279:
-

Github user coveralls commented on the issue:

https://github.com/apache/incubator-rocketmq/pull/159
  

[![Coverage 
Status](https://coveralls.io/builds/13082255/badge)](https://coveralls.io/builds/13082255)

Coverage decreased (-0.2%) to 38.511% when pulling 
**4b98ecac5ae00933cab648dc7b3446ea3c88e1bd on fuyou001:ROCKETMQ-2792** into 
**845830865fc37d0364a19cbd89ceaf8a30b37e1c on apache:develop**.



> add commit log data and consume queue data logic consistent check when start 
> mq
> ---
>
> Key: ROCKETMQ-279
> URL: https://issues.apache.org/jira/browse/ROCKETMQ-279
> Project: Apache RocketMQ
>  Issue Type: Bug
>  Components: rocketmq-store
>Affects Versions: 4.0.0-incubating, 4.1.0-incubating
>Reporter: yubaofu
>Assignee: yukon
> Fix For: 4.2.0-incubating
>
>
> commit log and consume data may logic inconsistent ,
> because  [ROCKETMQ-265|https://issues.apache.org/jira/browse/ROCKETMQ-265],
> so we add  data consistent check,when mq start.
> if check failed, start failed.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (ROCKETMQ-279) add commit log data and consume queue data logic consistent check when start mq

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/ROCKETMQ-279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1614#comment-1614
 ] 

ASF GitHub Bot commented on ROCKETMQ-279:
-

Github user fuyou001 commented on a diff in the pull request:

https://github.com/apache/incubator-rocketmq/pull/159#discussion_r136494489
  
--- Diff: store/src/main/java/org/apache/rocketmq/store/ConsumeQueue.java 
---
@@ -84,6 +84,58 @@ public boolean load() {
 return result;
 }
 
+public void checkCommitLogAndConsumeQueueConsistent() {
+
+String queueDir = this.storePath
++ File.separator + topic
++ File.separator + queueId + File.separator;
+
+long lastRecordOffset = Math.max(getMaxOffsetInQueue() - 1, 0);
+
+if (lastRecordOffset == 0) {
+return;
+}
+
+SelectMappedBufferResult consumeQueueBuffer = 
getIndexBuffer(lastRecordOffset);
+
+if (consumeQueueBuffer == null) {
+String errorMsg = String.format("Can't find consume queue 
mappedBuffer by offset %s,queueDir %s", lastRecordOffset, queueDir);
+log.error(errorMsg);
+throw new IllegalStateException(errorMsg);
+}
+
+//position latest consume queue offset,
+long offsetPy = consumeQueueBuffer.getByteBuffer().getLong();
+int sizePy = consumeQueueBuffer.getByteBuffer().getInt();
+consumeQueueBuffer.getByteBuffer().getLong();
--- End diff --

fixed


> add commit log data and consume queue data logic consistent check when start 
> mq
> ---
>
> Key: ROCKETMQ-279
> URL: https://issues.apache.org/jira/browse/ROCKETMQ-279
> Project: Apache RocketMQ
>  Issue Type: Bug
>  Components: rocketmq-store
>Affects Versions: 4.0.0-incubating, 4.1.0-incubating
>Reporter: yubaofu
>Assignee: yukon
> Fix For: 4.2.0-incubating
>
>
> commit log and consume data may logic inconsistent ,
> because  [ROCKETMQ-265|https://issues.apache.org/jira/browse/ROCKETMQ-265],
> so we add  data consistent check,when mq start.
> if check failed, start failed.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (ROCKETMQ-279) add commit log data and consume queue data logic consistent check when start mq

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/ROCKETMQ-279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16149969#comment-16149969
 ] 

ASF GitHub Bot commented on ROCKETMQ-279:
-

Github user zhouxinyu commented on a diff in the pull request:

https://github.com/apache/incubator-rocketmq/pull/159#discussion_r136491053
  
--- Diff: store/src/main/java/org/apache/rocketmq/store/ConsumeQueue.java 
---
@@ -84,6 +84,58 @@ public boolean load() {
 return result;
 }
 
+public void checkCommitLogAndConsumeQueueConsistent() {
+
+String queueDir = this.storePath
++ File.separator + topic
++ File.separator + queueId + File.separator;
+
+long lastRecordOffset = Math.max(getMaxOffsetInQueue() - 1, 0);
+
+if (lastRecordOffset == 0) {
+return;
+}
+
+SelectMappedBufferResult consumeQueueBuffer = 
getIndexBuffer(lastRecordOffset);
+
+if (consumeQueueBuffer == null) {
+String errorMsg = String.format("Can't find consume queue 
mappedBuffer by offset %s,queueDir %s", lastRecordOffset, queueDir);
+log.error(errorMsg);
+throw new IllegalStateException(errorMsg);
+}
+
+//position latest consume queue offset,
+long offsetPy = consumeQueueBuffer.getByteBuffer().getLong();
+int sizePy = consumeQueueBuffer.getByteBuffer().getInt();
+consumeQueueBuffer.getByteBuffer().getLong();
--- End diff --

Do we need the side effect of `getLong`?


> add commit log data and consume queue data logic consistent check when start 
> mq
> ---
>
> Key: ROCKETMQ-279
> URL: https://issues.apache.org/jira/browse/ROCKETMQ-279
> Project: Apache RocketMQ
>  Issue Type: Bug
>  Components: rocketmq-store
>Affects Versions: 4.0.0-incubating, 4.1.0-incubating
>Reporter: yubaofu
>Assignee: yukon
> Fix For: 4.2.0-incubating
>
>
> commit log and consume data may logic inconsistent ,
> because  [ROCKETMQ-265|https://issues.apache.org/jira/browse/ROCKETMQ-265],
> so we add  data consistent check,when mq start.
> if check failed, start failed.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (ROCKETMQ-279) add commit log data and consume queue data logic consistent check when start mq

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/ROCKETMQ-279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16149966#comment-16149966
 ] 

ASF GitHub Bot commented on ROCKETMQ-279:
-

Github user zhouxinyu commented on a diff in the pull request:

https://github.com/apache/incubator-rocketmq/pull/159#discussion_r136490566
  
--- Diff: store/src/main/java/org/apache/rocketmq/store/ConsumeQueue.java 
---
@@ -84,6 +84,58 @@ public boolean load() {
 return result;
 }
 
+public void checkCommitLogAndConsumeQueueConsistent() {
+
+String queueDir = this.storePath
--- End diff --

There is a helper to get queue dir: 
`StorePathConfigHelper#getStorePathConsumeQueue`


> add commit log data and consume queue data logic consistent check when start 
> mq
> ---
>
> Key: ROCKETMQ-279
> URL: https://issues.apache.org/jira/browse/ROCKETMQ-279
> Project: Apache RocketMQ
>  Issue Type: Bug
>  Components: rocketmq-store
>Affects Versions: 4.0.0-incubating, 4.1.0-incubating
>Reporter: yubaofu
>Assignee: yukon
> Fix For: 4.2.0-incubating
>
>
> commit log and consume data may logic inconsistent ,
> because  [ROCKETMQ-265|https://issues.apache.org/jira/browse/ROCKETMQ-265],
> so we add  data consistent check,when mq start.
> if check failed, start failed.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (ROCKETMQ-279) add commit log data and consume queue data logic consistent check when start mq

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/ROCKETMQ-279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16144776#comment-16144776
 ] 

ASF GitHub Bot commented on ROCKETMQ-279:
-

Github user coveralls commented on the issue:

https://github.com/apache/incubator-rocketmq/pull/159
  

[![Coverage 
Status](https://coveralls.io/builds/13028464/badge)](https://coveralls.io/builds/13028464)

Coverage decreased (-0.4%) to 38.427% when pulling 
**1ddd284118133cbb21be3076b73153131d9c7167 on fuyou001:ROCKETMQ-2792** into 
**629c3e9ea084d0a2947b2ab6f1cc8b0c6ed98502 on apache:develop**.



> add commit log data and consume queue data logic consistent check when start 
> mq
> ---
>
> Key: ROCKETMQ-279
> URL: https://issues.apache.org/jira/browse/ROCKETMQ-279
> Project: Apache RocketMQ
>  Issue Type: Bug
>  Components: rocketmq-store
>Affects Versions: 4.0.0-incubating, 4.1.0-incubating
>Reporter: yubaofu
>Assignee: yukon
> Fix For: 4.2.0-incubating
>
>
> commit log and consume data may logic inconsistent ,
> because  [ROCKETMQ-265|https://issues.apache.org/jira/browse/ROCKETMQ-265],
> so we add  data consistent check,when mq start.
> if check failed, start failed.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (ROCKETMQ-279) add commit log data and consume queue data logic consistent check when start mq

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/ROCKETMQ-279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16144772#comment-16144772
 ] 

ASF GitHub Bot commented on ROCKETMQ-279:
-

Github user coveralls commented on the issue:

https://github.com/apache/incubator-rocketmq/pull/159
  

[![Coverage 
Status](https://coveralls.io/builds/13028470/badge)](https://coveralls.io/builds/13028470)

Coverage decreased (-0.4%) to 38.468% when pulling 
**1ddd284118133cbb21be3076b73153131d9c7167 on fuyou001:ROCKETMQ-2792** into 
**629c3e9ea084d0a2947b2ab6f1cc8b0c6ed98502 on apache:develop**.



> add commit log data and consume queue data logic consistent check when start 
> mq
> ---
>
> Key: ROCKETMQ-279
> URL: https://issues.apache.org/jira/browse/ROCKETMQ-279
> Project: Apache RocketMQ
>  Issue Type: Bug
>  Components: rocketmq-store
>Affects Versions: 4.0.0-incubating, 4.1.0-incubating
>Reporter: yubaofu
>Assignee: yukon
> Fix For: 4.2.0-incubating
>
>
> commit log and consume data may logic inconsistent ,
> because  [ROCKETMQ-265|https://issues.apache.org/jira/browse/ROCKETMQ-265],
> so we add  data consistent check,when mq start.
> if check failed, start failed.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (ROCKETMQ-279) add commit log data and consume queue data logic consistent check when start mq

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/ROCKETMQ-279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16144738#comment-16144738
 ] 

ASF GitHub Bot commented on ROCKETMQ-279:
-

Github user fuyou001 commented on a diff in the pull request:

https://github.com/apache/incubator-rocketmq/pull/159#discussion_r135692614
  
--- Diff: store/src/main/java/org/apache/rocketmq/store/ConsumeQueue.java 
---
@@ -84,6 +84,59 @@ public boolean load() {
 return result;
 }
 
+public void checkCommitLogAndConsumeQueueConsistent() {
+
+String queueDir = this.storePath
++ File.separator + topic
++ File.separator + queueId + File.separator;
+
+long maxOffsetInQueue = getMaxOffsetInQueue();
--- End diff --

refactor


> add commit log data and consume queue data logic consistent check when start 
> mq
> ---
>
> Key: ROCKETMQ-279
> URL: https://issues.apache.org/jira/browse/ROCKETMQ-279
> Project: Apache RocketMQ
>  Issue Type: Bug
>  Components: rocketmq-store
>Affects Versions: 4.0.0-incubating, 4.1.0-incubating
>Reporter: yubaofu
>Assignee: yukon
> Fix For: 4.2.0-incubating
>
>
> commit log and consume data may logic inconsistent ,
> because  [ROCKETMQ-265|https://issues.apache.org/jira/browse/ROCKETMQ-265],
> so we add  data consistent check,when mq start.
> if check failed, start failed.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (ROCKETMQ-279) add commit log data and consume queue data logic consistent check when start mq

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/ROCKETMQ-279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16144713#comment-16144713
 ] 

ASF GitHub Bot commented on ROCKETMQ-279:
-

Github user zhouxinyu commented on a diff in the pull request:

https://github.com/apache/incubator-rocketmq/pull/159#discussion_r135689182
  
--- Diff: 
store/src/test/java/org/apache/rocketmq/store/ConsumeQueueTest.java ---
@@ -207,4 +226,34 @@ public void dispatch(DispatchRequest request) {
 UtilAll.deleteFile(new File(storePath));
 }
 }
+
+@Test
+public void test_checkCommitLogAndConsumeQueueConsistent() throws 
Exception {
+DefaultMessageStore master = gen();
+
+try {
+master.getDispatcherList().addFirst(new CommitLogDispatcher() {
+
+@Override
+public void dispatch(DispatchRequest request) {
+runCount++;
+}
+
+private int runCount = 0;
+});
+
+putMsg(master);
+// wait build consume queue
+Thread.sleep(2000);
+
+ConsumeQueue cq = 
master.getConsumeQueueTable().get(topic).get(queueId);
+cq.checkCommitLogAndConsumeQueueConsistent();
--- End diff --

It seems that this test only covers the consistent condition? 


> add commit log data and consume queue data logic consistent check when start 
> mq
> ---
>
> Key: ROCKETMQ-279
> URL: https://issues.apache.org/jira/browse/ROCKETMQ-279
> Project: Apache RocketMQ
>  Issue Type: Bug
>  Components: rocketmq-store
>Affects Versions: 4.0.0-incubating, 4.1.0-incubating
>Reporter: yubaofu
>Assignee: yukon
> Fix For: 4.2.0-incubating
>
>
> commit log and consume data may logic inconsistent ,
> because  [ROCKETMQ-265|https://issues.apache.org/jira/browse/ROCKETMQ-265],
> so we add  data consistent check,when mq start.
> if check failed, start failed.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (ROCKETMQ-279) add commit log data and consume queue data logic consistent check when start mq

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/ROCKETMQ-279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16143374#comment-16143374
 ] 

ASF GitHub Bot commented on ROCKETMQ-279:
-

Github user coveralls commented on the issue:

https://github.com/apache/incubator-rocketmq/pull/159
  

[![Coverage 
Status](https://coveralls.io/builds/13011345/badge)](https://coveralls.io/builds/13011345)

Coverage decreased (-0.09%) to 38.731% when pulling 
**bff62f4fd84b0084c655b05698c484381ee4e77f on fuyou001:ROCKETMQ-2792** into 
**629c3e9ea084d0a2947b2ab6f1cc8b0c6ed98502 on apache:develop**.



> add commit log data and consume queue data logic consistent check when start 
> mq
> ---
>
> Key: ROCKETMQ-279
> URL: https://issues.apache.org/jira/browse/ROCKETMQ-279
> Project: Apache RocketMQ
>  Issue Type: Bug
>  Components: rocketmq-store
>Affects Versions: 4.0.0-incubating, 4.1.0-incubating
>Reporter: yubaofu
>Assignee: yukon
> Fix For: 4.2.0-incubating
>
>
> commit log and consume data may logic inconsistent ,
> because  [ROCKETMQ-265|https://issues.apache.org/jira/browse/ROCKETMQ-265],
> so we add  data consistent check,when mq start.
> if check failed, start failed.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (ROCKETMQ-279) add commit log data and consume queue data logic consistent check when start mq

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/ROCKETMQ-279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16143324#comment-16143324
 ] 

ASF GitHub Bot commented on ROCKETMQ-279:
-

GitHub user fuyou001 opened a pull request:

https://github.com/apache/incubator-rocketmq/pull/159

[ROCKETMQ-279]  commit log data and consume queue data logic consistent

To avoid commit log  and consume queue logic inconsistent .
Motivation:
 add a check when mq start,if commit log  and consume queue logic 
inconsistent ,start failed.

fast failed if check failed.

(cherry picked from commit 6e77be0)

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/fuyou001/incubator-rocketmq ROCKETMQ-2792

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-rocketmq/pull/159.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #159


commit 0f0dfd35367402aea5ca3f40199387dd48cb1e7e
Author: 傅冲 
Date:   2017-08-28T02:21:49Z

[ROCKETMQ-279]  commit log data and consume queue data logic consistent

(cherry picked from commit 6e77be0)




> add commit log data and consume queue data logic consistent check when start 
> mq
> ---
>
> Key: ROCKETMQ-279
> URL: https://issues.apache.org/jira/browse/ROCKETMQ-279
> Project: Apache RocketMQ
>  Issue Type: Bug
>  Components: rocketmq-store
>Affects Versions: 4.0.0-incubating, 4.1.0-incubating
>Reporter: yubaofu
>Assignee: yukon
> Fix For: 4.2.0-incubating
>
>
> commit log and consume data may logic inconsistent ,
> because  [ROCKETMQ-265|https://issues.apache.org/jira/browse/ROCKETMQ-265],
> so we add  data consistent check,when mq start.
> if check failed, start failed.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)