from:"ASF GitHub Bot \(JIRA\)"

[jira] [Commented] (ROCKETMQ-294) Do flow control on the number and size dimensions when pull message

2017-09-25 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/ROCKETMQ-294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16178955#comment-16178955
 ] 

ASF GitHub Bot commented on ROCKETMQ-294:
-

Github user coveralls commented on the issue:

https://github.com/apache/incubator-rocketmq/pull/171
  

[![Coverage 
Status](https://coveralls.io/builds/13416107/badge)](https://coveralls.io/builds/13416107)

Coverage increased (+0.2%) to 38.988% when pulling 
**8878a21e2d0a87ac0d1ae59483eabcc34fdafb07 on zhouxinyu:ROCKETMQ-294** into 
**5ac45fc56da44e1244d2c0a15522debe9ed820a1 on apache:develop**.



> Do flow control on the number and size dimensions when pull message
> ---
>
> Key: ROCKETMQ-294
> URL: https://issues.apache.org/jira/browse/ROCKETMQ-294
> Project: Apache RocketMQ
>  Issue Type: Improvement
>  Components: rocketmq-client
>Reporter: yukon
>Assignee: yukon
> Fix For: 4.2.0-incubating
>
>
> *Motivations*
> Current flow control strategy only support on Queue level, each message queue 
> can cache 1000 message by default. 
> When lots of message queue are assigned to one consumer instance, the 
> consumer will cache too many messages, may cause OOM exception.
> On the other hand, only control the message amount is not enough, should 
> support do flow control on the number and size dimensions, on Topic and Queue 
> level.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (ROCKETMQ-294) Do flow control on the number and size dimensions when pull message

2017-09-25 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/ROCKETMQ-294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16178940#comment-16178940
 ] 

ASF GitHub Bot commented on ROCKETMQ-294:
-

GitHub user zhouxinyu opened a pull request:

https://github.com/apache/incubator-rocketmq/pull/171

[ROCKETMQ-294] Do flow control on the number and size dimensions when pull 
message

## What is the purpose of the change
Current flow control strategy only support on Queue level, each message 
queue can cache 1000 message by default.

When lots of message queue are assigned to one consumer instance, the 
consumer will cache too many messages, may cause OOM exception.

On the other hand, only control the message amount is not enough, should 
support do flow control on the number and size dimensions, on Topic and Queue 
level.

## Brief changelog

Do flow control on the number and size dimensions when pulling message

## Verifying this change

This PR added three parameters, plus the old one, there are 4 parameters 
for flow control:

* pullThresholdForQueue, flow control threshold on queue level, each 
message queue will cache at most 1000 messages by default.
* pullThresholdForTopic, flow control threshold on topic level, default 
value is -1(Unlimited)
* pullThresholdSizeForQueue, limit the cached message size on queue level, 
each message queue will cache at most 100 MiB messages by default.
* pullThresholdSizeForTopic, limit the cached message size on topic level, 
the default value is -1 MiB(Unlimited).

See the javadoc for details, you can change the value, and observe the log 
to ensure it works.

Follow this checklist to help us incorporate your contribution quickly and 
easily:

- [x] Make sure there is a [JIRA 
issue](https://issues.apache.org/jira/browse/ROCKETMQ-294) filed for the change 
(usually before you start working on it). Trivial changes like typos do not 
require a JIRA issue. Your pull request should address just this issue, without 
pulling in other changes - one PR resolves one issue. 
- [x] Format the pull request title like `[ROCKETMQ-XXX] Fix 
UnknownException when host config not exist`. Each commit in the pull request 
should have a meaningful subject line and body.
- [x] Write a pull request description that is detailed enough to 
understand what the pull request does, how, and why.
- [x] Write necessary unit-test to verify your logic correction, more mock 
a little better when cross module dependency exist. If the new feature or 
significant change is committed, please remember to add integration-test in 
[test module](https://github.com/apache/incubator-rocketmq/tree/master/test).
- [x] Run `mvn -B clean apache-rat:check findbugs:findbugs 
checkstyle:checkstyle` to make sure basic checks pass. Run `mvn clean install 
-DskipITs` to make sure unit-test pass. Run `mvn clean test-compile 
failsafe:integration-test`  to make sure integration-test pass.
- [x] If this contribution is large, please file an [Apache Individual 
Contributor License Agreement](http://www.apache.org/licenses/#clas).


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/zhouxinyu/incubator-rocketmq ROCKETMQ-294

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-rocketmq/pull/171.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #171


commit 8878a21e2d0a87ac0d1ae59483eabcc34fdafb07
Author: yukon 
Date:   2017-09-25T07:55:55Z

[ROCKETMQ-294] Do flow control on the number and size dimensions when pull 
message




> Do flow control on the number and size dimensions when pull message
> ---
>
> Key: ROCKETMQ-294
> URL: https://issues.apache.org/jira/browse/ROCKETMQ-294
> Project: Apache RocketMQ
>  Issue Type: Improvement
>  Components: rocketmq-client
>Reporter: yukon
>Assignee: yukon
> Fix For: 4.2.0-incubating
>
>
> *Motivations*
> Current flow control strategy only support on Queue level, each message queue 
> can cache 1000 message by default. 
> When lots of message queue are assigned to one consumer instance, the 
> consumer will cache too many messages, may cause OOM exception.
> On the other hand, only control the message amount is not enough, should 
> support do flow control on the number and size dimensions, on Topic and Queue 
> level.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (ROCKETMQ-239) Query message by time and get queue offset by time return is always less than want.

2017-09-24 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/ROCKETMQ-239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16178501#comment-16178501
 ] 

ASF GitHub Bot commented on ROCKETMQ-239:
-

Github user dongeforever commented on a diff in the pull request:

https://github.com/apache/incubator-rocketmq/pull/129#discussion_r140684767
  
--- Diff: store/src/main/java/org/apache/rocketmq/store/ConsumeQueue.java 
---
@@ -151,7 +151,7 @@ public void recover() {
 }
 }
 
-public long getOffsetInQueueByTime(final long timestamp) {
+public long getOffsetInQueueByTime(final long timestamp, boolean 
isGetTimeLast) {
 MappedFile mappedFile = 
this.mappedFileQueue.getMappedFileByTime(timestamp);
--- End diff --

It is not reliable to get mapped file by modification time, as if the 
dispatch service fall behind too much, it may get the wrong mapped file.
How about checking each file's first message and last message it pointed to?
This may cause performance loss, but it is acceptable, for such api is used 
only for some specific diagnoses. It is rarely used, but when using it, it'd 
better to return the exactly correct result. 


> Query message by time and get queue offset by time return is always less than 
> want. 
> 
>
> Key: ROCKETMQ-239
> URL: https://issues.apache.org/jira/browse/ROCKETMQ-239
> Project: Apache RocketMQ
>  Issue Type: Bug
>  Components: rocketmq-store
>Affects Versions: 4.1.0-incubating
>Reporter: lindzh
>Assignee: yukon
> Fix For: 4.2.0-incubating
>
>
> When using queryMessageByTime or searchOffsetByTime or resetOffsetByTime , 
> DefaultMessageStore return offset is not so accurate as we want.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (ROCKETMQ-23) MappedFileQueue#flush should return true when flushing is successful

2017-09-22 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/ROCKETMQ-23?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16176648#comment-16176648
 ] 

ASF GitHub Bot commented on ROCKETMQ-23:


Github user dongeforever commented on the issue:

https://github.com/apache/incubator-rocketmq/pull/20
  
@shroman It seems that last cleaning, for too long-time no-resolved prs,  
included this one.
If you feel it is necessary to reopen this PR, please feel free to do it.
you may also could create a new PR or launch new discussion in the mailing 
list.


> MappedFileQueue#flush should return true when flushing is successful
> 
>
> Key: ROCKETMQ-23
> URL: https://issues.apache.org/jira/browse/ROCKETMQ-23
> Project: Apache RocketMQ
>  Issue Type: Improvement
>  Components: rocketmq-store
>Affects Versions: 4.0.0-incubating
>Reporter: Roman Shtykh
>Assignee: Roman Shtykh
> Fix For: 4.2.0-incubating
>
>
> In the current implementation, MappedFileQueue#flush returns {{false}} when 
> flushing is successful.
> This is not intuitive and error prone.
> For instance, in {{CommitLog#run line:915-918}}
> {code}
> for (int i = 0; i < RETRY_TIMES_OVER && !result; i++) {
>result = CommitLog.this.mappedFileQueue.flush(0);
>// ...
> }
> {code}
> I believe retries has to be done when flushing is not successful. But with 
> the code above, it can try to flush only once on CommitLog termination and 
> not continue retrying.
> The same is for {{DefaultMessageStore#doFlush line:1551}}
> Or is this not retry on failure, but the number of times flushing has to be 
> done? Then, {{RETRY_TIMES_OVER}} should be renamed to something like 
> {{FLUSH_NUM}}.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (ROCKETMQ-249) Attempt to clear disk even if disk store path is found

2017-09-22 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/ROCKETMQ-249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16176057#comment-16176057
 ] 

ASF GitHub Bot commented on ROCKETMQ-249:
-

Github user coveralls commented on the issue:

https://github.com/apache/incubator-rocketmq/pull/133
  

[![Coverage 
Status](https://coveralls.io/builds/13384852/badge)](https://coveralls.io/builds/13384852)

Coverage increased (+0.3%) to 38.964% when pulling 
**ba0348b972cbab3f7c4f296039c85c27c5c1c1e2 on shroman:ROCKETMQ-249** into 
**845830865fc37d0364a19cbd89ceaf8a30b37e1c on apache:develop**.



> Attempt to clear disk even if disk store path is found
> --
>
> Key: ROCKETMQ-249
> URL: https://issues.apache.org/jira/browse/ROCKETMQ-249
> Project: Apache RocketMQ
>  Issue Type: Bug
>Reporter: Roman Shtykh
>Assignee: Roman Shtykh
>
> If disk usage estimates fail because the disk store path is not found 
> (doesn't exist), message store falsely decides the disk is full and tries to 
> delete.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (ROCKETMQ-249) Attempt to clear disk even if disk store path is found

2017-09-22 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/ROCKETMQ-249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16176040#comment-16176040
 ] 

ASF GitHub Bot commented on ROCKETMQ-249:
-

Github user shroman commented on a diff in the pull request:

https://github.com/apache/incubator-rocketmq/pull/133#discussion_r140425644
  
--- Diff: 
store/src/test/java/org/apache/rocketmq/store/CleanCommitLogServiceTest.java ---
@@ -0,0 +1,115 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.rocketmq.store;
+
+import java.lang.reflect.Field;
+import java.lang.reflect.Method;
+import org.apache.rocketmq.common.BrokerConfig;
+import org.apache.rocketmq.store.config.MessageStoreConfig;
+import org.junit.After;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+
+import static org.mockito.Mockito.spy;
+import static org.mockito.Mockito.when;
+
+/**
+ * Tests for CleanCommitLogService.
+ */
+public class CleanCommitLogServiceTest {
+DefaultMessageStore messageStore;
+DefaultMessageStore.CleanCommitLogService cleanCommitLogService;
+
+// CleanCommitLogService class.
+Class clazz;
+
+@Before
+public void setUp() throws Exception {
+clazz = 
Class.forName("org.apache.rocketmq.store.DefaultMessageStore$CleanCommitLogService");
+
+MessageStoreConfig messageStoreConfig = new MessageStoreConfig();
+messageStore = new DefaultMessageStore(
+messageStoreConfig, null, null, new BrokerConfig());
+
+Field cleanCommitLogServiceField = 
messageStore.getClass().getDeclaredField("cleanCommitLogService");
+cleanCommitLogServiceField.setAccessible(true);
+
+cleanCommitLogService = 
spy((DefaultMessageStore.CleanCommitLogService) 
(cleanCommitLogServiceField.get(messageStore)));
+}
+
+@After
+public void tearDown() throws Exception {
+}
+
+@Test
+public void isSpaceToDeleteNoCommitLog() throws Exception {
+Method m = clazz.getDeclaredMethod("isSpaceToDelete");
+m.setAccessible(true);
+
+Assert.assertFalse((boolean) m.invoke(cleanCommitLogService));
+}
+
+@Test
+public void isSpaceToDeleteWithCommitLogNoQueue() throws Exception {
+try {
+messageStore.start();
+messageStore.load();
+
+Method m = clazz.getDeclaredMethod("isSpaceToDelete");
+m.setAccessible(true);
+
+Assert.assertFalse((boolean) m.invoke(cleanCommitLogService));
+} finally {
+messageStore.shutdown();
--- End diff --

Good point. thanks.


> Attempt to clear disk even if disk store path is found
> --
>
> Key: ROCKETMQ-249
> URL: https://issues.apache.org/jira/browse/ROCKETMQ-249
> Project: Apache RocketMQ
>  Issue Type: Bug
>Reporter: Roman Shtykh
>Assignee: Roman Shtykh
>
> If disk usage estimates fail because the disk store path is not found 
> (doesn't exist), message store falsely decides the disk is full and tries to 
> delete.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (ROCKETMQ-249) Attempt to clear disk even if disk store path is found

2017-09-22 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/ROCKETMQ-249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16176039#comment-16176039
 ] 

ASF GitHub Bot commented on ROCKETMQ-249:
-

Github user shroman commented on a diff in the pull request:

https://github.com/apache/incubator-rocketmq/pull/133#discussion_r140425575
  
--- Diff: 
store/src/main/java/org/apache/rocketmq/store/DefaultMessageStore.java ---
@@ -1509,65 +1514,62 @@ private boolean isTimeToDelete() {
 return false;
 }
 
-private boolean isSpaceToDelete() {
-double ratio = 
DefaultMessageStore.this.getMessageStoreConfig().getDiskMaxUsedSpaceRatio() / 
100.0;
+double getDiskUsageRatio() {
+return UtilAll.getDiskPartitionSpaceUsedPercent(
+
DefaultMessageStore.this.getMessageStoreConfig().getStorePathCommitLog());
+}
 
-cleanImmediately = false;
+double getQueueSpace() {
+return 
UtilAll.getDiskPartitionSpaceUsedPercent(StorePathConfigHelper
+
.getStorePathConsumeQueue(DefaultMessageStore.this.getMessageStoreConfig().getStorePathRootDir()));
+}
 
-{
-String storePathPhysic = 
DefaultMessageStore.this.getMessageStoreConfig().getStorePathCommitLog();
-double physicRatio = 
UtilAll.getDiskPartitionSpaceUsedPercent(storePathPhysic);
-if (physicRatio > diskSpaceWarningLevelRatio) {
-boolean diskok = 
DefaultMessageStore.this.runningFlags.getAndMakeDiskFull();
-if (diskok) {
-DefaultMessageStore.log.error("physic disk maybe 
full soon " + physicRatio + ", so mark disk full");
-}
+/**
+ * Checks if cleaning on the disk is needed.
+ *
+ * @param usageRatio Usage ratio.
+ * @param allowedRatio Allowed ratio.
+ * @return True if cleaning is needed, otherwise 
false.
+ */
+private boolean needCleaning(double usageRatio, double 
allowedRatio) {
+if (usageRatio == -1)
--- End diff --

Thanks, fixed.


> Attempt to clear disk even if disk store path is found
> --
>
> Key: ROCKETMQ-249
> URL: https://issues.apache.org/jira/browse/ROCKETMQ-249
> Project: Apache RocketMQ
>  Issue Type: Bug
>Reporter: Roman Shtykh
>Assignee: Roman Shtykh
>
> If disk usage estimates fail because the disk store path is not found 
> (doesn't exist), message store falsely decides the disk is full and tries to 
> delete.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (ROCKETMQ-249) Attempt to clear disk even if disk store path is found

2017-09-22 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/ROCKETMQ-249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16176037#comment-16176037
 ] 

ASF GitHub Bot commented on ROCKETMQ-249:
-

Github user shroman commented on the issue:

https://github.com/apache/incubator-rocketmq/pull/133
  
@dongeforever Modified as you suggested, thanks!


> Attempt to clear disk even if disk store path is found
> --
>
> Key: ROCKETMQ-249
> URL: https://issues.apache.org/jira/browse/ROCKETMQ-249
> Project: Apache RocketMQ
>  Issue Type: Bug
>Reporter: Roman Shtykh
>Assignee: Roman Shtykh
>
> If disk usage estimates fail because the disk store path is not found 
> (doesn't exist), message store falsely decides the disk is full and tries to 
> delete.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (ROCKETMQ-249) Attempt to clear disk even if disk store path is found

2017-09-21 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/ROCKETMQ-249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16175815#comment-16175815
 ] 

ASF GitHub Bot commented on ROCKETMQ-249:
-

Github user dongeforever commented on a diff in the pull request:

https://github.com/apache/incubator-rocketmq/pull/133#discussion_r140402297
  
--- Diff: 
store/src/test/java/org/apache/rocketmq/store/CleanCommitLogServiceTest.java ---
@@ -0,0 +1,115 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.rocketmq.store;
+
+import java.lang.reflect.Field;
+import java.lang.reflect.Method;
+import org.apache.rocketmq.common.BrokerConfig;
+import org.apache.rocketmq.store.config.MessageStoreConfig;
+import org.junit.After;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+
+import static org.mockito.Mockito.spy;
+import static org.mockito.Mockito.when;
+
+/**
+ * Tests for CleanCommitLogService.
+ */
+public class CleanCommitLogServiceTest {
+DefaultMessageStore messageStore;
+DefaultMessageStore.CleanCommitLogService cleanCommitLogService;
+
+// CleanCommitLogService class.
+Class clazz;
+
+@Before
+public void setUp() throws Exception {
+clazz = 
Class.forName("org.apache.rocketmq.store.DefaultMessageStore$CleanCommitLogService");
+
+MessageStoreConfig messageStoreConfig = new MessageStoreConfig();
+messageStore = new DefaultMessageStore(
+messageStoreConfig, null, null, new BrokerConfig());
+
+Field cleanCommitLogServiceField = 
messageStore.getClass().getDeclaredField("cleanCommitLogService");
+cleanCommitLogServiceField.setAccessible(true);
+
+cleanCommitLogService = 
spy((DefaultMessageStore.CleanCommitLogService) 
(cleanCommitLogServiceField.get(messageStore)));
+}
+
+@After
+public void tearDown() throws Exception {
+}
+
+@Test
+public void isSpaceToDeleteNoCommitLog() throws Exception {
+Method m = clazz.getDeclaredMethod("isSpaceToDelete");
+m.setAccessible(true);
+
+Assert.assertFalse((boolean) m.invoke(cleanCommitLogService));
+}
+
+@Test
+public void isSpaceToDeleteWithCommitLogNoQueue() throws Exception {
+try {
+messageStore.start();
+messageStore.load();
+
+Method m = clazz.getDeclaredMethod("isSpaceToDelete");
+m.setAccessible(true);
+
+Assert.assertFalse((boolean) m.invoke(cleanCommitLogService));
+} finally {
+messageStore.shutdown();
--- End diff --

How about putting the finally block into teardown() ?
The test framework will handle it.


> Attempt to clear disk even if disk store path is found
> --
>
> Key: ROCKETMQ-249
> URL: https://issues.apache.org/jira/browse/ROCKETMQ-249
> Project: Apache RocketMQ
>  Issue Type: Bug
>Reporter: Roman Shtykh
>Assignee: Roman Shtykh
>
> If disk usage estimates fail because the disk store path is not found 
> (doesn't exist), message store falsely decides the disk is full and tries to 
> delete.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (ROCKETMQ-249) Attempt to clear disk even if disk store path is found

2017-09-21 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/ROCKETMQ-249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16175814#comment-16175814
 ] 

ASF GitHub Bot commented on ROCKETMQ-249:
-

Github user dongeforever commented on a diff in the pull request:

https://github.com/apache/incubator-rocketmq/pull/133#discussion_r140402195
  
--- Diff: 
store/src/main/java/org/apache/rocketmq/store/DefaultMessageStore.java ---
@@ -1509,65 +1514,62 @@ private boolean isTimeToDelete() {
 return false;
 }
 
-private boolean isSpaceToDelete() {
-double ratio = 
DefaultMessageStore.this.getMessageStoreConfig().getDiskMaxUsedSpaceRatio() / 
100.0;
+double getDiskUsageRatio() {
+return UtilAll.getDiskPartitionSpaceUsedPercent(
+
DefaultMessageStore.this.getMessageStoreConfig().getStorePathCommitLog());
+}
 
-cleanImmediately = false;
+double getQueueSpace() {
+return 
UtilAll.getDiskPartitionSpaceUsedPercent(StorePathConfigHelper
+
.getStorePathConsumeQueue(DefaultMessageStore.this.getMessageStoreConfig().getStorePathRootDir()));
+}
 
-{
-String storePathPhysic = 
DefaultMessageStore.this.getMessageStoreConfig().getStorePathCommitLog();
-double physicRatio = 
UtilAll.getDiskPartitionSpaceUsedPercent(storePathPhysic);
-if (physicRatio > diskSpaceWarningLevelRatio) {
-boolean diskok = 
DefaultMessageStore.this.runningFlags.getAndMakeDiskFull();
-if (diskok) {
-DefaultMessageStore.log.error("physic disk maybe 
full soon " + physicRatio + ", so mark disk full");
-}
+/**
+ * Checks if cleaning on the disk is needed.
+ *
+ * @param usageRatio Usage ratio.
+ * @param allowedRatio Allowed ratio.
+ * @return True if cleaning is needed, otherwise 
false.
+ */
+private boolean needCleaning(double usageRatio, double 
allowedRatio) {
+if (usageRatio == -1)
--- End diff --

It is not safe to use == for float value. In this case, usageRatio < 0 is OK


> Attempt to clear disk even if disk store path is found
> --
>
> Key: ROCKETMQ-249
> URL: https://issues.apache.org/jira/browse/ROCKETMQ-249
> Project: Apache RocketMQ
>  Issue Type: Bug
>Reporter: Roman Shtykh
>Assignee: Roman Shtykh
>
> If disk usage estimates fail because the disk store path is not found 
> (doesn't exist), message store falsely decides the disk is full and tries to 
> delete.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (ROCKETMQ-138) Add AuthenticationException class to remove hard coded Aliyun authentication class

2017-09-21 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/ROCKETMQ-138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16175810#comment-16175810
 ] 

ASF GitHub Bot commented on ROCKETMQ-138:
-

Github user dongeforever commented on the issue:

https://github.com/apache/incubator-rocketmq/pull/138
  
@lyy4j It is important to put the code of same logic in one place. For easy 
maintenance, it is suggested to reuse ConsistentHashRouter. If you think that 
ConsistentHashRouter is not clear and extensible enough, please feel free to 
polish it.


> Add AuthenticationException class to remove hard coded Aliyun authentication 
> class
> --
>
> Key: ROCKETMQ-138
> URL: https://issues.apache.org/jira/browse/ROCKETMQ-138
> Project: Apache RocketMQ
>  Issue Type: Improvement
>  Components: rocketmq-remoting
>Reporter: Jaskey Lam
>Assignee: Jaskey Lam
>Priority: Minor
> Fix For: 4.1.0-incubating
>
>
> in NettyRemotingAbstract.java
> a hard coded aliyun class is used 
> {code}
> catch (Throwable e) {
> if 
> (!"com.aliyun.openservices.ons.api.impl.authority.exception.AuthenticationException"
> .equals(e.getClass().getCanonicalName())) {
> PLOG.error("process request exception", e);
> PLOG.error(cmd.toString());
> }
> {code}
> A common AuthenticationException should be added to identify Authentication 
> failure.  Developers can throw this exception so that remoting component can 
> ignore it



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (ROCKETMQ-138) Add AuthenticationException class to remove hard coded Aliyun authentication class

2017-09-21 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/ROCKETMQ-138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16175808#comment-16175808
 ] 

ASF GitHub Bot commented on ROCKETMQ-138:
-

Github user dongeforever commented on a diff in the pull request:

https://github.com/apache/incubator-rocketmq/pull/138#discussion_r140401292
  
--- Diff: 
client/src/main/java/org/apache/rocketmq/client/producer/selector/SelectMessageQueueByConsistentHash.java
 ---
@@ -0,0 +1,132 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.client.producer.selector;
+
+import org.apache.rocketmq.client.producer.MessageQueueSelector;
+import org.apache.rocketmq.common.message.Message;
+import org.apache.rocketmq.common.message.MessageQueue;
+
+import java.util.HashMap;
+import java.util.List;
+import java.util.SortedMap;
+import java.util.TreeMap;
+
+/**
+* wanting to send a message with the same key combine with orderly 
consumer, user can 
+* use the implements of the MessageQueueSelector, which has one 
optimization compared with SelectMessageQueueByHash
+* when a broker crash , it can reduce the message Arriving out of the order
+*/
+public class SelectMessageQueueByConsistentHash implements 
MessageQueueSelector {
+
+private volatile SortedMap virtualNodes =
+new TreeMap();
+
+private static final int DEFAULT_VIRTUAL_NODES = 100;
+
+private final int virtualNodeNum;
+
+private volatile HashMap idToQueueMap = new 
HashMap();
+
+public SelectMessageQueueByConsistentHash() {
+this.virtualNodeNum = DEFAULT_VIRTUAL_NODES;
+}
+
+public SelectMessageQueueByConsistentHash(int virtualNodeNum) {
+this.virtualNodeNum = virtualNodeNum;
+}
+
+@Override
+public MessageQueue select(List mqs, Message msg, Object 
arg) {
+synchronized (this) {
+if (queueChange(mqs)) {
+reloadConsistentHash(mqs);
+}
+
+String uniqueQueueId = getMsgQueueIdBy(arg.toString());
+MessageQueue messageQueue = idToQueueMap.get(uniqueQueueId);
+return messageQueue;
+}
+}
+
+private boolean queueChange(List mqs) {
+if (mqs.size() != this.idToQueueMap.size()) {
+return true;
+}
+
+for (MessageQueue queue : mqs) {
+String id = queue.getTopic() + "_" + queue.getBrokerName() + 
"_" + queue.getQueueId();
+if (!this.idToQueueMap.containsKey(id)) {
+return true;
+}
+}
+
+return false;
+}
+
+private String getMsgQueueIdBy(String arg) {
+int hash = getHash(arg);
+SortedMap subMap = virtualNodes.tailMap(hash);
+
+Integer i;
+String virtualNode;
+
+if (subMap.size() == 0) {
+i = virtualNodes.firstKey();
+virtualNode = virtualNodes.get(i);
+} else {
+i = subMap.firstKey();
+virtualNode = subMap.get(i);
+}
+
+String result = virtualNode.substring(0, 
virtualNode.indexOf("&&"));
+return result;
+}
+
+private int getHash(String str) {
--- End diff --

Why not just use str.hashCode?


> Add AuthenticationException class to remove hard coded Aliyun authentication 
> class
> --
>
> Key: ROCKETMQ-138
> URL: https://issues.apache.org/jira/browse/ROCKETMQ-138
> Project: Apache RocketMQ
>  Issue Type: Improvement
>  Components: rocketmq-remoting
>Reporter: Jaskey Lam
>Assignee: Jaskey

[jira] [Commented] (ROCKETMQ-138) Add AuthenticationException class to remove hard coded Aliyun authentication class

2017-09-21 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/ROCKETMQ-138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16175807#comment-16175807
 ] 

ASF GitHub Bot commented on ROCKETMQ-138:
-

Github user dongeforever commented on a diff in the pull request:

https://github.com/apache/incubator-rocketmq/pull/138#discussion_r140401564
  
--- Diff: 
client/src/main/java/org/apache/rocketmq/client/producer/selector/SelectMessageQueueByConsistentHash.java
 ---
@@ -0,0 +1,132 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.client.producer.selector;
+
+import org.apache.rocketmq.client.producer.MessageQueueSelector;
+import org.apache.rocketmq.common.message.Message;
+import org.apache.rocketmq.common.message.MessageQueue;
+
+import java.util.HashMap;
+import java.util.List;
+import java.util.SortedMap;
+import java.util.TreeMap;
+
+/**
+* wanting to send a message with the same key combine with orderly 
consumer, user can 
+* use the implements of the MessageQueueSelector, which has one 
optimization compared with SelectMessageQueueByHash
+* when a broker crash , it can reduce the message Arriving out of the order
+*/
+public class SelectMessageQueueByConsistentHash implements 
MessageQueueSelector {
+
+private volatile SortedMap virtualNodes =
--- End diff --

how about  just using ?



> Add AuthenticationException class to remove hard coded Aliyun authentication 
> class
> --
>
> Key: ROCKETMQ-138
> URL: https://issues.apache.org/jira/browse/ROCKETMQ-138
> Project: Apache RocketMQ
>  Issue Type: Improvement
>  Components: rocketmq-remoting
>Reporter: Jaskey Lam
>Assignee: Jaskey Lam
>Priority: Minor
> Fix For: 4.1.0-incubating
>
>
> in NettyRemotingAbstract.java
> a hard coded aliyun class is used 
> {code}
> catch (Throwable e) {
> if 
> (!"com.aliyun.openservices.ons.api.impl.authority.exception.AuthenticationException"
> .equals(e.getClass().getCanonicalName())) {
> PLOG.error("process request exception", e);
> PLOG.error(cmd.toString());
> }
> {code}
> A common AuthenticationException should be added to identify Authentication 
> failure.  Developers can throw this exception so that remoting component can 
> ignore it



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (ROCKETMQ-293) Add high-availability support for rocketmq-mysql-replicator

2017-09-21 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/ROCKETMQ-293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16174448#comment-16174448
 ] 

ASF GitHub Bot commented on ROCKETMQ-293:
-

GitHub user zhaoqun911 opened a pull request:

https://github.com/apache/incubator-rocketmq-externals/pull/30

[ROCKETMQ-293] Add high-availability support for rocketmq-mysql-replicator

### What is the purpose of the change
Add high-availability support for rocketmq-mysql-replicator in order to 
ensure availability, if a replicator instance crashes, another instance will be 
started.
JIRA:https://issues.apache.org/jira/browse/ROCKETMQ-293

### Brief change log
* Add HA-mode for rocketmq-mysql-replicator.


### Verifying this change
* Start a zookeeper instance.
* Add the “zkAddr” property in the “rocketmq-mysql.conf” file.
* Start the replicator to verify.


Follow this checklist to help us incorporate your contribution quickly and 
easily:
- [x] Make sure there is a [JIRA issue](
https://issues.apache.org/jira/projects/ROCKETMQ/issues/) filed for the
change (usually before you start working on it). Trivial changes like typos
do not require a JIRA issue. Your pull request should address just this
issue, without pulling in other changes - one PR resolves one issue.
- [x] Format the pull request title like `[ROCKETMQ-XXX] Fix
UnknownException when host config not exist`. Each commit in the pull
request should have a meaningful subject line and body.
- [x] Write a pull request description that is detailed enough to
understand what the pull request does, how, and why.
- [x] Write necessary unit-test to verify your logic correction, more mock
a little better when cross module dependency exists. If the new feature or
significant change is committed, please remember to add integration-test in
[test module](https://github.com/apache/incubator-rocketmq/tree/master/test
).
- [x] Run `mvn -B clean apache-rat:check findbugs:findbugs
checkstyle:checkstyle` to make sure basic checks pass. Run `mvn clean
install -DskipITs` to make sure unit-test pass. Run `mvn clean test-compile
failsafe:integration-test`  to make sure integration-test pass.
- [x] If this contribution is large, please file an [Apache Individual
Contributor License Agreement](http://www.apache.org/licenses/#clas).

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/zhaoqun911/incubator-rocketmq-externals 
mysql-develop

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-rocketmq-externals/pull/30.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #30


commit 3906df66e1e7e6ce9b26a476a205c32a34182441
Author: zhaoqun007 <9...@zhaoqun911.cn>
Date:   2017-09-20T11:10:44Z

Add high-availability support for rocketmq-mysql-replicator using zookeeper.




> Add high-availability support for rocketmq-mysql-replicator
> ---
>
> Key: ROCKETMQ-293
> URL: https://issues.apache.org/jira/browse/ROCKETMQ-293
> Project: Apache RocketMQ
>  Issue Type: New Feature
>  Components: rocketmq-externals
>Reporter: Qun Zhao
>Assignee: dongeforever
>Priority: Minor
>
> Add high-availability support for rocketmq-mysql-replicator using zookeeper.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (ROCKETMQ-292) Delete system.exit in MQAdminStartup

2017-09-21 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/ROCKETMQ-292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16174421#comment-16174421
 ] 

ASF GitHub Bot commented on ROCKETMQ-292:
-

Github user coveralls commented on the issue:

https://github.com/apache/incubator-rocketmq/pull/169
  

[![Coverage 
Status](https://coveralls.io/builds/13367262/badge)](https://coveralls.io/builds/13367262)

Coverage increased (+0.02%) to 38.826% when pulling 
**a905b140d09d085d3c20aef4f08a6696e9b218bf on lindzh:fix_main_system.exit** 
into **27a678de3c592f4b87c4284bba64b2d50d966785 on apache:develop**.



> Delete system.exit in MQAdminStartup
> 
>
> Key: ROCKETMQ-292
> URL: https://issues.apache.org/jira/browse/ROCKETMQ-292
> Project: Apache RocketMQ
>  Issue Type: Improvement
>  Components: rocketmq-tools
>Affects Versions: 4.1.0-incubating
>Reporter: lindzh
>Assignee: vongosling
> Fix For: 4.2.0-incubating
>
>
> When using MQAdminStartup in a java process,some args problem may cause 
> MQAdminStartup to call system.ext and this lead to main java process 
> shutdown. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (ROCKETMQ-291) System.out.printf throws UnknownFormatConversionException

2017-09-21 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/ROCKETMQ-291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16174417#comment-16174417
 ] 

ASF GitHub Bot commented on ROCKETMQ-291:
-

Github user coveralls commented on the issue:

https://github.com/apache/incubator-rocketmq/pull/168
  

[![Coverage 
Status](https://coveralls.io/builds/13367180/badge)](https://coveralls.io/builds/13367180)

Coverage decreased (-0.2%) to 38.646% when pulling 
**144c270a84da144647b80c05df6cc287370e2963 on lindzh:fix_print** into 
**27a678de3c592f4b87c4284bba64b2d50d966785 on apache:develop**.



> System.out.printf throws UnknownFormatConversionException
> -
>
> Key: ROCKETMQ-291
> URL: https://issues.apache.org/jira/browse/ROCKETMQ-291
> Project: Apache RocketMQ
>  Issue Type: Bug
>  Components: rocketmq-tools
>Affects Versions: 4.1.0-incubating
>Reporter: lindzh
>Assignee: vongosling
> Fix For: 4.2.0-incubating
>
>
> Use mqadmin printmsg -t %RETRY%consumerid ，it throws an exception like this:
> Exception in thread "main" java.util.UnknownFormatConversionException: 
> Conversion = 'R'
>   at java.util.Formatter$FormatSpecifier.conversion(Formatter.java:2691)
>   at java.util.Formatter$FormatSpecifier.(Formatter.java:2720)
>   at java.util.Formatter.parse(Formatter.java:2560)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (ROCKETMQ-292) Delete system.exit in MQAdminStartup

2017-09-21 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/ROCKETMQ-292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16174408#comment-16174408
 ] 

ASF GitHub Bot commented on ROCKETMQ-292:
-

GitHub user lindzh opened a pull request:

https://github.com/apache/incubator-rocketmq/pull/169

[ROCKETMQ-292]Delete system.exit in MQAdminStartup

###  What is the purpose of the change 
When using MQAdminStartup in a java process,some args problem may cause 
MQAdminStartup to call system.ext and this lead to main java process shutdown. 
JIRA: https://issues.apache.org/jira/browse/ROCKETMQ-278.
### Brief change log
* delete system.exit in mqadmin.

### Verifying this change
* This change is a trivial rework / code cleanup without unit test coverage.
* execute `mqadmin testa` to verify.

Follow this checklist to help us incorporate your contribution quickly and 
easily:
- [x] Make sure there is a [JIRA 
issue](https://issues.apache.org/jira/projects/ROCKETMQ/issues/)  filed for the 
change (usually before you start working on it). Trivial changes like typos do 
not require a JIRA issue. Your pull request should address just this issue, 
without pulling in other changes - one PR resolves one issue. 
- [x]  Format the pull request title like [ROCKETMQ-XXX] Fix 
UnknownException when host config not exist. Each commit in the pull request 
should have a meaningful subject line and body.
- [x]  Write a pull request description that is detailed enough to 
understand the pull request :
  - What is the purpose of the change
  - Brief change log
  - Verifying this change
- [x] Write necessary unit-test to verify your logic correction, more mock 
a little better when cross module dependency exist. If the new feature or 
significant change is committed, please remember to add integration-test in 
[test module](https://github.com/apache/incubator-rocketmq/tree/master/test).
- [x]  Run mvn -B clean apache-rat:check findbugs:findbugs 
checkstyle:checkstyle to make sure basic checks pass. Run mvn clean install 
-DskipITs to make sure unit-test pass. Run mvn clean test-compile 
failsafe:integration-test to make sure integration-test pass. 
- [x]  If this contribution is large, please file an [Apache Individual 
Contributor License Agreement](http://www.apache.org/licenses/#clas).

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/lindzh/incubator-rocketmq fix_main_system.exit

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-rocketmq/pull/169.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #169


commit 1ae0693693da4a147f1bb3c122833f548c376b09
Author: lindzh 
Date:   2017-08-11T07:55:24Z

Merge branch 'develop' of github.com:apache/incubator-rocketmq into develop

commit f0e243c5a8899fedc157accef964a1d617d77d1b
Author: lindzh 
Date:   2017-08-11T10:02:36Z

Merge branch 'develop' of github.com:apache/incubator-rocketmq into develop

commit 1810be4c560767b83ff173e6fb6a96d4ac7c1428
Author: lindzh 
Date:   2017-08-14T02:11:27Z

Merge branch 'develop' of github.com:apache/incubator-rocketmq into develop

commit 53dcd8d0837eee419810f055a4035d3fdd72513f
Author: lindzh 
Date:   2017-08-22T12:34:49Z

Merge branch 'develop' of github.com:apache/incubator-rocketmq into develop

commit 4abfa4f326d0f5ac7a01f43b3cb882a7916fac59
Author: lindzh 
Date:   2017-08-28T10:41:53Z

Merge branch 'develop' of github.com:apache/incubator-rocketmq into develop

commit d576e38db79e4437dc2b304a6d6ef3a0e9337f74
Author: lindzh 
Date:   2017-08-30T06:38:39Z

Merge branch 'develop' of github.com:apache/incubator-rocketmq into develop

commit bb446c48fa54cb542708f1ee171d49b1f6d1cd64
Author: lindzh 
Date:   2017-09-21T07:15:43Z

Merge branch 'develop' of github.com:apache/incubator-rocketmq into develop

commit a905b140d09d085d3c20aef4f08a6696e9b218bf
Author: lindzh 
Date:   2017-09-21T07:18:43Z

fix main system.exit




> Delete system.exit in MQAdminStartup
> 
>
> Key: ROCKETMQ-292
> URL: https://issues.apache.org/jira/browse/ROCKETMQ-292
> Project: Apache RocketMQ
>  Issue Type: Improvement
>  Components: rocketmq-tools
>Affects Versions: 4.1.0-incubating
>Reporter: lindzh
>Assignee: vongosling
> Fix For: 4.2.0-incubating
>
>
> When using MQAdminStartup in a java process,some args problem may cause 
> MQAdminStartup to call system.ext and this lead to main java process 
> shutdown. 



--
This

[jira] [Commented] (ROCKETMQ-278) Add specified cluster for cluster info cmd

2017-09-21 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/ROCKETMQ-278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16174402#comment-16174402
 ] 

ASF GitHub Bot commented on ROCKETMQ-278:
-

GitHub user lindzh opened a pull request:

https://github.com/apache/incubator-rocketmq/pull/168

Fix System.out.printf throws UnknownFormatConversionException

###  What is the purpose of the change 
When using mqadmin command printmsg to print retry topic msgs,it throw an 
exception like this:
com.alibaba.rocketmq.client.exception.MQClientException: Can not find 
Message Queue for this 
```
topic, %%RETRY%%CID_MERSS_CONSUOUP_UAT
See http://rocketmq.apache.org/docs/faq/ for further details.
  at 
com.alibaba.rocketmq.client.impl.MQAdminImpl.fetchSubscribeMessageQueues(MQAdminImpl.java:161)
  at 
com.alibaba.rocketmq.client.impl.consumer.DefaultMQPullConsumerImpl.fetchSubscribeMessageQueues(DefaultMQPullConsumerImpl.java:136)
  at 
com.alibaba.rocketmq.client.consumer.DefaultMQPullConsumer.fetchSubscribeMessageQueues(DefaultMQPullConsumer.java:225)
  at 
com.alibaba.rocketmq.tools.command.message.PrintMessageSubCommand.execute(PrintMessageSubCommand.java:120)
  at 
com.alibaba.rocketmq.tools.command.MQAdminStartup.main0(MQAdminStartup.java:128)
,. JIRA: https://issues.apache.org/jira/browse/ROCKETMQ-278.
```
### Brief change log
* fix all system.out.prinf using in rocketmq.

### Verifying this change
* This change is a trivial rework / code cleanup with unitout test coverage.
* execute mqadmin printmsg -t %RETRY%CID_MERSS_CONSUOUP_UAT.

Follow this checklist to help us incorporate your contribution quickly and 
easily:
- [x] Make sure there is a [JIRA 
issue](https://issues.apache.org/jira/projects/ROCKETMQ/issues/)  filed for the 
change (usually before you start working on it). Trivial changes like typos do 
not require a JIRA issue. Your pull request should address just this issue, 
without pulling in other changes - one PR resolves one issue. 
- [x]  Format the pull request title like [ROCKETMQ-XXX] Fix 
UnknownException when host config not exist. Each commit in the pull request 
should have a meaningful subject line and body.
- [x]  Write a pull request description that is detailed enough to 
understand the pull request :
  - What is the purpose of the change
  - Brief change log
  - Verifying this change
- [x] Write necessary unit-test to verify your logic correction, more mock 
a little better when cross module dependency exist. If the new feature or 
significant change is committed, please remember to add integration-test in 
[test module](https://github.com/apache/incubator-rocketmq/tree/master/test).
- [x]  Run mvn -B clean apache-rat:check findbugs:findbugs 
checkstyle:checkstyle to make sure basic checks pass. Run mvn clean install 
-DskipITs to make sure unit-test pass. Run mvn clean test-compile 
failsafe:integration-test to make sure integration-test pass. 
- [x]  If this contribution is large, please file an [Apache Individual 
Contributor License Agreement](http://www.apache.org/licenses/#clas).

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/lindzh/incubator-rocketmq fix_print

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-rocketmq/pull/168.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #168


commit 1ae0693693da4a147f1bb3c122833f548c376b09
Author: lindzh 
Date:   2017-08-11T07:55:24Z

Merge branch 'develop' of github.com:apache/incubator-rocketmq into develop

commit f0e243c5a8899fedc157accef964a1d617d77d1b
Author: lindzh 
Date:   2017-08-11T10:02:36Z

Merge branch 'develop' of github.com:apache/incubator-rocketmq into develop

commit 1810be4c560767b83ff173e6fb6a96d4ac7c1428
Author: lindzh 
Date:   2017-08-14T02:11:27Z

Merge branch 'develop' of github.com:apache/incubator-rocketmq into develop

commit 53dcd8d0837eee419810f055a4035d3fdd72513f
Author: lindzh 
Date:   2017-08-22T12:34:49Z

Merge branch 'develop' of github.com:apache/incubator-rocketmq into develop

commit 4abfa4f326d0f5ac7a01f43b3cb882a7916fac59
Author: lindzh 
Date:   2017-08-28T10:41:53Z

Merge branch 'develop' of github.com:apache/incubator-rocketmq into develop

commit 845830865fc37d0364a19cbd89ceaf8a30b37e1c
Author: yukon 
Date:   2017-08-29T16:06:13Z

Polish merge script, use old title if input is null

commit 254d432496c424717d45b8dc9e44ae3bd78ab466
Author: vsair 
Date:   2017-08-29T16:14:27Z

[ROCKETMQ-284] ExpressionMessageFilter will pass some message.

Author:

[jira] [Commented] (ROCKETMQ-255) Offset store is null after consumer clients start()

2017-09-20 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/ROCKETMQ-255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16174204#comment-16174204
 ] 

ASF GitHub Bot commented on ROCKETMQ-255:
-

Github user dongeforever commented on the issue:

https://github.com/apache/incubator-rocketmq/pull/142
  
LGTM @zhouxinyu @vsair @Jaskey please help do a review?


> Offset store is null after consumer clients start()
> ---
>
> Key: ROCKETMQ-255
> URL: https://issues.apache.org/jira/browse/ROCKETMQ-255
> Project: Apache RocketMQ
>  Issue Type: Bug
>  Components: rocketmq-client
>Affects Versions: 4.1.0-incubating
>Reporter: Zhanhui Li
>Assignee: Zhanhui Li
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (ROCKETMQ-256) Manually clean disk to specified watermark with message consuming progress considered

2017-09-20 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/ROCKETMQ-256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16174185#comment-16174185
 ] 

ASF GitHub Bot commented on ROCKETMQ-256:
-

Github user dongeforever commented on the issue:

https://github.com/apache/incubator-rocketmq/pull/143
  
@lizhanhui This purge is not be safe for scheduled message.


> Manually clean disk to specified watermark with message consuming progress 
> considered
> -
>
> Key: ROCKETMQ-256
> URL: https://issues.apache.org/jira/browse/ROCKETMQ-256
> Project: Apache RocketMQ
>  Issue Type: Improvement
>  Components: rocketmq-broker, rocketmq-store, rocketmq-tools
>Affects Versions: 4.1.0-incubating
>Reporter: Zhanhui Li
>Assignee: Zhanhui Li
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (ROCKETMQ-256) Manually clean disk to specified watermark with message consuming progress considered

2017-09-20 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/ROCKETMQ-256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16174180#comment-16174180
 ] 

ASF GitHub Bot commented on ROCKETMQ-256:
-

Github user dongeforever commented on a diff in the pull request:

https://github.com/apache/incubator-rocketmq/pull/143#discussion_r140139906
  
--- Diff: 
store/src/main/java/org/apache/rocketmq/store/DefaultMessageStore.java ---
@@ -1392,6 +1397,14 @@ public void run() {
 }, 6, TimeUnit.SECONDS);
 }
 
+@Override
+public void purge(int watermark, long consumedPhysicalOffset, boolean 
force) {
+cleanCommitLogService.setPurgeForcefullyWhenManual(force);
+cleanCommitLogService.setDiskSpaceManuallyCleanRatio(watermark / 
100.0);
+
cleanCommitLogService.setConsumedPhysicalOffset(consumedPhysicalOffset);
+cleanCommitLogService.executeDeleteFilesManually();
--- End diff --

This is not thread safe.
How about just delivering the arguments into executeDeleteFilesManually() 
directly?


> Manually clean disk to specified watermark with message consuming progress 
> considered
> -
>
> Key: ROCKETMQ-256
> URL: https://issues.apache.org/jira/browse/ROCKETMQ-256
> Project: Apache RocketMQ
>  Issue Type: Improvement
>  Components: rocketmq-broker, rocketmq-store, rocketmq-tools
>Affects Versions: 4.1.0-incubating
>Reporter: Zhanhui Li
>Assignee: Zhanhui Li
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (ROCKETMQ-256) Manually clean disk to specified watermark with message consuming progress considered

2017-09-20 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/ROCKETMQ-256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16174181#comment-16174181
 ] 

ASF GitHub Bot commented on ROCKETMQ-256:
-

Github user dongeforever commented on a diff in the pull request:

https://github.com/apache/incubator-rocketmq/pull/143#discussion_r140139302
  
--- Diff: common/src/test/java/org/apache/rocketmq/common/UtilAllTest.java 
---
@@ -89,6 +89,14 @@ public void testGetDiskPartitionSpaceUsedPercent() {
 
assertThat(UtilAll.getDiskPartitionSpaceUsedPercent(tmpDir)).isNotCloseTo(-1, 
within(0.01));
 }
 
+
+@Test
+public void testGetDiskPartitionSpaceUsedPercent4Home() {
+String tmpDir = System.getProperty("user.home");
+double percent = UtilAll.getDiskPartitionSpaceUsedPercent(tmpDir);
+System.out.println(percent);
--- End diff --

What did this unit test mean for?


> Manually clean disk to specified watermark with message consuming progress 
> considered
> -
>
> Key: ROCKETMQ-256
> URL: https://issues.apache.org/jira/browse/ROCKETMQ-256
> Project: Apache RocketMQ
>  Issue Type: Improvement
>  Components: rocketmq-broker, rocketmq-store, rocketmq-tools
>Affects Versions: 4.1.0-incubating
>Reporter: Zhanhui Li
>Assignee: Zhanhui Li
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (ROCKETMQ-256) Manually clean disk to specified watermark with message consuming progress considered

2017-09-20 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/ROCKETMQ-256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16174179#comment-16174179
 ] 

ASF GitHub Bot commented on ROCKETMQ-256:
-

Github user dongeforever commented on a diff in the pull request:

https://github.com/apache/incubator-rocketmq/pull/143#discussion_r140139119
  
--- Diff: 
broker/src/main/java/org/apache/rocketmq/broker/plugin/AbstractPluginMessageStore.java
 ---
@@ -246,4 +247,14 @@ public void setConfirmOffset(long phyOffset) {
 public ConsumeQueue getConsumeQueue(String topic, int queueId) {
 return next.getConsumeQueue(topic, queueId);
 }
+
+@Override
+public PutMessageResult putMessages(MessageExtBatch messageExtBatch) {
+return null;
+}
--- End diff --

why return null  but not next.putMessages(messageExtBatch)


> Manually clean disk to specified watermark with message consuming progress 
> considered
> -
>
> Key: ROCKETMQ-256
> URL: https://issues.apache.org/jira/browse/ROCKETMQ-256
> Project: Apache RocketMQ
>  Issue Type: Improvement
>  Components: rocketmq-broker, rocketmq-store, rocketmq-tools
>Affects Versions: 4.1.0-incubating
>Reporter: Zhanhui Li
>Assignee: Zhanhui Li
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (ROCKETMQ-265) when os crash for some reasons, the broker consume queue’s data maybe repeat, consumer can’t pull the latest message, cause message lag

2017-09-20 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/ROCKETMQ-265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16174161#comment-16174161
 ] 

ASF GitHub Bot commented on ROCKETMQ-265:
-

Github user dongeforever commented on the issue:

https://github.com/apache/incubator-rocketmq/pull/146
  
@fuyou001 I want to merge this PR now. Could you please polish the unit 
tests?


> when os crash for some reasons, the broker consume queue’s data maybe repeat, 
> consumer can’t pull the latest message, cause message lag
> ---
>
> Key: ROCKETMQ-265
> URL: https://issues.apache.org/jira/browse/ROCKETMQ-265
> Project: Apache RocketMQ
>  Issue Type: Bug
>  Components: rocketmq-store
>Affects Versions: 4.0.0-incubating, 4.1.0-incubating
>Reporter: yubaofu
>Assignee: yukon
>Priority: Critical
>  Labels: bug
> Fix For: 4.2.0-incubating
>
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
> when os crash for some reasons, the broker consume queue’s data maybe repeat, 
> consumer can’t pull the latest message, cause message lag



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (ROCKETMQ-265) when os crash for some reasons, the broker consume queue’s data maybe repeat, consumer can’t pull the latest message, cause message lag

2017-09-20 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/ROCKETMQ-265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16174157#comment-16174157
 ] 

ASF GitHub Bot commented on ROCKETMQ-265:
-

Github user dongeforever commented on a diff in the pull request:

https://github.com/apache/incubator-rocketmq/pull/146#discussion_r140138721
  
--- Diff: 
store/src/test/java/org/apache/rocketmq/store/ConsumeQueueTest.java ---
@@ -131,6 +130,65 @@ protected void putMsg(DefaultMessageStore master) 
throws Exception {
 }
 }
 
+protected void deleteDirectory(String rootPath) {
+File file = new File(rootPath);
+deleteFile(file);
+}
+
+protected void deleteFile(File file) {
+File[] subFiles = file.listFiles();
+if (subFiles != null) {
+for (File sub : subFiles) {
+deleteFile(sub);
+}
+}
+
+file.delete();
+}
+
+@Test
+public void testPutMessagePositionInfo_buildCQRepeatedly() throws 
Exception {
+DefaultMessageStore messageStore = null;
+try {
+
+messageStore = gen();
+
+int totalMessages = 10;
+
+for (int i = 0; i < totalMessages; i++) {
+putMsg(messageStore);
+}
+Thread.sleep(5);
+
+ConsumeQueue cq = 
messageStore.getConsumeQueueTable().get(topic).get(queueId);
+Method method = 
cq.getClass().getDeclaredMethod("putMessagePositionInfo", long.class, 
int.class, long.class, long.class);
+
+assertThat(method).isNotNull();
+
+method.setAccessible(true);
+
+SelectMappedBufferResult result = 
messageStore.getCommitLog().getData(0);
+assertThat(result != null).isTrue();
+
+DispatchRequest dispatchRequest = 
messageStore.getCommitLog().checkMessageAndReturnSize(result.getByteBuffer(), 
false, false);
+
+assertThat(cq).isNotNull();
+
+Object dispatchResult = method.invoke(cq, 
dispatchRequest.getCommitLogOffset(),
+dispatchRequest.getMsgSize(), 
dispatchRequest.getTagsCode(), dispatchRequest.getConsumeQueueOffset());
+
+
assertThat(Boolean.parseBoolean(dispatchResult.toString())).isTrue();
+
+} finally {
+if (messageStore != null) {
--- End diff --

@fuyou001 It would look clear to remove try finally block from unit test. 
Better to do it in "@after".


> when os crash for some reasons, the broker consume queue’s data maybe repeat, 
> consumer can’t pull the latest message, cause message lag
> ---
>
> Key: ROCKETMQ-265
> URL: https://issues.apache.org/jira/browse/ROCKETMQ-265
> Project: Apache RocketMQ
>  Issue Type: Bug
>  Components: rocketmq-store
>Affects Versions: 4.0.0-incubating, 4.1.0-incubating
>Reporter: yubaofu
>Assignee: yukon
>Priority: Critical
>  Labels: bug
> Fix For: 4.2.0-incubating
>
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
> when os crash for some reasons, the broker consume queue’s data maybe repeat, 
> consumer can’t pull the latest message, cause message lag



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (ROCKETMQ-266) Can’t start consumer with a small “consumerThreadMax” number

2017-09-20 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/ROCKETMQ-266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16174144#comment-16174144
 ] 

ASF GitHub Bot commented on ROCKETMQ-266:
-

Github user dongeforever commented on the issue:

https://github.com/apache/incubator-rocketmq/pull/147
  
LGTM @zhouxinyu @vongosling  any view?


> Can’t start consumer with a small “consumerThreadMax” number
> 
>
> Key: ROCKETMQ-266
> URL: https://issues.apache.org/jira/browse/ROCKETMQ-266
> Project: Apache RocketMQ
>  Issue Type: Bug
>  Components: rocketmq-client
>Affects Versions: 4.1.0-incubating
>Reporter: Mark Yang
>Assignee: Xiaorui Wang
> Fix For: 4.2.0-incubating
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (ROCKETMQ-269) don’t need cal.setTimeInMillis(System.currentTimeMillis()) after Calendar.getInstance()

2017-09-20 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/ROCKETMQ-269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16174140#comment-16174140
 ] 

ASF GitHub Bot commented on ROCKETMQ-269:
-

Github user dongeforever commented on the issue:

https://github.com/apache/incubator-rocketmq/pull/148
  
@mark800 IMO, it is easy to add unit test for such change.
just test if the return value of computNextHourTimeMillis(and other 
methods) is expected.
I will merge this PR, if the unit test is added.



> don’t need cal.setTimeInMillis(System.currentTimeMillis()) after 
> Calendar.getInstance()
> ---
>
> Key: ROCKETMQ-269
> URL: https://issues.apache.org/jira/browse/ROCKETMQ-269
> Project: Apache RocketMQ
>  Issue Type: Improvement
>  Components: rocketmq-commons
>Reporter: Mark Yang
>Assignee: Jixiang Jin
>
> don’t need cal.setTimeInMillis(System.currentTimeMillis()) after 
> Calendar.getInstance()
> In JDK’s source code, innermost constructor for Calendar.getInstance() is:
> public GregorianCalendar(TimeZone zone, Locale aLocale) {
>  super(zone, aLocale); gdate = (BaseCalendar.Date) 
> gcal.newCalendarDate(zone); 
>  setTimeInMillis(System.currentTimeMillis()); 
> }



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (ROCKETMQ-136) Provide a handy message queue producer for order message sharding

2017-09-20 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/ROCKETMQ-136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16174137#comment-16174137
 ] 

ASF GitHub Bot commented on ROCKETMQ-136:
-

Github user dongeforever commented on the issue:

https://github.com/apache/incubator-rocketmq/pull/149
  
@zhouxinyu @vongosling any idea about this?


> Provide a handy message queue producer for order message sharding
> -
>
> Key: ROCKETMQ-136
> URL: https://issues.apache.org/jira/browse/ROCKETMQ-136
> Project: Apache RocketMQ
>  Issue Type: Improvement
>  Components: rocketmq-client
>Affects Versions: 4.1.0-incubating
>Reporter: Jaskey Lam
>Assignee: Jaskey Lam
>Priority: Minor
> Fix For: 4.2.0-incubating
>
>
> When order message is needed, users need to provide a message queue selector 
> to make sure that the messages which has the same shading key should be sent 
> to the same message queue.
> Actually this is a very common scenario with a common solutions, say 
> consistent hashing.
> We should provide a handy selector for them to easily do that, what they only 
> need to provide is a sharding key.
> A consistent hash selector will meet most of the user's need.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (ROCKETMQ-272) The config `syncFlushTimeout` doesn't work for SYNC_MASTER

2017-09-19 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/ROCKETMQ-272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16172755#comment-16172755
 ] 

ASF GitHub Bot commented on ROCKETMQ-272:
-

Github user evthoriz commented on the issue:

https://github.com/apache/incubator-rocketmq/pull/153
  
@dongeforever thanks for your reply. I tried, but it seems that there's no 
easy way to mock a test for this, since the ha mechanism involves lots of salve 
replicating behavior. Do you have any good ideas? 


> The config `syncFlushTimeout` doesn't work for SYNC_MASTER
> --
>
> Key: ROCKETMQ-272
> URL: https://issues.apache.org/jira/browse/ROCKETMQ-272
> Project: Apache RocketMQ
>  Issue Type: Bug
>  Components: rocketmq-broker
>Affects Versions: 4.1.0-incubating
>Reporter: Yu Kaiyuan
>Assignee: yukon
>
> It's quite frequent to get result as `sendStatus=FLUSH_SLAVE_TIMEOUT` when 
> sending big messages(>500k) in SYNC_MASTER/SLAVE scenario.
> The timeout value used by the sync process currently as I found, is the 
> config `syncFlushTimeout`. And its default value is 5000 milliseconds.
> But it shows that producer get the result as `FLUSH_SLAVE_TIMEOUT` less than 
> 1 second. 
> So why does the config not work as expected?
> Relevant code:
> {code:java}
> // CommitLog.java
> public void handleHA(AppendMessageResult result, PutMessageResult 
> putMessageResult, MessageExt messageExt) {
> if (BrokerRole.SYNC_MASTER == 
> this.defaultMessageStore.getMessageStoreConfig().getBrokerRole()) {
> HAService service = this.defaultMessageStore.getHaService();
> if (messageExt.isWaitStoreMsgOK()) {
> // Determine whether to wait
> if (service.isSlaveOK(result.getWroteOffset() + 
> result.getWroteBytes())) {
> GroupCommitRequest  request = new 
> GroupCommitRequest(result.getWroteOffset() + result.getWroteBytes());
> service.putRequest(request);
> service.getWaitNotifyObject().wakeupAll();
> boolean flushOK =
> 
> request.waitForFlush(this.defaultMessageStore.getMessageStoreConfig().getSyncFlushTimeout());
> if (!flushOK) {
> log.error("do sync transfer other node, wait return, but 
> failed, topic: " + messageExt.getTopic() + " tags: "
> + messageExt.getTags() + " client address: " + 
> messageExt.getBornHostNameString());
> 
> putMessageResult.setPutMessageStatus(PutMessageStatus.FLUSH_SLAVE_TIMEOUT);
> }
> }
> // Slave problem
> else {
> // Tell the producer, slave not available
> 
> putMessageResult.setPutMessageStatus(PutMessageStatus.SLAVE_NOT_AVAILABLE);
> }
> }
> }
> }
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (ROCKETMQ-273) return an expression when a function has no write operations

2017-09-19 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/ROCKETMQ-273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16172739#comment-16172739
 ] 

ASF GitHub Bot commented on ROCKETMQ-273:
-

Github user kevin-better commented on a diff in the pull request:

https://github.com/apache/incubator-rocketmq/pull/150#discussion_r139874392
  
--- Diff: 
store/src/main/java/org/apache/rocketmq/store/DefaultMessageStore.java ---
@@ -1094,34 +1090,15 @@ private boolean checkInDiskByCommitOffset(long 
offsetPy, long maxOffsetPy) {
 }
 
 private boolean isTheBatchFull(int sizePy, int maxMsgNums, int 
bufferTotal, int messageTotal, boolean isInDisk) {
-
-if (0 == bufferTotal || 0 == messageTotal) {
-return false;
-}
-
-if (maxMsgNums <= messageTotal) {
-return true;
-}
-
-if (isInDisk) {
-if ((bufferTotal + sizePy) > 
this.messageStoreConfig.getMaxTransferBytesOnMessageInDisk()) {
-return true;
-}
-
-if (messageTotal > 
this.messageStoreConfig.getMaxTransferCountOnMessageInDisk() - 1) {
-return true;
-}
-} else {
-if ((bufferTotal + sizePy) > 
this.messageStoreConfig.getMaxTransferBytesOnMessageInMemory()) {
-return true;
-}
-
-if (messageTotal > 
this.messageStoreConfig.getMaxTransferCountOnMessageInMemory() - 1) {
-return true;
-}
-}
-
-return false;
+return   (0 != bufferTotal && 0 != messageTotal)
+&&(
--- End diff --

It has been restored


> return an expression when a function has no write operations
> 
>
> Key: ROCKETMQ-273
> URL: https://issues.apache.org/jira/browse/ROCKETMQ-273
> Project: Apache RocketMQ
>  Issue Type: Improvement
>  Components: rocketmq-store
>Affects Versions: 4.2.0-incubating
>Reporter: wangkai
>Assignee: yukon
>Priority: Minor
>  Labels: Improvement
> Fix For: 4.2.0-incubating
>
>   Original Estimate: 2h
>  Remaining Estimate: 2h
>
> return an expression when a function has no write operations.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (ROCKETMQ-215) Use java 7 syntax to refator code base

2017-09-19 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/ROCKETMQ-215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16172734#comment-16172734
 ] 

ASF GitHub Bot commented on ROCKETMQ-215:
-

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-rocketmq/pull/117


> Use java 7 syntax to refator code base
> --
>
> Key: ROCKETMQ-215
> URL: https://issues.apache.org/jira/browse/ROCKETMQ-215
> Project: Apache RocketMQ
>  Issue Type: Improvement
>  Components: rocketmq-broker, rocketmq-namesrv, rocketmq-store
>Reporter: Jaskey Lam
>Assignee: yukon
>Priority: Minor
>
> code base may have some code like 
> {code}
> private ConcurrentMap Long>> offsetTable =
> new ConcurrentHashMap>(512);
> {code}
> Atucally, with java7, it can be simplied to 
> {code}
> private ConcurrentMap Long>> offsetTable =
> new ConcurrentHashMap<>(512);
> {code}
> Which is cleaner. 
> Since rocketmq broker require jdk which is greater or higher than java7, this 
> can be improved.
> But rocketmq client should still remains compatible with jdk 1.6



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (ROCKETMQ-23) MappedFileQueue#flush should return true when flushing is successful

2017-09-19 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/ROCKETMQ-23?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16172736#comment-16172736
 ] 

ASF GitHub Bot commented on ROCKETMQ-23:


Github user asfgit closed the pull request at:

https://github.com/apache/incubator-rocketmq/pull/20


> MappedFileQueue#flush should return true when flushing is successful
> 
>
> Key: ROCKETMQ-23
> URL: https://issues.apache.org/jira/browse/ROCKETMQ-23
> Project: Apache RocketMQ
>  Issue Type: Improvement
>  Components: rocketmq-store
>Affects Versions: 4.0.0-incubating
>Reporter: Roman Shtykh
>Assignee: Roman Shtykh
> Fix For: 4.2.0-incubating
>
>
> In the current implementation, MappedFileQueue#flush returns {{false}} when 
> flushing is successful.
> This is not intuitive and error prone.
> For instance, in {{CommitLog#run line:915-918}}
> {code}
> for (int i = 0; i < RETRY_TIMES_OVER && !result; i++) {
>result = CommitLog.this.mappedFileQueue.flush(0);
>// ...
> }
> {code}
> I believe retries has to be done when flushing is not successful. But with 
> the code above, it can try to flush only once on CommitLog termination and 
> not continue retrying.
> The same is for {{DefaultMessageStore#doFlush line:1551}}
> Or is this not retry on failure, but the number of times flushing has to be 
> done? Then, {{RETRY_TIMES_OVER}} should be renamed to something like 
> {{FLUSH_NUM}}.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (ROCKETMQ-44) Duplicated codes in DefaultMessageStore.getEarliestMessageTime and DefaultMessageStore.getMessageStoreTimeStamp

2017-09-19 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/ROCKETMQ-44?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16172732#comment-16172732
 ] 

ASF GitHub Bot commented on ROCKETMQ-44:


Github user asfgit closed the pull request at:

https://github.com/apache/incubator-rocketmq/pull/38


> Duplicated codes in DefaultMessageStore.getEarliestMessageTime and 
> DefaultMessageStore.getMessageStoreTimeStamp
> ---
>
> Key: ROCKETMQ-44
> URL: https://issues.apache.org/jira/browse/ROCKETMQ-44
> Project: Apache RocketMQ
>  Issue Type: Improvement
>Reporter: Wu Sheng
>Assignee: vongosling
>Priority: Minor
> Fix For: 4.3.0-incubating
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (ROCKETMQ-40) AUTO_CLIENT_ACKNOWNLEDGE & CLIENT_ACKNOWLEDGE support

2017-09-19 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/ROCKETMQ-40?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16172735#comment-16172735
 ] 

ASF GitHub Bot commented on ROCKETMQ-40:


Github user asfgit closed the pull request at:

https://github.com/apache/incubator-rocketmq/pull/40


> AUTO_CLIENT_ACKNOWNLEDGE & CLIENT_ACKNOWLEDGE support 
> --
>
> Key: ROCKETMQ-40
> URL: https://issues.apache.org/jira/browse/ROCKETMQ-40
> Project: Apache RocketMQ
>  Issue Type: New Feature
>  Components: rocketmq-client
>Reporter: Jaskey Lam
>Assignee: Xiaorui Wang
> Fix For: 4.3.0-incubating
>
>
> For now, push consumer only supports DUPS_OK_ACKNOWLEDGE .
> If user wants to AUTO_CLIENT_ACKNOWNLEDGE or CLIENT_ACKNOWLEDGE , he or she 
> must use pull consumer which is not easy enough to use.
> push consumer should support AUTO_CLIENT_ACKNOWNLEDGE  and CLIENT_ACKNOWLEDGE 
> .
> AUTO_CLIENT_ACKNOWNLEDGE  : auto ack after consume and  persist to broker
> CLIENT_ACKNOWLEDGE  : user should ack by itself, consumer will not ack even 
> after handle.
> ref: https://docs.oracle.com/cd/E19587-01/821-0029/aeqbk/index.html



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (ROCKETMQ-273) return an expression when a function has no write operations

2017-09-19 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/ROCKETMQ-273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16172720#comment-16172720
 ] 

ASF GitHub Bot commented on ROCKETMQ-273:
-

Github user kevin-better closed the pull request at:

https://github.com/apache/incubator-rocketmq/pull/150


> return an expression when a function has no write operations
> 
>
> Key: ROCKETMQ-273
> URL: https://issues.apache.org/jira/browse/ROCKETMQ-273
> Project: Apache RocketMQ
>  Issue Type: Improvement
>  Components: rocketmq-store
>Affects Versions: 4.2.0-incubating
>Reporter: wangkai
>Assignee: yukon
>Priority: Minor
>  Labels: Improvement
> Fix For: 4.2.0-incubating
>
>   Original Estimate: 2h
>  Remaining Estimate: 2h
>
> return an expression when a function has no write operations.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (ROCKETMQ-273) return an expression when a function has no write operations

2017-09-19 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/ROCKETMQ-273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16172716#comment-16172716
 ] 

ASF GitHub Bot commented on ROCKETMQ-273:
-

Github user dongeforever commented on the issue:

https://github.com/apache/incubator-rocketmq/pull/150
  
@kevin-better This PR has been merged. You may close it now.


> return an expression when a function has no write operations
> 
>
> Key: ROCKETMQ-273
> URL: https://issues.apache.org/jira/browse/ROCKETMQ-273
> Project: Apache RocketMQ
>  Issue Type: Improvement
>  Components: rocketmq-store
>Affects Versions: 4.2.0-incubating
>Reporter: wangkai
>Assignee: yukon
>Priority: Minor
>  Labels: Improvement
> Fix For: 4.2.0-incubating
>
>   Original Estimate: 2h
>  Remaining Estimate: 2h
>
> return an expression when a function has no write operations.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (ROCKETMQ-278) Add specified cluster for cluster info cmd

2017-09-19 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/ROCKETMQ-278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16172708#comment-16172708
 ] 

ASF GitHub Bot commented on ROCKETMQ-278:
-

Github user vongosling commented on a diff in the pull request:

https://github.com/apache/incubator-rocketmq/pull/152#discussion_r139871793
  
--- Diff: 
client/src/main/java/org/apache/rocketmq/client/impl/MQClientAPIImpl.java ---
@@ -1167,10 +1168,13 @@ public Properties getBrokerConfig(final String 
addr, final long timeoutMillis)
 throw new MQBrokerException(response.getCode(), 
response.getRemark());
 }
 
-public ClusterInfo getBrokerClusterInfo(
+public ClusterInfo getBrokerClusterInfo(String cluster,
--- End diff --

how to insure the api compatibility


> Add specified cluster for cluster info cmd
> --
>
> Key: ROCKETMQ-278
> URL: https://issues.apache.org/jira/browse/ROCKETMQ-278
> Project: Apache RocketMQ
>  Issue Type: Improvement
>  Components: rocketmq-client, rocketmq-namesrv, rocketmq-tools
>Affects Versions: 4.1.0-incubating
>Reporter: lindzh
>Assignee: Xiaorui Wang
> Fix For: 4.2.0-incubating
>
>
> When using mqadmin command clusterlist,it always display all cluster,i want 
> to display specified cluster.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (ROCKETMQ-278) Add specified cluster for cluster info cmd

2017-09-19 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/ROCKETMQ-278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16172707#comment-16172707
 ] 

ASF GitHub Bot commented on ROCKETMQ-278:
-

Github user vongosling commented on a diff in the pull request:

https://github.com/apache/incubator-rocketmq/pull/152#discussion_r139871898
  
--- Diff: 
tools/src/main/java/org/apache/rocketmq/tools/command/topic/TopicListSubCommand.java
 ---
@@ -65,7 +65,7 @@ public void execute(final CommandLine commandLine, final 
Options options,
 try {
 defaultMQAdminExt.start();
 if (commandLine.hasOption('c')) {
-ClusterInfo clusterInfo = 
defaultMQAdminExt.examineBrokerClusterInfo();
+ClusterInfo clusterInfo = 
defaultMQAdminExt.examineBrokerClusterInfo(null);
--- End diff --

null as parameter is not a good practice


> Add specified cluster for cluster info cmd
> --
>
> Key: ROCKETMQ-278
> URL: https://issues.apache.org/jira/browse/ROCKETMQ-278
> Project: Apache RocketMQ
>  Issue Type: Improvement
>  Components: rocketmq-client, rocketmq-namesrv, rocketmq-tools
>Affects Versions: 4.1.0-incubating
>Reporter: lindzh
>Assignee: Xiaorui Wang
> Fix For: 4.2.0-incubating
>
>
> When using mqadmin command clusterlist,it always display all cluster,i want 
> to display specified cluster.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (ROCKETMQ-278) Add specified cluster for cluster info cmd

2017-09-19 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/ROCKETMQ-278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16172706#comment-16172706
 ] 

ASF GitHub Bot commented on ROCKETMQ-278:
-

Github user vongosling commented on a diff in the pull request:

https://github.com/apache/incubator-rocketmq/pull/152#discussion_r139871853
  
--- Diff: 
namesrv/src/main/java/org/apache/rocketmq/namesrv/routeinfo/RouteInfoManager.java
 ---
@@ -63,11 +65,36 @@ public RouteInfoManager() {
 this.filterServerTable = new HashMap(256);
 }
 
-public byte[] getAllClusterInfo() {
-ClusterInfo clusterInfoSerializeWrapper = new ClusterInfo();
-
clusterInfoSerializeWrapper.setBrokerAddrTable(this.brokerAddrTable);
-
clusterInfoSerializeWrapper.setClusterAddrTable(this.clusterAddrTable);
-return clusterInfoSerializeWrapper.encode();
+public byte[] getAllClusterInfo(String cluster) {
--- End diff --

I agree @shroman here


> Add specified cluster for cluster info cmd
> --
>
> Key: ROCKETMQ-278
> URL: https://issues.apache.org/jira/browse/ROCKETMQ-278
> Project: Apache RocketMQ
>  Issue Type: Improvement
>  Components: rocketmq-client, rocketmq-namesrv, rocketmq-tools
>Affects Versions: 4.1.0-incubating
>Reporter: lindzh
>Assignee: Xiaorui Wang
> Fix For: 4.2.0-incubating
>
>
> When using mqadmin command clusterlist,it always display all cluster,i want 
> to display specified cluster.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (ROCKETMQ-278) Add specified cluster for cluster info cmd

2017-09-19 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/ROCKETMQ-278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16172701#comment-16172701
 ] 

ASF GitHub Bot commented on ROCKETMQ-278:
-

Github user dongeforever commented on the issue:

https://github.com/apache/incubator-rocketmq/pull/152
  
LGTM @zhouxinyu @vongosling 


> Add specified cluster for cluster info cmd
> --
>
> Key: ROCKETMQ-278
> URL: https://issues.apache.org/jira/browse/ROCKETMQ-278
> Project: Apache RocketMQ
>  Issue Type: Improvement
>  Components: rocketmq-client, rocketmq-namesrv, rocketmq-tools
>Affects Versions: 4.1.0-incubating
>Reporter: lindzh
>Assignee: Xiaorui Wang
> Fix For: 4.2.0-incubating
>
>
> When using mqadmin command clusterlist,it always display all cluster,i want 
> to display specified cluster.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (ROCKETMQ-271) add diagnosis tools

2017-09-19 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/ROCKETMQ-271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16172688#comment-16172688
 ] 

ASF GitHub Bot commented on ROCKETMQ-271:
-

Github user dongeforever commented on a diff in the pull request:

https://github.com/apache/incubator-rocketmq/pull/156#discussion_r139869366
  
--- Diff: 
client/src/main/java/org/apache/rocketmq/client/hook/TracerTimeSendMessageHook.java
 ---
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.rocketmq.client.hook;
+
+import java.lang.reflect.Method;
+import org.apache.rocketmq.common.ClientTracerTimeUtil;
+import org.apache.rocketmq.common.message.Message;
+import org.apache.rocketmq.common.message.MessageConst;
+
+public class TracerTimeSendMessageHook implements SendMessageHook {
+
+@Override
+public String hookName() {
+return "TracerTimeSendMessageHook";
+}
+
+@Override
+public void sendMessageBefore(SendMessageContext context) {
+tracerTimeIfNecessary(context.getMessage(), 
MessageConst.MESSAGE_SEND_TIME);
+}
+
+@Override
+public void sendMessageAfter(SendMessageContext context) {
+tracerTimeIfNecessary(context.getMessage(), 
MessageConst.RECEIVE_SEND_ACK_TIME);
+}
+
+public void tracerTimeIfNecessary(Message msg, String propertyKey) {
+if (ClientTracerTimeUtil.isEnableTracerTime()) {
+try {
+Method putPropertyMethod = 
msg.getClass().getDeclaredMethod("putProperty", String.class, String.class);
--- End diff --

Better to use MessageAccessor instead of reflection 


> add diagnosis tools
> ---
>
> Key: ROCKETMQ-271
> URL: https://issues.apache.org/jira/browse/ROCKETMQ-271
> Project: Apache RocketMQ
>  Issue Type: New Feature
>  Components: rocketmq-broker, rocketmq-client, rocketmq-commons, 
> rocketmq-remoting
>Reporter: yubaofu
>Assignee: yukon
>Priority: Minor
>   Original Estimate: 96h
>  Remaining Estimate: 96h
>
> when prodcurer send message timeout,how to find the root cause,it's difficult.
> so add the new diagnosis tools that record message lifetime ,eg message 
> create time,prodcurer send message to broker time,broker recive message 
> time,and send response to client time.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (ROCKETMQ-271) add diagnosis tools

2017-09-19 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/ROCKETMQ-271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16172689#comment-16172689
 ] 

ASF GitHub Bot commented on ROCKETMQ-271:
-

Github user dongeforever commented on a diff in the pull request:

https://github.com/apache/incubator-rocketmq/pull/156#discussion_r139868962
  
--- Diff: 
broker/src/main/java/org/apache/rocketmq/broker/ServerTracerTimeUtil.java ---
@@ -0,0 +1,108 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.rocketmq.broker;
+
+import com.google.common.cache.Cache;
+import com.google.common.cache.CacheBuilder;
+import java.util.concurrent.TimeUnit;
+import org.apache.rocketmq.common.ClientTracerTimeUtil;
+import org.apache.rocketmq.common.TracerTime;
+
+public class ServerTracerTimeUtil {
+
+public static Cache tracerTimeCache = 
CacheBuilder.newBuilder()
+.maximumSize(1)
+.expireAfterWrite(15, TimeUnit.MINUTES)
+.build();
+
+public static boolean isEnableTracerTime() {
+return ClientTracerTimeUtil.isEnableTracerTime();
+}
+
+public static void addMessageCreateTime(String messageTracerTimeId, 
String messageCreateTime) {
+if (messageCreateTime == null || messageCreateTime.length() < 1) {
+return;
+}
+
+TracerTime tracerTime = 
tracerTimeCache.getIfPresent(messageTracerTimeId);
+if (tracerTime == null) {
+tracerTime = new TracerTime();
--- End diff --

getIfPresent  may have concurrent problem

Maybe getOrDefault is OK



> add diagnosis tools
> ---
>
> Key: ROCKETMQ-271
> URL: https://issues.apache.org/jira/browse/ROCKETMQ-271
> Project: Apache RocketMQ
>  Issue Type: New Feature
>  Components: rocketmq-broker, rocketmq-client, rocketmq-commons, 
> rocketmq-remoting
>Reporter: yubaofu
>Assignee: yukon
>Priority: Minor
>   Original Estimate: 96h
>  Remaining Estimate: 96h
>
> when prodcurer send message timeout,how to find the root cause,it's difficult.
> so add the new diagnosis tools that record message lifetime ,eg message 
> create time,prodcurer send message to broker time,broker recive message 
> time,and send response to client time.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (ROCKETMQ-271) add diagnosis tools

2017-09-19 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/ROCKETMQ-271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16172677#comment-16172677
 ] 

ASF GitHub Bot commented on ROCKETMQ-271:
-

Github user dongeforever commented on the issue:

https://github.com/apache/incubator-rocketmq/pull/156
  
LGTM @zhouxinyu @vongosling 


> add diagnosis tools
> ---
>
> Key: ROCKETMQ-271
> URL: https://issues.apache.org/jira/browse/ROCKETMQ-271
> Project: Apache RocketMQ
>  Issue Type: New Feature
>  Components: rocketmq-broker, rocketmq-client, rocketmq-commons, 
> rocketmq-remoting
>Reporter: yubaofu
>Assignee: yukon
>Priority: Minor
>   Original Estimate: 96h
>  Remaining Estimate: 96h
>
> when prodcurer send message timeout,how to find the root cause,it's difficult.
> so add the new diagnosis tools that record message lifetime ,eg message 
> create time,prodcurer send message to broker time,broker recive message 
> time,and send response to client time.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (ROCKETMQ-286) Add a switch to decide whether to wake up commitLogService when put message

2017-09-18 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/ROCKETMQ-286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16171070#comment-16171070
 ] 

ASF GitHub Bot commented on ROCKETMQ-286:
-

Github user dongeforever commented on the issue:

https://github.com/apache/incubator-rocketmq/pull/162
  
@vsair @zhouxinyu another config flushCommitLogTimed has already handled 
such situation, and avoided SSD write amplification.
If flushCommitLogTimed is enabled, it will only wakeup the 
commitLogService, but could not wakeup the flushCommitLogService, for: 
```
 if (flushCommitLogTimed) {
  Thread.sleep(interval);
 } else {
  this.waitForRunning(interval);
}
```



> Add a switch to decide whether to wake up commitLogService when put message
> ---
>
> Key: ROCKETMQ-286
> URL: https://issues.apache.org/jira/browse/ROCKETMQ-286
> Project: Apache RocketMQ
>  Issue Type: Improvement
>  Components: rocketmq-store
>Affects Versions: 4.0.0-incubating, 4.1.0-incubating
>Reporter: Eric Liu
>Assignee: Eric Liu
> Fix For: 4.2.0-incubating
>
>
> Now, it will wake up commitLogService after putting message. This may have 
> impact on disk.
> Add a switch to control this feature.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (ROCKETMQ-27) Add storage partition online

2017-09-18 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/ROCKETMQ-27?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16171052#comment-16171052
 ] 

ASF GitHub Bot commented on ROCKETMQ-27:


Github user dongeforever commented on the issue:

https://github.com/apache/incubator-rocketmq/pull/28
  
@lizhanhui LGTM. Version 4.2.0 is going  to be released now. It will be 
nice to contain this feature. Could you please add one IT Test to maintain the 
quality?



> Add storage partition online
> 
>
> Key: ROCKETMQ-27
> URL: https://issues.apache.org/jira/browse/ROCKETMQ-27
> Project: Apache RocketMQ
>  Issue Type: New Feature
>  Components: rocketmq-store, rocketmq-tools
>Affects Versions: 4.1.0-incubating
>Reporter: Zhanhui Li
>Assignee: vongosling
> Fix For: 4.2.0-incubating
>
>
> RocketMQ broker stores all messages before they are expired, which requires 
> pretty much storage for a high load system. For now, the message store can be 
> configured to use only one partition. Quite often, we find broker runs out of 
> storage as business expands quickly. Online resizing a partition most of time 
> is not possible(with absence of LVM) whilst attaching a new disk/partition in 
> cloud environment is super easy. 
> It would be nice for RocketMQ to be capable of adding extra commit log store 
> path online, allocating commit log files onto partitions based on 
> free/available space. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (ROCKETMQ-23) MappedFileQueue#flush should return true when flushing is successful

2017-09-18 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/ROCKETMQ-23?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16171037#comment-16171037
 ] 

ASF GitHub Bot commented on ROCKETMQ-23:


Github user dongeforever commented on the issue:

https://github.com/apache/incubator-rocketmq/pull/20
  
@shroman could you please resolve the conflicts?
IMO, the return value of flush() and commit(), means whether there is data 
left to be flushed or committed, that is OK.
if you'd like to polish the semantic of the returned value, may you do the 
same polishing for commit() too, just to keep the same semantic for similar 
methods.


> MappedFileQueue#flush should return true when flushing is successful
> 
>
> Key: ROCKETMQ-23
> URL: https://issues.apache.org/jira/browse/ROCKETMQ-23
> Project: Apache RocketMQ
>  Issue Type: Improvement
>  Components: rocketmq-store
>Affects Versions: 4.0.0-incubating
>Reporter: Roman Shtykh
>Assignee: Roman Shtykh
> Fix For: 4.2.0-incubating
>
>
> In the current implementation, MappedFileQueue#flush returns {{false}} when 
> flushing is successful.
> This is not intuitive and error prone.
> For instance, in {{CommitLog#run line:915-918}}
> {code}
> for (int i = 0; i < RETRY_TIMES_OVER && !result; i++) {
>result = CommitLog.this.mappedFileQueue.flush(0);
>// ...
> }
> {code}
> I believe retries has to be done when flushing is not successful. But with 
> the code above, it can try to flush only once on CommitLog termination and 
> not continue retrying.
> The same is for {{DefaultMessageStore#doFlush line:1551}}
> Or is this not retry on failure, but the number of times flushing has to be 
> done? Then, {{RETRY_TIMES_OVER}} should be renamed to something like 
> {{FLUSH_NUM}}.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (ROCKETMQ-286) Add a switch to decide whether to wake up commitLogService when put message

2017-09-15 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/ROCKETMQ-286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16167477#comment-16167477
 ] 

ASF GitHub Bot commented on ROCKETMQ-286:
-

Github user zhouxinyu commented on the issue:

https://github.com/apache/incubator-rocketmq/pull/162
  
LGTM, wake up `commitLogService` is for real-time, but it may let broker 
suffer from SSD write amplification, so provide a switch is reasonable.


> Add a switch to decide whether to wake up commitLogService when put message
> ---
>
> Key: ROCKETMQ-286
> URL: https://issues.apache.org/jira/browse/ROCKETMQ-286
> Project: Apache RocketMQ
>  Issue Type: Improvement
>  Components: rocketmq-store
>Affects Versions: 4.0.0-incubating, 4.1.0-incubating
>Reporter: Eric Liu
>Assignee: Eric Liu
> Fix For: 4.2.0-incubating
>
>
> Now, it will wake up commitLogService after putting message. This may have 
> impact on disk.
> Add a switch to control this feature.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (ROCKETMQ-289) getBrokerRuntimeInfo method may be output fileReservedTime value

2017-09-15 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/ROCKETMQ-289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16167472#comment-16167472
 ] 

ASF GitHub Bot commented on ROCKETMQ-289:
-

Github user zhouxinyu commented on the issue:

https://github.com/apache/incubator-rocketmq/pull/163
  
Hi, thanks for your contribution. We could get FileReservedTime through 
`GetBrokerConfigCommand`, there is no need to add it to `runtimeInfo `.


> getBrokerRuntimeInfo method may be output fileReservedTime value 
> -
>
> Key: ROCKETMQ-289
> URL: https://issues.apache.org/jira/browse/ROCKETMQ-289
> Project: Apache RocketMQ
>  Issue Type: Bug
>  Components: rocketmq-broker
>Affects Versions: 4.0.0-incubating, 4.1.0-incubating, 4.2.0-incubating, 
> 4.3.0-incubating
>Reporter: yubaofu
>Assignee: yukon
> Fix For: 4.2.0-incubating, 4.3.0-incubating
>
>
> for some reasons, getBrokerRuntimeInfo method may be output fileReservedTime 
> value.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (ROCKETMQ-278) Add specified cluster for cluster info cmd

2017-09-15 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/ROCKETMQ-278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16167446#comment-16167446
 ] 

ASF GitHub Bot commented on ROCKETMQ-278:
-

Github user shroman commented on a diff in the pull request:

https://github.com/apache/incubator-rocketmq/pull/152#discussion_r139078626
  
--- Diff: 
namesrv/src/main/java/org/apache/rocketmq/namesrv/routeinfo/RouteInfoManager.java
 ---
@@ -63,11 +65,36 @@ public RouteInfoManager() {
 this.filterServerTable = new HashMap(256);
 }
 
-public byte[] getAllClusterInfo() {
-ClusterInfo clusterInfoSerializeWrapper = new ClusterInfo();
-
clusterInfoSerializeWrapper.setBrokerAddrTable(this.brokerAddrTable);
-
clusterInfoSerializeWrapper.setClusterAddrTable(this.clusterAddrTable);
-return clusterInfoSerializeWrapper.encode();
+public byte[] getAllClusterInfo(String cluster) {
--- End diff --

You change the signature of the public class here.
How about having another method -- `public byte[] getAllClusterInfo()`? And 
you won't need calling `getAllClusterInfo(null)` then too ;)


> Add specified cluster for cluster info cmd
> --
>
> Key: ROCKETMQ-278
> URL: https://issues.apache.org/jira/browse/ROCKETMQ-278
> Project: Apache RocketMQ
>  Issue Type: Improvement
>  Components: rocketmq-client, rocketmq-namesrv, rocketmq-tools
>Affects Versions: 4.1.0-incubating
>Reporter: lindzh
>Assignee: Xiaorui Wang
> Fix For: 4.2.0-incubating
>
>
> When using mqadmin command clusterlist,it always display all cluster,i want 
> to display specified cluster.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (ROCKETMQ-268) add rocketmq client4cpp

2017-09-13 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/ROCKETMQ-268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16165625#comment-16165625
 ] 

ASF GitHub Bot commented on ROCKETMQ-268:
-

Github user vincentWangKB commented on the issue:

https://github.com/apache/incubator-rocketmq-externals/pull/27
  
close pull request, as rocketmq cpp code had been merged


> add rocketmq client4cpp
> ---
>
> Key: ROCKETMQ-268
> URL: https://issues.apache.org/jira/browse/ROCKETMQ-268
> Project: Apache RocketMQ
>  Issue Type: New Feature
> Environment: Linux system
>Reporter: wang qiwei
>Assignee: vongosling
>Priority: Minor
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (ROCKETMQ-268) add rocketmq client4cpp

2017-09-13 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/ROCKETMQ-268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16165626#comment-16165626
 ] 

ASF GitHub Bot commented on ROCKETMQ-268:
-

Github user vincentWangKB closed the pull request at:

https://github.com/apache/incubator-rocketmq-externals/pull/27


> add rocketmq client4cpp
> ---
>
> Key: ROCKETMQ-268
> URL: https://issues.apache.org/jira/browse/ROCKETMQ-268
> Project: Apache RocketMQ
>  Issue Type: New Feature
> Environment: Linux system
>Reporter: wang qiwei
>Assignee: vongosling
>Priority: Minor
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (ROCKETMQ-286) Add a switch to decide whether to wake up commitLogService when put message

2017-09-13 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/ROCKETMQ-286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16165612#comment-16165612
 ] 

ASF GitHub Bot commented on ROCKETMQ-286:
-

Github user vsair commented on the issue:

https://github.com/apache/incubator-rocketmq/pull/162
  
@zhouxinyu  @vongosling  pls have a review ?


> Add a switch to decide whether to wake up commitLogService when put message
> ---
>
> Key: ROCKETMQ-286
> URL: https://issues.apache.org/jira/browse/ROCKETMQ-286
> Project: Apache RocketMQ
>  Issue Type: Improvement
>  Components: rocketmq-store
>Affects Versions: 4.0.0-incubating, 4.1.0-incubating
>Reporter: Eric Liu
>Assignee: Eric Liu
> Fix For: 4.2.0-incubating
>
>
> Now, it will wake up commitLogService after putting message. This may have 
> impact on disk.
> Add a switch to control this feature.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (ROCKETMQ-268) add rocketmq client4cpp

2017-09-13 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/ROCKETMQ-268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16164461#comment-16164461
 ] 

ASF GitHub Bot commented on ROCKETMQ-268:
-

Github user XadillaX commented on the issue:

https://github.com/apache/incubator-rocketmq-externals/pull/27
  
Consider removing boost? This dependency is too large for a SDK not a 
complete project.


> add rocketmq client4cpp
> ---
>
> Key: ROCKETMQ-268
> URL: https://issues.apache.org/jira/browse/ROCKETMQ-268
> Project: Apache RocketMQ
>  Issue Type: New Feature
> Environment: Linux system
>Reporter: wang qiwei
>Assignee: vongosling
>Priority: Minor
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (ROCKETMQ-289) getBrokerRuntimeInfo method may be output fileReservedTime value

2017-09-12 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/ROCKETMQ-289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16164148#comment-16164148
 ] 

ASF GitHub Bot commented on ROCKETMQ-289:
-

Github user coveralls commented on the issue:

https://github.com/apache/incubator-rocketmq/pull/163
  

[![Coverage 
Status](https://coveralls.io/builds/13246332/badge)](https://coveralls.io/builds/13246332)

Coverage increased (+0.1%) to 38.851% when pulling 
**c8079c3048c67b11ba9d78320a702be62aace523 on fuyou001:ROCKETMQ-289** into 
**368e7c86a0b06099f336c81672112dcb5143cf9e on apache:develop**.



> getBrokerRuntimeInfo method may be output fileReservedTime value 
> -
>
> Key: ROCKETMQ-289
> URL: https://issues.apache.org/jira/browse/ROCKETMQ-289
> Project: Apache RocketMQ
>  Issue Type: Bug
>  Components: rocketmq-broker
>Affects Versions: 4.0.0-incubating, 4.1.0-incubating, 4.2.0-incubating, 
> 4.3.0-incubating
>Reporter: yubaofu
>Assignee: yukon
> Fix For: 4.2.0-incubating, 4.3.0-incubating
>
>
> for some reasons, getBrokerRuntimeInfo method may be output fileReservedTime 
> value.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (ROCKETMQ-193) Develop rocketmq-redis-replicator component

2017-09-10 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/ROCKETMQ-193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16160316#comment-16160316
 ] 

ASF GitHub Bot commented on ROCKETMQ-193:
-

Github user leonchen83 commented on the issue:

https://github.com/apache/incubator-rocketmq-externals/pull/29
  
@vongosling you will see the result of integration-test at 
[incubator-rocketmq-externals](https://travis-ci.org/leonchen83/incubator-rocketmq-externals/branches)


> Develop rocketmq-redis-replicator component
> ---
>
> Key: ROCKETMQ-193
> URL: https://issues.apache.org/jira/browse/ROCKETMQ-193
> Project: Apache RocketMQ
>  Issue Type: Task
>Reporter: Rich Zhang
>Assignee: Rich Zhang
>Priority: Minor
> Fix For: 4.2.0-incubating
>
>
> Design:
>   Redis supplies an official replication mechanism , and slave communicates 
> to master with RESP protocol, so a natural way to design the 
> rocketmq-redis-replicator component is simulating itself as a slave, sending 
> commands to master and receiving datas from master timely, and then resending 
> to rocketmq broker.
>   If you are not familiar with redis replication mechanism, please learn this 
> section first [1]. After that, I will illustrate some key points ahead.
> 1. To make slave start from the point where it left off when it reconnects, 
> slave and master should agree on a master runId and a replication offset. 
> Slave acknowledges this offset to master periodically. In other words，slave 
> may received duplicate commands. Along with, the rocketmq-redis-replicator 
> component may send duplicate messages too. A good way to minimize the 
> duplicate time window is reducing the "ack period" to a smaller one, such as 
> 100ms.
> 2. If slave keeps offline for some time, it’s easy to use up backlog whose 
> default value is just 1M, especially for a high-traffic redis instance. 
> Unfortunatelly,if slave replication offset has already been covered in master 
> backlog, a full synchronization will have to execute, which is unacceptable 
> for rocketmq-redis-replicator component as a large number of messages will be 
> sent out intensively.
> 3. When synchronizing from master fully, master will generate a new rdb 
> file(the rdb file format [2])，and slave will receive this file,store in disk, 
> and last apply to memory. This strategy makes slave reaches a consistent 
> state with master as soon as possible, and hardly fail. For 
> rocketmq-redis-replicator component, it’s also a good way to prevent  
> synchronizing initial rdb file from failure in halfway. 
>   There already an open source project [3] which focuses on replicating redis 
> data, and provides api to handle data received [4]. The principal thoughts 
> are simulating itself as a slave , following official replication procedure, 
> communicating with master by RESP, and acking master with replication offset. 
> Base on this project to develop is a good idea, meanwhile some aspects should 
> also be enhanced and considered more robust. Here is some points: 
>   [High Available]
>Keeping the replication component's high availability is not difficult but 
> important, not only for providing an uninterruptible service. If component 
> leaves off for some time, a unacceptable full synchronization may be 
> triggered. 
>It’s also easy to reach high availability, including adopting master/slave 
> module, using zookeeper to coordinate and switch master/slave, storing data 
> onto zookeeper to keep component stateless. 
>   [Data Loss]
>Generally, data loss should be tried best to avoid. The key point is that 
> slave only acks replication offset to master after sending command to 
> rocketmq broker successfully.
>   [Data Stale]
>It also happened when slave reconnect. Consider case below:
>`time1` `time2`  `time3`
>set k=a set k=b set k=c
>If slave left off at time3, but the latest replication offset reported to 
> master is only at time1, when slave reconnected, it re-apply commands “set 
> k=b… set k=c”. In a small time window, “k” will equal the stale “b” until 
> “set k=c” command is applied. So the slave offline time shorter, the better.
>[Message Order]
>Redis uses single thread model to keep command execute in order, because 
> of its high performance. Replicating data with a single thread in slave is 
> also fine, as it is also totally memory operation. But sending all data to 
> rocketmq in a global order is a good choose? Producer should have no 
> performance issue, but consumer may not be able to consume messages in time, 
> especially redis was in a high load. 
>Hashing “KEY” to different rocketmq queue is a good strategy. Guarantee 
> the same key operation route to a unique queue, to keep partial ordered, and 
> the

[jira] [Commented] (ROCKETMQ-193) Develop rocketmq-redis-replicator component

2017-09-10 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/ROCKETMQ-193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16160315#comment-16160315
 ] 

ASF GitHub Bot commented on ROCKETMQ-193:
-

Github user leonchen83 commented on the issue:

https://github.com/apache/incubator-rocketmq-externals/pull/29
  
@vongosling I already added an integration-test to 
rocketmq-redis-replicator base on travis-ci.
please help review this commit



> Develop rocketmq-redis-replicator component
> ---
>
> Key: ROCKETMQ-193
> URL: https://issues.apache.org/jira/browse/ROCKETMQ-193
> Project: Apache RocketMQ
>  Issue Type: Task
>Reporter: Rich Zhang
>Assignee: Rich Zhang
>Priority: Minor
> Fix For: 4.2.0-incubating
>
>
> Design:
>   Redis supplies an official replication mechanism , and slave communicates 
> to master with RESP protocol, so a natural way to design the 
> rocketmq-redis-replicator component is simulating itself as a slave, sending 
> commands to master and receiving datas from master timely, and then resending 
> to rocketmq broker.
>   If you are not familiar with redis replication mechanism, please learn this 
> section first [1]. After that, I will illustrate some key points ahead.
> 1. To make slave start from the point where it left off when it reconnects, 
> slave and master should agree on a master runId and a replication offset. 
> Slave acknowledges this offset to master periodically. In other words，slave 
> may received duplicate commands. Along with, the rocketmq-redis-replicator 
> component may send duplicate messages too. A good way to minimize the 
> duplicate time window is reducing the "ack period" to a smaller one, such as 
> 100ms.
> 2. If slave keeps offline for some time, it’s easy to use up backlog whose 
> default value is just 1M, especially for a high-traffic redis instance. 
> Unfortunatelly,if slave replication offset has already been covered in master 
> backlog, a full synchronization will have to execute, which is unacceptable 
> for rocketmq-redis-replicator component as a large number of messages will be 
> sent out intensively.
> 3. When synchronizing from master fully, master will generate a new rdb 
> file(the rdb file format [2])，and slave will receive this file,store in disk, 
> and last apply to memory. This strategy makes slave reaches a consistent 
> state with master as soon as possible, and hardly fail. For 
> rocketmq-redis-replicator component, it’s also a good way to prevent  
> synchronizing initial rdb file from failure in halfway. 
>   There already an open source project [3] which focuses on replicating redis 
> data, and provides api to handle data received [4]. The principal thoughts 
> are simulating itself as a slave , following official replication procedure, 
> communicating with master by RESP, and acking master with replication offset. 
> Base on this project to develop is a good idea, meanwhile some aspects should 
> also be enhanced and considered more robust. Here is some points: 
>   [High Available]
>Keeping the replication component's high availability is not difficult but 
> important, not only for providing an uninterruptible service. If component 
> leaves off for some time, a unacceptable full synchronization may be 
> triggered. 
>It’s also easy to reach high availability, including adopting master/slave 
> module, using zookeeper to coordinate and switch master/slave, storing data 
> onto zookeeper to keep component stateless. 
>   [Data Loss]
>Generally, data loss should be tried best to avoid. The key point is that 
> slave only acks replication offset to master after sending command to 
> rocketmq broker successfully.
>   [Data Stale]
>It also happened when slave reconnect. Consider case below:
>`time1` `time2`  `time3`
>set k=a set k=b set k=c
>If slave left off at time3, but the latest replication offset reported to 
> master is only at time1, when slave reconnected, it re-apply commands “set 
> k=b… set k=c”. In a small time window, “k” will equal the stale “b” until 
> “set k=c” command is applied. So the slave offline time shorter, the better.
>[Message Order]
>Redis uses single thread model to keep command execute in order, because 
> of its high performance. Replicating data with a single thread in slave is 
> also fine, as it is also totally memory operation. But sending all data to 
> rocketmq in a global order is a good choose? Producer should have no 
> performance issue, but consumer may not be able to consume messages in time, 
> especially redis was in a high load. 
>Hashing “KEY” to different rocketmq queue is a good strategy. Guarantee 
> the same key operation route to a unique queue, to keep partial ordered, and 
> the downstream consumer could

[jira] [Commented] (ROCKETMQ-266) Can’t start consumer with a small “consumerThreadMax” number

2017-09-08 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/ROCKETMQ-266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16158516#comment-16158516
 ] 

ASF GitHub Bot commented on ROCKETMQ-266:
-

Github user coveralls commented on the issue:

https://github.com/apache/incubator-rocketmq/pull/147
  

[![Coverage 
Status](https://coveralls.io/builds/13183888/badge)](https://coveralls.io/builds/13183888)

Coverage increased (+0.03%) to 38.743% when pulling 
**7d2f899e15d7ed4127657a49034bdae892e4174b on mark800:master** into 
**845830865fc37d0364a19cbd89ceaf8a30b37e1c on apache:develop**.



> Can’t start consumer with a small “consumerThreadMax” number
> 
>
> Key: ROCKETMQ-266
> URL: https://issues.apache.org/jira/browse/ROCKETMQ-266
> Project: Apache RocketMQ
>  Issue Type: Bug
>  Components: rocketmq-client
>Affects Versions: 4.1.0-incubating
>Reporter: Mark Yang
>Assignee: Xiaorui Wang
> Fix For: 4.2.0-incubating
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (ROCKETMQ-271) add diagnosis tools

2017-09-07 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/ROCKETMQ-271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16157996#comment-16157996
 ] 

ASF GitHub Bot commented on ROCKETMQ-271:
-

Github user coveralls commented on the issue:

https://github.com/apache/incubator-rocketmq/pull/156
  

[![Coverage 
Status](https://coveralls.io/builds/13178521/badge)](https://coveralls.io/builds/13178521)

Coverage decreased (-1.6%) to 37.094% when pulling 
**94ffbc33084381858b13592ea7b0a068479b77ce on fuyou001:ROCKETMQ-271** into 
**368e7c86a0b06099f336c81672112dcb5143cf9e on apache:develop**.



> add diagnosis tools
> ---
>
> Key: ROCKETMQ-271
> URL: https://issues.apache.org/jira/browse/ROCKETMQ-271
> Project: Apache RocketMQ
>  Issue Type: New Feature
>  Components: rocketmq-broker, rocketmq-client, rocketmq-commons, 
> rocketmq-remoting
>Reporter: yubaofu
>Assignee: yukon
>Priority: Minor
>   Original Estimate: 96h
>  Remaining Estimate: 96h
>
> when prodcurer send message timeout,how to find the root cause,it's difficult.
> so add the new diagnosis tools that record message lifetime ,eg message 
> create time,prodcurer send message to broker time,broker recive message 
> time,and send response to client time.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (ROCKETMQ-280) improve dependency management in parent pom

2017-09-07 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/ROCKETMQ-280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16156733#comment-16156733
 ] 

ASF GitHub Bot commented on ROCKETMQ-280:
-

Github user qqeasonchen commented on the issue:

https://github.com/apache/incubator-rocketmq/pull/155
  
@zhouxinyu @lizhanhui @Jaskey 
can this be merged?


> improve dependency management in parent pom 
> 
>
> Key: ROCKETMQ-280
> URL: https://issues.apache.org/jira/browse/ROCKETMQ-280
> Project: Apache RocketMQ
>  Issue Type: Improvement
>  Components: build
>Affects Versions: 4.1.0-incubating
> Environment: compile
>Reporter: Eason Chen
>Assignee: Stevens Chew
>Priority: Minor
> Fix For: 4.2.0-incubating
>
>
> {code:java}
> as is:
> 
> org.apache.rocketmq
> rocketmq-filter
> ${project.version}
> 
> to be:
> 
> ${project.groupId}
> rocketmq-filter
> ${project.version}
> 
> 
> ${project.groupId}
> rocketmq-openmessaging
> ${project.version}
> 
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (ROCKETMQ-193) Develop rocketmq-redis-replicator component

2017-09-07 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/ROCKETMQ-193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16156714#comment-16156714
 ] 

ASF GitHub Bot commented on ROCKETMQ-193:
-

Github user leonchen83 commented on the issue:

https://github.com/apache/incubator-rocketmq-externals/pull/29
  
I recommend using travis ci to do integration test as other open source 
project.
but first need to add this sub-project to travis ci.
I have no permission to do this.


> Develop rocketmq-redis-replicator component
> ---
>
> Key: ROCKETMQ-193
> URL: https://issues.apache.org/jira/browse/ROCKETMQ-193
> Project: Apache RocketMQ
>  Issue Type: Task
>Reporter: Rich Zhang
>Assignee: Rich Zhang
>Priority: Minor
> Fix For: 4.2.0-incubating
>
>
> Design:
>   Redis supplies an official replication mechanism , and slave communicates 
> to master with RESP protocol, so a natural way to design the 
> rocketmq-redis-replicator component is simulating itself as a slave, sending 
> commands to master and receiving datas from master timely, and then resending 
> to rocketmq broker.
>   If you are not familiar with redis replication mechanism, please learn this 
> section first [1]. After that, I will illustrate some key points ahead.
> 1. To make slave start from the point where it left off when it reconnects, 
> slave and master should agree on a master runId and a replication offset. 
> Slave acknowledges this offset to master periodically. In other words，slave 
> may received duplicate commands. Along with, the rocketmq-redis-replicator 
> component may send duplicate messages too. A good way to minimize the 
> duplicate time window is reducing the "ack period" to a smaller one, such as 
> 100ms.
> 2. If slave keeps offline for some time, it’s easy to use up backlog whose 
> default value is just 1M, especially for a high-traffic redis instance. 
> Unfortunatelly,if slave replication offset has already been covered in master 
> backlog, a full synchronization will have to execute, which is unacceptable 
> for rocketmq-redis-replicator component as a large number of messages will be 
> sent out intensively.
> 3. When synchronizing from master fully, master will generate a new rdb 
> file(the rdb file format [2])，and slave will receive this file,store in disk, 
> and last apply to memory. This strategy makes slave reaches a consistent 
> state with master as soon as possible, and hardly fail. For 
> rocketmq-redis-replicator component, it’s also a good way to prevent  
> synchronizing initial rdb file from failure in halfway. 
>   There already an open source project [3] which focuses on replicating redis 
> data, and provides api to handle data received [4]. The principal thoughts 
> are simulating itself as a slave , following official replication procedure, 
> communicating with master by RESP, and acking master with replication offset. 
> Base on this project to develop is a good idea, meanwhile some aspects should 
> also be enhanced and considered more robust. Here is some points: 
>   [High Available]
>Keeping the replication component's high availability is not difficult but 
> important, not only for providing an uninterruptible service. If component 
> leaves off for some time, a unacceptable full synchronization may be 
> triggered. 
>It’s also easy to reach high availability, including adopting master/slave 
> module, using zookeeper to coordinate and switch master/slave, storing data 
> onto zookeeper to keep component stateless. 
>   [Data Loss]
>Generally, data loss should be tried best to avoid. The key point is that 
> slave only acks replication offset to master after sending command to 
> rocketmq broker successfully.
>   [Data Stale]
>It also happened when slave reconnect. Consider case below:
>`time1` `time2`  `time3`
>set k=a set k=b set k=c
>If slave left off at time3, but the latest replication offset reported to 
> master is only at time1, when slave reconnected, it re-apply commands “set 
> k=b… set k=c”. In a small time window, “k” will equal the stale “b” until 
> “set k=c” command is applied. So the slave offline time shorter, the better.
>[Message Order]
>Redis uses single thread model to keep command execute in order, because 
> of its high performance. Replicating data with a single thread in slave is 
> also fine, as it is also totally memory operation. But sending all data to 
> rocketmq in a global order is a good choose? Producer should have no 
> performance issue, but consumer may not be able to consume messages in time, 
> especially redis was in a high load. 
>Hashing “KEY” to different rocketmq queue is a good strategy. Guarantee 
> the same key operation route to a unique queue, to keep partial

[jira] [Commented] (ROCKETMQ-193) Develop rocketmq-redis-replicator component

2017-09-07 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/ROCKETMQ-193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16156687#comment-16156687
 ] 

ASF GitHub Bot commented on ROCKETMQ-193:
-

Github user vongosling commented on the issue:

https://github.com/apache/incubator-rocketmq-externals/pull/29
  
 Write necessary unit-test to verify your logic correction, more mock a 
little better when cross module dependency exist. If the new feature or 
significant change is committed, please remember to add integration-test in 
test module.


Do you have test module in this replicator ?


> Develop rocketmq-redis-replicator component
> ---
>
> Key: ROCKETMQ-193
> URL: https://issues.apache.org/jira/browse/ROCKETMQ-193
> Project: Apache RocketMQ
>  Issue Type: Task
>Reporter: Rich Zhang
>Assignee: Rich Zhang
>Priority: Minor
> Fix For: 4.2.0-incubating
>
>
> Design:
>   Redis supplies an official replication mechanism , and slave communicates 
> to master with RESP protocol, so a natural way to design the 
> rocketmq-redis-replicator component is simulating itself as a slave, sending 
> commands to master and receiving datas from master timely, and then resending 
> to rocketmq broker.
>   If you are not familiar with redis replication mechanism, please learn this 
> section first [1]. After that, I will illustrate some key points ahead.
> 1. To make slave start from the point where it left off when it reconnects, 
> slave and master should agree on a master runId and a replication offset. 
> Slave acknowledges this offset to master periodically. In other words，slave 
> may received duplicate commands. Along with, the rocketmq-redis-replicator 
> component may send duplicate messages too. A good way to minimize the 
> duplicate time window is reducing the "ack period" to a smaller one, such as 
> 100ms.
> 2. If slave keeps offline for some time, it’s easy to use up backlog whose 
> default value is just 1M, especially for a high-traffic redis instance. 
> Unfortunatelly,if slave replication offset has already been covered in master 
> backlog, a full synchronization will have to execute, which is unacceptable 
> for rocketmq-redis-replicator component as a large number of messages will be 
> sent out intensively.
> 3. When synchronizing from master fully, master will generate a new rdb 
> file(the rdb file format [2])，and slave will receive this file,store in disk, 
> and last apply to memory. This strategy makes slave reaches a consistent 
> state with master as soon as possible, and hardly fail. For 
> rocketmq-redis-replicator component, it’s also a good way to prevent  
> synchronizing initial rdb file from failure in halfway. 
>   There already an open source project [3] which focuses on replicating redis 
> data, and provides api to handle data received [4]. The principal thoughts 
> are simulating itself as a slave , following official replication procedure, 
> communicating with master by RESP, and acking master with replication offset. 
> Base on this project to develop is a good idea, meanwhile some aspects should 
> also be enhanced and considered more robust. Here is some points: 
>   [High Available]
>Keeping the replication component's high availability is not difficult but 
> important, not only for providing an uninterruptible service. If component 
> leaves off for some time, a unacceptable full synchronization may be 
> triggered. 
>It’s also easy to reach high availability, including adopting master/slave 
> module, using zookeeper to coordinate and switch master/slave, storing data 
> onto zookeeper to keep component stateless. 
>   [Data Loss]
>Generally, data loss should be tried best to avoid. The key point is that 
> slave only acks replication offset to master after sending command to 
> rocketmq broker successfully.
>   [Data Stale]
>It also happened when slave reconnect. Consider case below:
>`time1` `time2`  `time3`
>set k=a set k=b set k=c
>If slave left off at time3, but the latest replication offset reported to 
> master is only at time1, when slave reconnected, it re-apply commands “set 
> k=b… set k=c”. In a small time window, “k” will equal the stale “b” until 
> “set k=c” command is applied. So the slave offline time shorter, the better.
>[Message Order]
>Redis uses single thread model to keep command execute in order, because 
> of its high performance. Replicating data with a single thread in slave is 
> also fine, as it is also totally memory operation. But sending all data to 
> rocketmq in a global order is a good choose? Producer should have no 
> performance issue, but consumer may not be able to consume messages in time, 
> especially redis was in a high load. 
>Hashing “KEY” to different

[jira] [Commented] (ROCKETMQ-193) Develop rocketmq-redis-replicator component

2017-09-07 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/ROCKETMQ-193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16156673#comment-16156673
 ] 

ASF GitHub Bot commented on ROCKETMQ-193:
-

Github user vongosling commented on the issue:

https://github.com/apache/incubator-rocketmq-externals/pull/23
  
@Zhang-Ke Thanks, It is really good practice when discarded deserted pr.


> Develop rocketmq-redis-replicator component
> ---
>
> Key: ROCKETMQ-193
> URL: https://issues.apache.org/jira/browse/ROCKETMQ-193
> Project: Apache RocketMQ
>  Issue Type: Task
>Reporter: Rich Zhang
>Assignee: Rich Zhang
>Priority: Minor
> Fix For: 4.2.0-incubating
>
>
> Design:
>   Redis supplies an official replication mechanism , and slave communicates 
> to master with RESP protocol, so a natural way to design the 
> rocketmq-redis-replicator component is simulating itself as a slave, sending 
> commands to master and receiving datas from master timely, and then resending 
> to rocketmq broker.
>   If you are not familiar with redis replication mechanism, please learn this 
> section first [1]. After that, I will illustrate some key points ahead.
> 1. To make slave start from the point where it left off when it reconnects, 
> slave and master should agree on a master runId and a replication offset. 
> Slave acknowledges this offset to master periodically. In other words，slave 
> may received duplicate commands. Along with, the rocketmq-redis-replicator 
> component may send duplicate messages too. A good way to minimize the 
> duplicate time window is reducing the "ack period" to a smaller one, such as 
> 100ms.
> 2. If slave keeps offline for some time, it’s easy to use up backlog whose 
> default value is just 1M, especially for a high-traffic redis instance. 
> Unfortunatelly,if slave replication offset has already been covered in master 
> backlog, a full synchronization will have to execute, which is unacceptable 
> for rocketmq-redis-replicator component as a large number of messages will be 
> sent out intensively.
> 3. When synchronizing from master fully, master will generate a new rdb 
> file(the rdb file format [2])，and slave will receive this file,store in disk, 
> and last apply to memory. This strategy makes slave reaches a consistent 
> state with master as soon as possible, and hardly fail. For 
> rocketmq-redis-replicator component, it’s also a good way to prevent  
> synchronizing initial rdb file from failure in halfway. 
>   There already an open source project [3] which focuses on replicating redis 
> data, and provides api to handle data received [4]. The principal thoughts 
> are simulating itself as a slave , following official replication procedure, 
> communicating with master by RESP, and acking master with replication offset. 
> Base on this project to develop is a good idea, meanwhile some aspects should 
> also be enhanced and considered more robust. Here is some points: 
>   [High Available]
>Keeping the replication component's high availability is not difficult but 
> important, not only for providing an uninterruptible service. If component 
> leaves off for some time, a unacceptable full synchronization may be 
> triggered. 
>It’s also easy to reach high availability, including adopting master/slave 
> module, using zookeeper to coordinate and switch master/slave, storing data 
> onto zookeeper to keep component stateless. 
>   [Data Loss]
>Generally, data loss should be tried best to avoid. The key point is that 
> slave only acks replication offset to master after sending command to 
> rocketmq broker successfully.
>   [Data Stale]
>It also happened when slave reconnect. Consider case below:
>`time1` `time2`  `time3`
>set k=a set k=b set k=c
>If slave left off at time3, but the latest replication offset reported to 
> master is only at time1, when slave reconnected, it re-apply commands “set 
> k=b… set k=c”. In a small time window, “k” will equal the stale “b” until 
> “set k=c” command is applied. So the slave offline time shorter, the better.
>[Message Order]
>Redis uses single thread model to keep command execute in order, because 
> of its high performance. Replicating data with a single thread in slave is 
> also fine, as it is also totally memory operation. But sending all data to 
> rocketmq in a global order is a good choose? Producer should have no 
> performance issue, but consumer may not be able to consume messages in time, 
> especially redis was in a high load. 
>Hashing “KEY” to different rocketmq queue is a good strategy. Guarantee 
> the same key operation route to a unique queue, to keep partial ordered, and 
> the downstream consumer could consume messages concurrently. Of course, some 
> dependency

[jira] [Commented] (ROCKETMQ-193) Develop rocketmq-redis-replicator component

2017-09-07 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/ROCKETMQ-193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16156644#comment-16156644
 ] 

ASF GitHub Bot commented on ROCKETMQ-193:
-

Github user Zhang-Ke commented on the issue:

https://github.com/apache/incubator-rocketmq-externals/pull/23
  
Please see #29 instead, and this PR was discarded.


> Develop rocketmq-redis-replicator component
> ---
>
> Key: ROCKETMQ-193
> URL: https://issues.apache.org/jira/browse/ROCKETMQ-193
> Project: Apache RocketMQ
>  Issue Type: Task
>Reporter: Rich Zhang
>Assignee: Rich Zhang
>Priority: Minor
> Fix For: 4.2.0-incubating
>
>
> Design:
>   Redis supplies an official replication mechanism , and slave communicates 
> to master with RESP protocol, so a natural way to design the 
> rocketmq-redis-replicator component is simulating itself as a slave, sending 
> commands to master and receiving datas from master timely, and then resending 
> to rocketmq broker.
>   If you are not familiar with redis replication mechanism, please learn this 
> section first [1]. After that, I will illustrate some key points ahead.
> 1. To make slave start from the point where it left off when it reconnects, 
> slave and master should agree on a master runId and a replication offset. 
> Slave acknowledges this offset to master periodically. In other words，slave 
> may received duplicate commands. Along with, the rocketmq-redis-replicator 
> component may send duplicate messages too. A good way to minimize the 
> duplicate time window is reducing the "ack period" to a smaller one, such as 
> 100ms.
> 2. If slave keeps offline for some time, it’s easy to use up backlog whose 
> default value is just 1M, especially for a high-traffic redis instance. 
> Unfortunatelly,if slave replication offset has already been covered in master 
> backlog, a full synchronization will have to execute, which is unacceptable 
> for rocketmq-redis-replicator component as a large number of messages will be 
> sent out intensively.
> 3. When synchronizing from master fully, master will generate a new rdb 
> file(the rdb file format [2])，and slave will receive this file,store in disk, 
> and last apply to memory. This strategy makes slave reaches a consistent 
> state with master as soon as possible, and hardly fail. For 
> rocketmq-redis-replicator component, it’s also a good way to prevent  
> synchronizing initial rdb file from failure in halfway. 
>   There already an open source project [3] which focuses on replicating redis 
> data, and provides api to handle data received [4]. The principal thoughts 
> are simulating itself as a slave , following official replication procedure, 
> communicating with master by RESP, and acking master with replication offset. 
> Base on this project to develop is a good idea, meanwhile some aspects should 
> also be enhanced and considered more robust. Here is some points: 
>   [High Available]
>Keeping the replication component's high availability is not difficult but 
> important, not only for providing an uninterruptible service. If component 
> leaves off for some time, a unacceptable full synchronization may be 
> triggered. 
>It’s also easy to reach high availability, including adopting master/slave 
> module, using zookeeper to coordinate and switch master/slave, storing data 
> onto zookeeper to keep component stateless. 
>   [Data Loss]
>Generally, data loss should be tried best to avoid. The key point is that 
> slave only acks replication offset to master after sending command to 
> rocketmq broker successfully.
>   [Data Stale]
>It also happened when slave reconnect. Consider case below:
>`time1` `time2`  `time3`
>set k=a set k=b set k=c
>If slave left off at time3, but the latest replication offset reported to 
> master is only at time1, when slave reconnected, it re-apply commands “set 
> k=b… set k=c”. In a small time window, “k” will equal the stale “b” until 
> “set k=c” command is applied. So the slave offline time shorter, the better.
>[Message Order]
>Redis uses single thread model to keep command execute in order, because 
> of its high performance. Replicating data with a single thread in slave is 
> also fine, as it is also totally memory operation. But sending all data to 
> rocketmq in a global order is a good choose? Producer should have no 
> performance issue, but consumer may not be able to consume messages in time, 
> especially redis was in a high load. 
>Hashing “KEY” to different rocketmq queue is a good strategy. Guarantee 
> the same key operation route to a unique queue, to keep partial ordered, and 
> the downstream consumer could consume messages concurrently. Of course, some 
> dependency “KEY”s may need hash to a

[jira] [Commented] (ROCKETMQ-193) Develop rocketmq-redis-replicator component

2017-09-07 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/ROCKETMQ-193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16156531#comment-16156531
 ] 

ASF GitHub Bot commented on ROCKETMQ-193:
-

Github user Zhang-Ke closed the pull request at:

https://github.com/apache/incubator-rocketmq-externals/pull/23


> Develop rocketmq-redis-replicator component
> ---
>
> Key: ROCKETMQ-193
> URL: https://issues.apache.org/jira/browse/ROCKETMQ-193
> Project: Apache RocketMQ
>  Issue Type: Task
>Reporter: Rich Zhang
>Assignee: Rich Zhang
>Priority: Minor
> Fix For: 4.2.0-incubating
>
>
> Design:
>   Redis supplies an official replication mechanism , and slave communicates 
> to master with RESP protocol, so a natural way to design the 
> rocketmq-redis-replicator component is simulating itself as a slave, sending 
> commands to master and receiving datas from master timely, and then resending 
> to rocketmq broker.
>   If you are not familiar with redis replication mechanism, please learn this 
> section first [1]. After that, I will illustrate some key points ahead.
> 1. To make slave start from the point where it left off when it reconnects, 
> slave and master should agree on a master runId and a replication offset. 
> Slave acknowledges this offset to master periodically. In other words，slave 
> may received duplicate commands. Along with, the rocketmq-redis-replicator 
> component may send duplicate messages too. A good way to minimize the 
> duplicate time window is reducing the "ack period" to a smaller one, such as 
> 100ms.
> 2. If slave keeps offline for some time, it’s easy to use up backlog whose 
> default value is just 1M, especially for a high-traffic redis instance. 
> Unfortunatelly,if slave replication offset has already been covered in master 
> backlog, a full synchronization will have to execute, which is unacceptable 
> for rocketmq-redis-replicator component as a large number of messages will be 
> sent out intensively.
> 3. When synchronizing from master fully, master will generate a new rdb 
> file(the rdb file format [2])，and slave will receive this file,store in disk, 
> and last apply to memory. This strategy makes slave reaches a consistent 
> state with master as soon as possible, and hardly fail. For 
> rocketmq-redis-replicator component, it’s also a good way to prevent  
> synchronizing initial rdb file from failure in halfway. 
>   There already an open source project [3] which focuses on replicating redis 
> data, and provides api to handle data received [4]. The principal thoughts 
> are simulating itself as a slave , following official replication procedure, 
> communicating with master by RESP, and acking master with replication offset. 
> Base on this project to develop is a good idea, meanwhile some aspects should 
> also be enhanced and considered more robust. Here is some points: 
>   [High Available]
>Keeping the replication component's high availability is not difficult but 
> important, not only for providing an uninterruptible service. If component 
> leaves off for some time, a unacceptable full synchronization may be 
> triggered. 
>It’s also easy to reach high availability, including adopting master/slave 
> module, using zookeeper to coordinate and switch master/slave, storing data 
> onto zookeeper to keep component stateless. 
>   [Data Loss]
>Generally, data loss should be tried best to avoid. The key point is that 
> slave only acks replication offset to master after sending command to 
> rocketmq broker successfully.
>   [Data Stale]
>It also happened when slave reconnect. Consider case below:
>`time1` `time2`  `time3`
>set k=a set k=b set k=c
>If slave left off at time3, but the latest replication offset reported to 
> master is only at time1, when slave reconnected, it re-apply commands “set 
> k=b… set k=c”. In a small time window, “k” will equal the stale “b” until 
> “set k=c” command is applied. So the slave offline time shorter, the better.
>[Message Order]
>Redis uses single thread model to keep command execute in order, because 
> of its high performance. Replicating data with a single thread in slave is 
> also fine, as it is also totally memory operation. But sending all data to 
> rocketmq in a global order is a good choose? Producer should have no 
> performance issue, but consumer may not be able to consume messages in time, 
> especially redis was in a high load. 
>Hashing “KEY” to different rocketmq queue is a good strategy. Guarantee 
> the same key operation route to a unique queue, to keep partial ordered, and 
> the downstream consumer could consume messages concurrently. Of course, some 
> dependency “KEY”s may need hash to a unique queue too. We should supply 
> configuration or

[jira] [Commented] (ROCKETMQ-271) add diagnosis tools

2017-09-06 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/ROCKETMQ-271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16156512#comment-16156512
 ] 

ASF GitHub Bot commented on ROCKETMQ-271:
-

Github user coveralls commented on the issue:

https://github.com/apache/incubator-rocketmq/pull/156
  

[![Coverage 
Status](https://coveralls.io/builds/13161737/badge)](https://coveralls.io/builds/13161737)

Coverage decreased (-1.6%) to 37.136% when pulling 
**8a5b0340d8d2bd1ea4c5300e86028273d97535ef on fuyou001:ROCKETMQ-271** into 
**368e7c86a0b06099f336c81672112dcb5143cf9e on apache:develop**.



> add diagnosis tools
> ---
>
> Key: ROCKETMQ-271
> URL: https://issues.apache.org/jira/browse/ROCKETMQ-271
> Project: Apache RocketMQ
>  Issue Type: New Feature
>  Components: rocketmq-broker, rocketmq-client, rocketmq-commons, 
> rocketmq-remoting
>Reporter: yubaofu
>Assignee: yukon
>Priority: Minor
>   Original Estimate: 96h
>  Remaining Estimate: 96h
>
> when prodcurer send message timeout,how to find the root cause,it's difficult.
> so add the new diagnosis tools that record message lifetime ,eg message 
> create time,prodcurer send message to broker time,broker recive message 
> time,and send response to client time.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (ROCKETMQ-271) add diagnosis tools

2017-09-06 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/ROCKETMQ-271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16156358#comment-16156358
 ] 

ASF GitHub Bot commented on ROCKETMQ-271:
-

Github user coveralls commented on the issue:

https://github.com/apache/incubator-rocketmq/pull/156
  

[![Coverage 
Status](https://coveralls.io/builds/13160642/badge)](https://coveralls.io/builds/13160642)

Coverage decreased (-1.6%) to 37.152% when pulling 
**934d203bd724a4f7a2cc64af6c300ba8a35887b6 on fuyou001:ROCKETMQ-271** into 
**368e7c86a0b06099f336c81672112dcb5143cf9e on apache:develop**.



> add diagnosis tools
> ---
>
> Key: ROCKETMQ-271
> URL: https://issues.apache.org/jira/browse/ROCKETMQ-271
> Project: Apache RocketMQ
>  Issue Type: New Feature
>  Components: rocketmq-broker, rocketmq-client, rocketmq-commons, 
> rocketmq-remoting
>Reporter: yubaofu
>Assignee: yukon
>Priority: Minor
>   Original Estimate: 96h
>  Remaining Estimate: 96h
>
> when prodcurer send message timeout,how to find the root cause,it's difficult.
> so add the new diagnosis tools that record message lifetime ,eg message 
> create time,prodcurer send message to broker time,broker recive message 
> time,and send response to client time.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (ROCKETMQ-271) add diagnosis tools

2017-09-06 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/ROCKETMQ-271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16156354#comment-16156354
 ] 

ASF GitHub Bot commented on ROCKETMQ-271:
-

Github user coveralls commented on the issue:

https://github.com/apache/incubator-rocketmq/pull/156
  

[![Coverage 
Status](https://coveralls.io/builds/13160524/badge)](https://coveralls.io/builds/13160524)

Coverage decreased (-1.6%) to 37.152% when pulling 
**9f6ead962c510c69b5a29480e040a84d751aebf8 on fuyou001:ROCKETMQ-271** into 
**368e7c86a0b06099f336c81672112dcb5143cf9e on apache:develop**.



> add diagnosis tools
> ---
>
> Key: ROCKETMQ-271
> URL: https://issues.apache.org/jira/browse/ROCKETMQ-271
> Project: Apache RocketMQ
>  Issue Type: New Feature
>  Components: rocketmq-broker, rocketmq-client, rocketmq-commons, 
> rocketmq-remoting
>Reporter: yubaofu
>Assignee: yukon
>Priority: Minor
>   Original Estimate: 96h
>  Remaining Estimate: 96h
>
> when prodcurer send message timeout,how to find the root cause,it's difficult.
> so add the new diagnosis tools that record message lifetime ,eg message 
> create time,prodcurer send message to broker time,broker recive message 
> time,and send response to client time.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (ROCKETMQ-282) RocketMQ-GO-Client

2017-09-05 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/ROCKETMQ-282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16153742#comment-16153742
 ] 

ASF GitHub Bot commented on ROCKETMQ-282:
-

Github user zhouxinyu commented on the issue:

https://github.com/apache/incubator-rocketmq-externals/pull/28
  
cool~ 
Thanks @StyleTang 


> RocketMQ-GO-Client
> --
>
> Key: ROCKETMQ-282
> URL: https://issues.apache.org/jira/browse/ROCKETMQ-282
> Project: Apache RocketMQ
>  Issue Type: Task
>  Components: rocketmq-client
>Reporter: Jie.Tang
>Assignee: Jie.Tang
>
> RocketMQ-GO-Client



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (ROCKETMQ-265) when os crash for some reasons, the broker consume queue’s data maybe repeat, consumer can’t pull the latest message, cause message lag

2017-09-05 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/ROCKETMQ-265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16153501#comment-16153501
 ] 

ASF GitHub Bot commented on ROCKETMQ-265:
-

Github user coveralls commented on the issue:

https://github.com/apache/incubator-rocketmq/pull/146
  

[![Coverage 
Status](https://coveralls.io/builds/13127783/badge)](https://coveralls.io/builds/13127783)

Coverage increased (+0.07%) to 38.77% when pulling 
**ecb01f7650e85bfbcdc7aa148711bf075aea958b on fuyou001:ROCKETMQ-265** into 
**6a97d2884fe8ea23de8f231c8647b99ea5be1811 on apache:develop**.



> when os crash for some reasons, the broker consume queue’s data maybe repeat, 
> consumer can’t pull the latest message, cause message lag
> ---
>
> Key: ROCKETMQ-265
> URL: https://issues.apache.org/jira/browse/ROCKETMQ-265
> Project: Apache RocketMQ
>  Issue Type: Bug
>  Components: rocketmq-store
>Affects Versions: 4.0.0-incubating, 4.1.0-incubating
>Reporter: yubaofu
>Assignee: yukon
>Priority: Critical
>  Labels: bug
> Fix For: 4.2.0-incubating
>
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
> when os crash for some reasons, the broker consume queue’s data maybe repeat, 
> consumer can’t pull the latest message, cause message lag



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (ROCKETMQ-265) when os crash for some reasons, the broker consume queue’s data maybe repeat, consumer can’t pull the latest message, cause message lag

2017-09-05 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/ROCKETMQ-265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16153408#comment-16153408
 ] 

ASF GitHub Bot commented on ROCKETMQ-265:
-

Github user coveralls commented on the issue:

https://github.com/apache/incubator-rocketmq/pull/146
  

[![Coverage 
Status](https://coveralls.io/builds/13126602/badge)](https://coveralls.io/builds/13126602)

Coverage increased (+0.08%) to 38.781% when pulling 
**7320a82136061ac6f1ca565ba98f047d05c18d5a on fuyou001:ROCKETMQ-265** into 
**6a97d2884fe8ea23de8f231c8647b99ea5be1811 on apache:develop**.



> when os crash for some reasons, the broker consume queue’s data maybe repeat, 
> consumer can’t pull the latest message, cause message lag
> ---
>
> Key: ROCKETMQ-265
> URL: https://issues.apache.org/jira/browse/ROCKETMQ-265
> Project: Apache RocketMQ
>  Issue Type: Bug
>  Components: rocketmq-store
>Affects Versions: 4.0.0-incubating, 4.1.0-incubating
>Reporter: yubaofu
>Assignee: yukon
>Priority: Critical
>  Labels: bug
> Fix For: 4.2.0-incubating
>
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
> when os crash for some reasons, the broker consume queue’s data maybe repeat, 
> consumer can’t pull the latest message, cause message lag



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (ROCKETMQ-281) add check for preventing repeat start mq

2017-09-04 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/ROCKETMQ-281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16153057#comment-16153057
 ] 

ASF GitHub Bot commented on ROCKETMQ-281:
-

Github user coveralls commented on the issue:

https://github.com/apache/incubator-rocketmq/pull/158
  

[![Coverage 
Status](https://coveralls.io/builds/13122639/badge)](https://coveralls.io/builds/13122639)

Coverage decreased (-0.1%) to 38.731% when pulling 
**331775dccb458dbb52fa54942a28c4dc7b98a8a1 on fuyou001:ROCKETMQ-281** into 
**254d432496c424717d45b8dc9e44ae3bd78ab466 on apache:develop**.



> add check for preventing repeat start mq
> 
>
> Key: ROCKETMQ-281
> URL: https://issues.apache.org/jira/browse/ROCKETMQ-281
> Project: Apache RocketMQ
>  Issue Type: Improvement
>  Components: rocketmq-broker
>Affects Versions: 4.0.0-incubating, 4.1.0-incubating
>Reporter: yubaofu
>Assignee: yukon
> Fix For: 4.2.0-incubating, 4.3.0-incubating
>
>
> add check for preventing repeat start mq



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (ROCKETMQ-286) Add a switch to decide whether to wake up commitLogService when put message

2017-09-04 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/ROCKETMQ-286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16152223#comment-16152223
 ] 

ASF GitHub Bot commented on ROCKETMQ-286:
-

Github user coveralls commented on the issue:

https://github.com/apache/incubator-rocketmq/pull/162
  

[![Coverage 
Status](https://coveralls.io/builds/13111287/badge)](https://coveralls.io/builds/13111287)

Coverage decreased (-0.2%) to 38.636% when pulling 
**6bc0b58c04339c5f24eb306c60b25c6dc8ec626d on vsair:ROCKETMQ-286** into 
**254d432496c424717d45b8dc9e44ae3bd78ab466 on apache:develop**.



> Add a switch to decide whether to wake up commitLogService when put message
> ---
>
> Key: ROCKETMQ-286
> URL: https://issues.apache.org/jira/browse/ROCKETMQ-286
> Project: Apache RocketMQ
>  Issue Type: Improvement
>  Components: rocketmq-store
>Affects Versions: 4.0.0-incubating, 4.1.0-incubating
>Reporter: Eric Liu
>Assignee: Eric Liu
> Fix For: 4.2.0-incubating
>
>
> Now, it will wake up commitLogService after putting message. This may have 
> impact on disk.
> Add a switch to control this feature.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (ROCKETMQ-286) Add a switch to decide whether to wake up commitLogService when put message

2017-09-04 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/ROCKETMQ-286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16152195#comment-16152195
 ] 

ASF GitHub Bot commented on ROCKETMQ-286:
-

Github user vsair commented on the issue:

https://github.com/apache/incubator-rocketmq/pull/162
  
@zhouxinyu  @lizhanhui  @vongosling   pls have a review.


> Add a switch to decide whether to wake up commitLogService when put message
> ---
>
> Key: ROCKETMQ-286
> URL: https://issues.apache.org/jira/browse/ROCKETMQ-286
> Project: Apache RocketMQ
>  Issue Type: Improvement
>  Components: rocketmq-store
>Affects Versions: 4.0.0-incubating, 4.1.0-incubating
>Reporter: Eric Liu
>Assignee: Eric Liu
> Fix For: 4.2.0-incubating
>
>
> Now, it will wake up commitLogService after putting message. This may have 
> impact on disk.
> Add a switch to control this feature.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (ROCKETMQ-286) Add a switch to decide whether to wake up commitLogService when put message

2017-09-04 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/ROCKETMQ-286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16152194#comment-16152194
 ] 

ASF GitHub Bot commented on ROCKETMQ-286:
-

GitHub user vsair opened a pull request:

https://github.com/apache/incubator-rocketmq/pull/162

[ROCKETMQ-286] Add a switch to decide whether to wake up commitLogService 
when put message

https://issues.apache.org/jira/browse/ROCKETMQ-286

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/vsair/incubator-rocketmq ROCKETMQ-286

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-rocketmq/pull/162.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #162


commit 6bc0b58c04339c5f24eb306c60b25c6dc8ec626d
Author: vsair 
Date:   2017-09-04T07:09:54Z

[ROCKETMQ-286] Add a switch to decide whether to wake up commitLogService 
when put message




> Add a switch to decide whether to wake up commitLogService when put message
> ---
>
> Key: ROCKETMQ-286
> URL: https://issues.apache.org/jira/browse/ROCKETMQ-286
> Project: Apache RocketMQ
>  Issue Type: Improvement
>  Components: rocketmq-store
>Affects Versions: 4.0.0-incubating, 4.1.0-incubating
>Reporter: Eric Liu
>Assignee: Eric Liu
> Fix For: 4.2.0-incubating
>
>
> Now, it will wake up commitLogService after putting message. This may have 
> impact on disk.
> Add a switch to control this feature.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (ROCKETMQ-268) add rocketmq client4cpp

2017-09-03 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/ROCKETMQ-268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16152148#comment-16152148
 ] 

ASF GitHub Bot commented on ROCKETMQ-268:
-

Github user vongosling commented on the issue:

https://github.com/apache/incubator-rocketmq-externals/pull/27
  
It is a good practice for apache way :-)


> add rocketmq client4cpp
> ---
>
> Key: ROCKETMQ-268
> URL: https://issues.apache.org/jira/browse/ROCKETMQ-268
> Project: Apache RocketMQ
>  Issue Type: New Feature
> Environment: Linux system
>Reporter: wang qiwei
>Assignee: vongosling
>Priority: Minor
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (ROCKETMQ-268) add rocketmq client4cpp

2017-09-03 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/ROCKETMQ-268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16152126#comment-16152126
 ] 

ASF GitHub Bot commented on ROCKETMQ-268:
-

Github user lizhanhui commented on the issue:

https://github.com/apache/incubator-rocketmq-externals/pull/27
  
This Pull Request has been merged and it's safe to close this PR. 
@vincentWangKB 


> add rocketmq client4cpp
> ---
>
> Key: ROCKETMQ-268
> URL: https://issues.apache.org/jira/browse/ROCKETMQ-268
> Project: Apache RocketMQ
>  Issue Type: New Feature
> Environment: Linux system
>Reporter: wang qiwei
>Assignee: vongosling
>Priority: Minor
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (ROCKETMQ-268) add rocketmq client4cpp

2017-09-03 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/ROCKETMQ-268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16152115#comment-16152115
 ] 

ASF GitHub Bot commented on ROCKETMQ-268:
-

Github user lizhanhui commented on the issue:

https://github.com/apache/incubator-rocketmq-externals/pull/27
  
This patch looks good enough to be merged.  Thanks @vincentWangKB 


> add rocketmq client4cpp
> ---
>
> Key: ROCKETMQ-268
> URL: https://issues.apache.org/jira/browse/ROCKETMQ-268
> Project: Apache RocketMQ
>  Issue Type: New Feature
> Environment: Linux system
>Reporter: wang qiwei
>Assignee: vongosling
>Priority: Minor
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (ROCKETMQ-285) file test error when make link

2017-09-03 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/ROCKETMQ-285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16152074#comment-16152074
 ] 

ASF GitHub Bot commented on ROCKETMQ-285:
-

GitHub user zwillim reopened a pull request:

https://github.com/apache/incubator-rocketmq/pull/154

file test error when make link

line 62.
cannot test ${HOME}/tmpfs while making link to ./tmpfs
change "ln -s /dev/shm tmpfs" to "ln -s /dev/shm ${HOME}/tmpfs"

https://issues.apache.org/jira/browse/ROCKETMQ-285


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/zwillim/incubator-rocketmq patch-1

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-rocketmq/pull/154.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #154


commit 00d0fb673de8d8f5debe74a24179d6aa212d4ee4
Author: willim.z 
Date:   2017-08-25T02:35:57Z

logistic error

line 62.
cannot test ${HOME}/tmpfs while making link to ./tmpfs
change "ln -s /dev/shm tmpfs" to "ln -s /dev/shm ${HOME}/tmpfs"




> file test error when make link
> --
>
> Key: ROCKETMQ-285
> URL: https://issues.apache.org/jira/browse/ROCKETMQ-285
> Project: Apache RocketMQ
>  Issue Type: Bug
>  Components: build
>Affects Versions: 4.0.0-incubating, 4.1.0-incubating, 4.2.0-incubating, 
> 4.3.0-incubating
>Reporter: willim.z
>Assignee: Stevens Chew
>
> error file path: distribution/bin/os.sh.
> line 62.
> cannot test ${HOME}/tmpfs while making link to ./tmpfs
> change "ln -s /dev/shm tmpfs" to "ln -s /dev/shm ${HOME}/tmpfs"



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (ROCKETMQ-285) file test error when make link

2017-09-03 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/ROCKETMQ-285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16152073#comment-16152073
 ] 

ASF GitHub Bot commented on ROCKETMQ-285:
-

Github user zwillim closed the pull request at:

https://github.com/apache/incubator-rocketmq/pull/154


> file test error when make link
> --
>
> Key: ROCKETMQ-285
> URL: https://issues.apache.org/jira/browse/ROCKETMQ-285
> Project: Apache RocketMQ
>  Issue Type: Bug
>  Components: build
>Affects Versions: 4.0.0-incubating, 4.1.0-incubating, 4.2.0-incubating, 
> 4.3.0-incubating
>Reporter: willim.z
>Assignee: Stevens Chew
>
> error file path: distribution/bin/os.sh.
> line 62.
> cannot test ${HOME}/tmpfs while making link to ./tmpfs
> change "ln -s /dev/shm tmpfs" to "ln -s /dev/shm ${HOME}/tmpfs"



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (ROCKETMQ-279) add commit log data and consume queue data logic consistent check when start mq

2017-09-01 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/ROCKETMQ-279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16150277#comment-16150277
 ] 

ASF GitHub Bot commented on ROCKETMQ-279:
-

Github user fuyou001 closed the pull request at:

https://github.com/apache/incubator-rocketmq/pull/159


> add commit log data and consume queue data logic consistent check when start 
> mq
> ---
>
> Key: ROCKETMQ-279
> URL: https://issues.apache.org/jira/browse/ROCKETMQ-279
> Project: Apache RocketMQ
>  Issue Type: Bug
>  Components: rocketmq-store
>Affects Versions: 4.0.0-incubating, 4.1.0-incubating
>Reporter: yubaofu
>Assignee: yukon
> Fix For: 4.2.0-incubating
>
>
> commit log and consume data may logic inconsistent ,
> because  [ROCKETMQ-265|https://issues.apache.org/jira/browse/ROCKETMQ-265],
> so we add  data consistent check,when mq start.
> if check failed, start failed.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (ROCKETMQ-279) add commit log data and consume queue data logic consistent check when start mq

2017-08-31 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/ROCKETMQ-279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16150020#comment-16150020
 ] 

ASF GitHub Bot commented on ROCKETMQ-279:
-

Github user coveralls commented on the issue:

https://github.com/apache/incubator-rocketmq/pull/159
  

[![Coverage 
Status](https://coveralls.io/builds/13082255/badge)](https://coveralls.io/builds/13082255)

Coverage decreased (-0.2%) to 38.511% when pulling 
**4b98ecac5ae00933cab648dc7b3446ea3c88e1bd on fuyou001:ROCKETMQ-2792** into 
**845830865fc37d0364a19cbd89ceaf8a30b37e1c on apache:develop**.



> add commit log data and consume queue data logic consistent check when start 
> mq
> ---
>
> Key: ROCKETMQ-279
> URL: https://issues.apache.org/jira/browse/ROCKETMQ-279
> Project: Apache RocketMQ
>  Issue Type: Bug
>  Components: rocketmq-store
>Affects Versions: 4.0.0-incubating, 4.1.0-incubating
>Reporter: yubaofu
>Assignee: yukon
> Fix For: 4.2.0-incubating
>
>
> commit log and consume data may logic inconsistent ,
> because  [ROCKETMQ-265|https://issues.apache.org/jira/browse/ROCKETMQ-265],
> so we add  data consistent check,when mq start.
> if check failed, start failed.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (ROCKETMQ-279) add commit log data and consume queue data logic consistent check when start mq

2017-08-31 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/ROCKETMQ-279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1614#comment-1614
 ] 

ASF GitHub Bot commented on ROCKETMQ-279:
-

Github user fuyou001 commented on a diff in the pull request:

https://github.com/apache/incubator-rocketmq/pull/159#discussion_r136494489
  
--- Diff: store/src/main/java/org/apache/rocketmq/store/ConsumeQueue.java 
---
@@ -84,6 +84,58 @@ public boolean load() {
 return result;
 }
 
+public void checkCommitLogAndConsumeQueueConsistent() {
+
+String queueDir = this.storePath
++ File.separator + topic
++ File.separator + queueId + File.separator;
+
+long lastRecordOffset = Math.max(getMaxOffsetInQueue() - 1, 0);
+
+if (lastRecordOffset == 0) {
+return;
+}
+
+SelectMappedBufferResult consumeQueueBuffer = 
getIndexBuffer(lastRecordOffset);
+
+if (consumeQueueBuffer == null) {
+String errorMsg = String.format("Can't find consume queue 
mappedBuffer by offset %s,queueDir %s", lastRecordOffset, queueDir);
+log.error(errorMsg);
+throw new IllegalStateException(errorMsg);
+}
+
+//position latest consume queue offset,
+long offsetPy = consumeQueueBuffer.getByteBuffer().getLong();
+int sizePy = consumeQueueBuffer.getByteBuffer().getInt();
+consumeQueueBuffer.getByteBuffer().getLong();
--- End diff --

fixed


> add commit log data and consume queue data logic consistent check when start 
> mq
> ---
>
> Key: ROCKETMQ-279
> URL: https://issues.apache.org/jira/browse/ROCKETMQ-279
> Project: Apache RocketMQ
>  Issue Type: Bug
>  Components: rocketmq-store
>Affects Versions: 4.0.0-incubating, 4.1.0-incubating
>Reporter: yubaofu
>Assignee: yukon
> Fix For: 4.2.0-incubating
>
>
> commit log and consume data may logic inconsistent ,
> because  [ROCKETMQ-265|https://issues.apache.org/jira/browse/ROCKETMQ-265],
> so we add  data consistent check,when mq start.
> if check failed, start failed.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (ROCKETMQ-279) add commit log data and consume queue data logic consistent check when start mq

2017-08-31 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/ROCKETMQ-279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16149969#comment-16149969
 ] 

ASF GitHub Bot commented on ROCKETMQ-279:
-

Github user zhouxinyu commented on a diff in the pull request:

https://github.com/apache/incubator-rocketmq/pull/159#discussion_r136491053
  
--- Diff: store/src/main/java/org/apache/rocketmq/store/ConsumeQueue.java 
---
@@ -84,6 +84,58 @@ public boolean load() {
 return result;
 }
 
+public void checkCommitLogAndConsumeQueueConsistent() {
+
+String queueDir = this.storePath
++ File.separator + topic
++ File.separator + queueId + File.separator;
+
+long lastRecordOffset = Math.max(getMaxOffsetInQueue() - 1, 0);
+
+if (lastRecordOffset == 0) {
+return;
+}
+
+SelectMappedBufferResult consumeQueueBuffer = 
getIndexBuffer(lastRecordOffset);
+
+if (consumeQueueBuffer == null) {
+String errorMsg = String.format("Can't find consume queue 
mappedBuffer by offset %s,queueDir %s", lastRecordOffset, queueDir);
+log.error(errorMsg);
+throw new IllegalStateException(errorMsg);
+}
+
+//position latest consume queue offset,
+long offsetPy = consumeQueueBuffer.getByteBuffer().getLong();
+int sizePy = consumeQueueBuffer.getByteBuffer().getInt();
+consumeQueueBuffer.getByteBuffer().getLong();
--- End diff --

Do we need the side effect of `getLong`?


> add commit log data and consume queue data logic consistent check when start 
> mq
> ---
>
> Key: ROCKETMQ-279
> URL: https://issues.apache.org/jira/browse/ROCKETMQ-279
> Project: Apache RocketMQ
>  Issue Type: Bug
>  Components: rocketmq-store
>Affects Versions: 4.0.0-incubating, 4.1.0-incubating
>Reporter: yubaofu
>Assignee: yukon
> Fix For: 4.2.0-incubating
>
>
> commit log and consume data may logic inconsistent ,
> because  [ROCKETMQ-265|https://issues.apache.org/jira/browse/ROCKETMQ-265],
> so we add  data consistent check,when mq start.
> if check failed, start failed.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (ROCKETMQ-279) add commit log data and consume queue data logic consistent check when start mq

2017-08-31 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/ROCKETMQ-279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16149966#comment-16149966
 ] 

ASF GitHub Bot commented on ROCKETMQ-279:
-

Github user zhouxinyu commented on a diff in the pull request:

https://github.com/apache/incubator-rocketmq/pull/159#discussion_r136490566
  
--- Diff: store/src/main/java/org/apache/rocketmq/store/ConsumeQueue.java 
---
@@ -84,6 +84,58 @@ public boolean load() {
 return result;
 }
 
+public void checkCommitLogAndConsumeQueueConsistent() {
+
+String queueDir = this.storePath
--- End diff --

There is a helper to get queue dir: 
`StorePathConfigHelper#getStorePathConsumeQueue`


> add commit log data and consume queue data logic consistent check when start 
> mq
> ---
>
> Key: ROCKETMQ-279
> URL: https://issues.apache.org/jira/browse/ROCKETMQ-279
> Project: Apache RocketMQ
>  Issue Type: Bug
>  Components: rocketmq-store
>Affects Versions: 4.0.0-incubating, 4.1.0-incubating
>Reporter: yubaofu
>Assignee: yukon
> Fix For: 4.2.0-incubating
>
>
> commit log and consume data may logic inconsistent ,
> because  [ROCKETMQ-265|https://issues.apache.org/jira/browse/ROCKETMQ-265],
> so we add  data consistent check,when mq start.
> if check failed, start failed.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (ROCKETMQ-268) add rocketmq client4cpp

2017-08-29 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/ROCKETMQ-268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16146645#comment-16146645
 ] 

ASF GitHub Bot commented on ROCKETMQ-268:
-

Github user lizhanhui commented on the issue:

https://github.com/apache/incubator-rocketmq-externals/pull/27
  
Though there are still quite a few unused files, I am fine to merge this PR 
first and fix issues later on.


> add rocketmq client4cpp
> ---
>
> Key: ROCKETMQ-268
> URL: https://issues.apache.org/jira/browse/ROCKETMQ-268
> Project: Apache RocketMQ
>  Issue Type: New Feature
> Environment: Linux system
>Reporter: wang qiwei
>Assignee: vongosling
>Priority: Minor
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (ROCKETMQ-284) ExpressionMessageFilter will pass some message

2017-08-29 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/ROCKETMQ-284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16145494#comment-16145494
 ] 

ASF GitHub Bot commented on ROCKETMQ-284:
-

Github user vsair closed the pull request at:

https://github.com/apache/incubator-rocketmq/pull/160


> ExpressionMessageFilter will pass some message
> --
>
> Key: ROCKETMQ-284
> URL: https://issues.apache.org/jira/browse/ROCKETMQ-284
> Project: Apache RocketMQ
>  Issue Type: Bug
>  Components: rocketmq-broker
>Affects Versions: 4.1.0-incubating
>Reporter: Eric Liu
>Assignee: Eric Liu
> Fix For: 4.2.0-incubating
>
>   Original Estimate: 1h
>  Remaining Estimate: 1h
>
> ExpressionMessageFilter will pass some message with tag's hash code less than 
> 0.
> This is not correct. After sql filter feature added, the tag's hash code in 
> consume queue may be replaced by the address of consume queue ext. And, 
> differences between them are:
> 1. Tag's hash code is an integer, values is between Integer.MIN and 
> Integer.MAX.
> 2. Address of consume queue ext is a long, values is between Long.MIN and 
> Integer.MIN - 1.
> So, actually,  they are not conflict.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (ROCKETMQ-282) RocketMQ-GO-Client

2017-08-29 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/ROCKETMQ-282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16145362#comment-16145362
 ] 

ASF GitHub Bot commented on ROCKETMQ-282:
-

GitHub user StyleTang opened a pull request:

https://github.com/apache/incubator-rocketmq-externals/pull/28

[ROCKETMQ-282]RocketMQ-GO-Client

[ROCKETMQ-282 
RocketMQ-GO-Client](https://issues.apache.org/jira/browse/ROCKETMQ-282)

[Document](https://github.com/StyleTang/incubator-rocketmq-externals/blob/master/rocketmq-go/README.md)
Please review.
Your advice will help us improve it.
:-)

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/StyleTang/incubator-rocketmq-externals master

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-rocketmq-externals/pull/28.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #28


commit 2ebd1a02fc20089856e0157339d791c8c195c383
Author: tangjie 
Date:   2017-08-09T13:45:20Z

use localhost test

commit 5ae0244ded1934860f2ed4906519bacbcccfe76e
Author: tangjie 
Date:   2017-08-09T15:22:15Z

user only use
"github.com/apache/incubator-rocketmq-externals/rocketmq-go/api"
"github.com/apache/incubator-rocketmq-externals/rocketmq-go/api/model"

commit b115a9c3054079786a765014699bf8c604039148
Author: tangjie 
Date:   2017-08-09T15:29:22Z

consumer and producer can getConfig

commit 208f2a2be0f889ec281381b0445df6b36f1f5e45
Author: tangjie 
Date:   2017-08-10T12:52:28Z

add .travis.yml

commit 3e8fc27ed985462762bf5938c174efe68c4b6fa8
Author: tangjie 
Date:   2017-08-10T12:59:29Z

.travis.yml jdk

commit c3a99dbde1ec53831d281902220610c0a0f645c4
Author: tangjie 
Date:   2017-08-10T13:03:16Z

.travis.yml jdk

commit 903a3399368da10951926623389e1a0352f0ecf8
Author: tangjie 
Date:   2017-08-10T13:20:20Z

.travis.yml add go test

commit d8ef4cba93f0c01b0af95bed493121c7f4b4a7a4
Author: tangjie 
Date:   2017-08-10T13:22:27Z

.travis.yml remove java

commit 6ec7976dfe004d7f170990a1c7f487440767fa8a
Author: tangjie 
Date:   2017-08-10T13:54:31Z

.travis.yml test java

commit 6cec9d4f4b17fa0829d48ac9f677a9d59c79831a
Author: tangjie 
Date:   2017-08-10T14:00:40Z

.travis.yml test java

commit 82f7c87ce14f4c04abd9314720a967a8b3e96f1a
Author: tangjie 
Date:   2017-08-10T14:09:21Z

openjdk7

commit 713950086700f7c05e3a269f1fed072ea0eb8efd
Author: tangjie 
Date:   2017-08-10T14:12:48Z

  - os: linux
env: CUSTOM_JDK="oraclejdk7"

commit 7c4d36a8bcdf50543f8f3fcfbf20ed2e0bfd11ac
Author: tangjie 
Date:   2017-08-10T14:22:56Z

travis language: java

commit 9c43201ed790320e0707dcb6f3fe953dff3794a9
Author: tangjie 
Date:   2017-08-10T14:36:27Z

README.md

commit 119b6bc5de7ef0aa9e67cba9a533971726c9824d
Author: StyleTang 
Date:   2017-08-10T15:47:52Z

update go client code of api

commit 664caa2a14573f615bbfaf454cf7e2a59b304525
Author: StyleTang 
Date:   2017-08-12T06:36:58Z

CurrentTimeMillisInt64

commit 34760b647dffb3fa427ac898d475e8b814596e10
Author: StyleTang 
Date:   2017-08-12T07:05:49Z

go api config

commit 3d01651a2e571a00a05ee780b144d2985ff4b4d7
Author: StyleTang 
Date:   2017-08-12T10:00:03Z

update rocketmq's example

commit 5bc4e16dbef3ad496e82eae68e995ceccc61
Author: StyleTang 
Date:   2017-08-12T10:05:58Z

update rocketmq's example

commit 5301148c82e07a84742cd8649e5ce4e620f6aeec
Author: Rock1994 <806714...@qq.com>
Date:   2017-08-12T13:54:32Z

package util  unit test

commit ee9d4207d7eb2661287fb735ade013ea3cf354da
Author: StyleTang <396858...@qq.com>
Date:   2017-08-12T14:14:39Z

Merge pull request #1 from Rock1994/go-client-refactor

package util  unit test

commit 6165d4ea36c9c1c0992db7a9a111739b985dee55
Author: StyleTang 
Date:   2017-08-12T15:33:48Z

make SerializeType can be config

commit 925ab25aefb798263f4d209421159be36a091b3b
Author: StyleTang 
Date:   2017-08-13T12:39:58Z

update api

commit 275ae2ef2025adc18c0a850bb194eda8c56f313a
Author: StyleTang 
Date:   2017-08-13T14:05:40Z

refactor code

commit 5863f41f476020e47115526ea0d4289e2d98141b
Author: StyleTang 
Date:   2017-08-13T15:15:11Z

add test case:
retry message

[jira] [Commented] (ROCKETMQ-284) ExpressionMessageFilter will pass some message

2017-08-29 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/ROCKETMQ-284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16145195#comment-16145195
 ] 

ASF GitHub Bot commented on ROCKETMQ-284:
-

Github user vsair commented on a diff in the pull request:

https://github.com/apache/incubator-rocketmq/pull/160#discussion_r135773045
  
--- Diff: 
broker/src/test/java/org/apache/rocketmq/broker/filter/MessageStoreWithFilterTest.java
 ---
@@ -79,24 +79,17 @@
 try {
 StoreHost = new InetSocketAddress(InetAddress.getLocalHost(), 
8123);
 } catch (UnknownHostException e) {
-e.printStackTrace();
 }
 try {
 BornHost = new 
InetSocketAddress(InetAddress.getByName("127.0.0.1"), 0);
 } catch (UnknownHostException e) {
--- End diff --

Here,  don't  care about this exception, so not hide it .


> ExpressionMessageFilter will pass some message
> --
>
> Key: ROCKETMQ-284
> URL: https://issues.apache.org/jira/browse/ROCKETMQ-284
> Project: Apache RocketMQ
>  Issue Type: Bug
>  Components: rocketmq-broker
>Affects Versions: 4.1.0-incubating
>Reporter: Eric Liu
>Assignee: Eric Liu
> Fix For: 4.2.0-incubating
>
>   Original Estimate: 1h
>  Remaining Estimate: 1h
>
> ExpressionMessageFilter will pass some message with tag's hash code less than 
> 0.
> This is not correct. After sql filter feature added, the tag's hash code in 
> consume queue may be replaced by the address of consume queue ext. And, 
> differences between them are:
> 1. Tag's hash code is an integer, values is between Integer.MIN and 
> Integer.MAX.
> 2. Address of consume queue ext is a long, values is between Long.MIN and 
> Integer.MIN - 1.
> So, actually,  they are not conflict.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (ROCKETMQ-284) ExpressionMessageFilter will pass some message

2017-08-29 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/ROCKETMQ-284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16145173#comment-16145173
 ] 

ASF GitHub Bot commented on ROCKETMQ-284:
-

Github user vongosling commented on a diff in the pull request:

https://github.com/apache/incubator-rocketmq/pull/160#discussion_r135768140
  
--- Diff: 
broker/src/test/java/org/apache/rocketmq/broker/filter/MessageStoreWithFilterTest.java
 ---
@@ -79,24 +79,17 @@
 try {
 StoreHost = new InetSocketAddress(InetAddress.getLocalHost(), 
8123);
 } catch (UnknownHostException e) {
-e.printStackTrace();
 }
 try {
 BornHost = new 
InetSocketAddress(InetAddress.getByName("127.0.0.1"), 0);
 } catch (UnknownHostException e) {
-e.printStackTrace();
 }
 }
 
 @Before
-public void init() {
+public void init() throws Exception {
 filterManager = ConsumerFilterManagerTest.gen(topicCount, 
msgPerTopic);
-try {
-master = gen(filterManager);
-} catch (Exception e) {
-e.printStackTrace();
-assertThat(true).isFalse();
-}
--- End diff --

Excellent !


> ExpressionMessageFilter will pass some message
> --
>
> Key: ROCKETMQ-284
> URL: https://issues.apache.org/jira/browse/ROCKETMQ-284
> Project: Apache RocketMQ
>  Issue Type: Bug
>  Components: rocketmq-broker
>Affects Versions: 4.1.0-incubating
>Reporter: Eric Liu
>Assignee: Eric Liu
> Fix For: 4.2.0-incubating
>
>   Original Estimate: 1h
>  Remaining Estimate: 1h
>
> ExpressionMessageFilter will pass some message with tag's hash code less than 
> 0.
> This is not correct. After sql filter feature added, the tag's hash code in 
> consume queue may be replaced by the address of consume queue ext. And, 
> differences between them are:
> 1. Tag's hash code is an integer, values is between Integer.MIN and 
> Integer.MAX.
> 2. Address of consume queue ext is a long, values is between Long.MIN and 
> Integer.MIN - 1.
> So, actually,  they are not conflict.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (ROCKETMQ-284) ExpressionMessageFilter will pass some message

2017-08-29 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/ROCKETMQ-284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16145172#comment-16145172
 ] 

ASF GitHub Bot commented on ROCKETMQ-284:
-

Github user vongosling commented on a diff in the pull request:

https://github.com/apache/incubator-rocketmq/pull/160#discussion_r135768040
  
--- Diff: 
broker/src/test/java/org/apache/rocketmq/broker/filter/MessageStoreWithFilterTest.java
 ---
@@ -79,24 +79,17 @@
 try {
 StoreHost = new InetSocketAddress(InetAddress.getLocalHost(), 
8123);
 } catch (UnknownHostException e) {
-e.printStackTrace();
 }
 try {
 BornHost = new 
InetSocketAddress(InetAddress.getByName("127.0.0.1"), 0);
 } catch (UnknownHostException e) {
--- End diff --

no need to delete  catch part ?


> ExpressionMessageFilter will pass some message
> --
>
> Key: ROCKETMQ-284
> URL: https://issues.apache.org/jira/browse/ROCKETMQ-284
> Project: Apache RocketMQ
>  Issue Type: Bug
>  Components: rocketmq-broker
>Affects Versions: 4.1.0-incubating
>Reporter: Eric Liu
>Assignee: Eric Liu
> Fix For: 4.2.0-incubating
>
>   Original Estimate: 1h
>  Remaining Estimate: 1h
>
> ExpressionMessageFilter will pass some message with tag's hash code less than 
> 0.
> This is not correct. After sql filter feature added, the tag's hash code in 
> consume queue may be replaced by the address of consume queue ext. And, 
> differences between them are:
> 1. Tag's hash code is an integer, values is between Integer.MIN and 
> Integer.MAX.
> 2. Address of consume queue ext is a long, values is between Long.MIN and 
> Integer.MIN - 1.
> So, actually,  they are not conflict.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (ROCKETMQ-284) ExpressionMessageFilter will pass some message

2017-08-29 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/ROCKETMQ-284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16145120#comment-16145120
 ] 

ASF GitHub Bot commented on ROCKETMQ-284:
-

Github user coveralls commented on the issue:

https://github.com/apache/incubator-rocketmq/pull/160
  

[![Coverage 
Status](https://coveralls.io/builds/13031998/badge)](https://coveralls.io/builds/13031998)

Coverage increased (+0.1%) to 38.755% when pulling 
**3f78c4e2160f6bb9d8a08934e391b3c81db7b72e on vsair:ROCKETMQ-284** into 
**4c5e58b46acaf4541fbd741b97593618f027be2d on apache:develop**.



> ExpressionMessageFilter will pass some message
> --
>
> Key: ROCKETMQ-284
> URL: https://issues.apache.org/jira/browse/ROCKETMQ-284
> Project: Apache RocketMQ
>  Issue Type: Bug
>  Components: rocketmq-broker
>Affects Versions: 4.1.0-incubating
>Reporter: Eric Liu
>Assignee: Eric Liu
> Fix For: 4.2.0-incubating
>
>   Original Estimate: 1h
>  Remaining Estimate: 1h
>
> ExpressionMessageFilter will pass some message with tag's hash code less than 
> 0.
> This is not correct. After sql filter feature added, the tag's hash code in 
> consume queue may be replaced by the address of consume queue ext. And, 
> differences between them are:
> 1. Tag's hash code is an integer, values is between Integer.MIN and 
> Integer.MAX.
> 2. Address of consume queue ext is a long, values is between Long.MIN and 
> Integer.MIN - 1.
> So, actually,  they are not conflict.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (ROCKETMQ-284) ExpressionMessageFilter will pass some message

2017-08-29 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/ROCKETMQ-284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16145027#comment-16145027
 ] 

ASF GitHub Bot commented on ROCKETMQ-284:
-

Github user vsair commented on a diff in the pull request:

https://github.com/apache/incubator-rocketmq/pull/160#discussion_r135745011
  
--- Diff: 
broker/src/test/java/org/apache/rocketmq/broker/filter/MessageStoreWithFilterTest.java
 ---
@@ -369,4 +367,36 @@ public void testGetMessage_withFilterBitMap() {
 }
 }
 }
+
+@Test
+public void testGetMessage_withFilter_checkTagsCode() {
+try {
+putMsg(master, topicCount, msgPerTopic);
+// sleep to wait for consume queue has been constructed.
--- End diff --

sleep to replace sleep ? 


> ExpressionMessageFilter will pass some message
> --
>
> Key: ROCKETMQ-284
> URL: https://issues.apache.org/jira/browse/ROCKETMQ-284
> Project: Apache RocketMQ
>  Issue Type: Bug
>  Components: rocketmq-broker
>Affects Versions: 4.1.0-incubating
>Reporter: Eric Liu
>Assignee: Eric Liu
> Fix For: 4.2.0-incubating
>
>   Original Estimate: 1h
>  Remaining Estimate: 1h
>
> ExpressionMessageFilter will pass some message with tag's hash code less than 
> 0.
> This is not correct. After sql filter feature added, the tag's hash code in 
> consume queue may be replaced by the address of consume queue ext. And, 
> differences between them are:
> 1. Tag's hash code is an integer, values is between Integer.MIN and 
> Integer.MAX.
> 2. Address of consume queue ext is a long, values is between Long.MIN and 
> Integer.MIN - 1.
> So, actually,  they are not conflict.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (ROCKETMQ-284) ExpressionMessageFilter will pass some message

2017-08-29 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/ROCKETMQ-284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16144915#comment-16144915
 ] 

ASF GitHub Bot commented on ROCKETMQ-284:
-

Github user vongosling commented on a diff in the pull request:

https://github.com/apache/incubator-rocketmq/pull/160#discussion_r135722707
  
--- Diff: 
broker/src/test/java/org/apache/rocketmq/broker/filter/MessageStoreWithFilterTest.java
 ---
@@ -369,4 +367,36 @@ public void testGetMessage_withFilterBitMap() {
 }
 }
 }
+
+@Test
+public void testGetMessage_withFilter_checkTagsCode() {
+try {
+putMsg(master, topicCount, msgPerTopic);
+// sleep to wait for consume queue has been constructed.
+Thread.sleep(200);
+} catch (Exception e) {
--- End diff --

Could we using expected  for exception test ?


> ExpressionMessageFilter will pass some message
> --
>
> Key: ROCKETMQ-284
> URL: https://issues.apache.org/jira/browse/ROCKETMQ-284
> Project: Apache RocketMQ
>  Issue Type: Bug
>  Components: rocketmq-broker
>Affects Versions: 4.1.0-incubating
>Reporter: Eric Liu
>Assignee: Eric Liu
> Fix For: 4.2.0-incubating
>
>   Original Estimate: 1h
>  Remaining Estimate: 1h
>
> ExpressionMessageFilter will pass some message with tag's hash code less than 
> 0.
> This is not correct. After sql filter feature added, the tag's hash code in 
> consume queue may be replaced by the address of consume queue ext. And, 
> differences between them are:
> 1. Tag's hash code is an integer, values is between Integer.MIN and 
> Integer.MAX.
> 2. Address of consume queue ext is a long, values is between Long.MIN and 
> Integer.MIN - 1.
> So, actually,  they are not conflict.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (ROCKETMQ-284) ExpressionMessageFilter will pass some message

2017-08-29 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/ROCKETMQ-284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16144916#comment-16144916
 ] 

ASF GitHub Bot commented on ROCKETMQ-284:
-

Github user vongosling commented on a diff in the pull request:

https://github.com/apache/incubator-rocketmq/pull/160#discussion_r135724026
  
--- Diff: 
broker/src/test/java/org/apache/rocketmq/broker/filter/MessageStoreWithFilterTest.java
 ---
@@ -369,4 +367,36 @@ public void testGetMessage_withFilterBitMap() {
 }
 }
 }
+
+@Test
+public void testGetMessage_withFilter_checkTagsCode() {
+try {
+putMsg(master, topicCount, msgPerTopic);
+// sleep to wait for consume queue has been constructed.
--- End diff --

Must we use sleep to replace sleep ? Now ,we have polished our unit test 
within 3 minutes, thoughts @shroman 


> ExpressionMessageFilter will pass some message
> --
>
> Key: ROCKETMQ-284
> URL: https://issues.apache.org/jira/browse/ROCKETMQ-284
> Project: Apache RocketMQ
>  Issue Type: Bug
>  Components: rocketmq-broker
>Affects Versions: 4.1.0-incubating
>Reporter: Eric Liu
>Assignee: Eric Liu
> Fix For: 4.2.0-incubating
>
>   Original Estimate: 1h
>  Remaining Estimate: 1h
>
> ExpressionMessageFilter will pass some message with tag's hash code less than 
> 0.
> This is not correct. After sql filter feature added, the tag's hash code in 
> consume queue may be replaced by the address of consume queue ext. And, 
> differences between them are:
> 1. Tag's hash code is an integer, values is between Integer.MIN and 
> Integer.MAX.
> 2. Address of consume queue ext is a long, values is between Long.MIN and 
> Integer.MIN - 1.
> So, actually,  they are not conflict.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (ROCKETMQ-284) ExpressionMessageFilter will pass some message

2017-08-29 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/ROCKETMQ-284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16144871#comment-16144871
 ] 

ASF GitHub Bot commented on ROCKETMQ-284:
-

Github user vsair commented on the issue:

https://github.com/apache/incubator-rocketmq/pull/160
  
@vongosling  pls have a review.


> ExpressionMessageFilter will pass some message
> --
>
> Key: ROCKETMQ-284
> URL: https://issues.apache.org/jira/browse/ROCKETMQ-284
> Project: Apache RocketMQ
>  Issue Type: Bug
>  Components: rocketmq-broker
>Affects Versions: 4.1.0-incubating
>Reporter: Eric Liu
>Assignee: Eric Liu
> Fix For: 4.2.0-incubating
>
>   Original Estimate: 1h
>  Remaining Estimate: 1h
>
> ExpressionMessageFilter will pass some message with tag's hash code less than 
> 0.
> This is not correct. After sql filter feature added, the tag's hash code in 
> consume queue may be replaced by the address of consume queue ext. And, 
> differences between them are:
> 1. Tag's hash code is an integer, values is between Integer.MIN and 
> Integer.MAX.
> 2. Address of consume queue ext is a long, values is between Long.MIN and 
> Integer.MIN - 1.
> So, actually,  they are not conflict.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (ROCKETMQ-277) get localhost throw java.net.UnknownHostException,when server hostname not in /etc/hosts

2017-08-29 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/ROCKETMQ-277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16144837#comment-16144837
 ] 

ASF GitHub Bot commented on ROCKETMQ-277:
-

Github user fuyou001 closed the pull request at:

https://github.com/apache/incubator-rocketmq/pull/157


> get localhost throw java.net.UnknownHostException,when server hostname  not 
> in /etc/hosts
> -
>
> Key: ROCKETMQ-277
> URL: https://issues.apache.org/jira/browse/ROCKETMQ-277
> Project: Apache RocketMQ
>  Issue Type: Bug
>  Components: rocketmq-client, rocketmq-commons
>Affects Versions: 4.0.0-incubating, 4.1.0-incubating
>Reporter: yubaofu
>Assignee: Xiaorui Wang
> Fix For: 4.2.0-incubating
>
>
> if server hostname not in /etc/hosts, 
> org.apache.rocketmq.common.MixAll#localhost throw  UnknownHostException 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (ROCKETMQ-279) add commit log data and consume queue data logic consistent check when start mq

2017-08-28 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/ROCKETMQ-279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16144776#comment-16144776
 ] 

ASF GitHub Bot commented on ROCKETMQ-279:
-

Github user coveralls commented on the issue:

https://github.com/apache/incubator-rocketmq/pull/159
  

[![Coverage 
Status](https://coveralls.io/builds/13028464/badge)](https://coveralls.io/builds/13028464)

Coverage decreased (-0.4%) to 38.427% when pulling 
**1ddd284118133cbb21be3076b73153131d9c7167 on fuyou001:ROCKETMQ-2792** into 
**629c3e9ea084d0a2947b2ab6f1cc8b0c6ed98502 on apache:develop**.



> add commit log data and consume queue data logic consistent check when start 
> mq
> ---
>
> Key: ROCKETMQ-279
> URL: https://issues.apache.org/jira/browse/ROCKETMQ-279
> Project: Apache RocketMQ
>  Issue Type: Bug
>  Components: rocketmq-store
>Affects Versions: 4.0.0-incubating, 4.1.0-incubating
>Reporter: yubaofu
>Assignee: yukon
> Fix For: 4.2.0-incubating
>
>
> commit log and consume data may logic inconsistent ,
> because  [ROCKETMQ-265|https://issues.apache.org/jira/browse/ROCKETMQ-265],
> so we add  data consistent check,when mq start.
> if check failed, start failed.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (ROCKETMQ-279) add commit log data and consume queue data logic consistent check when start mq

2017-08-28 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/ROCKETMQ-279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16144772#comment-16144772
 ] 

ASF GitHub Bot commented on ROCKETMQ-279:
-

Github user coveralls commented on the issue:

https://github.com/apache/incubator-rocketmq/pull/159
  

[![Coverage 
Status](https://coveralls.io/builds/13028470/badge)](https://coveralls.io/builds/13028470)

Coverage decreased (-0.4%) to 38.468% when pulling 
**1ddd284118133cbb21be3076b73153131d9c7167 on fuyou001:ROCKETMQ-2792** into 
**629c3e9ea084d0a2947b2ab6f1cc8b0c6ed98502 on apache:develop**.



> add commit log data and consume queue data logic consistent check when start 
> mq
> ---
>
> Key: ROCKETMQ-279
> URL: https://issues.apache.org/jira/browse/ROCKETMQ-279
> Project: Apache RocketMQ
>  Issue Type: Bug
>  Components: rocketmq-store
>Affects Versions: 4.0.0-incubating, 4.1.0-incubating
>Reporter: yubaofu
>Assignee: yukon
> Fix For: 4.2.0-incubating
>
>
> commit log and consume data may logic inconsistent ,
> because  [ROCKETMQ-265|https://issues.apache.org/jira/browse/ROCKETMQ-265],
> so we add  data consistent check,when mq start.
> if check failed, start failed.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (ROCKETMQ-284) ExpressionMessageFilter will pass some message

2017-08-28 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/ROCKETMQ-284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16144751#comment-16144751
 ] 

ASF GitHub Bot commented on ROCKETMQ-284:
-

Github user coveralls commented on the issue:

https://github.com/apache/incubator-rocketmq/pull/160
  

[![Coverage 
Status](https://coveralls.io/builds/13028196/badge)](https://coveralls.io/builds/13028196)

Coverage increased (+0.02%) to 38.67% when pulling 
**0c9495f2458259bc419872a727403317de18bfb2 on vsair:ROCKETMQ-284** into 
**4c5e58b46acaf4541fbd741b97593618f027be2d on apache:develop**.



> ExpressionMessageFilter will pass some message
> --
>
> Key: ROCKETMQ-284
> URL: https://issues.apache.org/jira/browse/ROCKETMQ-284
> Project: Apache RocketMQ
>  Issue Type: Bug
>  Components: rocketmq-broker
>Affects Versions: 4.1.0-incubating
>Reporter: Eric Liu
>Assignee: Eric Liu
> Fix For: 4.2.0-incubating
>
>   Original Estimate: 1h
>  Remaining Estimate: 1h
>
> ExpressionMessageFilter will pass some message with tag's hash code less than 
> 0.
> This is not correct. After sql filter feature added, the tag's hash code in 
> consume queue may be replaced by the address of consume queue ext. And, 
> differences between them are:
> 1. Tag's hash code is an integer, values is between Integer.MIN and 
> Integer.MAX.
> 2. Address of consume queue ext is a long, values is between Long.MIN and 
> Integer.MIN - 1.
> So, actually,  they are not conflict.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (ROCKETMQ-279) add commit log data and consume queue data logic consistent check when start mq

2017-08-28 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/ROCKETMQ-279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16144738#comment-16144738
 ] 

ASF GitHub Bot commented on ROCKETMQ-279:
-

Github user fuyou001 commented on a diff in the pull request:

https://github.com/apache/incubator-rocketmq/pull/159#discussion_r135692614
  
--- Diff: store/src/main/java/org/apache/rocketmq/store/ConsumeQueue.java 
---
@@ -84,6 +84,59 @@ public boolean load() {
 return result;
 }
 
+public void checkCommitLogAndConsumeQueueConsistent() {
+
+String queueDir = this.storePath
++ File.separator + topic
++ File.separator + queueId + File.separator;
+
+long maxOffsetInQueue = getMaxOffsetInQueue();
--- End diff --

refactor


> add commit log data and consume queue data logic consistent check when start 
> mq
> ---
>
> Key: ROCKETMQ-279
> URL: https://issues.apache.org/jira/browse/ROCKETMQ-279
> Project: Apache RocketMQ
>  Issue Type: Bug
>  Components: rocketmq-store
>Affects Versions: 4.0.0-incubating, 4.1.0-incubating
>Reporter: yubaofu
>Assignee: yukon
> Fix For: 4.2.0-incubating
>
>
> commit log and consume data may logic inconsistent ,
> because  [ROCKETMQ-265|https://issues.apache.org/jira/browse/ROCKETMQ-265],
> so we add  data consistent check,when mq start.
> if check failed, start failed.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

1 2 3 4 5 6 7 8 9 10 >

1 - 100 of 1432 matches

Mail list logo