[jira] [Commented] (ROCKETMQ-281) add check for preventing repeat start mq

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/ROCKETMQ-281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16153057#comment-16153057
 ] 

ASF GitHub Bot commented on ROCKETMQ-281:
-

Github user coveralls commented on the issue:

https://github.com/apache/incubator-rocketmq/pull/158
  

[![Coverage 
Status](https://coveralls.io/builds/13122639/badge)](https://coveralls.io/builds/13122639)

Coverage decreased (-0.1%) to 38.731% when pulling 
**331775dccb458dbb52fa54942a28c4dc7b98a8a1 on fuyou001:ROCKETMQ-281** into 
**254d432496c424717d45b8dc9e44ae3bd78ab466 on apache:develop**.



> add check for preventing repeat start mq
> 
>
> Key: ROCKETMQ-281
> URL: https://issues.apache.org/jira/browse/ROCKETMQ-281
> Project: Apache RocketMQ
>  Issue Type: Improvement
>  Components: rocketmq-broker
>Affects Versions: 4.0.0-incubating, 4.1.0-incubating
>Reporter: yubaofu
>Assignee: yukon
> Fix For: 4.2.0-incubating, 4.3.0-incubating
>
>
> add check for preventing repeat start mq



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (ROCKETMQ-267) server may reject messages when pdflush write dirty data back info disk

[yukon (JIRA)]

[ 
https://issues.apache.org/jira/browse/ROCKETMQ-267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16153043#comment-16153043
 ] 

yukon commented on ROCKETMQ-267:


[~after_sss] Did you try this parameter `transientStorePoolEnable=true`? And it 
would be great if you could provide a solution for this issue -:)

> server may reject messages when pdflush write dirty data back info disk
> ---
>
> Key: ROCKETMQ-267
> URL: https://issues.apache.org/jira/browse/ROCKETMQ-267
> Project: Apache RocketMQ
>  Issue Type: Bug
>Affects Versions: 4.1.0-incubating
> Environment: linux
>Reporter: wenqi.huang
>Assignee: vongosling
>Priority: Critical
> Attachments: screenshot-1.png
>
>
> I found the following error in the client's log :
> 2017-08-10 09:06:57  ERROR [DubboServerHandler-10.28.109.45:20994-thread-475] 
> c.c.d.a.c.b.r.RocketMQMsgProducer[RocketMQMsgProducer.java:42] -> Send ons 
> msg failed, topic=TopicSubOrderDataSync, tag=TagActivityDataSync, 
> key=activity-order-411320385584760066, msg=411320385584760066
> org.apache.rocketmq.client.exception.MQBrokerException: CODE: 2  DESC: 
> [TIMEOUT_CLEAN_QUEUE]broker busy, start flow control for a while, period in 
> queue: 208ms, size of queue: 17
> For more information, please visit the url, 
> http://rocketmq.apache.org/docs/faq/
>   at 
> org.apache.rocketmq.client.impl.MQClientAPIImpl.processSendResponse(MQClientAPIImpl.java:531)
>  ~[rocketmq-client-4.0.0-incubating.jar!/:4.0.0-incubating]
>   at 
> org.apache.rocketmq.client.impl.MQClientAPIImpl.sendMessageSync(MQClientAPIImpl.java:345)
>  ~[rocketmq-client-4.0.0-incubating.jar!/:4.0.0-incubating]
>   at 
> org.apache.rocketmq.client.impl.MQClientAPIImpl.sendMessage(MQClientAPIImpl.java:327)
>  ~[rocketmq-client-4.0.0-incubating.jar!/:4.0.0-incubating]
>   at 
> org.apache.rocketmq.client.impl.MQClientAPIImpl.sendMessage(MQClientAPIImpl.java:290)
>  ~[rocketmq-client-4.0.0-incubating.jar!/:4.0.0-incubating]
>   at 
> org.apache.rocketmq.client.impl.producer.DefaultMQProducerImpl.sendKernelImpl(DefaultMQProducerImpl.java:688)
>  ~[rocketmq-client-4.0.0-incubating.jar!/:4.0.0-incubating]
>   at 
> org.apache.rocketmq.client.impl.producer.DefaultMQProducerImpl.sendDefaultImpl(DefaultMQProducerImpl.java:458)
>  ~[rocketmq-client-4.0.0-incubating.jar!/:4.0.0-incubating]
>   at 
> org.apache.rocketmq.client.impl.producer.DefaultMQProducerImpl.send(DefaultMQProducerImpl.java:1049)
>  ~[rocketmq-client-4.0.0-incubating.jar!/:4.0.0-incubating]
>   at 
> org.apache.rocketmq.client.impl.producer.DefaultMQProducerImpl.send(DefaultMQProducerImpl.java:1008)
>  ~[rocketmq-client-4.0.0-incubating.jar!/:4.0.0-incubating]
>   at 
> org.apache.rocketmq.client.producer.DefaultMQProducer.send(DefaultMQProducer.java:204)
>  ~[rocketmq-client-4.0.0-incubating.jar!/:4.0.0-incubating]
> and I look into store.log in RocketMQ Broker, and found that at the same 
> time, the following line is there:
> 2017-08-10 09:06:57 INFO FlushRealTimeService - Flush data to disk costs 1240 
> ms
> I look into the source code, and found that rocketmq have some index files, 
> which rocketmq will not write immediately（because it is wrote randomlly）, but 
> when a index file write finished(about 500MB), rocketmq finally force it into 
> disk, that means dirty pages will be about 500MB maximally， (I have executed 
> the bin/os.sh under RocketMQ,  which change the default behavior of linux 
> pdflush）.because the [vm.dirty_ratio] is 50, and the available memory is 
> about 1600MB at my linux machine, 500MB will not exceed 50% of 1600M, so 
> pdflush will not executed in this way. So I guess writeback will impact the 
> RT of  write.
> So I write a testcase and proved this, the code is:
> {code}
> public class MappedFileTest {
> public static void main(String[] args) throws IOException, 
> InterruptedException {
> //mock rocketmq's index file
> String indexFile = "/home/admin/rocketmq-data/index/index";
> int indexFileToWriteInMB = 180;
> FileChannel indexFileChannel = new RandomAccessFile(new 
> File(indexFile), "rw").getChannel();
> final MappedByteBuffer indexFileBuffer = 
> indexFileChannel.map(MapMode.READ_WRITE, 0, 1024*1024*500);//500M
> //put some dirty data， attention that the data size will not overflow 
> vm.dirty_background_ratio；
> // because we set vm.dirty_expire_centisecs to 3000，so after 30 
> seconds，pdflush will writeback the dirty data into disk.
> byte[] bs = new byte[1024*1024*indexFileToWriteInMB];//180m
> Arrays.fill(bs,(byte)1);
> indexFileBuffer.put(bs);
> //mock rocketmq's commitlog file
> String commitLogFile = 
> 

[jira] [Updated] (ROCKETMQ-267) server may reject messages when pdflush write dirty data back info disk

[yukon (JIRA)]

 [ 
https://issues.apache.org/jira/browse/ROCKETMQ-267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

yukon updated ROCKETMQ-267:
---
Description: 
I found the following error in the client's log :

2017-08-10 09:06:57  ERROR [DubboServerHandler-10.28.109.45:20994-thread-475] 
c.c.d.a.c.b.r.RocketMQMsgProducer[RocketMQMsgProducer.java:42] -> Send ons msg 
failed, topic=TopicSubOrderDataSync, tag=TagActivityDataSync, 
key=activity-order-411320385584760066, msg=411320385584760066
org.apache.rocketmq.client.exception.MQBrokerException: CODE: 2  DESC: 
[TIMEOUT_CLEAN_QUEUE]broker busy, start flow control for a while, period in 
queue: 208ms, size of queue: 17
For more information, please visit the url, http://rocketmq.apache.org/docs/faq/
at 
org.apache.rocketmq.client.impl.MQClientAPIImpl.processSendResponse(MQClientAPIImpl.java:531)
 ~[rocketmq-client-4.0.0-incubating.jar!/:4.0.0-incubating]
at 
org.apache.rocketmq.client.impl.MQClientAPIImpl.sendMessageSync(MQClientAPIImpl.java:345)
 ~[rocketmq-client-4.0.0-incubating.jar!/:4.0.0-incubating]
at 
org.apache.rocketmq.client.impl.MQClientAPIImpl.sendMessage(MQClientAPIImpl.java:327)
 ~[rocketmq-client-4.0.0-incubating.jar!/:4.0.0-incubating]
at 
org.apache.rocketmq.client.impl.MQClientAPIImpl.sendMessage(MQClientAPIImpl.java:290)
 ~[rocketmq-client-4.0.0-incubating.jar!/:4.0.0-incubating]
at 
org.apache.rocketmq.client.impl.producer.DefaultMQProducerImpl.sendKernelImpl(DefaultMQProducerImpl.java:688)
 ~[rocketmq-client-4.0.0-incubating.jar!/:4.0.0-incubating]
at 
org.apache.rocketmq.client.impl.producer.DefaultMQProducerImpl.sendDefaultImpl(DefaultMQProducerImpl.java:458)
 ~[rocketmq-client-4.0.0-incubating.jar!/:4.0.0-incubating]
at 
org.apache.rocketmq.client.impl.producer.DefaultMQProducerImpl.send(DefaultMQProducerImpl.java:1049)
 ~[rocketmq-client-4.0.0-incubating.jar!/:4.0.0-incubating]
at 
org.apache.rocketmq.client.impl.producer.DefaultMQProducerImpl.send(DefaultMQProducerImpl.java:1008)
 ~[rocketmq-client-4.0.0-incubating.jar!/:4.0.0-incubating]
at 
org.apache.rocketmq.client.producer.DefaultMQProducer.send(DefaultMQProducer.java:204)
 ~[rocketmq-client-4.0.0-incubating.jar!/:4.0.0-incubating]

and I look into store.log in RocketMQ Broker, and found that at the same time, 
the following line is there:

2017-08-10 09:06:57 INFO FlushRealTimeService - Flush data to disk costs 1240 ms

I look into the source code, and found that rocketmq have some index files, 
which rocketmq will not write immediately（because it is wrote randomlly）, but 
when a index file write finished(about 500MB), rocketmq finally force it into 
disk, that means dirty pages will be about 500MB maximally， (I have executed 
the bin/os.sh under RocketMQ,  which change the default behavior of linux 
pdflush）.because the [vm.dirty_ratio] is 50, and the available memory is about 
1600MB at my linux machine, 500MB will not exceed 50% of 1600M, so pdflush will 
not executed in this way. So I guess writeback will impact the RT of  write.

So I write a testcase and proved this, the code is:

{code}
public class MappedFileTest {
public static void main(String[] args) throws IOException, 
InterruptedException {
//mock rocketmq's index file
String indexFile = "/home/admin/rocketmq-data/index/index";
int indexFileToWriteInMB = 180;
FileChannel indexFileChannel = new RandomAccessFile(new 
File(indexFile), "rw").getChannel();
final MappedByteBuffer indexFileBuffer = 
indexFileChannel.map(MapMode.READ_WRITE, 0, 1024*1024*500);//500M

//put some dirty data， attention that the data size will not overflow 
vm.dirty_background_ratio；
// because we set vm.dirty_expire_centisecs to 3000，so after 30 
seconds，pdflush will writeback the dirty data into disk.
byte[] bs = new byte[1024*1024*indexFileToWriteInMB];//180m
Arrays.fill(bs,(byte)1);
indexFileBuffer.put(bs);

//mock rocketmq's commitlog file
String commitLogFile = "/home/admin/rocketmq-data/commitlog/commitlog";
FileChannel commitLogChannel = new RandomAccessFile(new 
File(commitLogFile), "rw").getChannel();
final MappedByteBuffer commitLogBuffer = 
commitLogChannel.map(MapMode.READ_WRITE, 0, 1024*1024*1024);//1G

final Object lockObj = new Object();

//mock FlushCommitLogService to writeback dirty data of commitLog into 
disk.
FlushCommitLogService commitLogService = new 
FlushCommitLogService(lockObj, commitLogBuffer);
commitLogService.start();

//mock messageReceived to write data into commitLogFile.
mockMessageReceived(lockObj, commitLogBuffer);

//wait for about 30 seconds(let linux pdflush to writeback dirty data 
of indexFile), then you will see some output like(this will block the thread 
that handle messages, then client will fail to send 

[jira] [Issue Comment Deleted] (ROCKETMQ-267) server may reject messages when pdflush write dirty data back info disk

[wenqi.huang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/ROCKETMQ-267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

wenqi.huang updated ROCKETMQ-267:
-
Comment: was deleted

(was: Few days ago, I move the logback directory(defaut ${user.home}/logs) and 
the index directory to another disk,  which means the commitlog is written into 
one disk, and logs or index files into another disk, and this problem 
disappeared. I don't know why, But this actually resolved my problem. Before 
doing this, the rocketmq client throw some send fail exception mentioned above 
for dozens of times, and after I do this, the send fail exception have been 
disappeared for four days. 
I think rocketmq's requirement for changing the linux kernel parameters 
 of write back is strange, it also affect the logback logs(If some other 
program is running at the same machine, it will also be affected, although this 
situation is rare). maybe you can consider rewrite the code of writing index 
file to use WAL or something else.)

> server may reject messages when pdflush write dirty data back info disk
> ---
>
> Key: ROCKETMQ-267
> URL: https://issues.apache.org/jira/browse/ROCKETMQ-267
> Project: Apache RocketMQ
>  Issue Type: Bug
>Affects Versions: 4.1.0-incubating
> Environment: linux
>Reporter: wenqi.huang
>Assignee: vongosling
>Priority: Critical
> Attachments: screenshot-1.png
>
>
> I found the following error in the client's log :
> 2017-08-10 09:06:57  ERROR [DubboServerHandler-10.28.109.45:20994-thread-475] 
> c.c.d.a.c.b.r.RocketMQMsgProducer[RocketMQMsgProducer.java:42] -> Send ons 
> msg failed, topic=TopicSubOrderDataSync, tag=TagActivityDataSync, 
> key=activity-order-411320385584760066, msg=411320385584760066
> org.apache.rocketmq.client.exception.MQBrokerException: CODE: 2  DESC: 
> [TIMEOUT_CLEAN_QUEUE]broker busy, start flow control for a while, period in 
> queue: 208ms, size of queue: 17
> For more information, please visit the url, 
> http://rocketmq.apache.org/docs/faq/
>   at 
> org.apache.rocketmq.client.impl.MQClientAPIImpl.processSendResponse(MQClientAPIImpl.java:531)
>  ~[rocketmq-client-4.0.0-incubating.jar!/:4.0.0-incubating]
>   at 
> org.apache.rocketmq.client.impl.MQClientAPIImpl.sendMessageSync(MQClientAPIImpl.java:345)
>  ~[rocketmq-client-4.0.0-incubating.jar!/:4.0.0-incubating]
>   at 
> org.apache.rocketmq.client.impl.MQClientAPIImpl.sendMessage(MQClientAPIImpl.java:327)
>  ~[rocketmq-client-4.0.0-incubating.jar!/:4.0.0-incubating]
>   at 
> org.apache.rocketmq.client.impl.MQClientAPIImpl.sendMessage(MQClientAPIImpl.java:290)
>  ~[rocketmq-client-4.0.0-incubating.jar!/:4.0.0-incubating]
>   at 
> org.apache.rocketmq.client.impl.producer.DefaultMQProducerImpl.sendKernelImpl(DefaultMQProducerImpl.java:688)
>  ~[rocketmq-client-4.0.0-incubating.jar!/:4.0.0-incubating]
>   at 
> org.apache.rocketmq.client.impl.producer.DefaultMQProducerImpl.sendDefaultImpl(DefaultMQProducerImpl.java:458)
>  ~[rocketmq-client-4.0.0-incubating.jar!/:4.0.0-incubating]
>   at 
> org.apache.rocketmq.client.impl.producer.DefaultMQProducerImpl.send(DefaultMQProducerImpl.java:1049)
>  ~[rocketmq-client-4.0.0-incubating.jar!/:4.0.0-incubating]
>   at 
> org.apache.rocketmq.client.impl.producer.DefaultMQProducerImpl.send(DefaultMQProducerImpl.java:1008)
>  ~[rocketmq-client-4.0.0-incubating.jar!/:4.0.0-incubating]
>   at 
> org.apache.rocketmq.client.producer.DefaultMQProducer.send(DefaultMQProducer.java:204)
>  ~[rocketmq-client-4.0.0-incubating.jar!/:4.0.0-incubating]
> and I look into store.log in RocketMQ Broker, and found that at the same 
> time, the following line is there:
> 2017-08-10 09:06:57 INFO FlushRealTimeService - Flush data to disk costs 1240 
> ms
> I look into the source code, and found that rocketmq have some index files, 
> which rocketmq will not write immediately（because it is wrote randomlly）, but 
> when a index file write finished(about 500MB), rocketmq finally force it into 
> disk, that means dirty pages will be about 500MB maximally， (I have executed 
> the bin/os.sh under RocketMQ,  which change the default behavior of linux 
> pdflush）.because the [vm.dirty_ratio] is 50, and the available memory is 
> about 1600MB at my linux machine, 500MB will not exceed 50% of 1600M, so 
> pdflush will not executed in this way. So I guess writeback will impact the 
> RT of  write.
> So I write a testcase and proved this, the code is:
> public class MappedFileTest {
> public static void main(String[] args) throws IOException, 
> InterruptedException {
> //mock rocketmq's index file
> String indexFile = "/home/admin/rocketmq-data/index/index";
> int indexFileToWriteInMB = 180;
> FileChannel indexFileChannel = new 

[jira] [Commented] (ROCKETMQ-286) Add a switch to decide whether to wake up commitLogService when put message

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/ROCKETMQ-286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16152223#comment-16152223
 ] 

ASF GitHub Bot commented on ROCKETMQ-286:
-

Github user coveralls commented on the issue:

https://github.com/apache/incubator-rocketmq/pull/162
  

[![Coverage 
Status](https://coveralls.io/builds/13111287/badge)](https://coveralls.io/builds/13111287)

Coverage decreased (-0.2%) to 38.636% when pulling 
**6bc0b58c04339c5f24eb306c60b25c6dc8ec626d on vsair:ROCKETMQ-286** into 
**254d432496c424717d45b8dc9e44ae3bd78ab466 on apache:develop**.



> Add a switch to decide whether to wake up commitLogService when put message
> ---
>
> Key: ROCKETMQ-286
> URL: https://issues.apache.org/jira/browse/ROCKETMQ-286
> Project: Apache RocketMQ
>  Issue Type: Improvement
>  Components: rocketmq-store
>Affects Versions: 4.0.0-incubating, 4.1.0-incubating
>Reporter: Eric Liu
>Assignee: Eric Liu
> Fix For: 4.2.0-incubating
>
>
> Now, it will wake up commitLogService after putting message. This may have 
> impact on disk.
> Add a switch to control this feature.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (ROCKETMQ-286) Add a switch to decide whether to wake up commitLogService when put message

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/ROCKETMQ-286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16152195#comment-16152195
 ] 

ASF GitHub Bot commented on ROCKETMQ-286:
-

Github user vsair commented on the issue:

https://github.com/apache/incubator-rocketmq/pull/162
  
@zhouxinyu  @lizhanhui  @vongosling   pls have a review.


> Add a switch to decide whether to wake up commitLogService when put message
> ---
>
> Key: ROCKETMQ-286
> URL: https://issues.apache.org/jira/browse/ROCKETMQ-286
> Project: Apache RocketMQ
>  Issue Type: Improvement
>  Components: rocketmq-store
>Affects Versions: 4.0.0-incubating, 4.1.0-incubating
>Reporter: Eric Liu
>Assignee: Eric Liu
> Fix For: 4.2.0-incubating
>
>
> Now, it will wake up commitLogService after putting message. This may have 
> impact on disk.
> Add a switch to control this feature.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (ROCKETMQ-286) Add a switch to decide whether to wake up commitLogService when put message

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/ROCKETMQ-286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16152194#comment-16152194
 ] 

ASF GitHub Bot commented on ROCKETMQ-286:
-

GitHub user vsair opened a pull request:

https://github.com/apache/incubator-rocketmq/pull/162

[ROCKETMQ-286] Add a switch to decide whether to wake up commitLogService 
when put message

https://issues.apache.org/jira/browse/ROCKETMQ-286

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/vsair/incubator-rocketmq ROCKETMQ-286

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-rocketmq/pull/162.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #162


commit 6bc0b58c04339c5f24eb306c60b25c6dc8ec626d
Author: vsair 
Date:   2017-09-04T07:09:54Z

[ROCKETMQ-286] Add a switch to decide whether to wake up commitLogService 
when put message




> Add a switch to decide whether to wake up commitLogService when put message
> ---
>
> Key: ROCKETMQ-286
> URL: https://issues.apache.org/jira/browse/ROCKETMQ-286
> Project: Apache RocketMQ
>  Issue Type: Improvement
>  Components: rocketmq-store
>Affects Versions: 4.0.0-incubating, 4.1.0-incubating
>Reporter: Eric Liu
>Assignee: Eric Liu
> Fix For: 4.2.0-incubating
>
>
> Now, it will wake up commitLogService after putting message. This may have 
> impact on disk.
> Add a switch to control this feature.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (ROCKETMQ-286) Add a switch to decide whether to wake up commitLogService when put message

[Eric Liu (JIRA)]

Eric Liu created ROCKETMQ-286:
-

 Summary: Add a switch to decide whether to wake up 
commitLogService when put message
 Key: ROCKETMQ-286
 URL: https://issues.apache.org/jira/browse/ROCKETMQ-286
 Project: Apache RocketMQ
  Issue Type: Improvement
  Components: rocketmq-store
Affects Versions: 4.1.0-incubating, 4.0.0-incubating
Reporter: Eric Liu
Assignee: Eric Liu
 Fix For: 4.2.0-incubating


Now, it will wake up commitLogService after putting message. This may have 
impact on disk.
Add a switch to control this feature.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)