[
https://issues.apache.org/jira/browse/ROCKETMQ-152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15976178#comment-15976178
]
yukon commented on ROCKETMQ-152:
--------------------------------
If you set a large sendMessageThreadPoolNums, please use ReentrantLock instead
of spin lock.
> lockForPutMessage CAS may cause system load very high and putMessage cost
> more than 500ms if the sendMessageThread num is very much
> --------------------------------------------------------------------------------------------------------------------------------------
>
> Key: ROCKETMQ-152
> URL: https://issues.apache.org/jira/browse/ROCKETMQ-152
> Project: Apache RocketMQ
> Issue Type: Improvement
> Components: rocketmq-store
> Affects Versions: 4.0.0-incubating
> Environment: sendMessageThreadPoolNums=128 and
> JDK jdk1.7.0_80 linux 2.6.32-504.el6.x86_64 and have more than 50 g free
> mem
> Reporter: zhaoziyan
> Assignee: yukon
> Attachments: QQ截图20170322152327.png
>
>
> broker store sendMessageThreadPoolNums=128
> 16 thread 1000bytes benchmark test ， System load to 100
> put message acquire the lock may spin，cause System load high and putMessage
> cost more than 500ms
> /**
> * Spin util acquired the lock.
> */
> private void lockForPutMessage() {
> if
> (this.defaultMessageStore.getMessageStoreConfig().isUseReentrantLockWhenPutMessage())
> {
> putMessageNormalLock.lock();
> } else {
> boolean flag;
> do {
> flag = this.putMessageSpinLock.compareAndSet(true, false);
> }
> while (!flag);
> }
> }
> 2017-03-22 14:40:16 WARN SendMessageThread_101 - putMessage not in lock
> eclipse time(ms)=1030, bodyLength=10013
> 2017-03-22 14:40:16 WARN SendMessageThread_34 - putMessage not in lock
> eclipse time(ms)=1932, bodyLength=10013
> 2017-03-22 14:40:16 WARN SendMessageThread_3 - putMessage not in lock eclipse
> time(ms)=581, bodyLength=10013
> 2017-03-22 14:40:16 WARN SendMessageThread_57 - putMessage not in lock
> eclipse time(ms)=583, bodyLength=10013
> 2017-03-22 14:40:16 WARN SendMessageThread_123 - putMessage not in lock
> eclipse time(ms)=2225, bodyLength=10013
> 2017-03-22 14:40:16 WARN SendMessageThread_1 - putMessage not in lock eclipse
> time(ms)=1642, bodyLength=10013
> 2017-03-22 14:40:16 WARN SendMessageThread_58 - putMessage not in lock
> eclipse time(ms)=587, bodyLength=10013
> 2017-03-22 14:40:16 WARN SendMessageThread_19 - putMessage not in lock
> eclipse time(ms)=1369, bodyLength=10013
> 2017-03-22 14:40:16 WARN SendMessageThread_32 - putMessage not in lock
> eclipse time(ms)=1896, bodyLength=10013
> 2017-03-22 14:40:16 WARN SendMessageThread_2 - putMessage not in lock eclipse
> time(ms)=1018, bodyLength=10013
> 2017-03-22 14:40:16 WARN SendMessageThread_122 - putMessage not in lock
> eclipse time(ms)=657, bodyLength=10013
> 2017-03-22 14:40:16 WARN SendMessageThread_120 - putMessage not in lock
> eclipse time(ms)=592, bodyLength=10013
> 2017-03-22 14:40:16 WARN SendMessageThread_88 - putMessage not in lock
> eclipse time(ms)=559, bodyLength=10013
> 2017-03-22 14:40:16 WARN SendMessageThread_44 - putMessage not in lock
> eclipse time(ms)=892, bodyLength=10013
> 2017-03-22 14:40:16 WARN SendMessageThread_79 - putMessage not in lock
> eclipse time(ms)=699, bodyLength=10013
> 2017-03-22 14:40:16 WARN SendMessageThread_84 - putMessage not in lock
> eclipse time(ms)=616, bodyLength=10013
> 2017-03-22 14:40:16 WARN SendMessageThread_112 - putMessage not in lock
> eclipse time(ms)=515, bodyLength=10013
> 2017-03-22 14:40:16 WARN SendMessageThread_59 - putMessage not in lock
> eclipse time(ms)=1301, bodyLength=10013
> 2017-03-22 14:40:16 WARN SendMessageThread_96 - putMessage not in lock
> eclipse time(ms)=635, bodyLength=10013
> 2017-03-22 14:40:16 WARN SendMessageThread_90 - ********lockForPutMessage
> cost:317
> 2017-03-22 14:40:16 WARN SendMessageThread_10 - ********lockForPutMessage
> cost:447
> 2017-03-22 14:40:16 WARN SendMessageThread_62 - ********lockForPutMessage
> cost:450
> 2017-03-22 14:40:16 WARN SendMessageThread_93 - ********lockForPutMessage
> cost:131
> 2017-03-22 14:40:16 WARN SendMessageThread_8 - ********lockForPutMessage
> cost:171
> 2017-03-22 14:40:16 WARN SendMessageThread_93 - ********lockForPutMessage
> cost:4
> 2017-03-22 14:40:16 WARN SendMessageThread_105 - ********lockForPutMessage
> cost:666
> 2017-03-22 14:40:16 WARN SendMessageThread_127 - ********lockForPutMessage
> cost:689
> 2017-03-22 14:40:16 WARN SendMessageThread_30 - ********lockForPutMessage
> cost:341
> 2017-03-22 14:40:16 WARN SendMessageThread_70 - ********lockForPutMessage
> cost:317
> 2017-03-22 14:40:16 WARN SendMessageThread_92 - ********lockForPutMessage
> cost:58
> 2017-03-22 14:40:16 WARN SendMessageThread_63 - ********lockForPutMessage
> cost:261
> 2017-03-22 14:40:16 WARN SendMessageThread_98 - ********lockForPutMessage
> cost:261
> 2017-03-22 14:40:16 WARN SendMessageThread_27 - ********lockForPutMessage
> cost:546
> 2017-03-22 14:40:16 WARN SendMessageThread_53 - ********lockForPutMessage
> cost:257
> 2017-03-22 14:40:16 WARN SendMessageThread_115 - ********lockForPutMessage
> cost:1047
> 2017-03-22 14:40:16 WARN SendMessageThread_82 - ********lockForPutMessage
> cost:1891
> 2017-03-22 14:40:16 WARN SendMessageThread_57 - ********lockForPutMessage
> cost:559
> 2017-03-22 14:40:16 WARN SendMessageThread_40 - ********lockForPutMessage
> cost:235
> 2017-03-22 14:40:16 WARN SendMessageThread_77 - ********lockForPutMessage
> cost:265
> 2017-03-22 14:40:16 WARN SendMessageThread_35 - ********lockForPutMessage
> cost:909
> 2017-03-22 14:40:16 WARN SendMessageThread_113 - ********lockForPutMessage
> cost:244
> 2017-03-22 14:40:16 WARN SendMessageThread_97 - ********lockForPutMessage
> cost:922
> 2017-03-22 14:40:16 WARN SendMessageThread_114 - ********lockForPutMessage
> cost:40
> 2017-03-22 14:40:16 WARN SendMessageThread_43 - ********lockForPutMessage
> cost:351
> 2017-03-22 14:40:16 WARN SendMessageThread_126 - ********lockForPutMessage
> cost:346
> 2017-03-22 14:40:16 WARN SendMessageThread_110 - ********lockForPutMessage
> cost:887
> 2017-03-22 14:40:16 WARN SendMessageThread_15 - ********lockForPutMessage
> cost:320
> 2017-03-22 14:40:16 WARN SendMessageThread_3 - ********lockForPutMessage
> cost:566
> 2017-03-22 14:40:16 WARN SendMessageThread_68 - ********lockForPutMessage
> cost:748
> 2017-03-22 14:40:16 WARN SendMessageThread_22 - ********lockForPutMessage
> cost:308
> 2017-03-22 14:40:16 WARN SendMessageThread_34 - ********lockForPutMessage
> cost:1926
> 2017-03-22 14:40:16 WARN SendMessageThread_91 - ********lockForPutMessage
> cost:358
> 2017-03-22 14:40:16 WARN SendMessageThread_25 - ********lockForPutMessage
> cost:107
> 2017-03-22 14:40:16 WARN SendMessageThread_101 - ********lockForPutMessage
> cost:1024
> 2017-03-22 14:40:16 WARN SendMessageThread_89 - ********lockForPutMessage
> cost:326
> 2017-03-22 14:40:16 WARN SendMessageThread_43 - ********lockForPutMessage
> cost:5
> 2017-03-22 14:40:16 WARN SendMessageThread_30 - ********lockForPutMessage
> cost:1
> 2017-03-22 14:40:16 WARN SendMessageThread_42 - ********lockForPutMessage
> cost:326
> 2017-03-22 14:40:16 WARN SendMessageThread_95 - ********lockForPutMessage
> cost:58
> 2017-03-22 14:40:16 WARN SendMessageThread_19 - ********lockForPutMessage
> cost:1363
> 2017-03-22 14:40:16 WARN SendMessageThread_86 - ********lockForPutMessage
> cost:92
> 2017-03-22 14:40:16 WARN SendMessageThread_1 - ********lockForPutMessage
> cost:1640
> 2017-03-22 14:40:16 WARN SendMessageThread_82 - ********lockForPutMessage
> cost:30
> 2017-03-22 14:40:16 WARN SendMessageThread_115 - ********lockForPutMessage
> cost:33
> 2017-03-22 14:40:16 WARN SendMessageThread_58 - ********lockForPutMessage
> cost:582
> 2017-03-22 14:40:16 WARN SendMessageThread_53 - ********lockForPutMessage
> cost:34
> 2017-03-22 14:40:16 WARN SendMessageThread_123 - ********lockForPutMessage
> cost:2224
> 2017-03-22 14:40:16 WARN SendMessageThread_78 - ********lockForPutMessage
> cost:286
> 2017-03-22 14:40:16 WARN SendMessageThread_63 - ********lockForPutMessage
> cost:27
> 2017-03-22 14:40:16 WARN SendMessageThread_32 - ********lockForPutMessage
> cost:1874
> 2017-03-22 14:40:16 WARN SendMessageThread_87 - ********lockForPutMessage
> cost:312
> 2017-03-22 14:40:16 WARN SendMessageThread_53 - ********lockForPutMessage
> cost:14
> 2017-03-22 14:40:16 WARN SendMessageThread_123 - ********lockForPutMessage
> cost:21
> 2017-03-22 14:40:16 WARN SendMessageThread_45 - ********lockForPutMessage
> cost:382
> 2017-03-22 14:40:16 WARN SendMessageThread_89 - ********lockForPutMessage
> cost:37
> 2017-03-22 14:40:16 WARN SendMessageThread_15 - ********lockForPutMessage
> cost:51
> 2017-03-22 14:40:16 WARN SendMessageThread_2 - ********lockForPutMessage
> cost:1015
> 2017-03-22 14:40:16 WARN SendMessageThread_68 - ********lockForPutMessage
> cost:50
> 2017-03-22 14:40:16 WARN SendMessageThread_22 - ********lockForPutMessage
> cost:51
> 2017-03-22 14:40:16 WARN SendMessageThread_84 - ********lockForPutMessage
> cost:606
> 2017-03-22 14:40:16 WARN SendMessageThread_3 - ********lockForPutMessage
> cost:42
> 2017-03-22 14:40:16 WARN SendMessageThread_122 - ********lockForPutMessage
> cost:652
> 2017-03-22 14:40:16 WARN SendMessageThread_115 - ********lockForPutMessage
> cost:19
> 2017-03-22 14:40:16 WARN SendMessageThread_126 - ********lockForPutMessage
> cost:24
> 2017-03-22 14:40:16 WARN SendMessageThread_57 - ********lockForPutMessage
> cost:3
> 2017-03-22 14:40:16 WARN SendMessageThread_110 - ********lockForPutMessage
> cost:3
> 2017-03-22 14:40:16 WARN SendMessageThread_70 - ********lockForPutMessage
> cost:40
> 2017-03-22 14:40:16 WARN SendMessageThread_118 - ********lockForPutMessage
> cost:279
> 2017-03-22 14:40:16 WARN SendMessageThread_103 - ********lockForPutMessage
> cost:301
> 2017-03-22 14:40:16 WARN SendMessageThread_36 - ********lockForPutMessage
> cost:384
> 2017-03-22 14:40:16 WARN SendMessageThread_120 - ********lockForPutMessage
> cost:587
> 2017-03-22 14:40:16 WARN SendMessageThread_88 - ********lockForPutMessage
> cost:554
> 2017-03-22 14:40:16 WARN SendMessageThread_124 - ********lockForPutMessage
> cost:324
> 2017-03-22 14:40:16 WARN SendMessageThread_44 - ********lockForPutMessage
> cost:888
> 2017-03-22 14:40:16 WARN SendMessageThread_49 - ********lockForPutMessage
> cost:311
> 2017-03-22 14:40:16 WARN SendMessageThread_41 - ********lockForPutMessage
> cost:312
> 2017-03-22 14:40:16 WARN SendMessageThread_5 - ********lockForPutMessage
> cost:115
> 2017-03-22 14:40:16 WARN SendMessageThread_4 - ********lockForPutMessage
> cost:151
> 2017-03-22 14:40:16 WARN SendMessageThread_119 - ********lockForPutMessage
> cost:79
> 2017-03-22 14:40:16 WARN SendMessageThread_65 - ********lockForPutMessage
> cost:107
> 2017-03-22 14:40:16 WARN SendMessageThread_128 - ********lockForPutMessage
> cost:112
> 2017-03-22 14:40:16 WARN SendMessageThread_96 - ********lockForPutMessage
> cost:609
> 2017-03-22 14:40:16 WARN SendMessageThread_79 - ********lockForPutMessage
> cost:697
> 2017-03-22 14:40:16 WARN SendMessageThread_7 - ********lockForPutMessage
> cost:417
> 2017-03-22 14:40:16 WARN SendMessageThread_94 - ********lockForPutMessage
> cost:419
> 2017-03-22 14:40:16 WARN SendMessageThread_72 - ********lockForPutMessage
> cost:208
> 2017-03-22 14:40:16 WARN SendMessageThread_8 - ********lockForPutMessage
> cost:42
> 2017-03-22 14:40:16 WARN SendMessageThread_99 - ********lockForPutMessage
> cost:237
> 2017-03-22 14:40:16 WARN SendMessageThread_16 - ********lockForPutMessage
> cost:123
> 2017-03-22 14:40:16 WARN SendMessageThread_85 - ********lockForPutMessage
> cost:133
> 2017-03-22 14:40:16 WARN SendMessageThread_111 - ********lockForPutMessage
> cost:139
> 2017-03-22 14:40:16 WARN SendMessageThread_112 - ********lockForPutMessage
> cost:510
> 2017-03-22 14:40:16 WARN SendMessageThread_45 - ********lockForPutMessage
> cost:2
> 2017-03-22 14:40:16 WARN SendMessageThread_2 - ********lockForPutMessage
> cost:3
> 2017-03-22 14:40:16 WARN SendMessageThread_59 - ********lockForPutMessage
> cost:1291
> 2017-03-22 14:40:16 WARN SendMessageThread_64 - ********lockForPutMessage
> cost:105
> 2017-03-22 14:40:16 WARN SendMessageThread_59 - ********lockForPutMessage
> cost:48
> 2017-03-22 14:40:16 WARN SendMessageThread_46 - ********lockForPutMessage
> cost:255
> 2017-03-22 14:40:16 WARN SendMessageThread_49 - ********lockForPutMessage
> cost:95
> 2017-03-22 14:40:16 WARN SendMessageThread_44 - ********lockForPutMessage
> cost:7
> 2017-03-22 14:40:16 WARN SendMessageThread_112 - ********lockForPutMessage
> cost:37
> 2017-03-22 14:40:16 WARN SendMessageThread_7 - ********lockForPutMessage
> cost:75
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
