回复：求助，RocketMQ broker01主机cpu异常飙高问题！

duhengforever Wed, 18 Jul 2018 23:06:35 -0700

从日志来看，是因为大量SendMessageThread竞争锁导致CAS时产生CPU空转，尝试将sendMessageThreadPoolNums设置为5试下。

发自网易邮箱大师

在2018年07月19日 13:59，hooligan<79015...@qq.com> 写道：

hi， all
现网运行的rocketmq-3.2.6集群，配置了两组broker：borker01和broker02，每组borker都是一主一备，都是24核、128G内存的机器，其中borker01主机的cpu特别高80%左右，boker02主机则只有20%左右，两台机器的生产消费流量差不多，broker01稍大一点，iowait也差不多，就是cpu使用率差异比较大，看broker.log日志，发现broker01的日志中有如下的WARN日志，请教一下大家这个问题要怎么解决，多谢

2018-07-19 13:54:15 INFO StoreStatsService - put_tps 10083.910572186434
2018-07-19 13:54:15 INFO StoreStatsService - get_found_tps 10138.794998105343
2018-07-19 13:54:15 INFO StoreStatsService - get_miss_tps 5010.140204622963
2018-07-19 13:54:15 INFO StoreStatsService - get_transfered_tps 17993.891625615764
2018-07-19 13:54:24 INFO FlushCommitLogService - how much disk fall behind memory, 9829
2018-07-19 13:54:24 WARN SendMessageThread_24 - putMessage in lock eclipse time(ms) 1757
2018-07-19 13:54:24 WARN SendMessageThread_24 - putMessage not in lock eclipse time(ms) 1757
2018-07-19 13:54:24 WARN SendMessageThread_10 - putMessage not in lock eclipse time(ms) 1759
2018-07-19 13:54:24 WARN SendMessageThread_9 - putMessage not in lock eclipse time(ms) 1761
2018-07-19 13:54:24 WARN SendMessageThread_4 - putMessage not in lock eclipse time(ms) 1761
2018-07-19 13:54:24 WARN SendMessageThread_3 - putMessage not in lock eclipse time(ms) 1763
2018-07-19 13:54:24 WARN SendMessageThread_21 - putMessage not in lock eclipse time(ms) 1765
2018-07-19 13:54:24 WARN SendMessageThread_6 - putMessage not in lock eclipse time(ms) 1769
2018-07-19 13:54:24 WARN SendMessageThread_20 - putMessage not in lock eclipse time(ms) 1771
2018-07-19 13:54:24 WARN SendMessageThread_15 - putMessage not in lock eclipse time(ms) 1771
2018-07-19 13:54:24 WARN SendMessageThread_23 - putMessage not in lock eclipse time(ms) 1773
2018-07-19 13:54:24 WARN SendMessageThread_22 - putMessage not in lock eclipse time(ms) 1773
2018-07-19 13:54:24 WARN SendMessageThread_2 - putMessage not in lock eclipse time(ms) 1776
2018-07-19 13:54:24 WARN SendMessageThread_16 - putMessage not in lock eclipse time(ms) 1778
2018-07-19 13:54:24 WARN SendMessageThread_8 - putMessage not in lock eclipse time(ms) 1778
2018-07-19 13:54:24 WARN SendMessageThread_7 - putMessage not in lock eclipse time(ms) 1778
2018-07-19 13:54:24 WARN SendMessageThread_11 - putMessage not in lock eclipse time(ms) 1780
2018-07-19 13:54:24 WARN SendMessageThread_17 - putMessage not in lock eclipse time(ms) 1782
2018-07-19 13:54:24 WARN SendMessageThread_12 - putMessage not in lock eclipse time(ms) 1784
2018-07-19 13:54:24 WARN SendMessageThread_13 - putMessage not in lock eclipse time(ms) 1787
2018-07-19 13:54:24 WARN SendMessageThread_14 - putMessage not in lock eclipse time(ms) 1790
2018-07-19 13:54:24 WARN SendMessageThread_19 - putMessage not in lock eclipse time(ms) 1790
2018-07-19 13:54:24 WARN SendMessageThread_5 - putMessage not in lock eclipse time(ms) 1791
2018-07-19 13:54:24 WARN SendMessageThread_18 - putMessage not in lock eclipse time(ms) 1792
2018-07-19 13:54:38 WARN SendMessageThread_18 - putMessage in lock eclipse time(ms) 1826
2018-07-19 13:54:38 WARN SendMessageThread_18 - putMessage not in lock eclipse time(ms) 1826
2018-07-19 13:54:38 WARN SendMessageThread_24 - putMessage not in lock eclipse time(ms) 1836
2018-07-19 13:54:38 WARN SendMessageThread_5 - putMessage not in lock eclipse time(ms) 1836
2018-07-19 13:54:38 WARN SendMessageThread_23 - putMessage not in lock eclipse time(ms) 1838
2018-07-19 13:54:38 WARN SendMessageThread_10 - putMessage not in lock eclipse time(ms) 1839
2018-07-19 13:54:38 WARN SendMessageThread_12 - putMessage not in lock eclipse time(ms) 1839
2018-07-19 13:54:38 WARN SendMessageThread_1 - putMessage not in lock eclipse time(ms) 1842
2018-07-19 13:54:38 WARN SendMessageThread_8 - putMessage not in lock eclipse time(ms) 1843
2018-07-19 13:54:38 WARN SendMessageThread_20 - putMessage not in lock eclipse time(ms) 1846
2018-07-19 13:54:38 WARN SendMessageThread_22 - putMessage not in lock eclipse time(ms) 1847
2018-07-19 13:54:38 WARN SendMessageThread_17 - putMessage not in lock eclipse time(ms) 1850
2018-07-19 13:54:38 WARN SendMessageThread_4 - putMessage not in lock eclipse time(ms) 1853
2018-07-19 13:54:38 WARN SendMessageThread_21 - putMessage not in lock eclipse time(ms) 1852
2018-07-19 13:54:38 WARN SendMessageThread_7 - putMessage not in lock eclipse time(ms) 1853
2018-07-19 13:54:39 WARN SendMessageThread_15 - putMessage not in lock eclipse time(ms) 1849
2018-07-19 13:54:39 WARN SendMessageThread_2 - putMessage not in lock eclipse time(ms) 1849
2018-07-19 13:54:39 WARN SendMessageThread_13 - putMessage not in lock eclipse time(ms) 1849
2018-07-19 13:54:39 WARN SendMessageThread_3 - putMessage not in lock eclipse time(ms) 1854
2018-07-19 13:54:39 WARN SendMessageThread_14 - putMessage not in lock eclipse time(ms) 1854
2018-07-19 13:54:39 WARN SendMessageThread_16 - putMessage not in lock eclipse time(ms) 1857
2018-07-19 13:54:39 WARN SendMessageThread_6 - putMessage not in lock eclipse time(ms) 1858
2018-07-19 13:54:39 WARN SendMessageThread_11 - putMessage not in lock eclipse time(ms) 1856
2018-07-19 13:54:39 WARN SendMessageThread_19 - putMessage not in lock eclipse time(ms) 1859

iostat日志：
iostat -x 2

avg-cpu: %user %nice %system %iowait %steal %idle
9.69 0.00 5.53 0.02 0.00 84.76

Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 252.63 0.93 233.58 55.05 2883.11 25.06 0.02 0.11 2.43 0.10 0.10 2.27

avg-cpu: %user %nice %system %iowait %steal %idle
48.03 0.00 23.91 0.13 0.00 27.93

Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 1124.00 0.00 1298.50 0.00 14552.00 22.41 0.15 0.12 0.00 0.12 0.11 14.60

avg-cpu: %user %nice %system %iowait %steal %idle
46.66 0.00 22.98 0.09 0.00 30.27

Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 1211.00 0.00 1379.00 0.00 15394.00 22.33 0.13 0.10 0.00 0.10 0.09 12.80

rocketmq配置文件：
brokerClusterName=MQ_sz
brokerName=broker-01
brokerId=0
deleteWhen=04
fileReservedTime=24
brokerRole=ASYNC_MASTER
flushDiskType=ASYNC_FLUSH
storePathRootDir=/data/rocketmq/store
storePathCommitLog=/data/rocketmq/store/commitlog
autoCreateTopicEnable=false
autoCreateSubscriptionGroup=false
messageDelayLevel=1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 15s 20s 30s 1m 2m 3m 4m 5m 6m 7m 8m 9m 10m 15m 20m 30m 1h 2h 12h 24h 48h 72h
sendMessageThreadPoolNums=24

------------------

hooligan

回复：求助，RocketMQ broker01主机cpu异常飙高问题！

Reply via email to