Here is another snapshot. I wonder if this write io wait is too big
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz
avgqu-sz await r_await w_await svctm %util
dm-14 0.00 0.00 0.00 23.00 0.00 336.00 29.22
0.34 14.74 0.00 14.74 2.87 6.60
dm-15 0.00 0.00 0.00 16.00 0.00 200.00 25.00
0.01 0.75 0.00 0.75 0.75 1.20
dm-16 0.00 0.00 0.00 17.00 0.00 276.00 32.47
0.25 14.94 0.00 14.94 3.35 5.70
dm-17 0.00 0.00 0.00 17.00 0.00 252.00 29.65
0.32 18.65 0.00 18.65 4.00 6.80
dm-18 0.00 0.00 0.00 15.00 0.00 152.00 20.27
0.25 16.80 0.00 16.80 4.07 6.10
dm-19 0.00 0.00 0.00 13.00 0.00 152.00 23.38
0.21 15.92 0.00 15.92 4.85 6.30
dm-20 0.00 0.00 0.00 20.00 0.00 248.00 24.80
0.27 13.60 0.00 13.60 3.25 6.50
dm-21 0.00 0.00 0.00 17.00 0.00 188.00 22.12
0.27 16.00 0.00 16.00 3.59 6.10
dm-22 0.00 0.00 0.00 20.00 0.00 156.00 15.60
0.11 5.55 0.00 5.55 2.95 5.90
dm-24 0.00 0.00 0.00 8.00 0.00 56.00 14.00
0.12 14.62 0.00 14.62 4.75 3.80
dm-25 0.00 0.00 0.00 19.00 0.00 200.00 21.05
0.21 10.89 0.00 10.89 2.74 5.20
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz
avgqu-sz await r_await w_await svctm %util
dm-14 0.00 0.00 0.00 11.00 0.00 136.00 24.73
0.11 9.73 0.00 9.73 1.82 2.00
dm-15 0.00 0.00 0.00 12.00 0.00 136.00 22.67
0.04 3.75 0.00 3.75 1.08 1.30
dm-16 0.00 0.00 0.00 9.00 0.00 104.00 23.11
0.09 10.44 0.00 10.44 2.44 2.20
dm-17 0.00 0.00 0.00 5.00 0.00 160.00 64.00
0.02 4.00 0.00 4.00 4.00 2.00
dm-18 0.00 0.00 0.00 5.00 0.00 52.00 20.80
0.03 5.80 0.00 5.80 3.60 1.80
dm-19 0.00 0.00 0.00 10.00 0.00 104.00 20.80
0.08 7.90 0.00 7.90 2.10 2.10
dm-20 0.00 0.00 0.00 9.00 0.00 132.00 29.33
0.10 11.22 0.00 11.22 2.56 2.30
dm-21 0.00 0.00 0.00 6.00 0.00 68.00 22.67
0.07 12.33 0.00 12.33 3.83 2.30
dm-22 0.00 0.00 0.00 3.00 0.00 20.00 13.33
0.01 3.67 0.00 3.67 3.67 1.10
dm-24 0.00 0.00 0.00 4.00 0.00 24.00 12.00
0.07 18.00 0.00 18.00 5.25 2.10
dm-25 0.00 0.00 0.00 6.00 0.00 64.00 21.33
0.06 10.33 0.00 10.33 3.67 2.20
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz
avgqu-sz await r_await w_await svctm %util
dm-14 0.00 0.00 0.00 5.00 0.00 140.00 56.00
0.08 15.20 0.00 15.20 5.40 2.70
dm-15 0.00 0.00 0.00 6.00 0.00 236.00 78.67
0.18 30.67 0.00 30.67 6.83 4.10
dm-16 0.00 0.00 0.00 8.00 0.00 84.00 21.00
0.06 7.25 0.00 7.25 1.62 1.30
dm-17 0.00 0.00 0.00 3.00 0.00 84.00 56.00
0.00 0.33 0.00 0.33 0.33 0.10
dm-18 0.00 0.00 0.00 2.00 0.00 20.00 20.00
0.02 12.00 0.00 12.00 12.00 2.40
dm-19 0.00 0.00 0.00 12.00 0.00 80.00 13.33
0.05 4.00 0.00 4.00 2.33 2.80
dm-20 0.00 0.00 0.00 16.00 0.00 256.00 32.00
0.00 0.06 0.00 0.06 0.06 0.10
dm-21 0.00 0.00 0.00 8.00 0.00 500.00 125.00
0.00 0.12 0.00 0.12 0.12 0.10
dm-22 0.00 0.00 0.00 1.00 0.00 8.00 16.00
0.00 0.00 0.00 0.00 0.00 0.00
dm-24 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00 0.00
dm-25 0.00 0.00 0.00 2.00 0.00 32.00 32.00
0.08 40.00 0.00 40.00 20.50 4.10
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz
avgqu-sz await r_await w_await svctm %util
dm-14 0.00 0.00 0.00 10.00 0.00 108.00 21.60
0.11 10.80 0.00 10.80 1.90 1.90
dm-15 0.00 0.00 0.00 5.00 0.00 60.00 24.00
0.03 6.20 0.00 6.20 3.40 1.70
dm-16 0.00 0.00 0.00 6.00 0.00 68.00 22.67
0.00 0.17 0.00 0.17 0.17 0.10
dm-17 0.00 0.00 0.00 7.00 0.00 100.00 28.57
0.09 12.29 0.00 12.29 2.86 2.00
dm-18 0.00 0.00 0.00 5.00 0.00 44.00 17.60
0.07 14.00 0.00 14.00 4.00 2.00
dm-19 0.00 0.00 0.00 2.00 0.00 24.00 24.00
0.03 14.00 0.00 14.00 8.50 1.70
dm-20 0.00 0.00 0.00 7.00 0.00 96.00 27.43
0.07 9.57 0.00 9.57 2.00 1.40
dm-21 0.00 0.00 0.00 7.00 0.00 84.00 24.00
0.12 16.57 0.00 16.57 4.86 3.40
dm-22 0.00 0.00 0.00 5.00 0.00 60.00 24.00
0.05 10.20 0.00 10.20 3.40 1.70
dm-24 0.00 0.00 0.00 2.00 0.00 16.00 16.00
0.03 17.00 0.00 17.00 9.00 1.80
dm-25 0.00 0.00 0.00 4.00 0.00 76.00 38.00
0.05 11.75 0.00 11.75 4.75 1.90
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz
avgqu-sz await r_await w_await svctm %util
dm-14 0.00 0.00 0.00 9.00 0.00 104.00 23.11
0.14 15.33 0.00 15.33 4.33 3.90
dm-15 0.00 0.00 0.00 11.00 0.00 128.00 23.27
0.01 0.82 0.00 0.82 0.82 0.90
dm-16 0.00 0.00 0.00 10.00 0.00 120.00 24.00
0.09 9.30 0.00 9.30 3.30 3.30
dm-17 0.00 0.00 0.00 5.00 0.00 72.00 28.80
0.04 8.20 0.00 8.20 8.20 4.10
dm-18 0.00 0.00 0.00 7.00 0.00 68.00 19.43
0.08 11.57 0.00 11.57 6.29 4.40
dm-19 0.00 0.00 0.00 7.00 0.00 76.00 21.71
0.10 13.86 0.00 13.86 6.29 4.40
dm-20 0.00 0.00 0.00 15.00 0.00 176.00 23.47
0.12 7.87 0.00 7.87 1.60 2.40
dm-21 0.00 0.00 0.00 9.00 0.00 112.00 24.89
0.17 18.56 0.00 18.56 6.56 5.90
dm-22 0.00 0.00 0.00 4.00 0.00 56.00 28.00
0.04 10.75 0.00 10.75 7.50 3.00
dm-24 0.00 0.00 0.00 5.00 0.00 532.00 212.80
0.02 4.00 0.00 4.00 3.60 1.80
dm-25 0.00 0.00 0.00 6.00 0.00 60.00 20.00
0.11 19.00 0.00 19.00 7.17 4.30
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz
avgqu-sz await r_await w_await svctm %util
dm-14 0.00 0.00 0.00 9.00 0.00 112.00 24.89
0.30 33.22 0.00 33.22 4.33 3.90
dm-15 0.00 0.00 0.00 8.00 0.00 96.00 24.00
0.08 9.88 0.00 9.88 2.62 2.10
dm-16 0.00 0.00 0.00 5.00 0.00 92.00 36.80
0.01 1.80 0.00 1.80 1.80 0.90
dm-17 0.00 0.00 0.00 4.00 0.00 104.00 52.00
0.04 10.00 0.00 10.00 10.00 4.00
dm-18 0.00 0.00 0.00 9.00 0.00 92.00 20.44
0.11 12.00 0.00 12.00 4.44 4.00
dm-19 0.00 0.00 0.00 7.00 0.00 64.00 18.29
0.10 14.43 0.00 14.43 4.00 2.80
dm-20 0.00 0.00 0.00 7.00 0.00 84.00 24.00
0.10 14.00 0.00 14.00 3.71 2.60
dm-21 0.00 0.00 0.00 2.00 0.00 16.00 16.00
0.08 39.50 0.00 39.50 20.50 4.10
dm-22 0.00 0.00 0.00 5.00 0.00 36.00 14.40
0.00 0.00 0.00 0.00 0.00 0.00
dm-24 0.00 0.00 0.00 5.00 0.00 52.00 20.80
0.09 17.20 0.00 17.20 5.40 2.70
dm-25 0.00 0.00 0.00 6.00 0.00 48.00 16.00
0.10 16.00 0.00 16.00 5.50 3.30
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz
avgqu-sz await r_await w_await svctm %util
dm-14 0.00 0.00 0.00 5.00 0.00 48.00 19.20
0.00 0.00 0.00 0.00 0.00 0.00
dm-15 0.00 0.00 0.00 5.00 0.00 44.00 17.60
0.01 3.00 0.00 3.00 3.00 1.50
dm-16 0.00 0.00 0.00 3.00 0.00 32.00 21.33
0.00 0.00 0.00 0.00 0.00 0.00
dm-17 0.00 0.00 0.00 4.00 0.00 40.00 20.00
0.00 0.00 0.00 0.00 0.00 0.00
dm-18 0.00 0.00 0.00 3.00 0.00 32.00 21.33
0.04 13.00 0.00 13.00 6.67 2.00
dm-19 0.00 0.00 0.00 3.00 0.00 28.00 18.67
0.00 0.00 0.00 0.00 0.00 0.00
dm-20 0.00 0.00 0.00 6.00 0.00 64.00 21.33
0.11 17.67 0.00 17.67 3.17 1.90
dm-21 0.00 0.00 0.00 1.00 0.00 12.00 24.00
0.00 0.00 0.00 0.00 0.00 0.00
dm-22 0.00 0.00 0.00 1.00 0.00 12.00 24.00
0.00 0.00 0.00 0.00 0.00 0.00
dm-24 0.00 0.00 0.00 3.00 0.00 20.00 13.33
0.00 0.33 0.00 0.33 0.33 0.10
dm-25 0.00 0.00 0.00 6.00 0.00 52.00 17.33
0.00 0.00 0.00 0.00 0.00 0.00
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz
avgqu-sz await r_await w_await svctm %util
dm-14 0.00 0.00 0.00 18.00 0.00 236.00 26.22
0.42 23.28 0.00 23.28 3.72 6.70
dm-15 0.00 0.00 0.00 15.00 0.00 240.00 32.00
0.12 8.27 0.00 8.27 2.00 3.00
dm-16 0.00 0.00 0.00 10.00 0.00 120.00 24.00
0.23 22.80 0.00 22.80 6.50 6.50
dm-17 0.00 0.00 0.00 6.00 0.00 68.00 22.67
0.06 9.83 0.00 9.83 5.00 3.00
dm-18 0.00 0.00 0.00 12.00 0.00 152.00 25.33
0.10 8.17 0.00 8.17 3.17 3.80
dm-19 0.00 0.00 0.00 12.00 0.00 136.00 22.67
0.14 11.58 0.00 11.58 3.42 4.10
dm-20 0.00 0.00 0.00 19.00 0.00 296.00 31.16
0.20 10.63 0.00 10.63 1.84 3.50
dm-21 0.00 0.00 0.00 14.00 0.00 132.00 18.86
0.17 12.14 0.00 12.14 3.71 5.20
dm-22 0.00 0.00 0.00 10.00 0.00 84.00 16.80
0.12 12.00 0.00 12.00 6.60 6.60
dm-24 0.00 0.00 0.00 5.00 0.00 40.00 16.00
0.10 19.40 0.00 19.40 9.80 4.90
dm-25 0.00 0.00 0.00 9.00 0.00 152.00 33.78
0.08 9.33 0.00 9.33 3.44 3.10
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz
avgqu-sz await r_await w_await svctm %util
dm-14 0.00 0.00 0.00 8.00 0.00 112.00 28.00
0.08 9.75 0.00 9.75 3.88 3.10
dm-15 0.00 0.00 0.00 14.00 0.00 156.00 22.29
0.13 9.43 0.00 9.43 2.71 3.80
dm-16 0.00 0.00 0.00 12.00 0.00 132.00 22.00
0.11 8.83 0.00 8.83 2.17 2.60
dm-17 0.00 0.00 0.00 8.00 0.00 116.00 29.00
0.08 10.12 0.00 10.12 4.25 3.40
dm-18 0.00 0.00 0.00 8.00 0.00 100.00 25.00
0.08 9.62 0.00 9.62 5.00 4.00
dm-19 0.00 0.00 0.00 20.00 0.00 212.00 21.20
0.25 12.75 0.00 12.75 2.60 5.20
dm-20 0.00 0.00 0.00 16.00 0.00 168.00 21.00
0.04 2.69 0.00 2.69 0.81 1.30
dm-21 0.00 0.00 0.00 5.00 0.00 48.00 19.20
0.05 10.80 0.00 10.80 7.80 3.90
dm-22 0.00 0.00 0.00 5.00 0.00 52.00 20.80
0.03 6.80 0.00 6.80 3.40 1.70
dm-24 0.00 0.00 0.00 8.00 0.00 60.00 15.00
0.09 11.75 0.00 11.75 2.25 1.80
dm-25 0.00 0.00 0.00 12.00 0.00 160.00 26.67
0.06 5.08 0.00 5.08 1.25 1.50
On 11/17/2018 10:19 PM, Alex Litvak wrote:
I stand corrected, I looked at the device iostat, but it was partitioned. Here
is a more correct picture of what is going on now.
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz
avgqu-sz await r_await w_await svctm %util
dm-14 0.00 0.00 0.00 19.00 0.00 4116.00 433.26
0.01 0.68 0.00 0.68 0.05 0.10
dm-15 0.00 0.00 0.00 35.00 0.00 8224.00 469.94
0.03 0.86 0.00 0.86 0.06 0.20
dm-16 0.00 0.00 0.00 53.00 0.00 12428.00 468.98
0.11 2.04 0.00 2.04 0.17 0.90
dm-17 0.00 0.00 0.00 43.00 0.00 8344.00 388.09
0.09 2.14 0.00 2.14 0.42 1.80
dm-18 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00 0.00
dm-19 0.00 0.00 0.00 75.00 0.00 16824.00 448.64
0.08 1.11 0.00 1.11 0.08 0.60
dm-20 0.00 0.00 0.00 70.00 0.00 16452.00 470.06
0.06 0.90 0.00 0.90 0.09 0.60
dm-21 0.00 0.00 0.00 18.00 0.00 4112.00 456.89
0.02 1.00 0.00 1.00 0.11 0.20
dm-22 0.00 0.00 0.00 53.00 0.00 12324.00 465.06
0.06 0.70 0.00 0.70 0.08 0.40
dm-24 0.00 0.00 0.00 18.00 0.00 4272.00 474.67
0.02 1.06 0.00 1.06 0.17 0.30
dm-25 0.00 0.00 0.00 74.00 0.00 16916.00 457.19
0.09 1.26 0.00 1.26 0.18 1.30
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz
avgqu-sz await r_await w_await svctm %util
dm-14 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00 0.00
dm-15 0.00 0.00 0.00 17.00 0.00 4108.00 483.29
0.02 1.00 0.00 1.00 0.06 0.10
dm-16 0.00 0.00 0.00 34.00 0.00 8208.00 482.82
0.03 1.00 0.00 1.00 0.06 0.20
dm-17 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00 0.00
dm-18 0.00 0.00 0.00 36.00 0.00 8220.00 456.67
0.05 1.33 0.00 1.33 0.08 0.30
dm-19 0.00 0.00 0.00 1.00 0.00 8.00 16.00
0.00 0.00 0.00 0.00 0.00 0.00
dm-20 0.00 0.00 0.00 36.00 0.00 8288.00 460.44
0.05 1.42 0.00 1.42 0.08 0.30
dm-21 0.00 0.00 0.00 34.00 0.00 8208.00 482.82
0.03 1.00 0.00 1.00 0.06 0.20
dm-22 0.00 0.00 0.00 18.00 0.00 4128.00 458.67
0.04 3.22 0.00 3.22 0.17 0.30
dm-24 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00 0.00
dm-25 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00 0.00
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz
avgqu-sz await r_await w_await svctm %util
dm-14 0.00 0.00 0.00 20.00 0.00 4032.00 403.20
0.00 0.00 0.00 0.00 0.00 0.00
dm-15 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00 0.00
dm-16 0.00 0.00 0.00 1.00 0.00 20.00 40.00
0.00 0.00 0.00 0.00 0.00 0.00
dm-17 0.00 0.00 0.00 4.00 0.00 28.00 14.00
0.00 0.00 0.00 0.00 0.00 0.00
dm-18 0.00 0.00 0.00 3.00 0.00 36.00 24.00
0.00 0.00 0.00 0.00 0.00 0.00
dm-19 0.00 0.00 0.00 2.00 0.00 20.00 20.00
0.01 2.50 0.00 2.50 2.50 0.50
dm-20 0.00 0.00 0.00 6.00 0.00 96.00 32.00
0.02 3.33 0.00 3.33 2.00 1.20
dm-21 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00 0.00
dm-22 0.00 0.00 0.00 2.00 0.00 32.00 32.00
0.00 0.00 0.00 0.00 0.00 0.00
dm-24 0.00 0.00 0.00 22.00 0.00 4184.00 380.36
0.10 4.59 0.00 4.59 0.95 2.10
dm-25 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00 0.00
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz
avgqu-sz await r_await w_await svctm %util
dm-14 0.00 0.00 0.00 8.00 0.00 1928.00 482.00
0.01 1.00 0.00 1.00 0.12 0.10
dm-15 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00 0.00
dm-16 0.00 0.00 0.00 3.00 0.00 312.00 208.00
0.00 0.33 0.00 0.33 0.33 0.10
dm-17 0.00 0.00 0.00 18.00 0.00 4264.00 473.78
0.03 1.67 0.00 1.67 0.11 0.20
dm-18 0.00 0.00 0.00 17.00 0.00 4104.00 482.82
0.03 1.82 0.00 1.82 0.12 0.20
dm-19 0.00 0.00 0.00 18.00 0.00 4112.00 456.89
0.02 1.06 0.00 1.06 0.11 0.20
dm-20 0.00 0.00 0.00 32.00 0.00 4308.00 269.25
0.03 0.81 0.00 0.81 0.34 1.10
dm-21 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00 0.00
dm-22 0.00 0.00 0.00 8.00 0.00 540.00 135.00
0.00 0.00 0.00 0.00 0.00 0.00
dm-24 0.00 0.00 0.00 35.00 0.00 8228.00 470.17
0.03 0.97 0.00 0.97 0.06 0.20
dm-25 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00 0.00
On 11/17/2018 5:27 PM, John Petrini wrote:
The iostat isn't very helpful because there are not many writes. I'd recommend disabling cstates entirely, not sure it's your problem but it's good practice and if your cluster goes as idle as your
iostat suggests it could be the culprit.
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
