I am not saying controller cache, you should check ssd disk caches.
On Sun, Nov 18, 2018 at 11:40 AM Alex Litvak
<alexander.v.lit...@gmail.com> wrote:
>
> All 3 nodes have this status for SSD mirror. Controller cache is on for all
> 3.
>
> Default Cache Policy: WriteBack, ReadAdaptive, Direct, No Write Cache if Bad
> BBU
> Current Cache Policy: WriteBack, ReadAdaptive, Direct, No Write Cache if Bad
> BBU
>
>
> On 11/18/2018 12:45 AM, Serkan Çoban wrote:
> > Does write cache on SSDs enabled on three servers? Can you check them?
> > On Sun, Nov 18, 2018 at 9:05 AM Alex Litvak
> > <alexander.v.lit...@gmail.com> wrote:
> >>
> >> Raid card for journal disks is Perc H730 (Megaraid), RAID 1, battery back
> >> cache is on
> >>
> >> Default Cache Policy: WriteBack, ReadAdaptive, Direct, No Write Cache if
> >> Bad BBU
> >> Current Cache Policy: WriteBack, ReadAdaptive, Direct, No Write Cache if
> >> Bad BBU
> >>
> >> I have 2 other nodes with older Perc H710 and similar SSDs with slightly
> >> higher wear (6.3% vs 5.18%) but from observation they hardly hit 1.5 ms on
> >> rear occasion
> >> Cache, RAID, and battery situation is the same.
> >>
> >> On 11/17/2018 11:38 PM, Serkan Çoban wrote:
> >>>> 10ms w_await for SSD is too much. How that SSD is connected to the
> >>>> system? Any raid card installed on this system? What is the raid mode?
> >>> On Sun, Nov 18, 2018 at 8:25 AM Alex Litvak
> >>> <alexander.v.lit...@gmail.com> wrote:
> >>>>
> >>>> Here is another snapshot. I wonder if this write io wait is too big
> >>>> Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s
> >>>> avgrq-sz avgqu-sz await r_await w_await svctm %util
> >>>> dm-14 0.00 0.00 0.00 23.00 0.00 336.00
> >>>> 29.22 0.34 14.74 0.00 14.74 2.87 6.60
> >>>> dm-15 0.00 0.00 0.00 16.00 0.00 200.00
> >>>> 25.00 0.01 0.75 0.00 0.75 0.75 1.20
> >>>> dm-16 0.00 0.00 0.00 17.00 0.00 276.00
> >>>> 32.47 0.25 14.94 0.00 14.94 3.35 5.70
> >>>> dm-17 0.00 0.00 0.00 17.00 0.00 252.00
> >>>> 29.65 0.32 18.65 0.00 18.65 4.00 6.80
> >>>> dm-18 0.00 0.00 0.00 15.00 0.00 152.00
> >>>> 20.27 0.25 16.80 0.00 16.80 4.07 6.10
> >>>> dm-19 0.00 0.00 0.00 13.00 0.00 152.00
> >>>> 23.38 0.21 15.92 0.00 15.92 4.85 6.30
> >>>> dm-20 0.00 0.00 0.00 20.00 0.00 248.00
> >>>> 24.80 0.27 13.60 0.00 13.60 3.25 6.50
> >>>> dm-21 0.00 0.00 0.00 17.00 0.00 188.00
> >>>> 22.12 0.27 16.00 0.00 16.00 3.59 6.10
> >>>> dm-22 0.00 0.00 0.00 20.00 0.00 156.00
> >>>> 15.60 0.11 5.55 0.00 5.55 2.95 5.90
> >>>> dm-24 0.00 0.00 0.00 8.00 0.00 56.00
> >>>> 14.00 0.12 14.62 0.00 14.62 4.75 3.80
> >>>> dm-25 0.00 0.00 0.00 19.00 0.00 200.00
> >>>> 21.05 0.21 10.89 0.00 10.89 2.74 5.20
> >>>>
> >>>> Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s
> >>>> avgrq-sz avgqu-sz await r_await w_await svctm %util
> >>>> dm-14 0.00 0.00 0.00 11.00 0.00 136.00
> >>>> 24.73 0.11 9.73 0.00 9.73 1.82 2.00
> >>>> dm-15 0.00 0.00 0.00 12.00 0.00 136.00
> >>>> 22.67 0.04 3.75 0.00 3.75 1.08 1.30
> >>>> dm-16 0.00 0.00 0.00 9.00 0.00 104.00
> >>>> 23.11 0.09 10.44 0.00 10.44 2.44 2.20
> >>>> dm-17 0.00 0.00 0.00 5.00 0.00 160.00
> >>>> 64.00 0.02 4.00 0.00 4.00 4.00 2.00
> >>>> dm-18 0.00 0.00 0.00 5.00 0.00 52.00
> >>>> 20.80 0.03 5.80 0.00 5.80 3.60 1.80
> >>>> dm-19 0.00 0.00 0.00 10.00 0.00 104.00
> >>>> 20.80 0.08 7.90 0.00 7.90 2.10 2.10
> >>>> dm-20 0.00 0.00 0.00 9.00 0.00 132.00
> >>>> 29.33 0.10 11.22 0.00 11.22 2.56 2.30
> >>>> dm-21 0.00 0.00 0.00 6.00 0.00 68.00
> >>>> 22.67 0.07 12.33 0.00 12.33 3.83 2.30
> >>>> dm-22 0.00 0.00 0.00 3.00 0.00 20.00
> >>>> 13.33 0.01 3.67 0.00 3.67 3.67 1.10
> >>>> dm-24 0.00 0.00 0.00 4.00 0.00 24.00
> >>>> 12.00 0.07 18.00 0.00 18.00 5.25 2.10
> >>>> dm-25 0.00 0.00 0.00 6.00 0.00 64.00
> >>>> 21.33 0.06 10.33 0.00 10.33 3.67 2.20
> >>>>
> >>>> Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s
> >>>> avgrq-sz avgqu-sz await r_await w_await svctm %util
> >>>> dm-14 0.00 0.00 0.00 5.00 0.00 140.00
> >>>> 56.00 0.08 15.20 0.00 15.20 5.40 2.70
> >>>> dm-15 0.00 0.00 0.00 6.00 0.00 236.00
> >>>> 78.67 0.18 30.67 0.00 30.67 6.83 4.10
> >>>> dm-16 0.00 0.00 0.00 8.00 0.00 84.00
> >>>> 21.00 0.06 7.25 0.00 7.25 1.62 1.30
> >>>> dm-17 0.00 0.00 0.00 3.00 0.00 84.00
> >>>> 56.00 0.00 0.33 0.00 0.33 0.33 0.10
> >>>> dm-18 0.00 0.00 0.00 2.00 0.00 20.00
> >>>> 20.00 0.02 12.00 0.00 12.00 12.00 2.40
> >>>> dm-19 0.00 0.00 0.00 12.00 0.00 80.00
> >>>> 13.33 0.05 4.00 0.00 4.00 2.33 2.80
> >>>> dm-20 0.00 0.00 0.00 16.00 0.00 256.00
> >>>> 32.00 0.00 0.06 0.00 0.06 0.06 0.10
> >>>> dm-21 0.00 0.00 0.00 8.00 0.00 500.00
> >>>> 125.00 0.00 0.12 0.00 0.12 0.12 0.10
> >>>> dm-22 0.00 0.00 0.00 1.00 0.00 8.00
> >>>> 16.00 0.00 0.00 0.00 0.00 0.00 0.00
> >>>> dm-24 0.00 0.00 0.00 0.00 0.00 0.00
> >>>> 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> >>>> dm-25 0.00 0.00 0.00 2.00 0.00 32.00
> >>>> 32.00 0.08 40.00 0.00 40.00 20.50 4.10
> >>>>
> >>>> Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s
> >>>> avgrq-sz avgqu-sz await r_await w_await svctm %util
> >>>> dm-14 0.00 0.00 0.00 10.00 0.00 108.00
> >>>> 21.60 0.11 10.80 0.00 10.80 1.90 1.90
> >>>> dm-15 0.00 0.00 0.00 5.00 0.00 60.00
> >>>> 24.00 0.03 6.20 0.00 6.20 3.40 1.70
> >>>> dm-16 0.00 0.00 0.00 6.00 0.00 68.00
> >>>> 22.67 0.00 0.17 0.00 0.17 0.17 0.10
> >>>> dm-17 0.00 0.00 0.00 7.00 0.00 100.00
> >>>> 28.57 0.09 12.29 0.00 12.29 2.86 2.00
> >>>> dm-18 0.00 0.00 0.00 5.00 0.00 44.00
> >>>> 17.60 0.07 14.00 0.00 14.00 4.00 2.00
> >>>> dm-19 0.00 0.00 0.00 2.00 0.00 24.00
> >>>> 24.00 0.03 14.00 0.00 14.00 8.50 1.70
> >>>> dm-20 0.00 0.00 0.00 7.00 0.00 96.00
> >>>> 27.43 0.07 9.57 0.00 9.57 2.00 1.40
> >>>> dm-21 0.00 0.00 0.00 7.00 0.00 84.00
> >>>> 24.00 0.12 16.57 0.00 16.57 4.86 3.40
> >>>> dm-22 0.00 0.00 0.00 5.00 0.00 60.00
> >>>> 24.00 0.05 10.20 0.00 10.20 3.40 1.70
> >>>> dm-24 0.00 0.00 0.00 2.00 0.00 16.00
> >>>> 16.00 0.03 17.00 0.00 17.00 9.00 1.80
> >>>> dm-25 0.00 0.00 0.00 4.00 0.00 76.00
> >>>> 38.00 0.05 11.75 0.00 11.75 4.75 1.90
> >>>>
> >>>> Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s
> >>>> avgrq-sz avgqu-sz await r_await w_await svctm %util
> >>>> dm-14 0.00 0.00 0.00 9.00 0.00 104.00
> >>>> 23.11 0.14 15.33 0.00 15.33 4.33 3.90
> >>>> dm-15 0.00 0.00 0.00 11.00 0.00 128.00
> >>>> 23.27 0.01 0.82 0.00 0.82 0.82 0.90
> >>>> dm-16 0.00 0.00 0.00 10.00 0.00 120.00
> >>>> 24.00 0.09 9.30 0.00 9.30 3.30 3.30
> >>>> dm-17 0.00 0.00 0.00 5.00 0.00 72.00
> >>>> 28.80 0.04 8.20 0.00 8.20 8.20 4.10
> >>>> dm-18 0.00 0.00 0.00 7.00 0.00 68.00
> >>>> 19.43 0.08 11.57 0.00 11.57 6.29 4.40
> >>>> dm-19 0.00 0.00 0.00 7.00 0.00 76.00
> >>>> 21.71 0.10 13.86 0.00 13.86 6.29 4.40
> >>>> dm-20 0.00 0.00 0.00 15.00 0.00 176.00
> >>>> 23.47 0.12 7.87 0.00 7.87 1.60 2.40
> >>>> dm-21 0.00 0.00 0.00 9.00 0.00 112.00
> >>>> 24.89 0.17 18.56 0.00 18.56 6.56 5.90
> >>>> dm-22 0.00 0.00 0.00 4.00 0.00 56.00
> >>>> 28.00 0.04 10.75 0.00 10.75 7.50 3.00
> >>>> dm-24 0.00 0.00 0.00 5.00 0.00 532.00
> >>>> 212.80 0.02 4.00 0.00 4.00 3.60 1.80
> >>>> dm-25 0.00 0.00 0.00 6.00 0.00 60.00
> >>>> 20.00 0.11 19.00 0.00 19.00 7.17 4.30
> >>>>
> >>>> Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s
> >>>> avgrq-sz avgqu-sz await r_await w_await svctm %util
> >>>> dm-14 0.00 0.00 0.00 9.00 0.00 112.00
> >>>> 24.89 0.30 33.22 0.00 33.22 4.33 3.90
> >>>> dm-15 0.00 0.00 0.00 8.00 0.00 96.00
> >>>> 24.00 0.08 9.88 0.00 9.88 2.62 2.10
> >>>> dm-16 0.00 0.00 0.00 5.00 0.00 92.00
> >>>> 36.80 0.01 1.80 0.00 1.80 1.80 0.90
> >>>> dm-17 0.00 0.00 0.00 4.00 0.00 104.00
> >>>> 52.00 0.04 10.00 0.00 10.00 10.00 4.00
> >>>> dm-18 0.00 0.00 0.00 9.00 0.00 92.00
> >>>> 20.44 0.11 12.00 0.00 12.00 4.44 4.00
> >>>> dm-19 0.00 0.00 0.00 7.00 0.00 64.00
> >>>> 18.29 0.10 14.43 0.00 14.43 4.00 2.80
> >>>> dm-20 0.00 0.00 0.00 7.00 0.00 84.00
> >>>> 24.00 0.10 14.00 0.00 14.00 3.71 2.60
> >>>> dm-21 0.00 0.00 0.00 2.00 0.00 16.00
> >>>> 16.00 0.08 39.50 0.00 39.50 20.50 4.10
> >>>> dm-22 0.00 0.00 0.00 5.00 0.00 36.00
> >>>> 14.40 0.00 0.00 0.00 0.00 0.00 0.00
> >>>> dm-24 0.00 0.00 0.00 5.00 0.00 52.00
> >>>> 20.80 0.09 17.20 0.00 17.20 5.40 2.70
> >>>> dm-25 0.00 0.00 0.00 6.00 0.00 48.00
> >>>> 16.00 0.10 16.00 0.00 16.00 5.50 3.30
> >>>>
> >>>> Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s
> >>>> avgrq-sz avgqu-sz await r_await w_await svctm %util
> >>>> dm-14 0.00 0.00 0.00 5.00 0.00 48.00
> >>>> 19.20 0.00 0.00 0.00 0.00 0.00 0.00
> >>>> dm-15 0.00 0.00 0.00 5.00 0.00 44.00
> >>>> 17.60 0.01 3.00 0.00 3.00 3.00 1.50
> >>>> dm-16 0.00 0.00 0.00 3.00 0.00 32.00
> >>>> 21.33 0.00 0.00 0.00 0.00 0.00 0.00
> >>>> dm-17 0.00 0.00 0.00 4.00 0.00 40.00
> >>>> 20.00 0.00 0.00 0.00 0.00 0.00 0.00
> >>>> dm-18 0.00 0.00 0.00 3.00 0.00 32.00
> >>>> 21.33 0.04 13.00 0.00 13.00 6.67 2.00
> >>>> dm-19 0.00 0.00 0.00 3.00 0.00 28.00
> >>>> 18.67 0.00 0.00 0.00 0.00 0.00 0.00
> >>>> dm-20 0.00 0.00 0.00 6.00 0.00 64.00
> >>>> 21.33 0.11 17.67 0.00 17.67 3.17 1.90
> >>>> dm-21 0.00 0.00 0.00 1.00 0.00 12.00
> >>>> 24.00 0.00 0.00 0.00 0.00 0.00 0.00
> >>>> dm-22 0.00 0.00 0.00 1.00 0.00 12.00
> >>>> 24.00 0.00 0.00 0.00 0.00 0.00 0.00
> >>>> dm-24 0.00 0.00 0.00 3.00 0.00 20.00
> >>>> 13.33 0.00 0.33 0.00 0.33 0.33 0.10
> >>>> dm-25 0.00 0.00 0.00 6.00 0.00 52.00
> >>>> 17.33 0.00 0.00 0.00 0.00 0.00 0.00
> >>>>
> >>>> Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s
> >>>> avgrq-sz avgqu-sz await r_await w_await svctm %util
> >>>> dm-14 0.00 0.00 0.00 18.00 0.00 236.00
> >>>> 26.22 0.42 23.28 0.00 23.28 3.72 6.70
> >>>> dm-15 0.00 0.00 0.00 15.00 0.00 240.00
> >>>> 32.00 0.12 8.27 0.00 8.27 2.00 3.00
> >>>> dm-16 0.00 0.00 0.00 10.00 0.00 120.00
> >>>> 24.00 0.23 22.80 0.00 22.80 6.50 6.50
> >>>> dm-17 0.00 0.00 0.00 6.00 0.00 68.00
> >>>> 22.67 0.06 9.83 0.00 9.83 5.00 3.00
> >>>> dm-18 0.00 0.00 0.00 12.00 0.00 152.00
> >>>> 25.33 0.10 8.17 0.00 8.17 3.17 3.80
> >>>> dm-19 0.00 0.00 0.00 12.00 0.00 136.00
> >>>> 22.67 0.14 11.58 0.00 11.58 3.42 4.10
> >>>> dm-20 0.00 0.00 0.00 19.00 0.00 296.00
> >>>> 31.16 0.20 10.63 0.00 10.63 1.84 3.50
> >>>> dm-21 0.00 0.00 0.00 14.00 0.00 132.00
> >>>> 18.86 0.17 12.14 0.00 12.14 3.71 5.20
> >>>> dm-22 0.00 0.00 0.00 10.00 0.00 84.00
> >>>> 16.80 0.12 12.00 0.00 12.00 6.60 6.60
> >>>> dm-24 0.00 0.00 0.00 5.00 0.00 40.00
> >>>> 16.00 0.10 19.40 0.00 19.40 9.80 4.90
> >>>> dm-25 0.00 0.00 0.00 9.00 0.00 152.00
> >>>> 33.78 0.08 9.33 0.00 9.33 3.44 3.10
> >>>>
> >>>> Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s
> >>>> avgrq-sz avgqu-sz await r_await w_await svctm %util
> >>>> dm-14 0.00 0.00 0.00 8.00 0.00 112.00
> >>>> 28.00 0.08 9.75 0.00 9.75 3.88 3.10
> >>>> dm-15 0.00 0.00 0.00 14.00 0.00 156.00
> >>>> 22.29 0.13 9.43 0.00 9.43 2.71 3.80
> >>>> dm-16 0.00 0.00 0.00 12.00 0.00 132.00
> >>>> 22.00 0.11 8.83 0.00 8.83 2.17 2.60
> >>>> dm-17 0.00 0.00 0.00 8.00 0.00 116.00
> >>>> 29.00 0.08 10.12 0.00 10.12 4.25 3.40
> >>>> dm-18 0.00 0.00 0.00 8.00 0.00 100.00
> >>>> 25.00 0.08 9.62 0.00 9.62 5.00 4.00
> >>>> dm-19 0.00 0.00 0.00 20.00 0.00 212.00
> >>>> 21.20 0.25 12.75 0.00 12.75 2.60 5.20
> >>>> dm-20 0.00 0.00 0.00 16.00 0.00 168.00
> >>>> 21.00 0.04 2.69 0.00 2.69 0.81 1.30
> >>>> dm-21 0.00 0.00 0.00 5.00 0.00 48.00
> >>>> 19.20 0.05 10.80 0.00 10.80 7.80 3.90
> >>>> dm-22 0.00 0.00 0.00 5.00 0.00 52.00
> >>>> 20.80 0.03 6.80 0.00 6.80 3.40 1.70
> >>>> dm-24 0.00 0.00 0.00 8.00 0.00 60.00
> >>>> 15.00 0.09 11.75 0.00 11.75 2.25 1.80
> >>>> dm-25 0.00 0.00 0.00 12.00 0.00 160.00
> >>>> 26.67 0.06 5.08 0.00 5.08 1.25 1.50
> >>>>
> >>>> On 11/17/2018 10:19 PM, Alex Litvak wrote:
> >>>>> I stand corrected, I looked at the device iostat, but it was
> >>>>> partitioned. Here is a more correct picture of what is going on now.
> >>>>>
> >>>>> Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s
> >>>>> avgrq-sz avgqu-sz await r_await w_await svctm %util
> >>>>> dm-14 0.00 0.00 0.00 19.00 0.00 4116.00
> >>>>> 433.26 0.01 0.68 0.00 0.68 0.05 0.10
> >>>>> dm-15 0.00 0.00 0.00 35.00 0.00 8224.00
> >>>>> 469.94 0.03 0.86 0.00 0.86 0.06 0.20
> >>>>> dm-16 0.00 0.00 0.00 53.00 0.00 12428.00
> >>>>> 468.98 0.11 2.04 0.00 2.04 0.17 0.90
> >>>>> dm-17 0.00 0.00 0.00 43.00 0.00 8344.00
> >>>>> 388.09 0.09 2.14 0.00 2.14 0.42 1.80
> >>>>> dm-18 0.00 0.00 0.00 0.00 0.00 0.00
> >>>>> 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> >>>>> dm-19 0.00 0.00 0.00 75.00 0.00 16824.00
> >>>>> 448.64 0.08 1.11 0.00 1.11 0.08 0.60
> >>>>> dm-20 0.00 0.00 0.00 70.00 0.00 16452.00
> >>>>> 470.06 0.06 0.90 0.00 0.90 0.09 0.60
> >>>>> dm-21 0.00 0.00 0.00 18.00 0.00 4112.00
> >>>>> 456.89 0.02 1.00 0.00 1.00 0.11 0.20
> >>>>> dm-22 0.00 0.00 0.00 53.00 0.00 12324.00
> >>>>> 465.06 0.06 0.70 0.00 0.70 0.08 0.40
> >>>>> dm-24 0.00 0.00 0.00 18.00 0.00 4272.00
> >>>>> 474.67 0.02 1.06 0.00 1.06 0.17 0.30
> >>>>> dm-25 0.00 0.00 0.00 74.00 0.00 16916.00
> >>>>> 457.19 0.09 1.26 0.00 1.26 0.18 1.30
> >>>>>
> >>>>> Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s
> >>>>> avgrq-sz avgqu-sz await r_await w_await svctm %util
> >>>>> dm-14 0.00 0.00 0.00 0.00 0.00 0.00
> >>>>> 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> >>>>> dm-15 0.00 0.00 0.00 17.00 0.00 4108.00
> >>>>> 483.29 0.02 1.00 0.00 1.00 0.06 0.10
> >>>>> dm-16 0.00 0.00 0.00 34.00 0.00 8208.00
> >>>>> 482.82 0.03 1.00 0.00 1.00 0.06 0.20
> >>>>> dm-17 0.00 0.00 0.00 0.00 0.00 0.00
> >>>>> 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> >>>>> dm-18 0.00 0.00 0.00 36.00 0.00 8220.00
> >>>>> 456.67 0.05 1.33 0.00 1.33 0.08 0.30
> >>>>> dm-19 0.00 0.00 0.00 1.00 0.00 8.00
> >>>>> 16.00 0.00 0.00 0.00 0.00 0.00 0.00
> >>>>> dm-20 0.00 0.00 0.00 36.00 0.00 8288.00
> >>>>> 460.44 0.05 1.42 0.00 1.42 0.08 0.30
> >>>>> dm-21 0.00 0.00 0.00 34.00 0.00 8208.00
> >>>>> 482.82 0.03 1.00 0.00 1.00 0.06 0.20
> >>>>> dm-22 0.00 0.00 0.00 18.00 0.00 4128.00
> >>>>> 458.67 0.04 3.22 0.00 3.22 0.17 0.30
> >>>>> dm-24 0.00 0.00 0.00 0.00 0.00 0.00
> >>>>> 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> >>>>> dm-25 0.00 0.00 0.00 0.00 0.00 0.00
> >>>>> 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> >>>>>
> >>>>> Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s
> >>>>> avgrq-sz avgqu-sz await r_await w_await svctm %util
> >>>>> dm-14 0.00 0.00 0.00 20.00 0.00 4032.00
> >>>>> 403.20 0.00 0.00 0.00 0.00 0.00 0.00
> >>>>> dm-15 0.00 0.00 0.00 0.00 0.00 0.00
> >>>>> 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> >>>>> dm-16 0.00 0.00 0.00 1.00 0.00 20.00
> >>>>> 40.00 0.00 0.00 0.00 0.00 0.00 0.00
> >>>>> dm-17 0.00 0.00 0.00 4.00 0.00 28.00
> >>>>> 14.00 0.00 0.00 0.00 0.00 0.00 0.00
> >>>>> dm-18 0.00 0.00 0.00 3.00 0.00 36.00
> >>>>> 24.00 0.00 0.00 0.00 0.00 0.00 0.00
> >>>>> dm-19 0.00 0.00 0.00 2.00 0.00 20.00
> >>>>> 20.00 0.01 2.50 0.00 2.50 2.50 0.50
> >>>>> dm-20 0.00 0.00 0.00 6.00 0.00 96.00
> >>>>> 32.00 0.02 3.33 0.00 3.33 2.00 1.20
> >>>>> dm-21 0.00 0.00 0.00 0.00 0.00 0.00
> >>>>> 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> >>>>> dm-22 0.00 0.00 0.00 2.00 0.00 32.00
> >>>>> 32.00 0.00 0.00 0.00 0.00 0.00 0.00
> >>>>> dm-24 0.00 0.00 0.00 22.00 0.00 4184.00
> >>>>> 380.36 0.10 4.59 0.00 4.59 0.95 2.10
> >>>>> dm-25 0.00 0.00 0.00 0.00 0.00 0.00
> >>>>> 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> >>>>>
> >>>>> Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s
> >>>>> avgrq-sz avgqu-sz await r_await w_await svctm %util
> >>>>> dm-14 0.00 0.00 0.00 8.00 0.00 1928.00
> >>>>> 482.00 0.01 1.00 0.00 1.00 0.12 0.10
> >>>>> dm-15 0.00 0.00 0.00 0.00 0.00 0.00
> >>>>> 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> >>>>> dm-16 0.00 0.00 0.00 3.00 0.00 312.00
> >>>>> 208.00 0.00 0.33 0.00 0.33 0.33 0.10
> >>>>> dm-17 0.00 0.00 0.00 18.00 0.00 4264.00
> >>>>> 473.78 0.03 1.67 0.00 1.67 0.11 0.20
> >>>>> dm-18 0.00 0.00 0.00 17.00 0.00 4104.00
> >>>>> 482.82 0.03 1.82 0.00 1.82 0.12 0.20
> >>>>> dm-19 0.00 0.00 0.00 18.00 0.00 4112.00
> >>>>> 456.89 0.02 1.06 0.00 1.06 0.11 0.20
> >>>>> dm-20 0.00 0.00 0.00 32.00 0.00 4308.00
> >>>>> 269.25 0.03 0.81 0.00 0.81 0.34 1.10
> >>>>> dm-21 0.00 0.00 0.00 0.00 0.00 0.00
> >>>>> 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> >>>>> dm-22 0.00 0.00 0.00 8.00 0.00 540.00
> >>>>> 135.00 0.00 0.00 0.00 0.00 0.00 0.00
> >>>>> dm-24 0.00 0.00 0.00 35.00 0.00 8228.00
> >>>>> 470.17 0.03 0.97 0.00 0.97 0.06 0.20
> >>>>> dm-25 0.00 0.00 0.00 0.00 0.00 0.00
> >>>>> 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> >>>>>
> >>>>>
> >>>>>
> >>>>> On 11/17/2018 5:27 PM, John Petrini wrote:
> >>>>>> The iostat isn't very helpful because there are not many writes. I'd
> >>>>>> recommend disabling cstates entirely, not sure it's your problem but
> >>>>>> it's good practice and if your cluster goes as idle as your
> >>>>>> iostat suggests it could be the culprit.
> >>>>>>
> >>>>>>
> >>>>>> _______________________________________________
> >>>>>> ceph-users mailing list
> >>>>>> ceph-users@lists.ceph.com
> >>>>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> >>>>>>
> >>>>
> >>>>
> >>>> _______________________________________________
> >>>> ceph-users mailing list
> >>>> ceph-users@lists.ceph.com
> >>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> >>
> >>
> >> _______________________________________________
> >> ceph-users mailing list
> >> ceph-users@lists.ceph.com
> >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> > _______________________________________________
> > ceph-users mailing list
> > ceph-users@lists.ceph.com
> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> >
>
>
> _______________________________________________
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
