Hello list,
Just curious if anyone has ever seen this behavior and might have some
ideas on how to troubleshoot it.
We're seeing very high iowait in iostat across all OSD's in on a single OSD
host. It's very spiky - dropping to zero and then shooting up to as high as
400 in some cases. Despite this it does not seem to be having a major
impact on the cluster performance as a whole.
Some more details:
3x OSD Nodes - Dell R730's: 24 cores @2.6GHz, 256GB RAM, 20x 1.2TB 10K SAS
OSD's per node.
We're running ceph hammer.
Here's the output of iostat. Note that this is from a period when the
cluster is not very busy but you can still see high spikes on a few OSD's.
It's much worse during high load.
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz
avgqu-sz await r_await w_await svctm %util
sda 0.00 0.00 0.00 0.50 0.00 6.00 24.00
0.00 8.00 0.00 8.00 8.00 0.40
sdb 0.00 0.00 0.00 60.00 0.00 808.00 26.93
0.00 0.07 0.00 0.07 0.03 0.20
sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00 0.00
sdd 0.00 0.00 0.00 67.00 0.00 1010.00 30.15
0.01 0.09 0.00 0.09 0.09 0.60
sde 0.00 0.00 0.00 93.00 0.00 868.00 18.67
0.00 0.04 0.00 0.04 0.04 0.40
sdf 0.00 0.00 0.00 57.50 0.00 572.00 19.90
0.00 0.03 0.00 0.03 0.03 0.20
sdg 0.00 1.00 0.00 3.50 0.00 22.00 12.57
0.75 16.00 0.00 16.00 2.86 1.00
sdh 0.00 0.00 1.50 25.50 6.00 458.50 34.41
2.03 75.26 0.00 79.69 3.04 8.20
sdi 0.00 0.00 0.00 30.50 0.00 384.50 25.21
2.36 77.51 0.00 77.51 3.28 10.00
sdj 0.00 1.00 1.50 105.00 6.00 925.75 17.50
10.85 101.84 8.00 103.18 2.35 25.00
sdl 0.00 0.00 2.00 0.00 320.00 0.00 320.00
0.01 3.00 3.00 0.00 2.00 0.40
sdk 0.00 1.00 0.00 55.00 0.00 334.50 12.16
7.92 136.91 0.00 136.91 2.51 13.80
sdm 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00 0.00
sdn 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00 0.00
sdo 0.00 0.00 1.00 0.00 4.00 0.00 8.00
0.00 4.00 4.00 0.00 4.00 0.40
sdp 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00 0.00
sdq 0.50 0.00 756.00 0.00 93288.00 0.00 246.79
1.47 1.95 1.95 0.00 1.17 88.60
sdr 0.00 0.00 1.00 0.00 4.00 0.00 8.00
0.00 4.00 4.00 0.00 4.00 0.40
sds 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00 0.00
sdt 0.00 0.00 0.00 36.50 0.00 643.50 35.26
3.49 95.73 0.00 95.73 2.63 9.60
sdu 0.00 0.00 0.00 21.00 0.00 323.25 30.79
0.78 37.24 0.00 37.24 2.95 6.20
sdv 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00 0.00
sdw 0.00 0.00 0.00 31.00 0.00 689.50 44.48
2.48 80.06 0.00 80.06 3.29 10.20
sdx 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00 0.00
dm-0 0.00 0.00 0.00 0.50 0.00 6.00 24.00
0.00 8.00 0.00 8.00 8.00 0.40
dm-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00 0.00
_______________________________________________
ceph-users mailing list
[email protected]
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
