Dear ceph-users,
I've recently started looking through our FileStore logs to better
understand the VM/RBD IO patterns, and noticed something interesting.
Here is a snapshot of the write lengths for one OSD server (with 24
OSDs) -- I've listed the top 10 write lengths ordered by number of
writes in one day:
Writes per length:
4096: 2011442
8192: 438259
4194304: 207293
12288: 175848
16384: 148274
20480: 69050
24576: 58961
32768: 54771
28672: 43627
65536: 34208
49152: 31547
40960: 28075
There were ~4000000 writes to that server on that day, so you see that
~50% of the writes were 4096 bytes, and then the distribution drops off
sharply before a peak again at 4MB (the object size, i.e. the max write
size). (For those interested, read lengths are below in the P.S.)
I'm trying to understand that distribution, and the best explanation
I've come up with is that these are ext4/xfs metadata updates, probably
atime updates. Based on that theory, I'm going to test noatime on a few
VMs and see if I notice a change in the distribution.
Did anyone already go through such an exercise, or does anyone already
enforce/recommend specific mount options for their clients' RBD volumes?
Of course I realize that noatime is a generally recommended mount option
for "performance", but I've never heard a discussion about noatime
specifically in relation to RBD volumes.
Best Regards, Dan
P.S. Reads per length:
524288: 1235401
4096: 675012
8192: 488194
516096: 342771
16384: 187577
65536: 87783
131072: 87279
12288: 66735
49152: 50170
24576: 47794
262144: 45199
466944: 23064
So reads are mostly 512kB, which is probably some default read-ahead size.
-- Dan van der Ster || Data & Storage Services || CERN IT Department --
_______________________________________________
ceph-users mailing list
[email protected]
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
