On 04/16/2014 12:14 PM, Gregory Farnum wrote:
On Wed, Apr 16, 2014 at 8:08 AM, Dan van der Ster
<[email protected]> wrote:
Dear ceph-users,
I've recently started looking through our FileStore logs to better
understand the VM/RBD IO patterns, and noticed something interesting. Here
is a snapshot of the write lengths for one OSD server (with 24 OSDs) -- I've
listed the top 10 write lengths ordered by number of writes in one day:
Writes per length:
4096: 2011442
8192: 438259
4194304: 207293
12288: 175848
16384: 148274
20480: 69050
24576: 58961
32768: 54771
28672: 43627
65536: 34208
49152: 31547
40960: 28075
There were ~4000000 writes to that server on that day, so you see that ~50%
of the writes were 4096 bytes, and then the distribution drops off sharply
before a peak again at 4MB (the object size, i.e. the max write size). (For
those interested, read lengths are below in the P.S.)
I'm trying to understand that distribution, and the best explanation I've
come up with is that these are ext4/xfs metadata updates, probably atime
updates. Based on that theory, I'm going to test noatime on a few VMs and
see if I notice a change in the distribution.
Did anyone already go through such an exercise, or does anyone already
enforce/recommend specific mount options for their clients' RBD volumes? Of
course I realize that noatime is a generally recommended mount option for
"performance", but I've never heard a discussion about noatime specifically
in relation to RBD volumes.
I don't think we have any standard recommendations, but Mark might
have more insight into this than I do. I forget which clients you're
using — is rbd caching enabled? (If it's atime updates that could
definitely still happen with ext4/xfs, but it's a bummer.) Still, it's
good we're getting a bunch of larger writes as well.
Nope, I haven't specifically looked at the difference between
enabling/disabling noatime on the rbd volumes myself. It would be a
good exercise though! Same thing with read_ahead_kb and the IO
elevator, especially in this context. Sounds like a fun project. :D
Mark
_______________________________________________
ceph-users mailing list
[email protected]
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
