Hi ceph-users,
We are using Ceph (radosgw) to store user generated images, as GET latency is
critical for us, most recently I did some investigation over the GET path to
understand where time spend.
I first confirmed that the latency came from OSD (read op), so that we
instrumented code to trace the GET request (read op at OSD side, to be more
specific, each object with size [512K + 4M * x] are splitted into [1 + x]
chunks, each chunk needs one read op ), for each read op, it needs to go
through the following steps:
1. Dispatch and take by a op thread to process (process not started).
0 – 20 ms, 94%
20 – 50 ms, 2%
50 – 100 ms, 2%
100ms+ , 2%
For those having 20ms+ latency, half of them are due to waiting for pg
lock (https://github.com/ceph/ceph/blob/dumpling/src/osd/OSD.cc#L7089), another
half are yet to be investigated.
2. Get file xattr (‘-‘), which open the file and populate fd cache
(https://github.com/ceph/ceph/blob/dumpling/src/os/FileStore.cc#L230).
0 – 20 ms, 80%
20 – 50 ms, 8%
50 – 100 ms, 7%
100ms+ , 5%
The latency either comes from (from more to less): file path lookup
(https://github.com/ceph/ceph/blob/dumpling/src/os/HashIndex.cc#L294), file
open, or fd cache lookup /add.
Currently objects are store in level 6 or level 7 folder (due to
http://tracker.ceph.com/issues/7207, I stopped folder splitting).
3. Get more xattrs, this is fast due to previous fd cache (rarely > 1ms).
4. Read the data.
0 – 20 ms, 84%
20 – 50 ms, 10%
50 – 100 ms, 4%
100ms+ , 2%
I decreased vfs_cache_pressure from its default value 100 to 5 to make VFS
favor dentry/inode cache over page cache, unfortunately it does not help.
Long story short, most of the long latency read op comes from file system call
(for cold data), as our workload mainly stores objects less than 500KB, so that
it generates a large bunch of objects.
I would like to ask if people experienced similar issue and if there is any
suggestion I can try to boost the GET performance. On the other hand, PUT could
be sacrificed.
Thanks,
Guang
_______________________________________________
ceph-users mailing list
[email protected]
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
