On Fri, Jan 18, 2019 at 2:12 PM <[email protected]> wrote: > Hi. > > We have the intention of using CephFS for some of our shares, which we'd > like to spool to tape as a part normal backup schedule. CephFS works nice > for large files but for "small" .. < 0.1MB .. there seem to be a > "overhead" on 20-40ms per file. I tested like this: > > root@abe:/nfs/home/jk# time cat /ceph/cluster/rsyncbackups/13kbfile > > /dev/null > > real 0m0.034s > user 0m0.001s > sys 0m0.000s > > And from local page-cache right after. > root@abe:/nfs/home/jk# time cat /ceph/cluster/rsyncbackups/13kbfile > > /dev/null > > real 0m0.002s > user 0m0.002s > sys 0m0.000s > > Giving a ~20ms overhead in a single file. > > This is about x3 higher than on our local filesystems (xfs) based on > same spindles. > > CephFS metadata is on SSD - everything else on big-slow HDD's (in both > cases). > > Is this what everyone else see? >
Pretty much. Reading a file from a pool of Filestore spinners: # time cat 13kb > /dev/null real 0m0.013s user 0m0.000s sys 0m0.003s That's after dropping the caches on the client however the file would have still been in the page cache on the OSD nodes as I just created it. If the file was coming straight off the spinners I'd expect to see something closer to your time. I guess if you wanted to improve the latency you would be looking at the usual stuff e.g (off the top of my head): - Faster network links/tuning your network - Turning down Ceph debugging - Trying a different striping layout on the dirs with the small files (unlikely to have much affect) - If you're using fuse mount try Kernel mount (or maybe vice versa) - Play with mount options - Tune CPU on MDS node Still even with all of that unlikely you'll get to local file-system performance, as Burkhard says you have the locking overhead. You'll probably need to look at getting more parallelism going in your rsyncs. > > Thanks > > -- > Jesper > > _______________________________________________ > ceph-users mailing list > [email protected] > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >
_______________________________________________ ceph-users mailing list [email protected] http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
