On Mon, Sep 4, 2017 at 4:27 PM, <[email protected]> wrote: > Hello! > > I'm validating IO performance of CephFS vs. NFS. > > Therefore I have mounted the relevant filesystems on the same client. > Then I start fio with the following parameters: > action = randwrite randrw > blocksize = 4k 128k 8m > rwmixreadread = 70 50 30 > 32 jobs run in parallel > > The NFS share is striping over 5 virtual disks with a 4+1 RAID5 > configuration; each disk has ~8TB. > The CephFS is configured on 2 MDS servers (1 up:active, 1 up:standby); > each MDS has 47 OSDs where 1 OSD is represented by single 8TB disk. > (The disks of RAID5 and OSD are identical.) >
So that's a 2 node cluster? I'm assuming filestore OSDs with journals on the OSDs, 2x or 3x replication. The NFS server on local storage is going to perform much better as you've found. > > What I can see is that the IO performance of blocksize 8m is slightly > better with CephFS, but worse (by factor 4-10) with blocksize 4k / 128k. Not surprising. You can > Here the stats for randrw with mix 30: > ld9930:/home # tail -n 3 ld9930-fio-test-cephfs-randrw30-8m > Run status group 0 (all jobs): > READ: bw=335MiB/s (351MB/s), 335MiB/s-335MiB/s (351MB/s-351MB/s), > io=19.7GiB (21.2GB), run=60099-60099msec > WRITE: bw=753MiB/s (789MB/s), 753MiB/s-753MiB/s (789MB/s-789MB/s), > io=44.2GiB (47.5GB), run=60099-60099msec > > ld9930:/home # tail -n 3 ld9930-fio-test-nfs-randrw30-8m > Run status group 0 (all jobs): > READ: bw=324MiB/s (340MB/s), 324MiB/s-324MiB/s (340MB/s-340MB/s), > io=19.0GiB (20.5GB), run=60052-60052msec > WRITE: bw=725MiB/s (760MB/s), 725MiB/s-725MiB/s (760MB/s-760MB/s), > io=42.6GiB (45.7GB), run=60052-60052msec > > ld9930:/home # tail -n 3 ld9930-fio-test-nfs-randrw30-128k > Run status group 0 (all jobs): > READ: bw=287MiB/s (301MB/s), 287MiB/s-287MiB/s (301MB/s-301MB/s), > io=16.9GiB (18.7GB), run=60006-60006msec > WRITE: bw=667MiB/s (700MB/s), 667MiB/s-667MiB/s (700MB/s-700MB/s), > io=39.1GiB (41.1GB), run=60006-60006msec > > ld9930:/home # tail -n 3 ld9930-fio-test-cephfs-randrw30-128k > Run status group 0 (all jobs): > READ: bw=69.2MiB/s (72.6MB/s), 69.2MiB/s-69.2MiB/s (72.6MB/s-72.6MB/s), > io=4172MiB (4375MB), run=60310-60310msec > WRITE: bw=161MiB/s (169MB/s), 161MiB/s-161MiB/s (169MB/s-169MB/s), > io=9732MiB (10.3GB), run=60310-60310msec > > ld9930:/home # tail -n 3 ld9930-fio-test-cephfs-randrw30-4k > Run status group 0 (all jobs): > READ: bw=5631KiB/s (5766kB/s), 5631KiB/s-5631KiB/s (5766kB/s-5766kB/s), > io=330MiB (346MB), run=60043-60043msec > WRITE: bw=12.8MiB/s (13.4MB/s), 12.8MiB/s-12.8MiB/s (13.4MB/s-13.4MB/s), > io=767MiB (804MB), run=60043-60043msec > > ld9930:/home # tail -n 3 ld9930-fio-test-nfs-randrw30-4k > Run status group 0 (all jobs): > READ: bw=77.2MiB/s (80.8MB/s), 77.2MiB/s-77.2MiB/s (80.8MB/s-80.8MB/s), > io=4621MiB (4846MB), run=60004-60004msec > WRITE: bw=180MiB/s (188MB/s), 180MiB/s-180MiB/s (188MB/s-188MB/s), > io=10.6GiB (11.4GB), run=60004-60004msec > > > This implies that for good IO performance only data with blocksize > 128k > (I guess > 1M) should be used. > Can anybody confirm this? > > THX > _______________________________________________ > ceph-users mailing list > [email protected] > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >
_______________________________________________ ceph-users mailing list [email protected] http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
