Hi again, On my testbed, I have 5 ceph nodes, each containing 23 OSDs (2TB btrfs drives). For these tests, I've setup a RAID0 on the 23 disks. For now, I'm not using SSDs as I discovered my vendor apparently decreased their perfs on purpose...
So : 5 server nodes of which 3 are MONS too. I also have 5 clients. All of them have a single 10G NIC, I'm not using a private network. I'm testing EC pools, with the failure domain set to hosts. The EC pool k/m is set to k=4/m=1 I'm testing EC pools using the giant release (ceph-0.87.1-0.el7.centos.x86_64) And... I just found out I had "limited" read performance. While I was watching the stats using dstat on one server node, I noticed that during the rados (read) bench, all the server nodes sent about 370MiB/s on the network, which is the average speed I get per server, but they also all received about 750-800MiB/s on that same network. And 800MB/s is about as much as you can get on a 10G link... I'm trying to understand why I see this inbound data flow ? - Why does a server node receive data at all during a read bench ? - Why is it about twice as much as the data the node is sending ? - Is this about verifying data integrity at read time ? I'm alone on the cluster, it's not used anywhere else. I will try tomorrow to see if adding a 2nd 10G port (with a private network this time) improves the performance, but I'm really curious here to understand what's the bottleneck and what's ceph doing... ? Looking at the write performance, I see the same kind of behavior : nodes send about half the amount of data they receive (600MB/300MB), but this might be because this time the client only sends the real data and the erasure coding happens behind the scenes (or not ?) Any idea ? Regards Frederic
_______________________________________________ ceph-users mailing list [email protected] http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
