The general advice floating around is that your want CPUs with high clock speeds rather than more cores to reduce latency and increase IOPS for SSD setups (see also http://www.sys-pro.co.uk/ceph-storage-fast-cpus-ssd-performance/) So something like a E5-2667V4 might bring better results in that situation. Also there was some talk about disabling the processor C states in order to bring latency down (something like this should be easy to test: https://stackoverflow.com/a/22482722/220986)
On Sat, Jun 24, 2017 at 1:28 AM, Kostas Paraskevopoulos < [email protected]> wrote: > Hello, > > We are in the process of evaluating the performance of a testing > cluster (3 nodes) with ceph jewel. Our setup consists of: > 3 monitors (VMs) > 2 physical servers each connected with 1 JBOD running Ubuntu Server 16.04 > > Each server has 32 threads @2.1GHz and 128GB RAM. > The disk distribution per server is: > 38 * HUS726020ALS210 (SAS rotational) > 2 * HUSMH8010BSS200 (SAS SSD for journals) > 2 * ST1920FM0043 (SAS SSD for data) > 1 * INTEL SSDPEDME012T4 (NVME measured with fio ~300K iops) > > Since we don't currently have a 10Gbit switch, we test the performance > with the cluster in a degraded state, the noout flag set and we mount > rbd images on the powered on osd node. We confirmed that the network > is not saturated during the tests. > > We ran tests on the NVME disk and the pool created on this disk where > we hoped to get the most performance without getting limited by the > hardware specs since we have more disks than CPU threads. > > The nvme disk was at first partitioned with one partition and the > journal on the same disk. The performance on random 4K reads was > topped at 50K iops. We then removed the osd and partitioned with 4 > data partitions and 4 journals on the same disk. The performance > didn't increase significantly. Also, since we run read tests, the > journals shouldn't cause performance issues. > > We then ran 4 fio processes in parallel on the same rbd mounted image > and the total iops reached 100K. More parallel fio processes didn't > increase the measured iops. > > Our ceph.conf is pretty basic (debug is set to 0/0 for everything) and > the crushmap just defines the different buckets/rules for the disk > separation (rotational, ssd, nvme) in order to create the required > pools > > Is the performance of 100.000 iops for random 4K read normal for a > disk that on the same benchmark runs at more than 300K iops on the > same hardware or are we missing something? > > Best regards, > Kostas > _______________________________________________ > ceph-users mailing list > [email protected] > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >
_______________________________________________ ceph-users mailing list [email protected] http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
