On Mon, Nov 4, 2019 at 11:44 PM Hermann Himmelbauer <herm...@qwer.tk> wrote: > > Hi, > I recently upgraded my 3-node cluster to proxmox 6 / debian-10 and > recreated my ceph cluster with a new release (14.2.4 bluestore) - > basically hoping to gain some I/O speed. > > The installation went flawlessly, reading is faster than before (~ 80 > MB/s), however, the write speed is still really slow (~ 3,5 MB/s). > > I wonder if I can do anything to speed things up? > > My Hardware is as the following: > > 3 Nodes with Supermicro X8DTT-HIBQF Mainboard each, > 2 OSD per node (2TB SATA harddisks, WDC WD2000F9YZ-0), > interconnected via Infiniband 40 > > The network should be reasonably fast, I measure ~ 16 GBit/s with iperf, > so this seems fine. > > I use ceph for RBD only, so my measurement is simply doing a very simple > "dd" read and write test within a virtual machine (Debian 8) like the > following: > > read: > dd if=/dev/vdb | pv | dd of=/dev/null > -> 80 MB/s > > > write: > dd if=/dev/zero | pv | dd of=/dev/vdb > -> 3.5 MB/s
you are mainly measuring latency, not bandwidth here. Use a larger block size (bs=4M) to measure bandwidth. Using /dev/zero as source can also be a bad benchmark if you are running with detect-zeroes=unmap (which is the proxmox default IIRC) Paul > > When I do the same on the virtual machine on a disk that is on a NFS > storage, I get something about 30 MB/s for reading and writing. > > If I disable the write cache on all OSD disks via "hdparm -W 0 > /dev/sdX", I gain a little bit of performance, write speed is then 4.3 MB/s. > > Thanks to your help from the list I plan to install a second ceph > cluster which is SSD based (Samsung PM1725b) which should be much > faster, however, I still wonder if there is any way to speed up my > harddisk based cluster? > > Thank you in advance for any help, > > Best Regards, > Hermann > > > -- > herm...@qwer.tk > PGP/GPG: 299893C7 (on keyservers) > _______________________________________________ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-le...@ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-le...@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
