Hello, in between I learned a lot from this group (thanks a lot) to solve many performance problems I initially faced with proxmox in VMs having their storage on CEPH rbds.
I parallelized access to many disks on a vm where possible, used iothreads and activated writeback cache. Running a bonnie++ I am now able to get about 300Mbytes/sec block write performance, which is a great value because it even scales out with ceph if I run the same bonnie++ on eg two machines. In this case I get 600MBytes/sec. Great. The last strangeness I am experiencing is read performance. The same bonnie on a VMs xfs_filesystem that yields 300MB write performance only gets a block read of about 90MB/sec. So on one of the pxa-hosts and later also on one of the ceph cluster nodes (nautilus 14.2.8, 144OSDs) I ran a rados bench test to see if ceph is slowing down reads. The results on both systems were very similar. So here is the test result from the pxa-host: # rados bench -p my-rbd 60 write --no-cleanup Total time run: 60.284332 Total writes made: 5376 Write size: 4194304 Object size: 4194304 Bandwidth (MB/sec): 356.71 Stddev Bandwidth: 46.8361 Max bandwidth (MB/sec): 424 Min bandwidth (MB/sec): 160 Average IOPS: 89 Stddev IOPS: 11 Max IOPS: 106 Min IOPS: 40 Average Latency(s): 0.179274 Stddev Latency(s): 0.105626 Max latency(s): 1.00746 Min latency(s): 0.0656261 # echo 3 > /proc/sys/vm/drop_caches # rados bench -p pxa-rbd 60 seq Total time run: 24.208097 Total reads made: 5376 Read size: 4194304 Object size: 4194304 Bandwidth (MB/sec): 888.298 Average IOPS: 222 Stddev IOPS: 33 Max IOPS: 249 Min IOPS: 92 Average Latency(s): 0.0714553 Max latency(s): 0.63154 Min latency(s): 0.0237746 According to these numbers the relation from write and read performance should be the other way round: writes should be slower than reads, but on a VM its exactly the other way round? Any idea why nevertheless writes on a VM are ~3 times faster then reads and what I could try to speed up reading? Thanks a lot Rainer -- Rainer Krienke, Uni Koblenz, Rechenzentrum, A22, Universitaetsstrasse 1 56070 Koblenz, Web: http://www.uni-koblenz.de/~krienke, Tel: +49261287 1312 PGP: http://www.uni-koblenz.de/~krienke/mypgp.html, Fax: +49261287 1001312 -- Rainer Krienke, Uni Koblenz, Rechenzentrum, A22, Universitaetsstrasse 1 56070 Koblenz, Web: http://www.uni-koblenz.de/~krienke, Tel: +49261287 1312 PGP: http://www.uni-koblenz.de/~krienke/mypgp.html, Fax: +49261287 1001312 _______________________________________________ pve-user mailing list pve-user@pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user