Re: [openstack-dev] [Ceph] Why performance of benchmarks with small blocks is extremely small?
On Sat, Sep 27, 2014 at 8:14 AM, Timur Nurlygayanov tnurlygaya...@mirantis.com wrote: Hello all, I installed OpenStack with Glance + Ceph OSD with replication factor 2 and now I can see the write operations are extremly slow. For example, I can see only 0.04 MB/s write speed when I run rados bench with 512b blocks: rados bench -p test 60 write --no-cleanup -t 1 -b 512 Maintaining 1 concurrent writes of 512 bytes for up to 60 seconds or 0 objects Object prefix: benchmark_data_node-17.domain.tld_15862 sec Cur ops started finishedavg MB/s cur MB/s last lat avg lat 0 0 0 0 00 - 0 1 183820.0400341 0.0400391 0.008465 0.0120985 2 1 169 168 0.04101110.0419922 0.080433 0.0118995 3 1 240 239 0.03889590.034668 0.008052 0.0125385 4 1 356 355 0.0433309 0.0566406 0.00837 0.0112662 5 1 472 471 0.0459919 0.0566406 0.008343 0.0106034 6 1 550 549 0.0446735 0.0380859 0.036639 0.0108791 7 1 581 580 0.0404538 0.0151367 0.008614 0.0120654 My test environment configuration: Hardware servers with 1Gb network interfaces, 64Gb RAM and 16 CPU cores per node, HDDs WDC WD5003ABYX-01WERA0. OpenStack with 1 controller, 1 compute and 2 ceph nodes (ceph on separate nodes). CentOS 6.5, kernel 2.6.32-431.el6.x86_64. I tested several config options for optimizations, like in /etc/ceph/ceph.conf: [default] ... osd_pool_default_pg_num = 1024 osd_pool_default_pgp_num = 1024 osd_pool_default_flag_hashpspool = true ... [osd] osd recovery max active = 1 osd max backfills = 1 filestore max sync interval = 30 filestore min sync interval = 29 filestore flusher = false filestore queue max ops = 1 filestore op threads = 16 osd op threads = 16 ... [client] rbd_cache = true rbd_cache_writethrough_until_flush = true and in /etc/cinder/cinder.conf: [DEFAULT] volume_tmp_dir=/tmp but in the result performance was increased only on ~30 % and it not looks like huge success. Non-default mount options and TCP optimization increase the speed in about 1%: [root@node-17 ~]# mount | grep ceph /dev/sda4 on /var/lib/ceph/osd/ceph-0 type xfs (rw,noexec,nodev,noatime,nodiratime,user_xattr,data=writeback,barrier=0) [root@node-17 ~]# cat /etc/sysctl.conf net.core.rmem_max = 16777216 net.core.wmem_max = 16777216 net.ipv4.tcp_rmem = 4096 87380 16777216 net.ipv4.tcp_wmem = 4096 65536 16777216 net.ipv4.tcp_window_scaling = 1 net.ipv4.tcp_timestamps = 1 net.ipv4.tcp_sack = 1 Do we have other ways to significantly improve CEPH storage performance? Any feedback and comments are welcome! This is entirely latency dominated and OpenStack configuration changes aren't going to be able to do much — you're getting 80 sequential ops a second out of a system that has to do two round trips over a network and hit two hard drives on every operation. You might want to spend some time looking at how latency, bandwidth, and concurrency are (often not) related. :) -Greg Software Engineer #42 @ http://inktank.com | http://ceph.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Ceph] Why performance of benchmarks with small blocks is extremely small?
Hello I have no experience with Ceph and this specific benchmark tool, anyway I have experience with several other performance benchmark tools and file systems and I can say it always happen to have very very low performance results when the file size is too small (i.e. 1MB). My suspect is that benchmark tools are not reliable for file size so small, since the time to write is so small that the overhead introduced by the test itself is not at all negligible. I saw that the default object size for rados is 4 MB, did you try your test without the option -b 512? I think the results should be different for several order of magnitude. BR On 09/27/14 17:14, Timur Nurlygayanov wrote: Hello all, I installed OpenStack with Glance + Ceph OSD with replication factor 2 and now I can see the write operations are extremly slow. For example, I can see only 0.04 MB/s write speed when I run rados bench with 512b blocks: rados bench -p test 60 write --no-cleanup -t 1 -b 512 Maintaining 1 concurrent writes of 512 bytes for up to 60 seconds or 0 objects Object prefix: benchmark_data_node-17.domain.tld_15862 sec Cur ops started finishedavg MB/s cur MB/s last lat avg lat 0 0 0 0 0 0 - 0 1 183820.0400341 0.0400391 0.008465 0.0120985 2 1 169 168 0.0410111 0.0419922 0.080433 0.0118995 3 1 240 239 0.0388959 0.034668 0.008052 0.0125385 4 1 356 355 0.0433309 0.0566406 0.00837 0.0112662 5 1 472 471 0.0459919 0.0566406 0.008343 0.0106034 6 1 550 549 0.0446735 0.0380859 0.036639 0.0108791 7 1 581 580 0.0404538 0.0151367 0.008614 0.0120654 My test environment configuration: Hardware servers with 1Gb network interfaces, 64Gb RAM and 16 CPU cores per node, HDDs WDC WD5003ABYX-01WERA0. OpenStack with 1 controller, 1 compute and 2 ceph nodes (ceph on separate nodes). CentOS 6.5, kernel 2.6.32-431.el6.x86_64. I tested several config options for optimizations, like in /etc/ceph/ceph.conf: [default] ... osd_pool_default_pg_num = 1024 osd_pool_default_pgp_num = 1024 osd_pool_default_flag_hashpspool = true ... [osd] osd recovery max active = 1 osd max backfills = 1 filestore max sync interval = 30 filestore min sync interval = 29 filestore flusher = false filestore queue max ops = 1 filestore op threads = 16 osd op threads = 16 ... [client] rbd_cache = true rbd_cache_writethrough_until_flush = true and in /etc/cinder/cinder.conf: [DEFAULT] volume_tmp_dir=/tmp but in the result performance was increased only on ~30 % and it not looks like huge success. Non-default mount options and TCP optimization increase the speed in about 1%: [root@node-17 ~]# mount | grep ceph /dev/sda4 on /var/lib/ceph/osd/ceph-0 type xfs (rw,noexec,nodev,noatime,nodiratime,user_xattr,data=writeback,barrier=0) [root@node-17 ~]# cat /etc/sysctl.conf net.core.rmem_max = 16777216 net.core.wmem_max = 16777216 net.ipv4.tcp_rmem = 4096 87380 16777216 net.ipv4.tcp_wmem = 4096 65536 16777216 net.ipv4.tcp_window_scaling = 1 net.ipv4.tcp_timestamps = 1 net.ipv4.tcp_sack = 1 Do we have other ways to significantly improve CEPH storage performance? Any feedback and comments are welcome! Thank you! -- Timur, QA Engineer OpenStack Projects Mirantis Inc ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Pasquale Porreca DEK Technologies Via dei Castelli Romani, 22 00040 Pomezia (Roma) Mobile +39 3394823805 Skype paskporr ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Ceph] Why performance of benchmarks with small blocks is extremely small?
I also have limited experience with Ceph and rados bench - but it looks like you're setting the number of worker threads to only 1? (-t 1) I think the default is 16, and most storage distributed storage systems designed for concurrency are going to do a bit better if you exercise more concurrent workers... so you might try turning that up until you see some diminishing returns. Be sure to watch for resource contention on the load generating server. -Clay On Mon, Sep 29, 2014 at 4:49 AM, Pasquale Porreca pasquale.porr...@dektech.com.au wrote: Hello I have no experience with Ceph and this specific benchmark tool, anyway I have experience with several other performance benchmark tools and file systems and I can say it always happen to have very very low performance results when the file size is too small (i.e. 1MB). My suspect is that benchmark tools are not reliable for file size so small, since the time to write is so small that the overhead introduced by the test itself is not at all negligible. I saw that the default object size for rados is 4 MB, did you try your test without the option -b 512? I think the results should be different for several order of magnitude. BR On 09/27/14 17:14, Timur Nurlygayanov wrote: Hello all, I installed OpenStack with Glance + Ceph OSD with replication factor 2 and now I can see the write operations are extremly slow. For example, I can see only 0.04 MB/s write speed when I run rados bench with 512b blocks: rados bench -p test 60 write --no-cleanup -t 1 -b 512 Maintaining 1 concurrent writes of 512 bytes for up to 60 seconds or 0 objects Object prefix: benchmark_data_node-17.domain.tld_15862 sec Cur ops started finishedavg MB/s cur MB/s last lat avg lat 0 0 0 0 0 0 - 0 1 183820.0400341 0.0400391 0.008465 0.0120985 2 1 169 168 0.04101110.0419922 0.080433 0.0118995 3 1 240 239 0.03889590.034668 0.008052 0.0125385 4 1 356 355 0.0433309 0.0566406 0.00837 0.0112662 5 1 472 471 0.0459919 0.0566406 0.008343 0.0106034 6 1 550 549 0.0446735 0.0380859 0.036639 0.0108791 7 1 581 580 0.0404538 0.0151367 0.008614 0.0120654 My test environment configuration: Hardware servers with 1Gb network interfaces, 64Gb RAM and 16 CPU cores per node, HDDs WDC WD5003ABYX-01WERA0. OpenStack with 1 controller, 1 compute and 2 ceph nodes (ceph on separate nodes). CentOS 6.5, kernel 2.6.32-431.el6.x86_64. I tested several config options for optimizations, like in /etc/ceph/ceph.conf: [default] ... osd_pool_default_pg_num = 1024 osd_pool_default_pgp_num = 1024 osd_pool_default_flag_hashpspool = true ... [osd] osd recovery max active = 1 osd max backfills = 1 filestore max sync interval = 30 filestore min sync interval = 29 filestore flusher = false filestore queue max ops = 1 filestore op threads = 16 osd op threads = 16 ... [client] rbd_cache = true rbd_cache_writethrough_until_flush = true and in /etc/cinder/cinder.conf: [DEFAULT] volume_tmp_dir=/tmp but in the result performance was increased only on ~30 % and it not looks like huge success. Non-default mount options and TCP optimization increase the speed in about 1%: [root@node-17 ~]# mount | grep ceph /dev/sda4 on /var/lib/ceph/osd/ceph-0 type xfs (rw,noexec,nodev,noatime,nodiratime,user_xattr,data=writeback,barrier=0) [root@node-17 ~]# cat /etc/sysctl.conf net.core.rmem_max = 16777216 net.core.wmem_max = 16777216 net.ipv4.tcp_rmem = 4096 87380 16777216 net.ipv4.tcp_wmem = 4096 65536 16777216 net.ipv4.tcp_window_scaling = 1 net.ipv4.tcp_timestamps = 1 net.ipv4.tcp_sack = 1 Do we have other ways to significantly improve CEPH storage performance? Any feedback and comments are welcome! Thank you! -- Timur, QA Engineer OpenStack Projects Mirantis Inc ___ OpenStack-dev mailing listOpenStack-dev@lists.openstack.orghttp://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Pasquale Porreca DEK Technologies Via dei Castelli Romani, 22 00040 Pomezia (Roma) Mobile +39 3394823805 Skype paskporr ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [Ceph] Why performance of benchmarks with small blocks is extremely small?
Hello all, I installed OpenStack with Glance + Ceph OSD with replication factor 2 and now I can see the write operations are extremly slow. For example, I can see only 0.04 MB/s write speed when I run rados bench with 512b blocks: rados bench -p test 60 write --no-cleanup -t 1 -b 512 Maintaining 1 concurrent writes of 512 bytes for up to 60 seconds or 0 objects Object prefix: benchmark_data_node-17.domain.tld_15862 sec Cur ops started finishedavg MB/s cur MB/s last lat avg lat 0 0 0 0 0 0 - 0 1 183820.0400341 0.0400391 0.008465 0.0120985 2 1 169 168 0.04101110.0419922 0.080433 0.0118995 3 1 240 239 0.03889590.034668 0.008052 0.0125385 4 1 356 355 0.0433309 0.0566406 0.00837 0.0112662 5 1 472 471 0.0459919 0.0566406 0.008343 0.0106034 6 1 550 549 0.0446735 0.0380859 0.036639 0.0108791 7 1 581 580 0.0404538 0.0151367 0.008614 0.0120654 My test environment configuration: Hardware servers with 1Gb network interfaces, 64Gb RAM and 16 CPU cores per node, HDDs WDC WD5003ABYX-01WERA0. OpenStack with 1 controller, 1 compute and 2 ceph nodes (ceph on separate nodes). CentOS 6.5, kernel 2.6.32-431.el6.x86_64. I tested several config options for optimizations, like in /etc/ceph/ceph.conf: [default] ... osd_pool_default_pg_num = 1024 osd_pool_default_pgp_num = 1024 osd_pool_default_flag_hashpspool = true ... [osd] osd recovery max active = 1 osd max backfills = 1 filestore max sync interval = 30 filestore min sync interval = 29 filestore flusher = false filestore queue max ops = 1 filestore op threads = 16 osd op threads = 16 ... [client] rbd_cache = true rbd_cache_writethrough_until_flush = true and in /etc/cinder/cinder.conf: [DEFAULT] volume_tmp_dir=/tmp but in the result performance was increased only on ~30 % and it not looks like huge success. Non-default mount options and TCP optimization increase the speed in about 1%: [root@node-17 ~]# mount | grep ceph /dev/sda4 on /var/lib/ceph/osd/ceph-0 type xfs (rw,noexec,nodev,noatime,nodiratime,user_xattr,data=writeback,barrier=0) [root@node-17 ~]# cat /etc/sysctl.conf net.core.rmem_max = 16777216 net.core.wmem_max = 16777216 net.ipv4.tcp_rmem = 4096 87380 16777216 net.ipv4.tcp_wmem = 4096 65536 16777216 net.ipv4.tcp_window_scaling = 1 net.ipv4.tcp_timestamps = 1 net.ipv4.tcp_sack = 1 Do we have other ways to significantly improve CEPH storage performance? Any feedback and comments are welcome! Thank you! -- Timur, QA Engineer OpenStack Projects Mirantis Inc ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
