Re: [openstack-dev] [Ceph] Why performance of benchmarks with small blocks is extremely small?

2014-09-30 Thread Gregory Farnum
On Sat, Sep 27, 2014 at 8:14 AM, Timur Nurlygayanov
tnurlygaya...@mirantis.com wrote:
 Hello all,

 I installed OpenStack with Glance + Ceph OSD with replication factor 2 and
 now I can see the write operations are extremly slow.
 For example, I can see only 0.04 MB/s write speed when I run rados bench
 with 512b blocks:

 rados bench -p test 60 write --no-cleanup -t 1 -b 512

  Maintaining 1 concurrent writes of 512 bytes for up to 60 seconds or 0
 objects
  Object prefix: benchmark_data_node-17.domain.tld_15862
sec Cur ops   started  finishedavg MB/s cur MB/s   last lat
 avg lat
  0   0 0 0  00
 -   0
  1   183820.0400341   0.0400391
 0.008465   0.0120985
  2   1   169   168  0.04101110.0419922
 0.080433   0.0118995
  3   1   240   239  0.03889590.034668
 0.008052   0.0125385
  4   1   356   355  0.0433309   0.0566406
 0.00837 0.0112662
  5   1   472   471  0.0459919   0.0566406
 0.008343   0.0106034
  6   1   550   549  0.0446735   0.0380859
 0.036639   0.0108791
  7   1   581   580  0.0404538   0.0151367
 0.008614   0.0120654


 My test environment configuration:
 Hardware servers with 1Gb network interfaces, 64Gb RAM and 16 CPU cores per
 node, HDDs WDC WD5003ABYX-01WERA0.
 OpenStack with 1 controller, 1 compute and 2 ceph nodes (ceph on separate
 nodes).
 CentOS 6.5, kernel 2.6.32-431.el6.x86_64.

 I tested several config options for optimizations, like in
 /etc/ceph/ceph.conf:

 [default]
 ...
 osd_pool_default_pg_num = 1024
 osd_pool_default_pgp_num = 1024
 osd_pool_default_flag_hashpspool = true
 ...
 [osd]
 osd recovery max active = 1
 osd max backfills = 1
 filestore max sync interval = 30
 filestore min sync interval = 29
 filestore flusher = false
 filestore queue max ops = 1
 filestore op threads = 16
 osd op threads = 16
 ...
 [client]
 rbd_cache = true
 rbd_cache_writethrough_until_flush = true

 and in /etc/cinder/cinder.conf:

 [DEFAULT]
 volume_tmp_dir=/tmp

 but in the result performance was increased only on ~30 % and it not looks
 like huge success.

 Non-default mount options and TCP optimization increase the speed in about
 1%:

 [root@node-17 ~]# mount | grep ceph
 /dev/sda4 on /var/lib/ceph/osd/ceph-0 type xfs
 (rw,noexec,nodev,noatime,nodiratime,user_xattr,data=writeback,barrier=0)

 [root@node-17 ~]# cat /etc/sysctl.conf
 net.core.rmem_max = 16777216
 net.core.wmem_max = 16777216
 net.ipv4.tcp_rmem = 4096 87380 16777216
 net.ipv4.tcp_wmem = 4096 65536 16777216
 net.ipv4.tcp_window_scaling = 1
 net.ipv4.tcp_timestamps = 1
 net.ipv4.tcp_sack = 1


 Do we have other ways to significantly improve CEPH storage performance?
 Any feedback and comments are welcome!

This is entirely latency dominated and OpenStack configuration changes
aren't going to be able to do much — you're getting 80 sequential ops
a second out of a system that has to do two round trips over a network
and hit two hard drives on every operation. You might want to spend
some time looking at how latency, bandwidth, and concurrency are
(often not) related. :)
-Greg
Software Engineer #42 @ http://inktank.com | http://ceph.com

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Ceph] Why performance of benchmarks with small blocks is extremely small?

2014-09-29 Thread Pasquale Porreca

Hello

I have no experience with Ceph and this specific benchmark tool, anyway 
I have experience with several other performance benchmark tools and 
file systems and I can say it always happen to have very very low 
performance results when the file size is too small (i.e.  1MB).


My suspect is that benchmark tools are not reliable for file size so 
small, since the time to write is so small that the overhead introduced 
by the test itself is not at all negligible.


I saw that the default object size for rados is 4 MB, did you try your 
test without the option -b 512? I think the results should be 
different for several order of magnitude.


BR

On 09/27/14 17:14, Timur Nurlygayanov wrote:

Hello all,

I installed OpenStack with Glance + Ceph OSD with replication factor 2 
and now I can see the write operations are extremly slow.
For example, I can see only 0.04 MB/s write speed when I run rados 
bench with 512b blocks:


rados bench -p test 60 write --no-cleanup -t 1 -b 512

 Maintaining 1 concurrent writes of 512 bytes for up to 60 seconds or 
0 objects

 Object prefix: benchmark_data_node-17.domain.tld_15862
   sec Cur ops   started  finishedavg MB/s cur MB/s   last 
lat  avg lat
 0   0 0 0 0
0   -   0
 1   183820.0400341 0.0400391  
0.008465   0.0120985
 2   1   169   168  0.0410111 0.0419922  
0.080433   0.0118995
 3   1   240   239  0.0388959 0.034668   
0.008052   0.0125385
 4   1   356   355  0.0433309 0.0566406  
0.00837 0.0112662
 5   1   472   471  0.0459919 0.0566406  
0.008343   0.0106034
 6   1   550   549  0.0446735 0.0380859  
0.036639   0.0108791
 7   1   581   580  0.0404538 0.0151367  
0.008614   0.0120654



My test environment configuration:
Hardware servers with 1Gb network interfaces, 64Gb RAM and 16 CPU 
cores per node, HDDs WDC WD5003ABYX-01WERA0.
OpenStack with 1 controller, 1 compute and 2 ceph nodes (ceph on 
separate nodes).

CentOS 6.5, kernel 2.6.32-431.el6.x86_64.

I tested several config options for optimizations, like in 
/etc/ceph/ceph.conf:


[default]
...
osd_pool_default_pg_num = 1024
osd_pool_default_pgp_num = 1024
osd_pool_default_flag_hashpspool = true
...
[osd]
osd recovery max active = 1
osd max backfills = 1
filestore max sync interval = 30
filestore min sync interval = 29
filestore flusher = false
filestore queue max ops = 1
filestore op threads = 16
osd op threads = 16
...
[client]
rbd_cache = true
rbd_cache_writethrough_until_flush = true

and in /etc/cinder/cinder.conf:

[DEFAULT]
volume_tmp_dir=/tmp

but in the result performance was increased only on ~30 % and it not 
looks like huge success.


Non-default mount options and TCP optimization increase the speed in 
about 1%:


[root@node-17 ~]# mount | grep ceph
/dev/sda4 on /var/lib/ceph/osd/ceph-0 type xfs 
(rw,noexec,nodev,noatime,nodiratime,user_xattr,data=writeback,barrier=0)


[root@node-17 ~]# cat /etc/sysctl.conf
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_wmem = 4096 65536 16777216
net.ipv4.tcp_window_scaling = 1
net.ipv4.tcp_timestamps = 1
net.ipv4.tcp_sack = 1


Do we have other ways to significantly improve CEPH storage performance?
Any feedback and comments are welcome!

Thank you!


--

Timur,
QA Engineer
OpenStack Projects
Mirantis Inc


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


--
Pasquale Porreca

DEK Technologies
Via dei Castelli Romani, 22
00040 Pomezia (Roma)

Mobile +39 3394823805
Skype paskporr

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Ceph] Why performance of benchmarks with small blocks is extremely small?

2014-09-29 Thread Clay Gerrard
I also have limited experience with Ceph and rados bench - but it looks
like you're setting the number of worker threads to only 1?  (-t 1)

I think the default is 16, and most storage distributed storage systems
designed for concurrency are going to do a bit better if you exercise more
concurrent workers... so you might try turning that up until you see some
diminishing returns.  Be sure to watch for resource contention on the load
generating server.

-Clay

On Mon, Sep 29, 2014 at 4:49 AM, Pasquale Porreca 
pasquale.porr...@dektech.com.au wrote:

  Hello

 I have no experience with Ceph and this specific benchmark tool, anyway I
 have experience with several other performance benchmark tools and file
 systems and I can say it always happen to have very very low performance
 results when the file size is too small (i.e.  1MB).

 My suspect is that benchmark tools are not reliable for file size so
 small, since the time to write is so small that the overhead introduced by
 the test itself is not at all negligible.

 I saw that the default object size for rados is 4 MB, did you try your
 test without the option -b 512? I think the results should be different
 for several order of magnitude.

 BR


 On 09/27/14 17:14, Timur Nurlygayanov wrote:

   Hello all,

  I installed OpenStack with Glance + Ceph OSD with replication factor 2
 and now I can see the write operations are extremly slow.
 For example, I can see only 0.04 MB/s write speed when I run rados bench
 with 512b blocks:

  rados bench -p test 60 write --no-cleanup -t 1 -b 512

  Maintaining 1 concurrent writes of 512 bytes for up to 60 seconds or 0
 objects
  Object prefix: benchmark_data_node-17.domain.tld_15862
sec Cur ops   started  finishedavg MB/s cur MB/s   last
 lat  avg lat
  0   0 0 0  0
 0   -   0
  1   183820.0400341   0.0400391
 0.008465   0.0120985
  2   1   169   168  0.04101110.0419922
 0.080433   0.0118995
  3   1   240   239  0.03889590.034668
 0.008052   0.0125385
  4   1   356   355  0.0433309   0.0566406
 0.00837 0.0112662
  5   1   472   471  0.0459919   0.0566406
 0.008343   0.0106034
  6   1   550   549  0.0446735   0.0380859
 0.036639   0.0108791
  7   1   581   580  0.0404538   0.0151367
 0.008614   0.0120654


 My test environment configuration:
  Hardware servers with 1Gb network interfaces, 64Gb RAM and 16 CPU cores
 per node, HDDs WDC WD5003ABYX-01WERA0.
  OpenStack with 1 controller, 1 compute and 2 ceph nodes (ceph on separate
 nodes).
 CentOS 6.5, kernel 2.6.32-431.el6.x86_64.

  I tested several config options for optimizations, like in
 /etc/ceph/ceph.conf:

  [default]
 ...
 osd_pool_default_pg_num = 1024
 osd_pool_default_pgp_num = 1024
 osd_pool_default_flag_hashpspool = true
 ...
 [osd]
 osd recovery max active = 1
 osd max backfills = 1
 filestore max sync interval = 30
 filestore min sync interval = 29
 filestore flusher = false
 filestore queue max ops = 1
 filestore op threads = 16
 osd op threads = 16
 ...
 [client]
 rbd_cache = true
 rbd_cache_writethrough_until_flush = true

  and in /etc/cinder/cinder.conf:

  [DEFAULT]
  volume_tmp_dir=/tmp

 but in the result performance was increased only on ~30 % and it not looks
 like huge success.

  Non-default mount options and TCP optimization increase the speed in
 about 1%:

 [root@node-17 ~]# mount | grep ceph
 /dev/sda4 on /var/lib/ceph/osd/ceph-0 type xfs
 (rw,noexec,nodev,noatime,nodiratime,user_xattr,data=writeback,barrier=0)

 [root@node-17 ~]# cat /etc/sysctl.conf
 net.core.rmem_max = 16777216
 net.core.wmem_max = 16777216
 net.ipv4.tcp_rmem = 4096 87380 16777216
 net.ipv4.tcp_wmem = 4096 65536 16777216
 net.ipv4.tcp_window_scaling = 1
 net.ipv4.tcp_timestamps = 1
 net.ipv4.tcp_sack = 1


 Do we have other ways to significantly improve CEPH storage performance?
  Any feedback and comments are welcome!

  Thank you!


  --

  Timur,
 QA Engineer
 OpenStack Projects
 Mirantis Inc


 ___
 OpenStack-dev mailing 
 listOpenStack-dev@lists.openstack.orghttp://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


 --
 Pasquale Porreca

 DEK Technologies
 Via dei Castelli Romani, 22
 00040 Pomezia (Roma)

 Mobile +39 3394823805
 Skype paskporr


 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev