Hello
I have no experience with Ceph and this specific benchmark tool, anyway
I have experience with several other performance benchmark tools and
file systems and I can say it always happen to have very very low
performance results when the file size is too small (i.e. < 1MB).
My suspect is that benchmark tools are not reliable for file size so
small, since the time to write is so small that the overhead introduced
by the test itself is not at all negligible.
I saw that the default object size for rados is 4 MB, did you try your
test without the option "-b 512"? I think the results should be
different for several order of magnitude.
BR
On 09/27/14 17:14, Timur Nurlygayanov wrote:
Hello all,
I installed OpenStack with Glance + Ceph OSD with replication factor 2
and now I can see the write operations are extremly slow.
For example, I can see only 0.04 MB/s write speed when I run rados
bench with 512b blocks:
rados bench -p test 60 write --no-cleanup -t 1 -b 512
Maintaining 1 concurrent writes of 512 bytes for up to 60 seconds or
0 objects
Object prefix: benchmark_data_node-17.domain.tld_15862
sec Cur ops started finished avg MB/s cur MB/s last
lat avg lat
0 0 0 0 0
0 - 0
1 1 83 82 0.0400341 0.0400391
0.008465 0.0120985
2 1 169 168 0.0410111 0.0419922
0.080433 0.0118995
3 1 240 239 0.0388959 0.034668
0.008052 0.0125385
4 1 356 355 0.0433309 0.0566406
0.00837 0.0112662
5 1 472 471 0.0459919 0.0566406
0.008343 0.0106034
6 1 550 549 0.0446735 0.0380859
0.036639 0.0108791
7 1 581 580 0.0404538 0.0151367
0.008614 0.0120654
My test environment configuration:
Hardware servers with 1Gb network interfaces, 64Gb RAM and 16 CPU
cores per node, HDDs WDC WD5003ABYX-01WERA0.
OpenStack with 1 controller, 1 compute and 2 ceph nodes (ceph on
separate nodes).
CentOS 6.5, kernel 2.6.32-431.el6.x86_64.
I tested several config options for optimizations, like in
/etc/ceph/ceph.conf:
[default]
...
osd_pool_default_pg_num = 1024
osd_pool_default_pgp_num = 1024
osd_pool_default_flag_hashpspool = true
...
[osd]
osd recovery max active = 1
osd max backfills = 1
filestore max sync interval = 30
filestore min sync interval = 29
filestore flusher = false
filestore queue max ops = 10000
filestore op threads = 16
osd op threads = 16
...
[client]
rbd_cache = true
rbd_cache_writethrough_until_flush = true
and in /etc/cinder/cinder.conf:
[DEFAULT]
volume_tmp_dir=/tmp
but in the result performance was increased only on ~30 % and it not
looks like huge success.
Non-default mount options and TCP optimization increase the speed in
about 1%:
[root@node-17 ~]# mount | grep ceph
/dev/sda4 on /var/lib/ceph/osd/ceph-0 type xfs
(rw,noexec,nodev,noatime,nodiratime,user_xattr,data=writeback,barrier=0)
[root@node-17 ~]# cat /etc/sysctl.conf
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_wmem = 4096 65536 16777216
net.ipv4.tcp_window_scaling = 1
net.ipv4.tcp_timestamps = 1
net.ipv4.tcp_sack = 1
Do we have other ways to significantly improve CEPH storage performance?
Any feedback and comments are welcome!
Thank you!
--
Timur,
QA Engineer
OpenStack Projects
Mirantis Inc
_______________________________________________
OpenStack-dev mailing list
[email protected]
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
--
Pasquale Porreca
DEK Technologies
Via dei Castelli Romani, 22
00040 Pomezia (Roma)
Mobile +39 3394823805
Skype paskporr
_______________________________________________
OpenStack-dev mailing list
[email protected]
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
