Hi,
I would propose to start with a OSD only benchmark (ceph tell osd.*
bench) to get an upper estimate of what the OSD itself is capabable of.
You also did not describe the network setup. 800MB/s is a good value if
the network connection is a 10GbE link (which has a theoretical upper
limit of 1.2 GB/s without protocol overhead). You may also be limited by
the client's CPU, so check CPU load, too.
As a comparison, our cluster (9 hosts with single 40 GbE links) allows
up to 1 GB/s in simple rados benchmarks runing on clients with 2x 10GbE
links).
Regards,
Burkhard
On 09/29/2016 12:05 PM, min fang wrote:
I used 2 copies, not 3, so should be 1000MB/s in theory. thanks.
2016-09-29 17:54 GMT+08:00 Nick Fisk <[email protected]
<mailto:[email protected]>>:
*From:*ceph-users [mailto:[email protected]
<mailto:[email protected]>] *On Behalf Of *min fang
*Sent:* 29 September 2016 10:34
*To:* ceph-users <[email protected]
<mailto:[email protected]>>
*Subject:* [ceph-users] ceph write performance issue
Hi, I created 40 osds ceph cluster with 8 PM863 960G SSD as
journal. One ssd is used by 5 osd drives as journal. The ssd 512
random write performance is about 450MB/s, but the whole cluster
sequential write throughput is only 800MB/s. Any suggestion on
improving sequential write performance? thanks.
Take a conservative figure of 50MB/s for each disk as writing in
Ceph is not just straight sequential writes, there is a slight
random nature to it.
(40x50MB/s)/3 = 666MB/s. Seems fine to me.
Testing result is here:
rados bench -p libvirt-pool 10 write --no-cleanup
Maintaining 16 concurrent writes of 4194304 bytes to objects of
size 4194304 for up to 10 seconds or 0 objects
Object prefix: benchmark_data_redpower-sh-04_16462
sec Cur ops started finished avg MB/s cur MB/s last lat(s)
avg lat(s)
0 0 0 0 0 0 - 0
1 15 189 174 695.968 696 0.0359122
0.082477
2 16 395 379 757.938 820 0.0634079
0.0826266
3 16 582 566 754.601 748 0.0401129
0.0830207
4 16 796 780 779.934 856 0.0374938
0.0816794
5 16 977 961 768.735 724 0.0489886
0.0827479
6 16 1172 1156 770.601 780 0.0428639
0.0812062
7 16 1387 1371 783.362 860 0.0461826
0.0811803
8 16 1545 1529 764.433 632 0.238497
0.0831018
9 16 1765 1749 777.265 880 0.0557358
0.0814399
10 16 1971 1955 781.931 824 0.0321333
0.0814144
Total time run: 10.044813
Total writes made: 1972
Write size: 4194304
Object size: 4194304
Bandwidth (MB/sec): 785.281
Stddev Bandwidth: 80.8235
Max bandwidth (MB/sec): 880
Min bandwidth (MB/sec): 632
Average IOPS: 196
Stddev IOPS: 20
Max IOPS: 220
Min IOPS: 158
Average Latency(s): 0.081415
Stddev Latency(s): 0.0554568
Max latency(s): 0.345111
Min latency(s): 0.0230153
my ceph osd configuration:
sd_mkfs_type = xfs
osd_mount_options_xfs = rw,noatime,inode64,logbsize=256k
osd_mkfs_options_xfs = -f -i size=2048
filestore_max_inline_xattr_size = 254
filestore_max_inline_xattrs = 6
osd_op_threads = 20
filestore_queue_max_ops = 25000
journal_max_write_entries=10000
journal_queue_max_ops=50000
objecter_inflight_ops=10240
filestore_queue_max_bytes=1048576000
filestore_queue_committing_max_bytes =1048576000
journal_max_write_bytes=1073714824
journal_queue_max_bytes=10485760000
ms_dispatch_throttle_bytes=1048576000
objecter_infilght_op_bytes=1048576000
filestore_max_sync_interval=20
filestore_flusher=false
filestore_flush_min=0
filestore_sync_flush=true
journal_block_align = true
journal_dio = true
journal_aio = true
journal_force_aio = true
osd_op_num_shards=8
osd_op_num_threads_per_shard=2
filestore_wbthrottle_enable=false
filestore_fd_cache_size=1024
filestore_omap_header_cache_size=1024
_______________________________________________
ceph-users mailing list
[email protected]
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
--
Dr. rer. nat. Burkhard Linke
Bioinformatics and Systems Biology
Justus-Liebig-University Giessen
35392 Giessen, Germany
Phone: (+49) (0)641 9935810
_______________________________________________
ceph-users mailing list
[email protected]
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
