I used 2 copies, not 3, so should be 1000MB/s in theory. thanks. 2016-09-29 17:54 GMT+08:00 Nick Fisk <[email protected]>:
> *From:* ceph-users [mailto:[email protected]] *On Behalf > Of *min fang > *Sent:* 29 September 2016 10:34 > *To:* ceph-users <[email protected]> > *Subject:* [ceph-users] ceph write performance issue > > > > Hi, I created 40 osds ceph cluster with 8 PM863 960G SSD as journal. One > ssd is used by 5 osd drives as journal. The ssd 512 random write > performance is about 450MB/s, but the whole cluster sequential write > throughput is only 800MB/s. Any suggestion on improving sequential write > performance? thanks. > > > > Take a conservative figure of 50MB/s for each disk as writing in Ceph is > not just straight sequential writes, there is a slight random nature to it. > > (40x50MB/s)/3 = 666MB/s. Seems fine to me. > > > > > Testing result is here: > rados bench -p libvirt-pool 10 write --no-cleanup > Maintaining 16 concurrent writes of 4194304 bytes to objects of size > 4194304 for up to 10 seconds or 0 objects > Object prefix: benchmark_data_redpower-sh-04_16462 > sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg > lat(s) > 0 0 0 0 0 0 > - 0 > 1 15 189 174 695.968 696 0.0359122 > 0.082477 > 2 16 395 379 757.938 820 0.0634079 > 0.0826266 > 3 16 582 566 754.601 748 0.0401129 > 0.0830207 > 4 16 796 780 779.934 856 0.0374938 > 0.0816794 > 5 16 977 961 768.735 724 0.0489886 > 0.0827479 > 6 16 1172 1156 770.601 780 0.0428639 > 0.0812062 > 7 16 1387 1371 783.362 860 0.0461826 > 0.0811803 > 8 16 1545 1529 764.433 632 0.238497 > 0.0831018 > 9 16 1765 1749 777.265 880 0.0557358 > 0.0814399 > 10 16 1971 1955 781.931 824 0.0321333 > 0.0814144 > Total time run: 10.044813 > Total writes made: 1972 > Write size: 4194304 > Object size: 4194304 > Bandwidth (MB/sec): 785.281 > Stddev Bandwidth: 80.8235 > Max bandwidth (MB/sec): 880 > Min bandwidth (MB/sec): 632 > Average IOPS: 196 > Stddev IOPS: 20 > Max IOPS: 220 > Min IOPS: 158 > Average Latency(s): 0.081415 > Stddev Latency(s): 0.0554568 > Max latency(s): 0.345111 > Min latency(s): 0.0230153 > > my ceph osd configuration: > sd_mkfs_type = xfs > osd_mount_options_xfs = rw,noatime,inode64,logbsize=256k > osd_mkfs_options_xfs = -f -i size=2048 > filestore_max_inline_xattr_size = 254 > filestore_max_inline_xattrs = 6 > osd_op_threads = 20 > filestore_queue_max_ops = 25000 > journal_max_write_entries=10000 > journal_queue_max_ops=50000 > objecter_inflight_ops=10240 > filestore_queue_max_bytes=1048576000 > filestore_queue_committing_max_bytes =1048576000 > journal_max_write_bytes=1073714824 > journal_queue_max_bytes=10485760000 > ms_dispatch_throttle_bytes=1048576000 > objecter_infilght_op_bytes=1048576000 > filestore_max_sync_interval=20 > filestore_flusher=false > filestore_flush_min=0 > filestore_sync_flush=true > journal_block_align = true > journal_dio = true > journal_aio = true > journal_force_aio = true > osd_op_num_shards=8 > osd_op_num_threads_per_shard=2 > filestore_wbthrottle_enable=false > filestore_fd_cache_size=1024 > filestore_omap_header_cache_size=1024 > >
_______________________________________________ ceph-users mailing list [email protected] http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
