Hi, I created 40 osds ceph cluster with 8 PM863 960G SSD as journal. One
ssd is used by 5 osd drives as journal. The ssd 512 random write
performance is about 450MB/s, but the whole cluster sequential write
throughput is only 800MB/s. Any suggestion on improving sequential write
performance? thanks.
Testing result is here:
rados bench -p libvirt-pool 10 write --no-cleanup
Maintaining 16 concurrent writes of 4194304 bytes to objects of size
4194304 for up to 10 seconds or 0 objects
Object prefix: benchmark_data_redpower-sh-04_16462
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg
lat(s)
0 0 0 0 0 0 -
0
1 15 189 174 695.968 696 0.0359122
0.082477
2 16 395 379 757.938 820 0.0634079
0.0826266
3 16 582 566 754.601 748 0.0401129
0.0830207
4 16 796 780 779.934 856 0.0374938
0.0816794
5 16 977 961 768.735 724 0.0489886
0.0827479
6 16 1172 1156 770.601 780 0.0428639
0.0812062
7 16 1387 1371 783.362 860 0.0461826
0.0811803
8 16 1545 1529 764.433 632 0.238497
0.0831018
9 16 1765 1749 777.265 880 0.0557358
0.0814399
10 16 1971 1955 781.931 824 0.0321333
0.0814144
Total time run: 10.044813
Total writes made: 1972
Write size: 4194304
Object size: 4194304
Bandwidth (MB/sec): 785.281
Stddev Bandwidth: 80.8235
Max bandwidth (MB/sec): 880
Min bandwidth (MB/sec): 632
Average IOPS: 196
Stddev IOPS: 20
Max IOPS: 220
Min IOPS: 158
Average Latency(s): 0.081415
Stddev Latency(s): 0.0554568
Max latency(s): 0.345111
Min latency(s): 0.0230153
my ceph osd configuration:
sd_mkfs_type = xfs
osd_mount_options_xfs = rw,noatime,inode64,logbsize=256k
osd_mkfs_options_xfs = -f -i size=2048
filestore_max_inline_xattr_size = 254
filestore_max_inline_xattrs = 6
osd_op_threads = 20
filestore_queue_max_ops = 25000
journal_max_write_entries=10000
journal_queue_max_ops=50000
objecter_inflight_ops=10240
filestore_queue_max_bytes=1048576000
filestore_queue_committing_max_bytes =1048576000
journal_max_write_bytes=1073714824
journal_queue_max_bytes=10485760000
ms_dispatch_throttle_bytes=1048576000
objecter_infilght_op_bytes=1048576000
filestore_max_sync_interval=20
filestore_flusher=false
filestore_flush_min=0
filestore_sync_flush=true
journal_block_align = true
journal_dio = true
journal_aio = true
journal_force_aio = true
osd_op_num_shards=8
osd_op_num_threads_per_shard=2
filestore_wbthrottle_enable=false
filestore_fd_cache_size=1024
filestore_omap_header_cache_size=1024
_______________________________________________
ceph-users mailing list
[email protected]
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
