As matter of interest, when I ran the test, the network throughput reached
3.98Gb/s:
ens2f0 / traffic statistics
rx | tx
--------------------------------------+------------------
bytes 2.59 GiB | 4.63 GiB
--------------------------------------+------------------
max 2.29 Gbit/s | 3.98 Gbit/s
average 905.58 Mbit/s | 1.62 Gbit/s
min 203 kbit/s | 186 kbit/s
--------------------------------------+------------------
packets 1980792 | 3354372
--------------------------------------+------------------
max 207630 p/s | 342902 p/s
average 82533 p/s | 139765 p/s
min 51 p/s | 56 p/s
--------------------------------------+------------------
time 24 seconds
Some more stats:
root@virt2:~# rados bench -p Data 10 seq
hints = 1
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg
lat(s)
0 0 0 0 0 0 -
0
1 16 402 386 1543.69 1544 0.00182802
0.0395421
2 16 773 757 1513.71 1484 0.00243911
0.0409455
Total time run: 2.340037
Total reads made: 877
Read size: 4194304
Object size: 4194304
Bandwidth (MB/sec): 1499.12
Average IOPS: 374
Stddev IOPS: 10
Max IOPS: 386
Min IOPS: 371
Average Latency(s): 0.0419036
Max latency(s): 0.176739
Min latency(s): 0.00161271
root@virt2:~# rados bench -p Data 10 rand
hints = 1
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg
lat(s)
0 0 0 0 0 0 -
0
1 16 376 360 1439.71 1440 0.0356502
0.0409024
2 16 752 736 1471.74 1504 0.0163304
0.0419063
3 16 1134 1118 1490.43 1528 0.059643
0.0417043
4 16 1515 1499 1498.78 1524 0.0502131
0.0416087
5 15 1880 1865 1491.79 1464 0.017407
0.0414158
6 16 2254 2238 1491.79 1492 0.0657474
0.0420471
7 15 2509 2494 1424.95 1024 0.00182097
0.0440063
8 15 2873 2858 1428.81 1456 0.0302541
0.0439319
9 15 3243 3228 1434.47 1480 0.108037
0.0438106
10 16 3616 3600 1439.81 1488 0.0295953
0.0436184
Total time run: 10.058519
Total reads made: 3616
Read size: 4194304
Object size: 4194304
Bandwidth (MB/sec): 1437.99
Average IOPS: 359
Stddev IOPS: 37
Max IOPS: 382
Min IOPS: 256
Average Latency(s): 0.0438002
Max latency(s): 0.664223
Min latency(s): 0.00156885
On Mon, Nov 20, 2017 at 12:38 PM, Rudi Ahlers <[email protected]> wrote:
> Hi,
>
> Can someone please help me, how do I improve performance on ou CEPH
> cluster?
>
> The hardware in use are as follows:
> 3x SuperMicro servers with the following configuration
> 12Core Dual XEON 2.2Ghz
> 128GB RAM
> 2x 400GB Intel DC SSD drives
> 4x 8TB Seagate 7200rpm 6Gbps SATA HDD's
> 1x SuperMicro DOM for Proxmox / Debian OS
> 4x Port 10Gbe NIC
> Cisco 10Gbe switch.
>
>
> root@virt2:~# rados bench -p Data 10 write --no-cleanup
> hints = 1
> Maintaining 16 concurrent writes of 4194304 bytes to objects of size
> 4194304 for up to 10 seconds or 0 objects
> Object prefix: benchmark_data_virt2_39099
> sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg
> lat(s)
> 0 0 0 0 0 0 -
> 0
> 1 16 85 69 275.979 276 0.185576
> 0.204146
> 2 16 171 155 309.966 344 0.0625409
> 0.193558
> 3 16 243 227 302.633 288 0.0547129
> 0.19835
> 4 16 330 314 313.965 348 0.0959492
> 0.199825
> 5 16 413 397 317.565 332 0.124908
> 0.196191
> 6 16 494 478 318.633 324 0.1556
> 0.197014
> 7 15 591 576 329.109 392 0.136305
> 0.192192
> 8 16 670 654 326.965 312 0.0703808
> 0.190643
> 9 16 757 741 329.297 348 0.165211
> 0.192183
> 10 16 828 812 324.764 284 0.0935803
> 0.194041
> Total time run: 10.120215
> Total writes made: 829
> Write size: 4194304
> Object size: 4194304
> Bandwidth (MB/sec): 327.661
> Stddev Bandwidth: 35.8664
> Max bandwidth (MB/sec): 392
> Min bandwidth (MB/sec): 276
> Average IOPS: 81
> Stddev IOPS: 8
> Max IOPS: 98
> Min IOPS: 69
> Average Latency(s): 0.195191
> Stddev Latency(s): 0.0830062 <083%200062>
> Max latency(s): 0.481448
> Min latency(s): 0.0414858
> root@virt2:~# hdparm -I /dev/sda
>
>
>
> root@virt2:~# ceph osd tree
> ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
> -1 72.78290 root default
> -3 29.11316 host virt1
> 1 hdd 7.27829 osd.1 up 1.00000 1.00000
> 2 hdd 7.27829 osd.2 up 1.00000 1.00000
> 3 hdd 7.27829 osd.3 up 1.00000 1.00000
> 4 hdd 7.27829 osd.4 up 1.00000 1.00000
> -5 21.83487 host virt2
> 5 hdd 7.27829 osd.5 up 1.00000 1.00000
> 6 hdd 7.27829 osd.6 up 1.00000 1.00000
> 7 hdd 7.27829 osd.7 up 1.00000 1.00000
> -7 21.83487 host virt3
> 8 hdd 7.27829 osd.8 up 1.00000 1.00000
> 9 hdd 7.27829 osd.9 up 1.00000 1.00000
> 10 hdd 7.27829 osd.10 up 1.00000 1.00000
> 0 0 osd.0 down 0 1.00000
>
>
> root@virt2:~# ceph -s
> cluster:
> id: 278a2e9c-0578-428f-bd5b-3bb348923c27
> health: HEALTH_OK
>
> services:
> mon: 3 daemons, quorum virt1,virt2,virt3
> mgr: virt1(active)
> osd: 11 osds: 10 up, 10 in
>
> data:
> pools: 1 pools, 512 pgs
> objects: 6084 objects, 24105 MB
> usage: 92822 MB used, 74438 GB / 74529 GB avail
> pgs: 512 active+clean
>
> root@virt2:~# ceph -w
> cluster:
> id: 278a2e9c-0578-428f-bd5b-3bb348923c27
> health: HEALTH_OK
>
> services:
> mon: 3 daemons, quorum virt1,virt2,virt3
> mgr: virt1(active)
> osd: 11 osds: 10 up, 10 in
>
> data:
> pools: 1 pools, 512 pgs
> objects: 6084 objects, 24105 MB
> usage: 92822 MB used, 74438 GB / 74529 GB avail
> pgs: 512 active+clean
>
>
> 2017-11-20 12:32:08.199450 mon.virt1 [INF] mon.1 10.10.10.82:6789/0
>
>
>
> The SSD drives are used as journal drives:
>
> root@virt3:~# ceph-disk list | grep /dev/sde | grep osd
> /dev/sdb1 ceph data, active, cluster ceph, osd.8, block /dev/sdb2,
> block.db /dev/sde1
> root@virt3:~# ceph-disk list | grep /dev/sdf | grep osd
> /dev/sdc1 ceph data, active, cluster ceph, osd.9, block /dev/sdc2,
> block.db /dev/sdf1
> /dev/sdd1 ceph data, active, cluster ceph, osd.10, block /dev/sdd2,
> block.db /dev/sdf2
>
>
>
> I see now /dev/sda doesn't have a journal, though it should have. Not sure
> why.
> This is the command I used to create it:
>
>
> pveceph createosd /dev/sda -bluestore 1 -journal_dev /dev/sde
>
>
> --
> Kind Regards
> Rudi Ahlers
> Website: http://www.rudiahlers.co.za
>
--
Kind Regards
Rudi Ahlers
Website: http://www.rudiahlers.co.za
_______________________________________________
ceph-users mailing list
[email protected]
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
