Re: [ceph-users] troubleshooting ceph performance
On 2018-01-31 08:14, Manuel Sopena Ballesteros wrote:
> Dear Ceph community,
>
> I have a very small ceph cluster for testing with this configuration:
>
> · 2x compute nodes each with:
>
> · dual port of 25 nic
>
> · 2x socket (56 cores with hyperthreading)
>
> · X10 intel nvme DC P3500 drives
>
> · 512 GB RAM
>
> One of the nodes is also running as a monitor.
>
> Installation has been done using ceph-ansible.
>
> Ceph version: jewel
>
> Storage engine: filestore
>
> Performance test below:
>
> [root@zeus-59 ceph-block-device]# ceph osd pool ls detail
>
> pool 0 'rbd' replicated size 2 min_size 2 crush_ruleset 0 object_hash
> rjenkins pg_num 64 pgp_num 64 last_change 115 flags hashpspool stripe_width 0
>
> pool 1 'images' replicated size 2 min_size 2 crush_ruleset 0 object_hash
> rjenkins pg_num 128 pgp_num 128 last_change 118 flags hashpspool stripe_width
> 0
>
> removed_snaps [1~3,7~4]
>
> pool 3 'backups' replicated size 2 min_size 2 crush_ruleset 0 object_hash
> rjenkins pg_num 128 pgp_num 128 last_change 120 flags hashpspool stripe_width
> 0
>
> pool 4 'vms' replicated size 2 min_size 2 crush_ruleset 0 object_hash
> rjenkins pg_num 128 pgp_num 128 last_change 122 flags hashpspool stripe_width
> 0
>
> removed_snaps [1~7]
>
> pool 5 'volumes' replicated size 2 min_size 2 crush_ruleset 0 object_hash
> rjenkins pg_num 128 pgp_num 128 last_change 124 flags hashpspool stripe_width
> 0
>
> removed_snaps [1~3]
>
> pool 6 'scbench' replicated size 2 min_size 2 crush_ruleset 0 object_hash
> rjenkins pg_num 100 pgp_num 100 last_change 126 flags hashpspool stripe_width
> 0
>
> pool 7 'rbdbench' replicated size 2 min_size 2 crush_ruleset 0 object_hash
> rjenkins pg_num 100 pgp_num 100 last_change 128 flags hashpspool stripe_width
> 0
>
> removed_snaps [1~3]
>
> [root@zeus-59 ceph-block-device]# ceph osd tree
>
> ID WEIGHT TYPE NAMEUP/DOWN REWEIGHT PRIMARY-AFFINITY
>
> -1 36.17371 root default
>
> -2 18.08685 host zeus-58
>
> 0 1.80869 osd.0 up 1.0 1.0
>
> 2 1.80869 osd.2 up 1.0 1.0
>
> 4 1.80869 osd.4 up 1.0 1.0
>
> 6 1.80869 osd.6 up 1.0 1.0
>
> 8 1.80869 osd.8 up 1.0 1.0
>
> 10 1.80869 osd.10up 1.0 1.0
>
> 12 1.80869 osd.12up 1.0 1.0
>
> 14 1.80869 osd.14up 1.0 1.0
>
> 16 1.80869 osd.16up 1.0 1.0
>
> 18 1.80869 osd.18up 1.0 1.0
>
> -3 18.08685 host zeus-59
>
> 1 1.80869 osd.1 up 1.0 1.0
>
> 3 1.80869 osd.3 up 1.0 1.0
>
> 5 1.80869 osd.5 up 1.0 1.0
>
> 7 1.80869 osd.7 up 1.0 1.0
>
> 9 1.80869 osd.9 up 1.0 1.0
>
> 11 1.80869 osd.11up 1.0 1.0
>
> 13 1.80869 osd.13up 1.0 1.0
>
> 15 1.80869 osd.15up 1.0 1.0
>
> 17 1.80869 osd.17up 1.0 1.0
>
> 19 1.80869 osd.19up 1.0 1.0
>
> [root@zeus-59 ceph-block-device]# ceph status
>
> cluster 8e930b6c-455e-4328-872d-cb9f5c0359ae
>
> health HEALTH_OK
>
> monmap e1: 1 mons at {zeus-59=10.0.32.59:6789/0}
>
> election epoch 3, quorum 0 zeus-59
>
> osdmap e129: 20 osds: 20 up, 20 in
>
> flags sortbitwise,require_jewel_osds
>
> pgmap v1166945: 776 pgs, 7 pools, 1183 GB data, 296 kobjects
>
> 2363 GB used, 34678 GB / 37042 GB avail
>
> 775 active+clean
>
> 1 active+clean+scrubbing+deep
>
> [root@zeus-59 ceph-block-device]# rados bench -p scbench 10 write
> --no-cleanup
>
> Maintaining 16 concurrent writes of 4194304 bytes to objects of size 4194304
> for up to 10 seconds or 0 objects
>
> Object prefix: benchmark_data_zeus-59.localdomain_2844050
>
> sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
>
> 0 0 0 0 0 0 - 0
>
> 1 16 644 6282511.4 2512 0.02102730.025206
>
> 2 16 1319 1303 2605.49 2700 0.0238678 0.0243974
>
> 3 16 2003 1987 2648.89 2736 0.0201334 0.0240726
>
> 4 16 2669 2653 2652.59 2664 0.0258618 0.0240468
>
> 5 16 3349 2666.01 2720 0.0189464 0.0239484
>
> 6 16 4026 4010 2672.96 2708 0.02215 0.0238954
>
> 7 16 4697 4681 2674.49 2684 0.0217258 0.0238887
>
> 8 16 5358 5342 2670.64
[ceph-users] troubleshooting ceph performance
Dear Ceph community,
I have a very small ceph cluster for testing with this configuration:
* 2x compute nodes each with:
* dual port of 25 nic
* 2x socket (56 cores with hyperthreading)
* X10 intel nvme DC P3500 drives
* 512 GB RAM
One of the nodes is also running as a monitor.
Installation has been done using ceph-ansible.
Ceph version: jewel
Storage engine: filestore
Performance test below:
[root@zeus-59 ceph-block-device]# ceph osd pool ls detail
pool 0 'rbd' replicated size 2 min_size 2 crush_ruleset 0 object_hash rjenkins
pg_num 64 pgp_num 64 last_change 115 flags hashpspool stripe_width 0
pool 1 'images' replicated size 2 min_size 2 crush_ruleset 0 object_hash
rjenkins pg_num 128 pgp_num 128 last_change 118 flags hashpspool stripe_width 0
removed_snaps [1~3,7~4]
pool 3 'backups' replicated size 2 min_size 2 crush_ruleset 0 object_hash
rjenkins pg_num 128 pgp_num 128 last_change 120 flags hashpspool stripe_width 0
pool 4 'vms' replicated size 2 min_size 2 crush_ruleset 0 object_hash rjenkins
pg_num 128 pgp_num 128 last_change 122 flags hashpspool stripe_width 0
removed_snaps [1~7]
pool 5 'volumes' replicated size 2 min_size 2 crush_ruleset 0 object_hash
rjenkins pg_num 128 pgp_num 128 last_change 124 flags hashpspool stripe_width 0
removed_snaps [1~3]
pool 6 'scbench' replicated size 2 min_size 2 crush_ruleset 0 object_hash
rjenkins pg_num 100 pgp_num 100 last_change 126 flags hashpspool stripe_width 0
pool 7 'rbdbench' replicated size 2 min_size 2 crush_ruleset 0 object_hash
rjenkins pg_num 100 pgp_num 100 last_change 128 flags hashpspool stripe_width 0
removed_snaps [1~3]
[root@zeus-59 ceph-block-device]# ceph osd tree
ID WEIGHT TYPE NAMEUP/DOWN REWEIGHT PRIMARY-AFFINITY
-1 36.17371 root default
-2 18.08685 host zeus-58
0 1.80869 osd.0 up 1.0 1.0
2 1.80869 osd.2 up 1.0 1.0
4 1.80869 osd.4 up 1.0 1.0
6 1.80869 osd.6 up 1.0 1.0
8 1.80869 osd.8 up 1.0 1.0
10 1.80869 osd.10up 1.0 1.0
12 1.80869 osd.12up 1.0 1.0
14 1.80869 osd.14up 1.0 1.0
16 1.80869 osd.16up 1.0 1.0
18 1.80869 osd.18up 1.0 1.0
-3 18.08685 host zeus-59
1 1.80869 osd.1 up 1.0 1.0
3 1.80869 osd.3 up 1.0 1.0
5 1.80869 osd.5 up 1.0 1.0
7 1.80869 osd.7 up 1.0 1.0
9 1.80869 osd.9 up 1.0 1.0
11 1.80869 osd.11up 1.0 1.0
13 1.80869 osd.13up 1.0 1.0
15 1.80869 osd.15up 1.0 1.0
17 1.80869 osd.17up 1.0 1.0
19 1.80869 osd.19up 1.0 1.0
[root@zeus-59 ceph-block-device]# ceph status
cluster 8e930b6c-455e-4328-872d-cb9f5c0359ae
health HEALTH_OK
monmap e1: 1 mons at {zeus-59=10.0.32.59:6789/0}
election epoch 3, quorum 0 zeus-59
osdmap e129: 20 osds: 20 up, 20 in
flags sortbitwise,require_jewel_osds
pgmap v1166945: 776 pgs, 7 pools, 1183 GB data, 296 kobjects
2363 GB used, 34678 GB / 37042 GB avail
775 active+clean
1 active+clean+scrubbing+deep
[root@zeus-59 ceph-block-device]# rados bench -p scbench 10 write --no-cleanup
Maintaining 16 concurrent writes of 4194304 bytes to objects of size 4194304
for up to 10 seconds or 0 objects
Object prefix: benchmark_data_zeus-59.localdomain_2844050
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
0 0 0 0 0 0 - 0
1 16 644 6282511.4 2512 0.02102730.025206
2 16 1319 1303 2605.49 2700 0.0238678 0.0243974
3 16 2003 1987 2648.89 2736 0.0201334 0.0240726
4 16 2669 2653 2652.59 2664 0.0258618 0.0240468
5 16 3349 2666.01 2720 0.0189464 0.0239484
6 16 4026 4010 2672.96 2708 0.02215 0.0238954
7 16 4697 4681 2674.49 2684 0.0217258 0.0238887
8 16 5358 5342 2670.64 2644 0.0265384 0.0239066
9 16 6043 60272678.3 2740 0.0260798 0.0238637
10 16 6731 6715 2685.64 2752 0.0174624 0.0237982
Total time run: 10.026091
Total writes made: 6731
Write size: 4194304
Object size:4194304
Bandwidth (MB/sec): 2685.39
