Hello Community
Currently, I operate a CEPH Cluster utilizing Ceph Octopus version 1.5.2.7,
installed through Ansible. The challenge I'm encountering is that, during
scrubbing, OSD latency spikes to 300-600 ms, resulting in sluggish
performance for all VMs.
Additionally, some OSDs fail during the scrubbing process. In such
instances, promptly halting the scrubbing resolves the issue.
*Summary of CEPH Version*:
CEPH Version Number of Nodes Node Networking OSD OSD Total pools PG Size
15.2.7 12
(6 SSD node + 6 HDD node) All nodes are connected through 10G bonded link,
i.e. 10Gx2=20GB for each node. 64 SSD
42 HDD 106 one-ssd 256 active+clean
one-hdd 512 active+clean
cloudstack.hdd 512 active+clean
I intend to enlarge the PG size for the "one-ssd" configuration. Please
provide the PG number, and suggest the optimal approach to increase the PG
size without causing any slowdown or service disruptions to the VMs.
Your expertise and guidance on this matter would be highly valuable, and
I'm eager to benefit from the collective knowledge of the Ceph community.
Thank you in advance for your time and assistance. I look forward to
hearing from you.
*CEPH Health status:*
root@mon1:~# ceph -s
cluster:
id: f8096ec7-51db-4557-85e6-57d7fdfe9423
health: HEALTH_WARN
nodeep-scrub flag(s) set
656 pgs not deep-scrubbed in time
services:
mon: 3 daemons, quorum ceph2,mon1,ceph6 (age 4d)
mgr: ceph4(active, since 2w), standbys: mon1, ceph3, ceph6, ceph1
mds: cephfs:1 {0=ceph8=up:active} 1 up:standby
osd: 107 osds: 105 up (since 3h), 105 in (since 3d)
flags nodeep-scrub
rgw: 4 daemons active (ceph10.rgw0, ceph7.rgw0, ceph9.rgw0,
mon1.rgw0)
rgw-nfs: 2 daemons active (ceph7, ceph9)
task status:
data:
pools: 13 pools, 2057 pgs
objects: 9.40M objects, 35 TiB
usage: 106 TiB used, 154 TiB / 259 TiB avail
pgs: 2057 active+clean
io:
client: 14 MiB/s rd, 30 MiB/s wr, 1.50k op/s rd, 1.53k op/s wr
root@ceph1:~# ceph df
--- RAW STORAGE ---
CLASS SIZE AVAIL USED RAW USED %RAW USED
hdd 151 TiB 78 TiB 72 TiB 73 TiB 48.04
ssd 110 TiB 78 TiB 32 TiB 32 TiB 29.42
TOTAL 261 TiB 156 TiB 104 TiB 105 TiB 40.19
--- POOLS ---
POOL ID PGS STORED OBJECTS USED %USED MAX
AVAIL
cephfs_data 1 64 3.8 KiB 0 11 KiB 0
23 TiB
cephfs_metadata 2 8 228 MiB 79 685 MiB 0
23 TiB
.rgw.root 3 32 6.0 KiB 8 1.5 MiB 0
23 TiB
default.rgw.control 4 32 0 B 8 0 B 0
23 TiB
default.rgw.meta 5 32 12 KiB 48 7.5 MiB 0
23 TiB
default.rgw.log 6 32 4.8 KiB 207 6.0 MiB 0
23 TiB
default.rgw.buckets.index 7 32 410 MiB 15 1.2 GiB 0
23 TiB
default.rgw.buckets.data 8 512 4.6 TiB 1.29M 14 TiB 16.59
23 TiB
default.rgw.buckets.non-ec 9 32 1.0 MiB 676 130 MiB 0
23 TiB
one-hdd 10 512 9.2 TiB 2.45M 28 TiB 28.69
23 TiB
device_health_metrics 11 1 9.5 MiB 113 28 MiB 0
23 TiB
one-ssd 12 256 11 TiB 2.88M 32 TiB 31.37
23 TiB
cloudstack.hdd 15 512 10 TiB 2.72M 31 TiB 30.94
23 TiB
Regards
Mosharaf Hossain
Manager, Product Development
IT Division
Bangladesh Export Import Company Ltd.
Level-8, SAM Tower, Plot #4, Road #22, Gulshan-1, Dhaka-1212,Bangladesh
Tel: +880 9609 000 999, +880 2 5881 5559, Ext: 14191, Fax: +880 2 9895757
Cell: +8801787680828, Email: [email protected], Web:
www.bol-online.com
<https://www.google.com/url?q=http://www.bol-online.com&sa=D&source=hangouts&ust=1557908951423000&usg=AFQjCNGMxIuHSHsD3qO6y5JddpEZ0S592A>
_______________________________________________
ceph-users mailing list -- [email protected]
To unsubscribe send an email to [email protected]
