Hello Laura
I have created the tracker and you can find it on
https://tracker.ceph.com/issues/62836
Please find the OSD map as below on the cluster
*root@cph2n1:/# ceph -s* cluster:
id: e5f5ec6e-0b1b-11ec-adc5-35a84c0db1fb
health: HEALTH_WARN
65 pgs not deep-scrubbed in time
services:
mon: 5 daemons, quorum cph2n1,cph2n2,cph2n3,cph2n5,cph2n4 (age 17h)
mgr: cph2n6.kijcmg(active, since 17h), standbys: cph2n5.czgzpy,
cph2n2.dqlmbn, cph2n4.gppcct, cph2n1.ezsddw
osd: 56 osds: 56 up (since 8h), 56 in (since 8d)
data:
pools: 12 pools, 1913 pgs
objects: 5.04M objects, 19 TiB
usage: 57 TiB used, 98 TiB / 154 TiB avail
pgs: 1913 active+clean
io:
client: 13 KiB/s rd, 2.8 MiB/s wr, 33 op/s rd, 44 op/s wr
*root@cph2n1:/# ceph osd tree*ID CLASS WEIGHT TYPE NAME
STATUS REWEIGHT PRI-AFF
-1 154.42514 root default
-3 24.59732 host cph2n1
6 ent_ssd 3.49309 osd.6 up 1.00000 1.00000
7 ent_ssd 3.49309 osd.7 up 1.00000 1.00000
8 ent_ssd 3.49309 osd.8 up 1.00000 1.00000
9 ent_ssd 3.49309 osd.9 up 1.00000 1.00000
32 ent_ssd 3.49309 osd.32 up 1.00000 1.00000
37 ent_ssd 3.49309 osd.37 up 1.00000 1.00000
0 nvme_ssd 1.81940 osd.0 up 1.00000 1.00000
1 nvme_ssd 1.81940 osd.1 up 1.00000 1.00000
-5 24.59732 host cph2n2
10 ent_ssd 3.49309 osd.10 up 1.00000 1.00000
11 ent_ssd 3.49309 osd.11 up 1.00000 1.00000
12 ent_ssd 3.49309 osd.12 up 1.00000 1.00000
13 ent_ssd 3.49309 osd.13 up 1.00000 1.00000
31 ent_ssd 3.49309 osd.31 up 1.00000 1.00000
35 ent_ssd 3.49309 osd.35 up 1.00000 1.00000
2 nvme_ssd 1.81940 osd.2 up 1.00000 1.00000
3 nvme_ssd 1.81940 osd.3 up 1.00000 1.00000
-7 24.59732 host cph2n3
14 ent_ssd 3.49309 osd.14 up 1.00000 1.00000
15 ent_ssd 3.49309 osd.15 up 1.00000 1.00000
16 ent_ssd 3.49309 osd.16 up 1.00000 1.00000
17 ent_ssd 3.49309 osd.17 up 1.00000 1.00000
30 ent_ssd 3.49309 osd.30 up 1.00000 1.00000
36 ent_ssd 3.49309 osd.36 up 1.00000 1.00000
4 nvme_ssd 1.81940 osd.4 up 1.00000 1.00000
5 nvme_ssd 1.81940 osd.5 up 1.00000 1.00000
-21 28.09041 host cph2n4
22 ent_ssd 3.49309 osd.22 up 1.00000 1.00000
24 ent_ssd 3.49309 osd.24 up 1.00000 1.00000
26 ent_ssd 3.49309 osd.26 up 1.00000 1.00000
29 ent_ssd 3.49309 osd.29 up 1.00000 1.00000
33 ent_ssd 3.49309 osd.33 up 1.00000 1.00000
39 ent_ssd 3.49309 osd.39 up 1.00000 1.00000
41 ent_ssd 3.49309 osd.41 up 1.00000 1.00000
20 nvme_ssd 1.81940 osd.20 up 1.00000 1.00000
21 nvme_ssd 1.81940 osd.21 up 1.00000 1.00000
-17 28.09041 host cph2n5
23 ent_ssd 3.49309 osd.23 up 1.00000 1.00000
25 ent_ssd 3.49309 osd.25 up 1.00000 1.00000
27 ent_ssd 3.49309 osd.27 up 1.00000 1.00000
28 ent_ssd 3.49309 osd.28 up 1.00000 1.00000
34 ent_ssd 3.49309 osd.34 up 1.00000 1.00000
38 ent_ssd 3.49309 osd.38 up 1.00000 1.00000
40 ent_ssd 3.49309 osd.40 up 1.00000 1.00000
18 nvme_ssd 1.81940 osd.18 up 1.00000 1.00000
19 nvme_ssd 1.81940 osd.19 up 1.00000 1.00000
-25 12.22618 host cph2n6
42 ent_ssd 1.74660 osd.42 up 1.00000 1.00000
43 ent_ssd 1.74660 osd.43 up 1.00000 1.00000
44 ent_ssd 1.74660 osd.44 up 1.00000 1.00000
45 ent_ssd 1.74660 osd.45 up 1.00000 1.00000
49 ent_ssd 1.74660 osd.49 up 1.00000 1.00000
50 ent_ssd 1.74660 osd.50 up 1.00000 1.00000
51 ent_ssd 1.74660 osd.51 up 1.00000 1.00000
-29 12.22618 host cph2n7
52 ent_ssd 1.74660 osd.52 up 1.00000 1.00000
53 ent_ssd 1.74660 osd.53 up 1.00000 1.00000
54 ent_ssd 1.74660 osd.54 up 1.00000 1.00000
59 ent_ssd 1.74660 osd.59 up 1.00000 1.00000
60 ent_ssd 1.74660 osd.60 up 1.00000 1.00000
61 ent_ssd 1.74660 osd.61 up 1.00000 1.00000
62 ent_ssd 1.74660 osd.62 up 1.00000 1.00000
Regards
Mosharaf Hossain
Manager, Product Development
IT Division
Bangladesh Export Import Company Ltd.
Level-8, SAM Tower, Plot #4, Road #22, Gulshan-1, Dhaka-1212,Bangladesh
Tel: +880 9609 000 999, +880 2 5881 5559, Ext: 14191, Fax: +880 2 9895757
Cell: +8801787680828, Email: [email protected], Web:
www.bol-online.com
<https://www.google.com/url?q=http://www.bol-online.com&sa=D&source=hangouts&ust=1557908951423000&usg=AFQjCNGMxIuHSHsD3qO6y5JddpEZ0S592A>
On Thu, Sep 14, 2023 at 4:38 AM Laura Flores <[email protected]> wrote:
> Link the tracker on this list if you have it. You can create one under the
> RADOS project: https://tracker.ceph.com/projects/rados
>
> Thanks,
> Laura
>
> On Wed, Sep 13, 2023 at 4:35 PM Laura Flores <[email protected]> wrote:
>
>> Hi Mosharaf,
>>
>> Can you please create a tracker issue and attach a copy of your osdmap?
>> Also, please include any other output that characterizes the slowdown in
>> client I/O operations you're noticing in your cluster. I can take a look
>> once I have that information,
>>
>> Thanks,
>> Laura
>>
>> On Wed, Sep 13, 2023 at 5:23 AM Mosharaf Hossain <
>> [email protected]> wrote:
>>
>>> Hello Folks
>>> We've recently performed an upgrade on our Cephadm cluster, transitioning
>>> from Ceph Quiency to Reef. However, following the manual implementation
>>> of
>>> a read balancer in the Reef cluster, we've experienced a significant
>>> slowdown in client I/O operations within the Ceph cluster, affecting both
>>> client bandwidth and overall cluster performance.
>>>
>>> This slowdown has resulted in unresponsiveness across all virtual
>>> machines
>>> within the cluster, despite the fact that the cluster exclusively
>>> utilizes
>>> SSD storage."
>>>
>>> Kindly guide us to move forward.
>>>
>>>
>>>
>>> Regards
>>> Mosharaf Hossain
>>> _______________________________________________
>>> ceph-users mailing list -- [email protected]
>>> To unsubscribe send an email to [email protected]
>>>
>>>
>>
>> --
>>
>> Laura Flores
>>
>> She/Her/Hers
>>
>> Software Engineer, Ceph Storage <https://ceph.io>
>>
>> Chicago, IL
>>
>> [email protected] | [email protected] <[email protected]>
>> M: +17087388804
>>
>>
>>
>
> --
>
> Laura Flores
>
> She/Her/Hers
>
> Software Engineer, Ceph Storage <https://ceph.io>
>
> Chicago, IL
>
> [email protected] | [email protected] <[email protected]>
> M: +17087388804
>
>
>
_______________________________________________
ceph-users mailing list -- [email protected]
To unsubscribe send an email to [email protected]
