[ceph-users] Re: How check local network

[David C.]

Hello Albert,

this should return you the sockets used on the network cluster :

ceph report | jq '.osdmap.osds[] | .cluster_addrs.addrvec[] | .addr'



Cordialement,

*David CASIER*





Le lun. 29 janv. 2024 à 22:52, Albert Shih  a écrit :

> Le 29/01/2024 à 22:43:46+0100, Albert Shih a écrit
> > Hi
> >
> > When I deploy my cluster I didn't notice on two of my servers the private
> > network was not working (wrong vlan), now it's working, but how can I
> check
> > the it's indeed working (currently I don't have data).
>
> I mean...ceph going to use it...sorry.
>
> --
> Albert SHIH 嶺 
> France
> Heure locale/Local time:
> lun. 29 janv. 2024 22:50:36 CET
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: How check local network

[Albert Shih]

Le 29/01/2024 à 22:43:46+0100, Albert Shih a écrit
> Hi
> 
> When I deploy my cluster I didn't notice on two of my servers the private
> network was not working (wrong vlan), now it's working, but how can I check
> the it's indeed working (currently I don't have data). 

I mean...ceph going to use it...sorry.

-- 
Albert SHIH 嶺 
France
Heure locale/Local time:
lun. 29 janv. 2024 22:50:36 CET
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] How check local network

[Albert Shih]

Hi

When I deploy my cluster I didn't notice on two of my servers the private
network was not working (wrong vlan), now it's working, but how can I check
the it's indeed working (currently I don't have data). 

Regards

-- 
Albert SHIH 嶺 
France
Heure locale/Local time:
lun. 29 janv. 2024 22:36:01 CET
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] pacific 16.2.15 QE validation status

[Yuri Weinstein]

Details of this release are summarized here:

https://tracker.ceph.com/issues/64151#note-1

Seeking approvals/reviews for:

rados - Radek, Laura, Travis, Ernesto, Adam King
rgw - Casey
fs - Venky
rbd - Ilya
krbd - in progress

upgrade/nautilus-x (pacific) - Casey PTL (regweed tests failed)
upgrade/octopus-x (pacific) - Casey PTL (regweed tests failed)

upgrade/pacific-x (quincy) - in progress
upgrade/pacific-p2p - Ilya PTL (maybe rbd related?)

ceph-volume - Guillaume

TIA
YuriW
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Unsetting maintenance mode for failed host

[Eugen Block]

Hi,

if you just want the cluster to drain this host but bring it back  
online soon I would just remove the noout flag:


ceph osd rm-noout osd1

This flag is set when entering maintenance mode (ceph osd add-noout  
). But it would not remove the health warning (host is in  
maintenance) until the host is back. One (potentially dangerous) way  
to clear that warning would be to remove the host from the host list  
(ceph orch host rm ), but this can potentially cause data loss  
since the entire host will be removed from the crushmap. So I would  
only choose that path after all PGs have been backfilled successfully  
and your attempts to bring that host back takes longer than expected.


Maybe there are more options which I haven't thought of, but these two  
came to mind.


Regards,
Eugen

Zitat von Bryce Nicholls :


Hi

We put a host in maintenance and had issues bringing it back.
Is there a safe way of exiting maintenance while the host is  
unreachable / offline?
We would like the cluster to rebalance while we are working to get  
this host back online.


Maintenance was set using:
ceph orch host maintenance enter osd1

I tried exiting using:
ceph orch host maintenance exit osd1

but got the below stacktrace.

root@mon1 ~ # ceph orch host maintenance exit osd1

Error EINVAL: Traceback (most recent call last):
  File "/usr/share/ceph/mgr/mgr_module.py", line 1756, in _handle_command
return self.handle_command(inbuf, cmd)
  File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 171,  
in handle_command

return dispatch[cmd['prefix']].call(self, cmd, inbuf)
  File "/usr/share/ceph/mgr/mgr_module.py", line 462, in call
return self.func(mgr, **kwargs)
  File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 107,  
in 
wrapper_copy = lambda *l_args, **l_kwargs: wrapper(*l_args,  
**l_kwargs)  # noqa: E731

  File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 96, in wrapper
return func(*args, **kwargs)
  File "/usr/share/ceph/mgr/orchestrator/module.py", line 455, in  
_host_maintenance_exit

raise_if_exception(completion)
  File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 225,  
in raise_if_exception

e = pickle.loads(c.serialized_exception)
TypeError: __init__() missing 2 required positional arguments:  
'hostname' and 'addr'



Thanks
Bryce


Bryce Nicholls
OpenStack Engineer
bryce.nicholl...@thehutgroup.com
[THG Ingenuity Logo]
[https://i.imgur.com/wbpVRW6.png]  [https://i.imgur.com/c3040tr.png]  


___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io



___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: 6 pgs not deep-scrubbed in time

[Frank Schilder]

You will have to look at the output of "ceph df" and make a decision to balance 
"objects per PG" and "GB per PG". Increase he PG count for the pools with the 
worst of these two numbers most such that it balances out as much as possible. 
If you have pools that see significantly more user-IO than others, prioritise 
these. 

You will have to find out for your specific cluster, we can only give general 
guidelines. Make changes, run benchmarks, re-evaluate. Take the time for it. 
The better you know your cluster and your users, the better the end result will 
be.

Best regards,
=
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14


From: Michel Niyoyita 
Sent: Monday, January 29, 2024 2:04 PM
To: Janne Johansson
Cc: Frank Schilder; E Taka; ceph-users
Subject: Re: [ceph-users] Re: 6 pgs not deep-scrubbed in time

This is how it is set , if you suggest to make some changes please advises.

Thank you.


ceph osd pool ls detail
pool 1 'device_health_metrics' replicated size 3 min_size 2 crush_rule 0 
object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode on last_change 1407 
flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application 
mgr_devicehealth
pool 2 '.rgw.root' replicated size 3 min_size 2 crush_rule 0 object_hash 
rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 1393 flags 
hashpspool stripe_width 0 application rgw
pool 3 'default.rgw.log' replicated size 3 min_size 2 crush_rule 0 object_hash 
rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 1394 flags 
hashpspool stripe_width 0 application rgw
pool 4 'default.rgw.control' replicated size 3 min_size 2 crush_rule 0 
object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 1395 
flags hashpspool stripe_width 0 application rgw
pool 5 'default.rgw.meta' replicated size 3 min_size 2 crush_rule 0 object_hash 
rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 1396 flags 
hashpspool stripe_width 0 pg_autoscale_bias 4 application rgw
pool 6 'volumes' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins 
pg_num 128 pgp_num 128 autoscale_mode on last_change 108802 lfor 0/0/14812 
flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd
removed_snaps_queue 
[22d7~3,11561~2,11571~1,11573~1c,11594~6,1159b~f,115b0~1,115b3~1,115c3~1,115f3~1,115f5~e,11613~6,1161f~c,11637~1b,11660~1,11663~2,11673~1,116d1~c,116f5~10,11721~c]
pool 7 'images' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins 
pg_num 32 pgp_num 32 autoscale_mode on last_change 94609 flags 
hashpspool,selfmanaged_snaps stripe_width 0 application rbd
pool 8 'backups' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins 
pg_num 32 pgp_num 32 autoscale_mode on last_change 1399 flags hashpspool 
stripe_width 0 application rbd
pool 9 'vms' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins 
pg_num 32 pgp_num 32 autoscale_mode on last_change 108783 lfor 0/561/559 flags 
hashpspool,selfmanaged_snaps stripe_width 0 application rbd
removed_snaps_queue [3fa~1,3fc~3,400~1,402~1]
pool 10 'testbench' replicated size 3 min_size 2 crush_rule 0 object_hash 
rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 20931 lfor 
0/20931/20929 flags hashpspool stripe_width 0


On Mon, Jan 29, 2024 at 2:09 PM Michel Niyoyita 
mailto:mico...@gmail.com>> wrote:
Thank you Janne ,

no need of setting some flags like ceph osd set nodeep-scrub  ???

Thank you

On Mon, Jan 29, 2024 at 2:04 PM Janne Johansson 
mailto:icepic...@gmail.com>> wrote:
Den mån 29 jan. 2024 kl 12:58 skrev Michel Niyoyita 
mailto:mico...@gmail.com>>:
>
> Thank you Frank ,
>
> All disks are HDDs . Would like to know if I can increase the number of PGs
> live in production without a negative impact to the cluster. if yes which
> commands to use .

Yes. "ceph osd pool set  pg_num "
where the number usually should be a power of two that leads to a
number of PGs per OSD between 100-200.

--
May the most significant bit of your life be positive.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: 6 pgs not deep-scrubbed in time

[Frank Schilder]

Setting osd_max_scrubs = 2 for HDD OSDs was a mistake I made. The result was 
that PGs needed a bit more than twice as long to deep-scrub. Net effect: high 
scrub load, much less user IO and, last but not least, the "not deep-scrubbed 
in time" problem got worse, because (2+eps)/2 > 1.

For spinners a consideration looking at the actually available drive 
performance is required, plus a few things more, like PG count, distribution 
etc.

Best regards,
=
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14


From: Wesley Dillingham 
Sent: Monday, January 29, 2024 7:14 PM
To: Michel Niyoyita
Cc: Josh Baergen; E Taka; ceph-users
Subject: [ceph-users] Re: 6 pgs not deep-scrubbed in time

Respond back with "ceph versions" output

If your sole goal is to eliminate the not scrubbed in time errors you can
increase the aggressiveness of scrubbing by setting:
osd_max_scrubs = 2

The default in pacific is 1.

if you are going to start tinkering manually with the pg_num you will want
to turn off the pg autoscaler on the pools you are touching.
reducing the size of your PGs may make sense and help with scrubbing but if
the pool has a lot of data it will take a long long time to finish.





Respectfully,

*Wes Dillingham*
w...@wesdillingham.com
LinkedIn 


On Mon, Jan 29, 2024 at 10:08 AM Michel Niyoyita  wrote:

> I am running ceph pacific , version 16 , ubuntu 20 OS , deployed using
> ceph-ansible.
>
> Michel
>
> On Mon, Jan 29, 2024 at 4:47 PM Josh Baergen 
> wrote:
>
> > Make sure you're on a fairly recent version of Ceph before doing this,
> > though.
> >
> > Josh
> >
> > On Mon, Jan 29, 2024 at 5:05 AM Janne Johansson 
> > wrote:
> > >
> > > Den mån 29 jan. 2024 kl 12:58 skrev Michel Niyoyita  >:
> > > >
> > > > Thank you Frank ,
> > > >
> > > > All disks are HDDs . Would like to know if I can increase the number
> > of PGs
> > > > live in production without a negative impact to the cluster. if yes
> > which
> > > > commands to use .
> > >
> > > Yes. "ceph osd pool set  pg_num "
> > > where the number usually should be a power of two that leads to a
> > > number of PGs per OSD between 100-200.
> > >
> > > --
> > > May the most significant bit of your life be positive.
> > > ___
> > > ceph-users mailing list -- ceph-users@ceph.io
> > > To unsubscribe send an email to ceph-users-le...@ceph.io
> >
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: 6 pgs not deep-scrubbed in time

[Wesley Dillingham]

Respond back with "ceph versions" output

If your sole goal is to eliminate the not scrubbed in time errors you can
increase the aggressiveness of scrubbing by setting:
osd_max_scrubs = 2

The default in pacific is 1.

if you are going to start tinkering manually with the pg_num you will want
to turn off the pg autoscaler on the pools you are touching.
reducing the size of your PGs may make sense and help with scrubbing but if
the pool has a lot of data it will take a long long time to finish.





Respectfully,

*Wes Dillingham*
w...@wesdillingham.com
LinkedIn 


On Mon, Jan 29, 2024 at 10:08 AM Michel Niyoyita  wrote:

> I am running ceph pacific , version 16 , ubuntu 20 OS , deployed using
> ceph-ansible.
>
> Michel
>
> On Mon, Jan 29, 2024 at 4:47 PM Josh Baergen 
> wrote:
>
> > Make sure you're on a fairly recent version of Ceph before doing this,
> > though.
> >
> > Josh
> >
> > On Mon, Jan 29, 2024 at 5:05 AM Janne Johansson 
> > wrote:
> > >
> > > Den mån 29 jan. 2024 kl 12:58 skrev Michel Niyoyita  >:
> > > >
> > > > Thank you Frank ,
> > > >
> > > > All disks are HDDs . Would like to know if I can increase the number
> > of PGs
> > > > live in production without a negative impact to the cluster. if yes
> > which
> > > > commands to use .
> > >
> > > Yes. "ceph osd pool set  pg_num "
> > > where the number usually should be a power of two that leads to a
> > > number of PGs per OSD between 100-200.
> > >
> > > --
> > > May the most significant bit of your life be positive.
> > > ___
> > > ceph-users mailing list -- ceph-users@ceph.io
> > > To unsubscribe send an email to ceph-users-le...@ceph.io
> >
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Unsetting maintenance mode for failed host

[Bryce Nicholls]

Hi

We put a host in maintenance and had issues bringing it back.
Is there a safe way of exiting maintenance while the host is unreachable / 
offline?
We would like the cluster to rebalance while we are working to get this host 
back online.

Maintenance was set using:
ceph orch host maintenance enter osd1

I tried exiting using:
ceph orch host maintenance exit osd1

but got the below stacktrace.

root@mon1 ~ # ceph orch host maintenance exit osd1

Error EINVAL: Traceback (most recent call last):
  File "/usr/share/ceph/mgr/mgr_module.py", line 1756, in _handle_command
return self.handle_command(inbuf, cmd)
  File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 171, in 
handle_command
return dispatch[cmd['prefix']].call(self, cmd, inbuf)
  File "/usr/share/ceph/mgr/mgr_module.py", line 462, in call
return self.func(mgr, **kwargs)
  File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 107, in 
wrapper_copy = lambda *l_args, **l_kwargs: wrapper(*l_args, **l_kwargs)  # 
noqa: E731
  File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 96, in wrapper
return func(*args, **kwargs)
  File "/usr/share/ceph/mgr/orchestrator/module.py", line 455, in 
_host_maintenance_exit
raise_if_exception(completion)
  File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 225, in 
raise_if_exception
e = pickle.loads(c.serialized_exception)
TypeError: __init__() missing 2 required positional arguments: 'hostname' and 
'addr'


Thanks
Bryce


Bryce Nicholls
OpenStack Engineer
bryce.nicholl...@thehutgroup.com
[THG Ingenuity Logo]
[https://i.imgur.com/wbpVRW6.png]
  [https://i.imgur.com/c3040tr.png] 
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: 6 pgs not deep-scrubbed in time

[Josh Baergen]

You need to be running at least 16.2.11 on the OSDs so that you have
the fix for https://tracker.ceph.com/issues/55631.

On Mon, Jan 29, 2024 at 8:07 AM Michel Niyoyita  wrote:
>
> I am running ceph pacific , version 16 , ubuntu 20 OS , deployed using 
> ceph-ansible.
>
> Michel
>
> On Mon, Jan 29, 2024 at 4:47 PM Josh Baergen  
> wrote:
>>
>> Make sure you're on a fairly recent version of Ceph before doing this, 
>> though.
>>
>> Josh
>>
>> On Mon, Jan 29, 2024 at 5:05 AM Janne Johansson  wrote:
>> >
>> > Den mån 29 jan. 2024 kl 12:58 skrev Michel Niyoyita :
>> > >
>> > > Thank you Frank ,
>> > >
>> > > All disks are HDDs . Would like to know if I can increase the number of 
>> > > PGs
>> > > live in production without a negative impact to the cluster. if yes which
>> > > commands to use .
>> >
>> > Yes. "ceph osd pool set  pg_num "
>> > where the number usually should be a power of two that leads to a
>> > number of PGs per OSD between 100-200.
>> >
>> > --
>> > May the most significant bit of your life be positive.
>> > ___
>> > ceph-users mailing list -- ceph-users@ceph.io
>> > To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] January Ceph Science Virtual User Group

[Kevin Hrpcek]

Hey All,

We will be having a Ceph science/research/big cluster call on Wednesday 
January 31st. If anyone wants to discuss something specific they can add 
it to the pad linked below. If you have questions or comments you can 
contact me.


This is an informal open call of community members mostly from 
hpc/htc/research/big cluster environments (though anyone is welcome) 
where we discuss whatever is on our minds regarding ceph. Updates, 
outages, features, maintenance, etc...there is no set presenter but I do 
attempt to keep the conversation lively.


Pad URL:
https://pad.ceph.com/p/Ceph_Science_User_Group_20240131

Virtual event details:
January 31, 2024
15:00 UTC
4pm Central European
9am Central US

Description: Main pad for discussions: 
https://pad.ceph.com/p/Ceph_Science_User_Group_Index

Meetings will be recorded and posted to the Ceph Youtube channel.

To join the meeting on a computer or mobile phone: 
https://meet.jit.si/ceph-science-wg


Kevin

--
Kevin Hrpcek
NASA VIIRS Atmosphere SIPS/TROPICS
Space Science & Engineering Center
University of Wisconsin-Madison
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: 6 pgs not deep-scrubbed in time

[Michel Niyoyita]

I am running ceph pacific , version 16 , ubuntu 20 OS , deployed using
ceph-ansible.

Michel

On Mon, Jan 29, 2024 at 4:47 PM Josh Baergen 
wrote:

> Make sure you're on a fairly recent version of Ceph before doing this,
> though.
>
> Josh
>
> On Mon, Jan 29, 2024 at 5:05 AM Janne Johansson 
> wrote:
> >
> > Den mån 29 jan. 2024 kl 12:58 skrev Michel Niyoyita :
> > >
> > > Thank you Frank ,
> > >
> > > All disks are HDDs . Would like to know if I can increase the number
> of PGs
> > > live in production without a negative impact to the cluster. if yes
> which
> > > commands to use .
> >
> > Yes. "ceph osd pool set  pg_num "
> > where the number usually should be a power of two that leads to a
> > number of PGs per OSD between 100-200.
> >
> > --
> > May the most significant bit of your life be positive.
> > ___
> > ceph-users mailing list -- ceph-users@ceph.io
> > To unsubscribe send an email to ceph-users-le...@ceph.io
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: 6 pgs not deep-scrubbed in time

[Josh Baergen]

Make sure you're on a fairly recent version of Ceph before doing this, though.

Josh

On Mon, Jan 29, 2024 at 5:05 AM Janne Johansson  wrote:
>
> Den mån 29 jan. 2024 kl 12:58 skrev Michel Niyoyita :
> >
> > Thank you Frank ,
> >
> > All disks are HDDs . Would like to know if I can increase the number of PGs
> > live in production without a negative impact to the cluster. if yes which
> > commands to use .
>
> Yes. "ceph osd pool set  pg_num "
> where the number usually should be a power of two that leads to a
> number of PGs per OSD between 100-200.
>
> --
> May the most significant bit of your life be positive.
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: RadosGW manual deployment

[Janne Johansson]

> If there is a (planned) documentation of manual rgw bootstrapping,
> it would be nice to have also the names of required pools listed there.

It will depend on several things, like if you enable swift users, I
think they get a pool of their own, so I guess one would need to look
in the source for a full list of potential pools made by rgw.

> # verify it works
> curl http://127.0.0.1:8088
> ceph osd pool ls
> # should print at least the following pools:
> # .rgw.root
> # default.rgw.log
> # default.rgw.control
> # default.rgw.meta
> # ... and maybe also these, after some buckets are created:
> # default.rgw.buckets.index
> # default.rgw.buckets.non-ec
> # default.rgw.buckets.data
> 

My oldest cluster seems to have these ones: (some may not be relevant anymore)

.rgw.root 5  10.0KiB 0
10.8TiB   23
default.rgw.control   6   0B 0
10.8TiB8
default.rgw.data.root 7  64.7KiB 0
10.8TiB  186
default.rgw.gc8   0B 0
10.8TiB   32
default.rgw.log   9   685GiB  5.85
10.8TiB   159903
default.rgw.users.uid 10 5.32KiB 0
10.8TiB   28
default.rgw.usage 11  0B 0
10.8TiB   13
default.rgw.users.keys12459B 0
10.8TiB   14
default.rgw.meta  13  330KiB 0
10.8TiB  923
default.rgw.buckets.index 14  0B 0
10.8TiB  184
default.rgw.buckets.non-ec15  0B 0
10.8TiB  828
default.rgw.buckets.data  16 6.39TiB 13.09
42.5TiB  2777682
default.rgw.users.email   23115B 0
10.8TiB4

so .log and .usage probably need some traffic before they appear.

-- 
May the most significant bit of your life be positive.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: RadosGW manual deployment

[Eugen Block]

Hi,

I was just curious what your intentions are, not meaning to critisize  
it. ;-) There are different reasons why that could be a better choice.  
And as I already mentioned previously, you only would have stray  
daemons warnings if you deployed the RGWs on hosts which already have  
cephadm managed daemons.


Thanks,
Eugen

Zitat von Jan Kasprzak :


Hello, Eugen,

Eugen Block wrote:

Janne was a bit quicker than me, so I'll skip my short instructions
how to deploy it manually. But your (cephadm managed) cluster will
complain about "stray daemons". There doesn't seem to be a way to
deploy rgw daemons manually with the cephadm tool so it wouldn't be
stray. Is there a specific reason not to use the orchestrator for
rgw deployment?


I don't want to start an opinion war here :-), but the reason is
more or less that I want to be able to see under the hood (at least
to some degree :-). So for now I want to run a plain, non-containerized
non-orchestrated setup.

Thanks all for your help, it works for me now.

-Yenya

--
| Jan "Yenya" Kasprzak  |
| https://www.fi.muni.cz/~kas/GPG: 4096R/A45477D5 |
We all agree on the necessity of compromise. We just can't agree on
when it's necessary to compromise. --Larry Wall



___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: RadosGW manual deployment

[Jan Kasprzak]

Hello, Eugen,

Eugen Block wrote:
> Janne was a bit quicker than me, so I'll skip my short instructions
> how to deploy it manually. But your (cephadm managed) cluster will
> complain about "stray daemons". There doesn't seem to be a way to
> deploy rgw daemons manually with the cephadm tool so it wouldn't be
> stray. Is there a specific reason not to use the orchestrator for
> rgw deployment?

I don't want to start an opinion war here :-), but the reason is
more or less that I want to be able to see under the hood (at least
to some degree :-). So for now I want to run a plain, non-containerized
non-orchestrated setup.

Thanks all for your help, it works for me now.

-Yenya

-- 
| Jan "Yenya" Kasprzak  |
| https://www.fi.muni.cz/~kas/GPG: 4096R/A45477D5 |
We all agree on the necessity of compromise. We just can't agree on
when it's necessary to compromise. --Larry Wall
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: RadosGW manual deployment

[Jan Kasprzak]

Hello, Janne,

Janne Johansson wrote:
> Den mån 29 jan. 2024 kl 08:11 skrev Jan Kasprzak :
> >
> > Is it possible to install a new radosgw instance manually?
> > If so, how can I do it?
> 
> We are doing it, and I found the same docs issue recently, so Zac
> pushed me to provide a skeleton (at least) for such a page. I have
> recently made a quincy cluster manual install with RGWs so I will
> condense what I did to something that can be used for docs later on
> (I'll leave it to Zac to format and merge).
> 
> Really short version for you:
> Install radosgw debs/rpms on the rgw box(es)
> 
> On one of the mons or a box with admin ceph auth run
>ceph auth get-or-create client.short-hostname-of-rgw mon 'allow rw'
> osd 'allow rwx'

OK, I was looking for something like ... mon 'allow profile radosgw'.
Which can be set, but does not work. This was my main problem, apparently.
mon 'allow rw' works.

> On each of the rgw box(es)
>   create a ceph-user owned dir, for instance like this
>   install -d -o ceph -g ceph /var/lib/ceph/radosgw/ceph-$(hostname -s)
>   inside this dir, put the key (or the first two lines of it) you got
> from the above ceph auth get-or-create
>   vi /var/lib/ceph/radosgw/ceph-$(hostname -s)/keyring
>   Figure out what URL rgw should answer to and all that in the config
> parts, but that would be the same
>   for manual and ceph-adm/orchestrated installs.
>   and now you should be able to start the service with
>   systemctl start ceph-radosgw@$(hostname -s).service

Works for me, thanks.

> The last part may or may not act up a bit due to two things, one is
> that it may have tried starting lots of times after the deb/rpm got
> installed, but long before you added they usable key for it, so doing
> a slight boxing match with systemd might be in order, to stop the
> service, reset-failed on the service and then restarting it. (and
> check that it is enabled, so it starts on next boot also)

No problem with that on my systems.

> Secondly, I also tend to run into this issue* where rgw (and other
> parts of ceph!) can't create pools if they don't specify PG numbers,
> which rgw doesn't do any longer, and if you get this error, you end up
> having to create all the pools manually yourself (from a mon/admin
> host or the rgw, but doing it from the rgw requires a lot more
> specifying username and keyfile locations than the default admin-key
> hosts)
> 
> *) https://tracker.ceph.com/issues/62770
>This ticket has a VERY SIMPLE method of testing if ceph versions
> has this problem or not, just
>run "ceph osd pool create some-name" and see how it fails unless
> you add a number behind
>it or not.
> 
>The help is quite clear that all other parameters are meant to be optional:
> 
> osd pool create  [] []
> [] [] []
> [] [] []
> [] [] [--bulk]
> [] [] :  create pool

Also OK on my system.

If there is a (planned) documentation of manual rgw bootstrapping,
it would be nice to have also the names of required pools listed there.

So, thanks for a heplful reply! To sum it up, the following
worked for me (on AlmaLinux 9 host with client.admin.keyring and Ceph Reef):


RGWNAME=`hostname -s`
echo "RGW name is $RGWNAME"

cat >> /etc/ceph/ceph.conf < /var/lib/ceph/radosgw/ceph-$RGWNAME/keyring
chown -R ceph:ceph /var/lib/ceph/radosgw/ceph-$RGWNAME

systemctl enable --now ceph-radosgw@$RGWNAME

# verify it works
curl http://127.0.0.1:8088
ceph osd pool ls
# should print at least the following pools:
# .rgw.root
# default.rgw.log
# default.rgw.control
# default.rgw.meta
# ... and maybe also these, after some buckets are created:
# default.rgw.buckets.index
# default.rgw.buckets.non-ec
# default.rgw.buckets.data


-Yenya

-- 
| Jan "Yenya" Kasprzak  |
| https://www.fi.muni.cz/~kas/GPG: 4096R/A45477D5 |
We all agree on the necessity of compromise. We just can't agree on
when it's necessary to compromise. --Larry Wall
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: 6 pgs not deep-scrubbed in time

[Michel Niyoyita]

This is how it is set , if you suggest to make some changes please advises.

Thank you.


ceph osd pool ls detail
pool 1 'device_health_metrics' replicated size 3 min_size 2 crush_rule 0
object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode on last_change 1407
flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application
mgr_devicehealth
pool 2 '.rgw.root' replicated size 3 min_size 2 crush_rule 0 object_hash
rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 1393 flags
hashpspool stripe_width 0 application rgw
pool 3 'default.rgw.log' replicated size 3 min_size 2 crush_rule 0
object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change
1394 flags hashpspool stripe_width 0 application rgw
pool 4 'default.rgw.control' replicated size 3 min_size 2 crush_rule 0
object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change
1395 flags hashpspool stripe_width 0 application rgw
pool 5 'default.rgw.meta' replicated size 3 min_size 2 crush_rule 0
object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change
1396 flags hashpspool stripe_width 0 pg_autoscale_bias 4 application rgw
pool 6 'volumes' replicated size 3 min_size 2 crush_rule 0 object_hash
rjenkins pg_num 128 pgp_num 128 autoscale_mode on last_change 108802 lfor
0/0/14812 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd
removed_snaps_queue
[22d7~3,11561~2,11571~1,11573~1c,11594~6,1159b~f,115b0~1,115b3~1,115c3~1,115f3~1,115f5~e,11613~6,1161f~c,11637~1b,11660~1,11663~2,11673~1,116d1~c,116f5~10,11721~c]
pool 7 'images' replicated size 3 min_size 2 crush_rule 0 object_hash
rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 94609 flags
hashpspool,selfmanaged_snaps stripe_width 0 application rbd
pool 8 'backups' replicated size 3 min_size 2 crush_rule 0 object_hash
rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 1399 flags
hashpspool stripe_width 0 application rbd
pool 9 'vms' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins
pg_num 32 pgp_num 32 autoscale_mode on last_change 108783 lfor 0/561/559
flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd
removed_snaps_queue [3fa~1,3fc~3,400~1,402~1]
pool 10 'testbench' replicated size 3 min_size 2 crush_rule 0 object_hash
rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 20931 lfor
0/20931/20929 flags hashpspool stripe_width 0


On Mon, Jan 29, 2024 at 2:09 PM Michel Niyoyita  wrote:

> Thank you Janne ,
>
> no need of setting some flags like ceph osd set nodeep-scrub  ???
>
> Thank you
>
> On Mon, Jan 29, 2024 at 2:04 PM Janne Johansson 
> wrote:
>
>> Den mån 29 jan. 2024 kl 12:58 skrev Michel Niyoyita :
>> >
>> > Thank you Frank ,
>> >
>> > All disks are HDDs . Would like to know if I can increase the number of
>> PGs
>> > live in production without a negative impact to the cluster. if yes
>> which
>> > commands to use .
>>
>> Yes. "ceph osd pool set  pg_num "
>> where the number usually should be a power of two that leads to a
>> number of PGs per OSD between 100-200.
>>
>> --
>> May the most significant bit of your life be positive.
>>
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: easy way to find out the number of allocated objects for a RBD image

[Ilya Dryomov]

On Sat, Nov 25, 2023 at 7:01 PM Tony Liu  wrote:
>
> Thank you Eugen! "rbd du" is it.
> The used_size from "rbd du" is object count times object size.
> That's the actual storage taken by the image in backend.

Somebody just quoted this sentence out of context, so I feel like
I need to elaborate.

Earlier in this thread, Tony said that

> An allocated object could be used partially, but that's fine,
> no need to be 100% accurate. To get the object count and
> times object size, that should be sufficient.

For this purpose, USED as reported by "rbd du" is fine.  But it's not
the actual storage taken by the image on the OSDs.  As a counterexample,
one could have an image with 100% USED (i.e. all objects allocated),
but with only 4096 bytes of actual storage taken per object, so "rbd
du" could be off by three orders of magnitude: 4k * number of allocated
objects vs 4M * number of allocated objects (assuming default object
size and ignoring replication factor).

"rbd du" really just gives an upper bound.  Reporting a precise number
would be slow and currently there is no easy way to get it.

Thanks,

Ilya

>
> For export, it actually flattens and also sparsifies the image.
> In case of many small data pieces, the export size is smaller than du size.
>
>
> Thanks!
> Tony
> 
> From: Eugen Block 
> Sent: November 25, 2023 12:17 AM
> To: ceph-users@ceph.io
> Subject: [ceph-users] Re: easy way to find out the number of allocated 
> objects for a RBD image
>
> Maybe I misunderstand, but isn’t ’rbd du‘ what you're looking for?
>
> Zitat von Tony Liu :
>
> > Hi,
> >
> > Other than get all objects of the pool and filter by image ID,
> > is there any easier way to get the number of allocated objects for
> > a RBD image?
> >
> > What I really want to know is the actual usage of an image.
> > An allocated object could be used partially, but that's fine,
> > no need to be 100% accurate. To get the object count and
> > times object size, that should be sufficient.
> >
> > "rbd export" exports actual used data, but to get the actual usage
> > by exporting the image seems too much. This brings up another
> > question, is there any way to know the export size before running it?
> >
> >
> > Thanks!
> > Tony
> > ___
> > ceph-users mailing list -- ceph-users@ceph.io
> > To unsubscribe send an email to ceph-users-le...@ceph.io
>
>
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Debian 12 (bookworm) / Reef 18.2.1 problems

[Chris Palmer]

I have logged this as https://tracker.ceph.com/issues/64213

On 16/01/2024 14:18, DERUMIER, Alexandre wrote:

Hi,


ImportError: PyO3 modules may only be initialized once per
interpreter
process

and ceph -s reports "Module 'dashboard' has failed dependency: PyO3
modules may only be initialized once per interpreter process

We have the same problem on proxmox8 (based on debian12) with ceph
quincy or reef.

It seem to be related to python version on debian12

(we have no fix for this currently)




___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: 6 pgs not deep-scrubbed in time

[Michel Niyoyita]

Thank you Janne ,

no need of setting some flags like ceph osd set nodeep-scrub  ???

Thank you

On Mon, Jan 29, 2024 at 2:04 PM Janne Johansson  wrote:

> Den mån 29 jan. 2024 kl 12:58 skrev Michel Niyoyita :
> >
> > Thank you Frank ,
> >
> > All disks are HDDs . Would like to know if I can increase the number of
> PGs
> > live in production without a negative impact to the cluster. if yes which
> > commands to use .
>
> Yes. "ceph osd pool set  pg_num "
> where the number usually should be a power of two that leads to a
> number of PGs per OSD between 100-200.
>
> --
> May the most significant bit of your life be positive.
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: 6 pgs not deep-scrubbed in time

[Janne Johansson]

Den mån 29 jan. 2024 kl 12:58 skrev Michel Niyoyita :
>
> Thank you Frank ,
>
> All disks are HDDs . Would like to know if I can increase the number of PGs
> live in production without a negative impact to the cluster. if yes which
> commands to use .

Yes. "ceph osd pool set  pg_num "
where the number usually should be a power of two that leads to a
number of PGs per OSD between 100-200.

-- 
May the most significant bit of your life be positive.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: 6 pgs not deep-scrubbed in time

[Michel Niyoyita]

Thank you Frank ,

All disks are HDDs . Would like to know if I can increase the number of PGs
live in production without a negative impact to the cluster. if yes which
commands to use .

Thank you very much for your prompt reply.

Michel

On Mon, Jan 29, 2024 at 10:59 AM Frank Schilder  wrote:

> Hi Michel,
>
> are your OSDs HDD or SSD? If they are HDD, its possible that they can't
> handle the deep-scrub load with default settings. In that case, have a look
> at this post
> https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/YUHWQCDAKP5MPU6ODTXUSKT7RVPERBJF/
> for some basic tuning info and a script to check your scrub stamp
> distribution.
>
> You should also get rid of slow/failing ones. Look at smartctl output and
> throw out disks with remapped sectors, uncorrectable r/w errors or
> unusually many corrected read-write-errors (assuming you have disks with
> ECC).
>
> A basic calculation for deep-scrub is as follows: max number of PGs that
> can be scrubbed at the same time: A=#OSDs/replication factor (rounded
> down). Take the B=deep-scrub times from the OSD logs (grep for deep-scrub)
> in minutes. Your pool can deep-scrub at a max A*24*(60/B) PGs per day. For
> reasonable operations you should not do more than 50% of that. With that
> you can calculate how many days it needs to deep-scrub your PGs.
>
> Usual reasons for slow deep-scrub progress is too few PGs. With
> replication factor 3 and 48 OSDs you have a PG budget of ca. 3200 (ca
> 200/OSD) but use only 385. You should consider increasing the PG count for
> pools with lots of data. This should already relax the situation somewhat.
> Then do the calc above and tune deep-scrub times per pool such that they
> match with disk performance.
>
> Best regards,
> =
> Frank Schilder
> AIT Risø Campus
> Bygning 109, rum S14
>
> 
> From: Michel Niyoyita 
> Sent: Monday, January 29, 2024 7:42 AM
> To: E Taka
> Cc: ceph-users
> Subject: [ceph-users] Re: 6 pgs not deep-scrubbed in time
>
> Now they are increasing , Friday I tried to deep-scrubbing manually and
> they have been successfully done , but Monday morning I found that they are
> increasing to 37 , is it the best to deep-scrubbing manually while we are
> using the cluster? if not what is the best to do in order to address that .
>
> Best Regards.
>
> Michel
>
>  ceph -s
>   cluster:
> id: cb0caedc-eb5b-42d1-a34f-96facfda8c27
> health: HEALTH_WARN
> 37 pgs not deep-scrubbed in time
>
>   services:
> mon: 3 daemons, quorum ceph-mon1,ceph-mon2,ceph-mon3 (age 11M)
> mgr: ceph-mon2(active, since 11M), standbys: ceph-mon3, ceph-mon1
> osd: 48 osds: 48 up (since 11M), 48 in (since 11M)
> rgw: 6 daemons active (6 hosts, 1 zones)
>
>   data:
> pools:   10 pools, 385 pgs
> objects: 6.00M objects, 23 TiB
> usage:   151 TiB used, 282 TiB / 433 TiB avail
> pgs: 381 active+clean
>  4   active+clean+scrubbing+deep
>
>   io:
> client:   265 MiB/s rd, 786 MiB/s wr, 3.87k op/s rd, 699 op/s wr
>
> On Sun, Jan 28, 2024 at 6:14 PM E Taka <0eta...@gmail.com> wrote:
>
> > 22 is more often there than the others. Other operations may be blocked
> > because of a deep-scrub is not finished yet. I would remove OSD 22, just
> to
> > be sure about this: ceph orch osd rm osd.22
> >
> > If this does not help, just add it again.
> >
> > Am Fr., 26. Jan. 2024 um 08:05 Uhr schrieb Michel Niyoyita <
> > mico...@gmail.com>:
> >
> >> It seems that are different OSDs as shown here . how have you managed to
> >> sort this out?
> >>
> >> ceph pg dump | grep -F 6.78
> >> dumped all
> >> 6.78   44268   0 0  00
> >> 1786796401180   0  10099 10099
> >>  active+clean  2024-01-26T03:51:26.781438+0200  107547'115445304
> >> 107547:225274427  [12,36,37]  12  [12,36,37]  12
> >> 106977'114532385  2024-01-24T08:37:53.597331+0200  101161'109078277
> >> 2024-01-11T16:07:54.875746+0200  0
> >> root@ceph-osd3:~# ceph pg dump | grep -F 6.60
> >> dumped all
> >> 6.60   9   0 0  00
> >> 179484338742  716  36  10097 10097
> >>  active+clean  2024-01-26T03:50:44.579831+0200  107547'153238805
> >> 107547:287193139   [32,5,29]  32   [32,5,29]  32
> >> 107231'152689835  2024-01-25T02:34:01.849966+0200  102171'147920798
> >> 2024-01-13T19:44:26.922000+0200  0
> >> 6.3a   44807   0 0  00
> >> 1809690056940   0  10093 10093
> >>  active+clean  2024-01-26T03:53:28.837685+0200  107547'114765984
> >> 107547:238170093  [22,13,11]  22  [22,13,11]  22
> >> 106945'113739877  2024-01-24T04:10:17.224982+0200  102863'109559444
> >> 2024-01-15T05:31:36.606478+0200  0
> >> root@ceph-osd3:~# ceph pg dump | grep -F 6.5c
> >> 6.5c   44277  

[ceph-users] Re: RadosGW manual deployment

[Janne Johansson]

Den mån 29 jan. 2024 kl 10:38 skrev Eugen Block :
>
> Ah, you probably have dedicated RGW servers, right?

They are VMs, but yes.

-- 
May the most significant bit of your life be positive.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: RadosGW manual deployment

[Eugen Block]

Ah, you probably have dedicated RGW servers, right?

Zitat von Janne Johansson :


Den mån 29 jan. 2024 kl 09:35 skrev Eugen Block :
 But your (cephadm managed) cluster will

complain about "stray daemons". There doesn't seem to be a way to
deploy rgw daemons manually with the cephadm tool so it wouldn't be
stray. Is there a specific reason not to use the orchestrator for rgw
deployment?

@Janne: how do you deal with the stray daemons?


We don't get that problem at all. We do all installs 100% manually
(via ansible but still)
and don't see this.

--
May the most significant bit of your life be positive.



___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: RadosGW manual deployment

[Janne Johansson]

Den mån 29 jan. 2024 kl 09:35 skrev Eugen Block :
 But your (cephadm managed) cluster will
> complain about "stray daemons". There doesn't seem to be a way to
> deploy rgw daemons manually with the cephadm tool so it wouldn't be
> stray. Is there a specific reason not to use the orchestrator for rgw
> deployment?
>
> @Janne: how do you deal with the stray daemons?

We don't get that problem at all. We do all installs 100% manually
(via ansible but still)
and don't see this.

-- 
May the most significant bit of your life be positive.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: 1 clients failing to respond to cache pressure (quincy:17.2.6)

[Eugen Block]

I'm not sure if I understand correctly:


I decided to distribute subvolumes across multiple pools instead of
multi-active-mds.
With this method I will have multiple MDS and [1x cephfs clients for each
pool / Host]


Those two statements contradict each other, either you have  
multi-active MDS or not.
Great that you were able to tune your clients, that's really  
interesting although I haven't looked too deep into your results. But  
do the actual clients reflect the same improvement (if you already  
tested that) or was the improvement only for you fio tests?  
Neverthelesse, quite good IOPS!


Zitat von Özkan Göksu :


Thank you Frank.

My focus is actually performance tuning.
After your mail, I started to investigate client-side.

I think the kernel tunings work great now.
After the tunings I didn't get any warning again.

Now I will continue with performance tunings.
I decided to distribute subvolumes across multiple pools instead of
multi-active-mds.
With this method I will have multiple MDS and [1x cephfs clients for each
pool / Host]

To hide subvolume uuids, I'm using "mount --bind kernel links" and I wonder
is it able to create performance issues on cephfs clients?

Best regards.



Frank Schilder , 27 Oca 2024 Cmt, 12:34 tarihinde şunu yazdı:


Hi Özkan,

> ... The client is actually at idle mode and there is no reason to fail
at all. ...

if you re-read my message, you will notice that I wrote that

- its not the client failing, its a false positive error flag that
- is not cleared for idle clients.

You seem to encounter exactly this situation and a simple

echo 3 > /proc/sys/vm/drop_caches

would probably have cleared the warning. There is nothing wrong with your
client, its an issue with the client-MDS communication protocol that is
probably still under review. You will encounter these warnings every now
and then until its fixed.

Best regards,
=
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io


___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io



___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: 6 pgs not deep-scrubbed in time

[Frank Schilder]

Hi Michel,

are your OSDs HDD or SSD? If they are HDD, its possible that they can't handle 
the deep-scrub load with default settings. In that case, have a look at this 
post 
https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/YUHWQCDAKP5MPU6ODTXUSKT7RVPERBJF/
 for some basic tuning info and a script to check your scrub stamp distribution.

You should also get rid of slow/failing ones. Look at smartctl output and throw 
out disks with remapped sectors, uncorrectable r/w errors or unusually many 
corrected read-write-errors (assuming you have disks with ECC).

A basic calculation for deep-scrub is as follows: max number of PGs that can be 
scrubbed at the same time: A=#OSDs/replication factor (rounded down). Take the 
B=deep-scrub times from the OSD logs (grep for deep-scrub) in minutes. Your 
pool can deep-scrub at a max A*24*(60/B) PGs per day. For reasonable operations 
you should not do more than 50% of that. With that you can calculate how many 
days it needs to deep-scrub your PGs.

Usual reasons for slow deep-scrub progress is too few PGs. With replication 
factor 3 and 48 OSDs you have a PG budget of ca. 3200 (ca 200/OSD) but use only 
385. You should consider increasing the PG count for pools with lots of data. 
This should already relax the situation somewhat. Then do the calc above and 
tune deep-scrub times per pool such that they match with disk performance.

Best regards,
=
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14


From: Michel Niyoyita 
Sent: Monday, January 29, 2024 7:42 AM
To: E Taka
Cc: ceph-users
Subject: [ceph-users] Re: 6 pgs not deep-scrubbed in time

Now they are increasing , Friday I tried to deep-scrubbing manually and
they have been successfully done , but Monday morning I found that they are
increasing to 37 , is it the best to deep-scrubbing manually while we are
using the cluster? if not what is the best to do in order to address that .

Best Regards.

Michel

 ceph -s
  cluster:
id: cb0caedc-eb5b-42d1-a34f-96facfda8c27
health: HEALTH_WARN
37 pgs not deep-scrubbed in time

  services:
mon: 3 daemons, quorum ceph-mon1,ceph-mon2,ceph-mon3 (age 11M)
mgr: ceph-mon2(active, since 11M), standbys: ceph-mon3, ceph-mon1
osd: 48 osds: 48 up (since 11M), 48 in (since 11M)
rgw: 6 daemons active (6 hosts, 1 zones)

  data:
pools:   10 pools, 385 pgs
objects: 6.00M objects, 23 TiB
usage:   151 TiB used, 282 TiB / 433 TiB avail
pgs: 381 active+clean
 4   active+clean+scrubbing+deep

  io:
client:   265 MiB/s rd, 786 MiB/s wr, 3.87k op/s rd, 699 op/s wr

On Sun, Jan 28, 2024 at 6:14 PM E Taka <0eta...@gmail.com> wrote:

> 22 is more often there than the others. Other operations may be blocked
> because of a deep-scrub is not finished yet. I would remove OSD 22, just to
> be sure about this: ceph orch osd rm osd.22
>
> If this does not help, just add it again.
>
> Am Fr., 26. Jan. 2024 um 08:05 Uhr schrieb Michel Niyoyita <
> mico...@gmail.com>:
>
>> It seems that are different OSDs as shown here . how have you managed to
>> sort this out?
>>
>> ceph pg dump | grep -F 6.78
>> dumped all
>> 6.78   44268   0 0  00
>> 1786796401180   0  10099 10099
>>  active+clean  2024-01-26T03:51:26.781438+0200  107547'115445304
>> 107547:225274427  [12,36,37]  12  [12,36,37]  12
>> 106977'114532385  2024-01-24T08:37:53.597331+0200  101161'109078277
>> 2024-01-11T16:07:54.875746+0200  0
>> root@ceph-osd3:~# ceph pg dump | grep -F 6.60
>> dumped all
>> 6.60   9   0 0  00
>> 179484338742  716  36  10097 10097
>>  active+clean  2024-01-26T03:50:44.579831+0200  107547'153238805
>> 107547:287193139   [32,5,29]  32   [32,5,29]  32
>> 107231'152689835  2024-01-25T02:34:01.849966+0200  102171'147920798
>> 2024-01-13T19:44:26.922000+0200  0
>> 6.3a   44807   0 0  00
>> 1809690056940   0  10093 10093
>>  active+clean  2024-01-26T03:53:28.837685+0200  107547'114765984
>> 107547:238170093  [22,13,11]  22  [22,13,11]  22
>> 106945'113739877  2024-01-24T04:10:17.224982+0200  102863'109559444
>> 2024-01-15T05:31:36.606478+0200  0
>> root@ceph-osd3:~# ceph pg dump | grep -F 6.5c
>> 6.5c   44277   0 0  00
>> 1787649782300   0  10051 10051
>>  active+clean  2024-01-26T03:55:23.339584+0200  107547'126480090
>> 107547:264432655  [22,37,30]  22  [22,37,30]  22
>> 107205'125858697  2024-01-24T22:32:10.365869+0200  101941'120957992
>> 2024-01-13T09:07:24.780936+0200  0
>> dumped all
>> root@ceph-osd3:~# ceph pg dump | grep -F 4.12
>> dumped all
>> 4.12   0   0 

[ceph-users] Re: RadosGW manual deployment

[Eugen Block]

Good morning,

Janne was a bit quicker than me, so I'll skip my short instructions  
how to deploy it manually. But your (cephadm managed) cluster will  
complain about "stray daemons". There doesn't seem to be a way to  
deploy rgw daemons manually with the cephadm tool so it wouldn't be  
stray. Is there a specific reason not to use the orchestrator for rgw  
deployment?


@Janne: how do you deal with the stray daemons?

Zitat von Jan Kasprzak :


Hi all,

how can radosgw be deployed manually? For Ceph cluster deployment,
there is still (fortunately!) a documented method which works flawlessly
even in Reef:

https://docs.ceph.com/en/latest/install/manual-deployment/#monitor-bootstrapping

But as for radosgw, there is no such description, unless I am missing
something. Even going back to the oldest docs still available at
docs.ceph.com (mimic), the radosgw installation is described
only using ceph-deploy:

https://docs.ceph.com/en/mimic/install/install-ceph-gateway/

Is it possible to install a new radosgw instance manually?
If so, how can I do it?

Thanks!

-Yenya

--
| Jan "Yenya" Kasprzak  |
| https://www.fi.muni.cz/~kas/GPG: 4096R/A45477D5 |
We all agree on the necessity of compromise. We just can't agree on
when it's necessary to compromise. --Larry Wall
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io



___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: RadosGW manual deployment

[Janne Johansson]

Den mån 29 jan. 2024 kl 08:11 skrev Jan Kasprzak :
>
> Hi all,
>
> how can radosgw be deployed manually? For Ceph cluster deployment,
> there is still (fortunately!) a documented method which works flawlessly
> even in Reef:
>
> https://docs.ceph.com/en/latest/install/manual-deployment/#monitor-bootstrapping
>
> But as for radosgw, there is no such description, unless I am missing
> something. Even going back to the oldest docs still available at
> docs.ceph.com (mimic), the radosgw installation is described
> only using ceph-deploy:
>
> https://docs.ceph.com/en/mimic/install/install-ceph-gateway/
>
> Is it possible to install a new radosgw instance manually?
> If so, how can I do it?

We are doing it, and I found the same docs issue recently, so Zac
pushed me to provide a skeleton (at least) for such a page. I have
recently made a quincy cluster manual install with RGWs so I will
condense what I did to something that can be used for docs later on
(I'll leave it to Zac to format and merge).

Really short version for you:
Install radosgw debs/rpms on the rgw box(es)

On one of the mons or a box with admin ceph auth run
   ceph auth get-or-create client.short-hostname-of-rgw mon 'allow rw'
osd 'allow rwx'
On each of the rgw box(es)
  create a ceph-user owned dir, for instance like this
  install -d -o ceph -g ceph /var/lib/ceph/radosgw/ceph-$(hostname -s)
  inside this dir, put the key (or the first two lines of it) you got
from the above ceph auth get-or-create
  vi /var/lib/ceph/radosgw/ceph-$(hostname -s)/keyring
  Figure out what URL rgw should answer to and all that in the config
parts, but that would be the same
  for manual and ceph-adm/orchestrated installs.
  and now you should be able to start the service with
  systemctl start ceph-radosgw@$(hostname -s).service

The last part may or may not act up a bit due to two things, one is
that it may have tried starting lots of times after the deb/rpm got
installed, but long before you added they usable key for it, so doing
a slight boxing match with systemd might be in order, to stop the
service, reset-failed on the service and then restarting it. (and
check that it is enabled, so it starts on next boot also)

Secondly, I also tend to run into this issue* where rgw (and other
parts of ceph!) can't create pools if they don't specify PG numbers,
which rgw doesn't do any longer, and if you get this error, you end up
having to create all the pools manually yourself (from a mon/admin
host or the rgw, but doing it from the rgw requires a lot more
specifying username and keyfile locations than the default admin-key
hosts)

*) https://tracker.ceph.com/issues/62770
   This ticket has a VERY SIMPLE method of testing if ceph versions
has this problem or not, just
   run "ceph osd pool create some-name" and see how it fails unless
you add a number behind
   it or not.

   The help is quite clear that all other parameters are meant to be optional:

osd pool create  [] []
[] [] []
[] [] []
[] [] [--bulk]
[] [] :  create pool



--
May the most significant bit of your life be positive.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io