I have had similar issues specifying what I'm actually trying to do.
One idea I had, but it may be a bit much, would be to have an
interactive radosgw-admin prompt. Similar to the way switch/router
prompts work. e.g., you would enter a realm by selecting it, (the same
way you enter an interface on a switch and then modify it). Then, modify
that realm. Any modifications apply only to whatever realm you are in.
And then, from there, enter a zonegroup, but the only ones available are
the ones that exist in that realm. Same with zones, etc etc. Query
commands can be performed from anywhere, but modifications must be done
within the intended realm/zonegroup/zone.
radosgw-admin> enable
radosgw-admin> configure terminal
radosgw-admin> realm my_realm
radosgw-admin my_realm> no default
radosgw-admin my_realm> zonegroup my_zonegroup
radosgw-admin my_realm my_zonegroup> no default
radosgw-admin my_realm my_zonegroup> zone my_zone
radosgw-admin my_realm my_zonegroup my_zone> no default
radosgw-admin my_realm my_zonegroup my_zone> exit
radosgw-admin> rgw pending
realm my_realm
no default
zonegroup my_zonegroup
no default
zone my_zone
no default
radosgw-admin> rgw commit
Granted, this would require a lot of logic/work. And may be more
restrictive than folks want/need.
Just an idea.
On 5/27/25 18:27, Michel Jouvin wrote:
Hi Casey,
Thanks for you answer. Good news to know that in Squid it is possible to
unset the default realm definition. In fact I was running Reef when I
(or my colleagues) had all the last bad experiences but we upgraded to
Squid yesterday! Nonetheless, I just add a look at `radosgw-admin --
help` and the command `realm default rm` doesn't seems to be documented...
Improvements to the documentation are probably the easiest and here are
a few suggestions:
- Warn that, despite there is normally a hierarchical relationship
between realm, zonegroup and zone, radosgw-admin doesn't implement this
hierarchy with the result that for a zone or zonegroup specifying a
realm will not have any effect to restrict/select one (and the same with
zonegroup and zone).
- Mention that defining a default realm/zonegroup/zone is discouraged if
you have several of thems for the reason above. This can lead to
updating an object different from what you had in mind.
- Remove the --default from suggested commands. When you are new you
tend to add it, it you miss the note or didn't fully understand all the
things it implies.
As for small improvements to radosgw-admin that may avoid running into
troubles, I could suggest:
- Have a "default rm" command for zonegroup and zone too and document it
in the help for all the objects that have it
- Rewrite the help (may be not a small change in fact) so that you could
have a hierarchical help giving more details than the summary on a given
command (for example `realm zone modify --help` would tell you that if
you use --master, this will modify the zonegroup the zone belongs to and
that you have to mention it or the default zone will be used)
- Do a minimum of validation when modifications are done to prevent have
a configuration/period that cannot be commited anymore
Really RGW are a very nice service in Ceph, easy to deploy in various
scenarios when you use cephadm, but the radosgw-admin part of the
configuration is very risky. This is not the case in other parts of Ceph
and I think something should be done to put this part at the same level
as other Ceph management commands/tools (I don't have in mind any where
you can just by a small mistake break your configuration as there is
enough validation done to prevent it and the famous --yes-i-really-want-
it for the things that can make sense but are potentially dangerous). An
issue has been open by Frederic after the discussion in another thread
(https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/
thread/6SD24SR7JBHFZ4E5YTTTQ7NZULHYVHRA/) around this idea: https://
tracker.ceph.com/issues/71406.
Best regards,
Michel
Le 27/05/2025 à 17:53, Casey Bodley a écrit :
On Mon, May 26, 2025 at 7:38 AM Michel Jouvin
<michel.jou...@ijclab.in2p3.fr> wrote:
Hi,
We have a Ceph cluster hosting ~10 RGW realms and in our experience,
radosgw-admin is making the Ceph admins life very hard... The opposite
experience to other Ceph management tools.
In addition to the lack of verifications before applying modifications
that leads to the issues describe in another thread were you can have a
configuration that cannot be commited and at the same time cannot be
fixed, I have the feeling that one of the major issue is that
radosgw-admin predates the rgw/zonegroup/zone paradigm and manages these
3 different kinds of objects as completely independent from the others.
It is made worst by the fact that radosgw-admin defines a default object
in each category, used implicitely (and slilently) if you don't select
one specific object. The consequence of these 2 problems is that if you
modify a zone, you can endup modifying the zonegroup of another realm
than the one the zone belongs to (as there is no concept of a zone
belonging to a zonegroup in radosgw-admin). It happens for example if
you modify a zone (with zone modify) to make it the master zone (of a
zonegroup in fact) and you don't specify the zonegroup the zone belongs
to/is associated with: it will in fact update the default zonegroup that
can be in another realm or at least not be the zonegroup the zone is
associated with. A less impactful but nevertheless unexpected result is
if you do something like: 'zonegroup list --rgw-realm my_realm'. You
would expect to see only the zonegroups associated with the realm you
specified where you'll get the list of all the zonegroups defined.
I can imagine that chaning this is difficult without breaking backward
compatibility. IMO, an improvement would be the ability not to have a
default realm, zonegroup, zone. At least you would have an error if you
don't specify explicitely the object to modify.
thanks Michel,
this 'default' stuff was meant to be a convenience for clusters with a
single zone each, but has indeed turned out to be a nightmare. prior
to squid, it wasn't even possible to leave the 'default realm' unset.
quoting the release note:
The radosgw-admin commands 'realm create' and 'realm pull' no longer
set the default realm without --default.
there's a 'radosgw-admin realm default rm' command that can clear that
default realm, if set. i don't know of a way to unset the default zone
or zonegroup, though you at least had to opt in to those with
--default
Am I the only one struggling with radosgw-admin? Is there any plan to
improve this?
as you say, it's hard to make significant changes here without
breaking existing deployments/automation. so there aren't any concrete
plans, but we're open to ideas
in the meantime, would you suggest any improvements to the
documentation? for example, the note about 'default' behavior under
https://docs.ceph.com/en/latest/radosgw/multisite/#create-a-realm
Best regards,
Michel
_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
