Not sure if it can help in your case but the rgw mgr module provides some automation around multisite configuration:
https://docs.ceph.com/en/quincy/mgr/rgw/ On Mon, Sep 15, 2025 at 4:45 PM Kevin Hrpcek <khrp...@ucar.edu> wrote: > I went through multiple deletions and rebuilds but the steps I usually took > were: > - ceph orch rm the rgw daemons > - remove secondary realm > - commit period > - Remove primary realm > - Remove zongroup on both > - Remove realm on both > - Delete realm related pools > > On the last delete when I wanted to ensure I removed everything and attempt > a completely fresh start I also deleted the .rgw.root and any default realm > pools (that I never used) on both clusters. Though I believe I skipped a > couple steps after the realm removal because I thought deleting the > .rgw.root pools would give it a fresh start. > > Kevin > > > On Sun, Sep 14, 2025 at 2:48 AM Eugen Block <ebl...@nde.ag> wrote: > > > How exactly did you clean up the RGW config? > > > > Zitat von Kevin Hrpcek <khrp...@ucar.edu>: > > > > > Hey all, > > > > > > I've been having some problems with a relatively basic rgw multisite > set > > > up. I initially had it working a few weeks ago with a simple config but > > > rebuilt it with ssl enabled for the endpoints and then started having > > some > > > problems to work through. I eventually had that working but then tore > it > > > all down so I could verify the recreation steps but now I'm having > > problems > > > with the multisite. There are a few different errors.... > > > > > > I set up the master zone without problems, but if I use the rgw module > > with > > > a token and spec it fails to sync the sync user and the primary zone > > can't > > > sync back from the secondary zone. A permission denied error for data > > > syncing primary->secondary but secondary-> primary works because it has > > the > > > credentials. > > > > > > So i used to be able to manually create the user on the secondary zone > > and > > > get data syncing but the metadata wouldn't sync with the master showing > > an > > > error message but when I check the mdlog on the primary it looks okay. > > > > > > 2025-09-11T15:15:18.771+0000 7fc29e96c640 1 req 14061042141473454829 > > > 0.001000005s failed to read mdlog history: (2) No such file or > directory > > > > > > I ended up trying a full clean reset of rgw on both clusters so I > deleted > > > the daemons and all related pools. The primary starts up but I still > have > > > the problem of the sync user not being created on the secondary and now > > > when I try to create a user on the secondary for syncing or just for > the > > > dashboard to work I get errors that I've tracked down to > > > > > > https://github.com/ceph/ceph/blob/62bcf65e8c0995783bb3e368909716346874ad62/src/rgw/radosgw-admin/radosgw-admin.cc#L4631 > > > but am not quite following whats all going on in the source to > determine > > > why. > > > > > > # radosgw-admin user create --uid=dashboard --display-name=dashboard > > > --system > > > couldn't init storage provider > > > # radosgw-admin user create --uid=sysuser-asdf > > --display-name=sysuser-asdf > > > --access-key=xxxxxxxxxxxxx --secret-key=yyyyyyyyyyyyyyy --system > > > --rgw-realm=asdf > > > Unable to initialize site config. > > > > > > Any thoughts? It seems like I may have messed something up in the ceph > > > config database from multiple tear downs and restarts of the multisite > > > setup but I'm not seeing what it could be. > > > > > > ceph 19.2.1 using cephadm on alma 9 > > > > > > Thanks! > > > Kevin > > > _______________________________________________ > > > ceph-users mailing list -- ceph-users@ceph.io > > > To unsubscribe send an email to ceph-users-le...@ceph.io > > > > > > _______________________________________________ > > ceph-users mailing list -- ceph-users@ceph.io > > To unsubscribe send an email to ceph-users-le...@ceph.io > > > _______________________________________________ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-le...@ceph.io > _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
