Thank you so much! This worked. This issue kept me busy for about a week, since the rados objects backing the old pubsub framework are not really documented. I'm impressed that I got a highly technical and correct answer so quickly.
Since other people might be using multisite the same way we did (with creating the same topic on both sites), this might be worth mentioning in the docs somewhere. Regards, György Lázár Csütörtök, December 04, 2025 15:30 CET dátummal, Casey Bodley via ceph-users <[email protected]> ezt írta: > (cc Yuval and Ali) > > the problem is that v1 topics did not support multisite, so i'm > guessing you had to recreate those same topics on each zone. the v2 > migration code can only run on the primary zone, so multisite can't > automatically reconcile those v1 topics on your secondary zone > > > WARNING: topic migration in process. please try again later > > you're seeing this warning on the secondary zone because the v1 topics > still exist there. that's also why "radosgw-admin topic list" is only > showing the v1 topics > > all v1 topics were stored in a single rados object per tenant, named > "pubsub.{tenant}" in the site2.rgw.log pool. if you're not using > tenants, that would just be "pubsub." > > since the v2 topic metadata has already replicated, it should be safe > to delete any of those v1 rados objects manually to complete that v2 > migration: > > 1. first make a local backup: > > $ rados -p site2.rgw.log get pubsub. pubsub.bak > > 2. delete the rados object: > > $ rados -p site2.rgw.log rm pubsub. > > 3. verify that v2 topics are now visible to radosgw-admin: > > $ radosgw-admin topic list > > 4. restart radosgw instances that are still caching the v1 topic metadata > > repeat steps 1 and 2 for other tenant names as needed. if necessary, > you can restore the v1 metadata with: > > $ rados -p site2.rgw.log put pubsub. pubsub.bak > > > On Thu, Dec 4, 2025 at 6:08 AM Lázár György via ceph-users > <[email protected]> wrote: > > > > Hello, > > I have an object multisite setup. I'm on squid 19.2.3 (upgraded, not a > > greenfield deployment - both clusters are on the same builds), so I > > upgraded to notification_v2 as well. However, while the primary zone > > converted the notifications and topics to v2 easily, the secondary zone > > seems to have issues with this (topics and notifications are not synced to > > the secondary, but the rados objects are synced, shown via rados -p > > site2.rgw.log -N notif ls. The radosgw-admin topic list command also gives > > back the v1 format of the json output on the secondary zone, while having > > the v2 json output on the primary). What's (almost) certain is, that one of > > the clusters (the primary) wants to use v2, and the other the v1 > > notification style, and mixing these two causes issues (topics randomly > > breaking, etc). Also, since 19.2.3, in theory, if I create a topic on the > > secondary zone, the operation should be forwarded to the primary zone. > > While this happens, the topic creation still fails with an 503 error, while > > the topic still gets created. The rgw on the secondary site after this > > half-failed operation also spits out logs like: > > rgw debug 2025-11-26T11:51:14.241+0000 7ffa54948640 1 req > > 12442569609481506669 0.000000000s sns:pubsub_topic_create ERROR: topic not > > found > > rgw debug 2025-11-26T11:51:14.254+0000 7ffa2e8fc640 1 req > > 12442569609481506669 0.013000983s sns:pubsub_topic_create WARNING: topic > > migration in process. please try again later > > I tried to create another multisite setup (same clusters, different realm), > > and there the notification v2 scheme works without an issue. I think if I > > would delete the secondary zone and re-create it (including the pools), it > > would fix the issue, but since that operation has a relatively large blast > > radius (especially if something goes wrong, but there are some services > > that use only the secondary zone, so they should be redirected beforehand), > > I'm a bit hesitant to do it (before the christmas holidays). I also tried > > searching for these error messages on google, with no luck. I also tried > > searching for these error messages in the RGW source code, and still no > > luck. I can find them in the compiled RGW binary at least. > > > > Any help would be greatly appreciated! I would be also very happy if I > > wouldn't have to nuke the secondary zone. > > > > Redards, > > György Lázár > > _______________________________________________ > > ceph-users mailing list -- [email protected] > > To unsubscribe send an email to [email protected] > _______________________________________________ > ceph-users mailing list -- [email protected] > To unsubscribe send an email to [email protected] _______________________________________________ ceph-users mailing list -- [email protected] To unsubscribe send an email to [email protected]
