Hi Bron, Yes, I have looked at the Cyrus Replication documentation and at the different options around sync_client.
My concern is less about whether Cyrus Replication provides stronger consistency than rsync — I agree that it does — and more about the operational trade-off in our specific environment. I have been looking at the operational constraints within the SI, as well as feedback from deployments discussed publicly, and one of the concerns I have identified with sync_client is the operational complexity around synchronization failures between nodes. It also introduces additional continuous processing and load on the Cyrus instances. The purpose of rsync in our design is actually quite different from active Cyrus replication. We are not trying to maintain a continuously synchronized hot standby. The rsync copy is intended to provide a recent copy of the Cyrus metadata that can be used to recover a mailbox on another node if the original VM is lost. The SI requirement allows an RPO of up to 4 hours. Therefore, if a VM is lost and the most recent rsync copy is, for example, two hours old, this is still within the contractual recovery objective. This changes the trade-off considerably. With rsync, if the source VM is lost during an rsync operation, we can simply discard the incomplete copy and use the previous consistent backup/snapshot. We may lose up to the permitted RPO, but we do not necessarily introduce a synchronization incident that requires operational intervention. With Cyrus Replication, we obviously gain much stronger consistency and integrity guarantees, but we also introduce another continuously running synchronization mechanism. If synchronization is interrupted or gets into an inconsistent state around a node failure, the production team may have to intervene, investigate the replication state and potentially run reconstruct/recovery operations. So the question I am trying to answer is not really: "Which mechanism is technically stronger?" Cyrus Replication clearly has advantages there. The question is rather: "Given a 4-hour RPO, is the additional consistency and near-real-time replication provided by sync_client worth the additional operational complexity and potential synchronization incidents?" Our objective is to choose the solution that meets the SI requirements while minimizing the amount of operational intervention required in production. This is also why I was interested in your opinion. Based on your experience operating Cyrus at scale, do you think the operational risk of sync_client is significant enough that you would still recommend it for this kind of RPO, or would a simpler asynchronous metadata backup approach such as ours be reasonable? I am particularly interested in your practical experience here, rather than just the theoretical capabilities of the two mechanisms. Best regards, Seif Allah Khalifa Systems Architect On Thu, Sep 17, 2026 at 4:53 PM Bron Gondwana via Info < [email protected]> wrote: > Have you read the documents about how Fastmail does it, using Cyrus > Replication protocol rather than rsync. You get real benefits of the > integrity checks during the replication phase. > > Anyway, your architecture seems good in the happy case. The real question > is what happens when one of your nodes dies half way through an rsync - you > wind up with some interesting recovery challenges with cyrus.index files > not lining up to the files on disk. Cyrus has pretty decent recovery tools > for it, but you wind up doing more manual work than you'd want for disaster > recovery. > > Bron > > On Thu, Sep 17, 2026, at 10:25, Seif Allah Khalifa wrote: > > Hi Bron, > > I’m a systems architect currently working on a messaging project for an > administrative client with approximately 140K mailboxes. > > I have analyzed the different Cyrus deployment models in order to identify > an architecture that meets the client’s SI requirements while remaining as > close as possible to Cyrus’ native design. > > The architecture we are currently considering is based on the following > principles: > > - > > a routing/proxy layer to direct each mailbox to its Cyrus owner; > - > > Redis as the distributed mailbox-to-server routing and coordination > layer; > - > > Cyrus metadata/indexes kept locally on each Cyrus server; > - > > metadata synchronization between Cyrus instances using rsync; > - > > one dedicated NFS export per Cyrus server for the message spool/data; > - > > the NAS remains the central storage layer for the message data. > > > The principle is therefore that a given mailbox is owned by a single Cyrus > server. The server owns and manages its metadata/indexes locally, while the > message data is stored on the NAS through its dedicated NFS export. > > I have also been looking carefully at the limitations of NFS semantics > compared with a local filesystem. My understanding is that Cyrus > metadata/index files are not something that can be treated as ordinary > shared NAS data with full filesystem semantics. > > That said, with the metadata kept local, aggressive caching, and NFSv4 for > the message data, I believe this architecture can remain functional while > significantly limiting the amount of filesystem interaction going through > NFS. > > The objective here is not to ask you to endorse or commit to this > architecture, but rather to get your expert opinion based on your > experience with Cyrus. > > In particular, I would be very interested in knowing whether you see any > fundamental issue with this model, or any specific Cyrus > behavior/constraint that we should take into account before proceeding > further. > > I have attached the architecture diagram for context. > > Thank you for your time and for any feedback you may be able to provide. > > Best regards, > Seif Allah Khalifa > Systems Architect > > > On Wed, Sep 16, 2026 at 10:50 PM Bron Gondwana via Info < > [email protected]> wrote: > > > On Wed, Sep 16, 2026, at 15:47, Seif Allah Khalifa wrote: > > Hi Ricardo, > > I appreciate your candid response regarding the support limitations. I > fully understand that we are operating outside the "standard" environment > and that the project cannot be expected to provide fixes for issues arising > from such setups. > > My interest, however, remains purely architectural. Even if the project > doesn't officially sanction it, I am interested in your technical > perspective on whether modern strategies effectively address the historical > problems: > Separation of Concerns: By keeping Cyrus metadata (indexes/headers) on > local NVMe/SSD storage and only offloading bulk data (message > bodies/attachments) to NFS, does that mitigate the primary > locking/concurrency dangers you usually see? > > > Yep, it would. We don't lock spool files. You'd PROBABLY be OK with the > cyrus.cache file on the external drive too. > > Access Control: If we use a mail proxy to ensure that each mailbox is > accessed by only one server instance at a time, does this resolve the > concurrency conflicts that typically make NFS problematic for Cyrus? > > > Don't even think about it like this. The index files are part of the data > structure, you don't get to access the spool from somewhere that doesn't > have them. "mail proxy" whatever - a particular path on the NFS server > belongs to ONE server only. > > Obviously, you get to keep all the bits. But I don't see a reason why > this wouldn't work. > > Bron. > > -- > Bron Gondwana, CEO, Fastmail Pty Ltd / Fastmail US LLC > [email protected] > > > *Attachments:* > > - Architecture Cyrus NAS.jpeg > > > -- > Bron Gondwana, CEO, Fastmail Pty Ltd / Fastmail US LLC > [email protected] > > *Cyrus <https://cyrus.topicbox.com/latest>* / Info / see discussions > <https://cyrus.topicbox.com/groups/info> + participants > <https://cyrus.topicbox.com/groups/info/members> + delivery options > <https://cyrus.topicbox.com/groups/info/subscription> Permalink > <https://cyrus.topicbox.com/groups/info/Tcf711fc6f0ce8498-M442dcc4d2968b013c242cda0> > ------------------------------------------ Cyrus: Info Permalink: https://cyrus.topicbox.com/groups/info/Tcf711fc6f0ce8498-Mee8d8a8fe728d7f894b7b5c7 Delivery options: https://cyrus.topicbox.com/groups/info/subscription
