I mean, I have no practical experience of running this, because we haven't used rsync.
I will tell you, rsync has a consistency problem, because it's not integrated with the Cyrus locking, so you can get an index file part way through a rewrite and have CRC errors on re-read. You can get around that with the global runlock in very recent Cyrus - but rsync is too slow, you'd need to use a filesystem with snapshots really -- take a snapshot under the lock, and then rsync the snapshot. Bron. On Thu, Sep 17, 2026, at 11:07, Seif Allah Khalifa wrote: > Hi Bron, > > Yes, I have looked at the Cyrus Replication documentation and at the > different options around sync_client. > > My concern is less about whether Cyrus Replication provides stronger > consistency than rsync — I agree that it does — and more about the > operational trade-off in our specific environment. > > I have been looking at the operational constraints within the SI, as well as > feedback from deployments discussed publicly, and one of the concerns I have > identified with sync_client is the operational complexity around > synchronization failures between nodes. It also introduces additional > continuous processing and load on the Cyrus instances. > > The purpose of rsync in our design is actually quite different from active > Cyrus replication. > > We are not trying to maintain a continuously synchronized hot standby. The > rsync copy is intended to provide a recent copy of the Cyrus metadata that > can be used to recover a mailbox on another node if the original VM is lost. > > The SI requirement allows an RPO of up to 4 hours. Therefore, if a VM is lost > and the most recent rsync copy is, for example, two hours old, this is still > within the contractual recovery objective. > > This changes the trade-off considerably. > > With rsync, if the source VM is lost during an rsync operation, we can simply > discard the incomplete copy and use the previous consistent backup/snapshot. > We may lose up to the permitted RPO, but we do not necessarily introduce a > synchronization incident that requires operational intervention. > > With Cyrus Replication, we obviously gain much stronger consistency and > integrity guarantees, but we also introduce another continuously running > synchronization mechanism. If synchronization is interrupted or gets into an > inconsistent state around a node failure, the production team may have to > intervene, investigate the replication state and potentially run > reconstruct/recovery operations. > > So the question I am trying to answer is not really: > > "Which mechanism is technically stronger?" > > Cyrus Replication clearly has advantages there. > > The question is rather: > > "Given a 4-hour RPO, is the additional consistency and near-real-time > replication provided by sync_client worth the additional operational > complexity and potential synchronization incidents?" > > Our objective is to choose the solution that meets the SI requirements while > minimizing the amount of operational intervention required in production. > > This is also why I was interested in your opinion. Based on your experience > operating Cyrus at scale, do you think the operational risk of sync_client is > significant enough that you would still recommend it for this kind of RPO, or > would a simpler asynchronous metadata backup approach such as ours be > reasonable? > > I am particularly interested in your practical experience here, rather than > just the theoretical capabilities of the two mechanisms. > > Best regards, > > Seif Allah Khalifa > Systems Architect > > > On Thu, Sep 17, 2026 at 4:53 PM Bron Gondwana via Info > <[email protected]> wrote: >> __ >> Have you read the documents about how Fastmail does it, using Cyrus >> Replication protocol rather than rsync. You get real benefits of the >> integrity checks during the replication phase. >> >> Anyway, your architecture seems good in the happy case. The real question >> is what happens when one of your nodes dies half way through an rsync - you >> wind up with some interesting recovery challenges with cyrus.index files not >> lining up to the files on disk. Cyrus has pretty decent recovery tools for >> it, but you wind up doing more manual work than you'd want for disaster >> recovery. >> >> Bron >> >> On Thu, Sep 17, 2026, at 10:25, Seif Allah Khalifa wrote: >>> Hi Bron, >>> >>> I’m a systems architect currently working on a messaging project for an >>> administrative client with approximately 140K mailboxes. >>> >>> I have analyzed the different Cyrus deployment models in order to identify >>> an architecture that meets the client’s SI requirements while remaining as >>> close as possible to Cyrus’ native design. >>> >>> The architecture we are currently considering is based on the following >>> principles: >>> >>> • a routing/proxy layer to direct each mailbox to its Cyrus owner; >>> >>> • Redis as the distributed mailbox-to-server routing and coordination >>> layer; >>> >>> • Cyrus metadata/indexes kept locally on each Cyrus server; >>> >>> • metadata synchronization between Cyrus instances using rsync; >>> >>> • one dedicated NFS export per Cyrus server for the message spool/data; >>> >>> • the NAS remains the central storage layer for the message data. >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> The principle is therefore that a given mailbox is owned by a single Cyrus >>> server. The server owns and manages its metadata/indexes locally, while the >>> message data is stored on the NAS through its dedicated NFS export. >>> >>> I have also been looking carefully at the limitations of NFS semantics >>> compared with a local filesystem. My understanding is that Cyrus >>> metadata/index files are not something that can be treated as ordinary >>> shared NAS data with full filesystem semantics. >>> >>> That said, with the metadata kept local, aggressive caching, and NFSv4 for >>> the message data, I believe this architecture can remain functional while >>> significantly limiting the amount of filesystem interaction going through >>> NFS. >>> >>> The objective here is not to ask you to endorse or commit to this >>> architecture, but rather to get your expert opinion based on your >>> experience with Cyrus. >>> >>> In particular, I would be very interested in knowing whether you see any >>> fundamental issue with this model, or any specific Cyrus >>> behavior/constraint that we should take into account before proceeding >>> further. >>> >>> I have attached the architecture diagram for context. >>> >>> Thank you for your time and for any feedback you may be able to provide. >>> >>> Best regards, >>> Seif Allah Khalifa >>> Systems Architect >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> On Wed, Sep 16, 2026 at 10:50 PM Bron Gondwana via Info >>> <[email protected]> wrote: >>>> __ >>>> On Wed, Sep 16, 2026, at 15:47, Seif Allah Khalifa wrote: >>>>> Hi Ricardo, >>>>> >>>>> I appreciate your candid response regarding the support limitations. I >>>>> fully understand that we are operating outside the "standard" environment >>>>> and that the project cannot be expected to provide fixes for issues >>>>> arising from such setups. >>>>> >>>>> My interest, however, remains purely architectural. Even if the project >>>>> doesn't officially sanction it, I am interested in your technical >>>>> perspective on whether modern strategies effectively address the >>>>> historical problems: >>>>> Separation of Concerns: By keeping Cyrus metadata (indexes/headers) on >>>>> local NVMe/SSD storage and only offloading bulk data (message >>>>> bodies/attachments) to NFS, does that mitigate the primary >>>>> locking/concurrency dangers you usually see? >>>> >>>> Yep, it would. We don't lock spool files. You'd PROBABLY be OK with the >>>> cyrus.cache file on the external drive too. >>>> >>>>> Access Control: If we use a mail proxy to ensure that each mailbox is >>>>> accessed by only one server instance at a time, does this resolve the >>>>> concurrency conflicts that typically make NFS problematic for Cyrus? >>>> >>>> Don't even think about it like this. The index files are part of the data >>>> structure, you don't get to access the spool from somewhere that doesn't >>>> have them. "mail proxy" whatever - a particular path on the NFS server >>>> belongs to ONE server only. >>>> >>>> Obviously, you get to keep all the bits. But I don't see a reason why >>>> this wouldn't work. >>>> >>>> Bron. >>>> >>>> -- >>>> Bron Gondwana, CEO, Fastmail Pty Ltd / Fastmail US LLC >>>> [email protected] >>>> >>> >>> *Attachments:* >>> • Architecture Cyrus NAS.jpeg >> >> -- >> Bron Gondwana, CEO, Fastmail Pty Ltd / Fastmail US LLC >> [email protected] >> > *Cyrus <https://cyrus.topicbox.com/latest>* / Info / see discussions > <https://cyrus.topicbox.com/groups/info> + participants > <https://cyrus.topicbox.com/groups/info/members> + delivery options > <https://cyrus.topicbox.com/groups/info/subscription> Permalink > <https://cyrus.topicbox.com/groups/info/Tcf711fc6f0ce8498-Mee8d8a8fe728d7f894b7b5c7> -- Bron Gondwana, CEO, Fastmail Pty Ltd / Fastmail US LLC [email protected] ------------------------------------------ Cyrus: Info Permalink: https://cyrus.topicbox.com/groups/info/Tcf711fc6f0ce8498-M59d374481606ffa4d9fb8962 Delivery options: https://cyrus.topicbox.com/groups/info/subscription
