Have you read the documents about how Fastmail does it, using Cyrus Replication protocol rather than rsync. You get real benefits of the integrity checks during the replication phase.
Anyway, your architecture seems good in the happy case. The real question is what happens when one of your nodes dies half way through an rsync - you wind up with some interesting recovery challenges with cyrus.index files not lining up to the files on disk. Cyrus has pretty decent recovery tools for it, but you wind up doing more manual work than you'd want for disaster recovery. Bron On Thu, Sep 17, 2026, at 10:25, Seif Allah Khalifa wrote: > Hi Bron, > > I’m a systems architect currently working on a messaging project for an > administrative client with approximately 140K mailboxes. > > I have analyzed the different Cyrus deployment models in order to identify an > architecture that meets the client’s SI requirements while remaining as close > as possible to Cyrus’ native design. > > The architecture we are currently considering is based on the following > principles: > > • a routing/proxy layer to direct each mailbox to its Cyrus owner; > > • Redis as the distributed mailbox-to-server routing and coordination layer; > > • Cyrus metadata/indexes kept locally on each Cyrus server; > > • metadata synchronization between Cyrus instances using rsync; > > • one dedicated NFS export per Cyrus server for the message spool/data; > > • the NAS remains the central storage layer for the message data. > > The principle is therefore that a given mailbox is owned by a single Cyrus > server. The server owns and manages its metadata/indexes locally, while the > message data is stored on the NAS through its dedicated NFS export. > > I have also been looking carefully at the limitations of NFS semantics > compared with a local filesystem. My understanding is that Cyrus > metadata/index files are not something that can be treated as ordinary shared > NAS data with full filesystem semantics. > > That said, with the metadata kept local, aggressive caching, and NFSv4 for > the message data, I believe this architecture can remain functional while > significantly limiting the amount of filesystem interaction going through NFS. > > The objective here is not to ask you to endorse or commit to this > architecture, but rather to get your expert opinion based on your experience > with Cyrus. > > In particular, I would be very interested in knowing whether you see any > fundamental issue with this model, or any specific Cyrus behavior/constraint > that we should take into account before proceeding further. > > I have attached the architecture diagram for context. > > Thank you for your time and for any feedback you may be able to provide. > > Best regards, > Seif Allah Khalifa > Systems Architect > > > On Wed, Sep 16, 2026 at 10:50 PM Bron Gondwana via Info > <[email protected]> wrote: >> __ >> On Wed, Sep 16, 2026, at 15:47, Seif Allah Khalifa wrote: >>> Hi Ricardo, >>> >>> I appreciate your candid response regarding the support limitations. I >>> fully understand that we are operating outside the "standard" environment >>> and that the project cannot be expected to provide fixes for issues arising >>> from such setups. >>> >>> My interest, however, remains purely architectural. Even if the project >>> doesn't officially sanction it, I am interested in your technical >>> perspective on whether modern strategies effectively address the historical >>> problems: >>> Separation of Concerns: By keeping Cyrus metadata (indexes/headers) on >>> local NVMe/SSD storage and only offloading bulk data (message >>> bodies/attachments) to NFS, does that mitigate the primary >>> locking/concurrency dangers you usually see? >> >> Yep, it would. We don't lock spool files. You'd PROBABLY be OK with the >> cyrus.cache file on the external drive too. >> >>> Access Control: If we use a mail proxy to ensure that each mailbox is >>> accessed by only one server instance at a time, does this resolve the >>> concurrency conflicts that typically make NFS problematic for Cyrus? >> >> Don't even think about it like this. The index files are part of the data >> structure, you don't get to access the spool from somewhere that doesn't >> have them. "mail proxy" whatever - a particular path on the NFS server >> belongs to ONE server only. >> >> Obviously, you get to keep all the bits. But I don't see a reason why this >> wouldn't work. >> >> Bron. >> >> -- >> Bron Gondwana, CEO, Fastmail Pty Ltd / Fastmail US LLC >> [email protected] >> > *Cyrus <https://cyrus.topicbox.com/latest>* / Info / see discussions > <https://cyrus.topicbox.com/groups/info> + participants > <https://cyrus.topicbox.com/groups/info/members> + delivery options > <https://cyrus.topicbox.com/groups/info/subscription> Permalink > <https://cyrus.topicbox.com/groups/info/Tcf711fc6f0ce8498-Ma3290c47a6afe477405d4ef8> > > *Attachments:* > • Architecture Cyrus NAS.jpeg -- Bron Gondwana, CEO, Fastmail Pty Ltd / Fastmail US LLC [email protected] ------------------------------------------ Cyrus: Info Permalink: https://cyrus.topicbox.com/groups/info/Tcf711fc6f0ce8498-M442dcc4d2968b013c242cda0 Delivery options: https://cyrus.topicbox.com/groups/info/subscription
