Hi Bron, all.
Thanks for the update and for the support of the project. That's great
we'll see the 3.0 release soon!
Replying to your last paragraph in the blog post about the community
needs, I believe that what's good for FM is mostly good for the
community too. The FM team is probably the largest operator of the
project and has a better view / face issues and special needs more
frequently than anyone else, so your vision should suit well other
project users too.
A few areas where I see the FM needs probably don't exactly match the
needs of the community are the following 3.
*1. **Small (SMB) deployments* with a single server and somehow limited
physical resources (e.g. disk space).
Here as an example comes the excellent backup mechanism Ellie
implemented that suits well the needs of medium to large deployments,
but IMO that's not the best approach for small deployments, as it
requires a separate server or, if ran at the same server just for the
safe data-to-disk synchronization, twice the disk space.
A better approach for small deployments, as I see it (and I believe it's
highly demanded by the community), would be to have an executable that
would instruct Cyrus daemon to synchronize to disk all the internal
structures and lock (stop writing to disk) for a defined period. The
lock could be implemented by hanging on network write requests or by
writing them to temporary files, or by accumulating the changes in
memory (the latter approach has a potential for data loss).
Once the flush is performed and the lock is applied, a (custom) backup
script could create a snapshot of the partition that would hold the
Cyrus data in a safe-to-backup state. Immediately after creating the
snapshot, the lock would be released and the daemon would continue its
normal operation. Then the backup script would be able to safely backup
the data, e.g. create an incremental backup and upload it to some
external storage, then destroy the snapshot.
Usage example: cyrus_sync2disk --lock=5 -> returns 0 when the data is
synced and a lock for 5 seconds is obtained. cyrus_sync2disk --unlock ->
returns 0 if the lock has been released and 1 if there was no active
lock (e.g a previous lock has expired), so the backup script knows if it
performed the required operations with the lock still in place or if it
should perform the lock-snapshot-unlock operation again. The short
timeout is to protect the daemon from an infinite lock if a backup
script fails to unlock it.
*2. **Small sysadmin tasks* for typical configurations that now require
manual actions or writing one's own scripts. An example: new mailbox
creation with particular flags (\Sent, \Junk, \Trash) set for
special-use folders (that could be implemented as an extended
functionality of the autocreate_inbox_folders option).
At FM you have everything automated for sure with your own customs
scripts, but sysadmins with little experience with Cyrus or those that
don't write scripts with ease would find some tasks difficult to
accomplish, for others that's just an overhead/additional points of
failure that could be avoided with small built-in automations.
*3. **New deployments* (vs ongoing upgrades/maintenance). How easy and
straightforward it is to setup a new deployment (possibly migrating from
other email servers). Here I'm referring to both the initial
configuration, tools and documentation.
*Push* is an area that is well implemented at FM, but there's no
considerable advance in the Cyrus repository, and I believe the
community needs in this area are mostly the same as the FM's.
The 3.0 release includes Apple push notifications support
(XAPPLEPUSHSERVICE) and that's a good start. I haven't tried it yet and
I understand that some effort would be required to make it work (the
part that talks to the APS is not included and should be implemented
independently). I do wonder why wouldn't FM share the notifier code &
some documentation about how to make everything work? The only thing
that'd be different in each deployment are the certificates. And it
would be really exciting to have working apple push in Cyrus just after
some typical setup steps.
If there are some impediments for the FM team to share their
implementation details on mail and caldav/carddav push notifications,
I'll try to make this feature work in my deployments in the near future
and contribute to the project a detailed howto and the APS notifier code
(but your assistance would be great).
And a general area that would benefit everyone, but that wasn't
specifically mentioned in the blog post, is *Security*.
I don't mean Cyrus is insecure, and I do know that the FM team pays
special attention to security of their infrastructure as a whole. Rather
I would like to suggest that a special emphasis could be placed on Cyrus
security from a development POV, e.g. to document in detail (and keep
updated) the entire project's code base and its architecture, to follow
most of the security development best-practices, to re-implement with
security in mind some old/hacky parts of the system (they would become
apparent during the documentation phase), to apply general hardening
tactics (like chroot) or even to re-engineer the overall architecture
for security, to perform internal security code reviews on a regular basis.
FM already had a security audit in 2014 (according to your previous blog
posts), but you don't specify any details of how deep it was and what
aspects it covered. Maybe an independent in-depth security audit with
public results just for the Cyrus code base could be sponsored in
collaboration with the community?
As for me as a member of the community, I have an intention to implement
the chroot functionality for the daemon (late chroot like in OpenVPN).
I've already discussed it briefly with Ellie and was hoping to make it
ready for the 3.0 release, but had no time for it yet. To implement it
correctly, first some important changes should be applied to the
initialization logic (the moment of dropping the privs, it should be
inside newly started processes, rather than in the master). This change
should be carefully analyzed and it's a significant effort, I hope to be
able to contribute it during the Q1/17. Once this change is implemented
(which in itself wouldn't change almost any functionality, so it would
be easy to test and deploy), the chroot functionality would be some 15
lines of code.
Merry Christmas and Happy New Year!
Anatoli
*From:* Bron Gondwana Via Cyrus-devel
*Sent:* Thursday, December 22, 2016 03:15
*To:* Cyrus Devel, Info Cyrus
*Subject:* Release plan blog post
I posted on the FastMail advent about our plans for releasing Cyrus 3.0 - it's
a bit roundabout doing it this way rather than here first, but hey - we talked
about it on Monday night's regular meeting.
Here's the blog post:
https://blog.fastmail.com/2016/12/22/cyrus-development-and-release-plans/
tl;dr, Ellie recently released 3.0beta6. We're going to do a release candidate
on Jan 13th and then release for real soon afterwards, so get testing!
There are no major changes expected before release. I'll be doing a couple of
small JMAP changes to align with the latest spec and possibly to add
getMessageListUpdates if I can manage it in time.
Other than that, I'm looking a reverse UniqueId indexing similar to the RACL
support - it's already in testing and might get added behind a default-off
config switch.
We'll be assessing all the defaults. I'm really tempted to turn RACL on, but
it needs group support if your site uses groups, and that's not done yet, so
I'd need someone willing to test it!
Bron.
