On 12/21/2016 05:55 PM, Nicola Nye wrote:
So it looks like the release notes going to 2.5 need to understand the implications of using the dot separator. Please go ahead and update the release notes for 2.5.X accordingly!Of course, if you're going to move to the (soon to be released) 3.0, we ditch netnews in favour of unixhs anyway. (Now more fully documented http://cyrusimap.org/dev/imap/concepts/features/namespaces.html)But we do need to check that the docs don't contain any more references to poor Phabricator as it's now defunct. Long may it RIP.
Nicola,Actually, that snippet from Jeroen's commit had nothing at all to do with T16, as I had previously reported. He must have typo'd that. However, Bron helpfully mined the Internet archive to find the original T16, which included this text:
When upgrading a Cyrus IMAP Murder (discrete), 2.4 backends do not advertise the MOVE capability, but 2.5 frontends will -- and happily proxy the UID MOVE command despite the fact that the backends do not support RFC 6851 <https://web.archive.org/web/20150406213645/https://tools.ietf.org/html/rfc6851> So the fix is to disable MOVE on the frontend by suppressing the capability.But, what users have reported on the list is that the approach of using "suppress_capabilities" may not do the trick. So, I have updated the 2.5.0 Release Notes thus:
Cyrus IMAP Murder Topologies
Environments that run a Cyrus IMAP Murder topology will want to
upgrade their backends before they upgrade their frontends:
When upgrading a Cyrus IMAP Murder (discrete), 2.4 backends do
not advertise the |MOVE| capability, but 2.5 frontends will –
and happily proxy the |UID MOVE| command despite the fact that
the backends do not support *RFC 6851*
<https://tools.ietf.org/html/rfc6851.html> So the fix is to
disable |MOVE| on the frontend by suppressing the capability.
However, subsequent to these notes, users in the field have found
the use of |suppress_capabilities| on frontends may not be a
suitable fix in all situations.
It is unclear to me how to effect this change to github, however, as the
file referenced on the active website doesn't appear in the same place
in my checkouts. So, I am attaching it herein for you to figure out how
to apply. :-)
Cheers,
-nic
--
Nic Bernstein n...@onlight.com
Onlight, Inc. www.onlight.com
6525 W Bluemound Road, Suite 24 v. 414.272.4477
Milwaukee, Wisconsin 53213-4073
:tocdepth: 3
==============================
Cyrus IMAP 2.5.0 Release Notes
==============================
.. IMPORTANT::
Make sure to read the :ref:`imap-relnotes-2.5.0-upgrading` notes
(all of them).
* :ref:`relnotes-2.5.0-new-features`
* :ref:`relnotes-2.5.0-config-option-changes`
* :ref:`relnotes-2.5.0-development-autoconf`
* :ref:`relnotes-2.5.0-development-phabricator`
* :ref:`relnotes-2.5.0-development-documentation`
Download via HTTP:
* http://www.cyrusimap.org/releases/old/cyrus-imapd-2.5.0.tar.gz
* http://www.cyrusimap.org/releases/old/cyrus-imapd-2.5.0.tar.gz.sig
Download via FTP:
* ftp://ftp.cyrusimap.org/cyrus-imapd/OLD-VERSIONS/cyrus-imapd-2.5.0.tar.gz
* ftp://ftp.cyrusimap.org/cyrus-imapd/OLD-VERSIONS/cyrus-imapd-2.5.0.tar.gz.sig
.. _imap-relnotes-2.5.0-upgrading:
Upgrading to Cyrus IMAP 2.5.0
=============================
It is strongly recommended to shut down the Cyrus IMAP services before
performing the upgrade, as the newer binaries will end up writing to
``mailboxes.db`` in a way that is not compatible with the older binaries
(that would otherwise still be running).
You can start the server immediately after upgrading, however.
Underscores in ``cmd`` Names in :cyrusman:`cyrus.conf(5)`
---------------------------------------------------------
Underscores (the ``_`` character) are no longer allowed in the
``START``, ``SERVICES`` and ``EVENTS`` sections of
:cyrusman:`cyrus.conf(5)`, as they interfere with configuration options
in :cyrusman:`imapd.conf(5)` being prefixed by service names and an
underscore (``_``) character.
Database Format Upgrade for Mailboxes
-------------------------------------
Upgrading to Cyrus IMAP 2.5.0 recommends the upgrade of database
formats, for which the reconstruct utility has been modified allowing
administrators to run:
.. parsed-literal::
# :command:`/usr/lib/cyrus-imapd/reconstruct -V max` [options]
This command can be run at the administrator's convenience, and while
running the database format upgrade process is not strictly required, it
is certainly strongly recommended.
Releases prior to 2.5.0, namely the 2.4 series, upgraded ``cyrus.index``
database formats in-line (i.e. as the mailbox in question as being
used). This may have caused an I/O storm in some situations, which 2.5.0
aims to address.
The minor version of the ``cyrus.index`` file in 2.5.0 is **13**. While
mailboxes with an older format will continue to work "forever", newer
features such as annotation quotas will fail, as there is no space in
the database formats prior to version 13 to store the new feature's
information.
Until you upgrade the database format, you may experience slightly
inconsistent search results, due to the ``cyrus.cache`` format in Cyrus
IMAP versions prior to 2.4.0 running words together and not obeying word
boundaries (as well as versions in the 2.4 and 2.5 series).
If you were running Cyrus 2.3.x with ``expunge_mode: delayed``, then the
contents of the ``cyrus.expunge`` file will be removed the first time
the mailbox is opened, and the mailbox will act as if ``expunge_mode``
was set to ``immediate`` -- until it is upgraded.
This is because version 2.3 mailboxes used the ``cyrus.expunge`` file,
and it's not worth the complexity to try to recreate that file.
This upgrade path supports upgrades from Cyrus IMAP version 2.2.12 and
later versions.
.. IMPORTANT::
The above command can run while the Cyrus IMAP services are in
operation, and does **not** interfere with the service's operations.
.. NOTE::
Additional options can be specified to :cyrusman:`reconstruct(8)` to
more granularly specify which mailboxes' :file:`cyrus.index` is to
be upgraded.
Quota Fixes and Enhancements
----------------------------
After all mailboxes are upgraded, you should run ``quota -f`` to make
all message and folder counts are correct. The new quota types that
Cyrus IMAP 2.5.0 supports require the counts to need to be accurate.
Cyrus IMAP Murder Topologies
----------------------------
Environments that run a Cyrus IMAP Murder topology will want to upgrade
their backends before they upgrade their frontends:
When upgrading a Cyrus IMAP Murder (discrete), 2.4 backends do not
advertise the ``MOVE`` capability, but 2.5 frontends will -- and happily proxy
the ``UID MOVE`` command despite the fact that the backends do not support
:rfc:`6851`
So the fix is to disable ``MOVE`` on the frontend by suppressing the capability.
However, subsequent to these notes, users in the field have found the use of
``suppress_capabilities`` on frontends may not be a suitable fix in all situations.
.. _relnotes-2.5.0-new-features:
New Features
============
.. _relnotes-2.5.0-index-namelock-release:
Index Namelock Release
----------------------
Long-running (idling) connections may have previously intervened with
mailbox deletions and general cleanup work, resulting in one connection
blocking another from, for example, deleting and recreating a mailbox:
.. code-block:: bash
:linenos:
C1: SELECT "Foo"
C2: DELETE "Foo"
C2: CREATE "Foo"
The ``CREATE`` command on line 3 (by client 2, while client 1 still has
the ``Foo`` mailbox selected) would have failed before, but does now
succeed.
Extended Quota Types
--------------------
New ways to restrict resource usage:
* Number of Folders,
* Number of Messages,
* Number of Annotations
CalDAV and CardDAV Support
--------------------------
CalDAV and CardDAV support no longer live out-of-tree in the same GIT
repository, but are now mainstream and included in Cyrus IMAP 2.5.0.
This has been a major effort by Carnegie Mellon University and FastMail,
and continues to be, enriching the experience of Cyrus IMAP users
globally.
If upgrading to Cyrus IMAP 2.5 from one of the 2.4.17-caldav-beta
releases, you MUST run the ``dav_reconstruct`` utility for each of
your CalDAV users.
Support for RFC 5464: IMAP METADATA
-----------------------------------
Cyrus IMAP now fully supports :rfc:`5464`, *The IMAP METADATA
Extension*.
This also means the support for the ANNOTATEMORE draft for IMAP will
ultimately be dropped.
Aside from the trusted folder metadata, this also introduces message
annotations. Users will need to be given the ``n`` right to allow them
to set message annotations.
Event Notifications
-------------------
Various events occuring in Cyrus IMAP, among which mailbox, message and
access events, can now be pushed out through a side-channel and notify
client applications or provide other infrastructure with detailed
information.
Mailbox Distribution Enhancements: Backend and Partition Selection
------------------------------------------------------------------
Thanks to the work of Julien Coloos and colleagues, a new mode is
available for server and partition selection upon mailbox creation.
Prior to Cyrus IMAP 2.5.0, the server and/or partition on which to
create a new mailbox was selected by detecting the largest amount of
absolute free disk space on all servers and partitions. The mailbox
distribution feature allows for more intelligent and flexible routines
to be used in the selection. Please see our Administrator Guide for
more details.
New Database Format for ``mailboxes.db``
----------------------------------------
The database format for mailboxes.db has been upgraded, adding;
* A new mailbox type for deleted mailboxes.
In versions of Cyrus IMAP prior to 2.5.0, mailboxes that were
deleted may have become unavailable for actual cleanup expecting
another session on the same mailbox to clean up the directories and
files. See also :ref:`relnotes-2.5.0-index-namelock-release`.
* A key-value storage format is used, allowing for faster and better
parsing of :file:`mailboxes.db`, more granular updates to runtime
environments, and more sustainable future upgrades.
New Database Format ``twoskip``
-------------------------------
A new database format has been added, called ``twoskip`` [#]_.
This new database format is reputedly better, faster, safer and 64-bit
capable, as opposed to our former favorite ``skiplist``.
``twoskip`` can be used for the following databases:
* ``annotation_db``
* ``duplicate_db``
* ``mboxkey_db``
* ``mboxlist_db``
* ``ptscache_db``
* ``quota_db``
* ``seenstate_db``
* ``subscription_db``
* ``statuscache_db``
* ``tls_sessions_db``
* ``user_deny_db``
Miscellaneous
-------------
Allowing Undefined Annotations
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Cyrus IMAP 2.5.0 allows administrators to configure that undefined
annotations should be allowed, using a new
``annotation_allow_undefined`` setting in :cyrusman:`imapd.conf(5)`.
Catchall Mailbox for LMTP
^^^^^^^^^^^^^^^^^^^^^^^^^
Thanks to the work by Carsten Hoeger and Ralf Haferkamp, this new
feature enables administrators to configure a target mailbox for mail
delivered through LMTP to targetted mailboxes that do not exist.
For example, a mail that LMTP would deliver to ``user/bovik``, which
for the sake of argument does not exist in this example, setting
``lmtp_catchall_mailbox`` to ``admin`` will instead deliver the mail
to ``user/admin``.
.. NOTE::
**Mailbox name, not Email Address**
Note that **lmtp_catchall_mailbox** must be a user mailbox name,
not an email address. Also note that the **user/** namespace
indicator as well as the hierarchy separator are to be omitted.
Does this impact lmtp_fuzzy_mailbox_match?
++++++++++++++++++++++++++++++++++++++++++
Environments that have ``lmtp_fuzzy_mailbox_match`` enabled, in order
to have LMTP seek from the targetted, non-existent mailbox sub-folder
(example: ``user/bovik/spam/probably``) all the way to the toplevel
mailbox folder (i.e. ``user/bovik``) until it finds a mailbox
(sub-)folder that does exist (example: ``user/bovik/spam``), are not
impacted by this setting.
Can the lmtp_catchall_mailbox include the path to a sub-folder of a target mailbox?
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
UNCONFIRMED
Can the lmtp_catchall_mailbox be a shared folder?
+++++++++++++++++++++++++++++++++++++++++++++++++
UNCONFIRMED
Callout for SETMETADATA
^^^^^^^^^^^^^^^^^^^^^^^
A callout program can be called when annotations are set, configured
through ``annotation_callout``.
Host & User Login Restrictions
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Logins can now be restricted on a per host (source IP address) or per
user basis, using the settings ``maxlogins_per_host`` and
``maxlogins_per_user``.
.. _relnotes-2.5.0-config-option-changes:
Configuration Option Changes and Enhancements
=============================================
.. IMPORTANT::
While it is not mandatory to update your configuration file with
these new settings, not doing so may have undesired side-effects,
including but not limited to deprecation warnings in log messages.
Option Name Changes for ``autocreate``
--------------------------------------
The options related to automatic creation of user mailboxes and
sub-folders (aka. *autocreate*) have been changed to hold a prefix of
``autocreate_``.
The following *autocreate* options are now available:
**autocreate_inbox_folders** (was: ``autocreateinboxfolders``)
``autocreate_inbox_folders`` controls which folders to create in
addition to the INBOX folder.
Separate the folder names by ``|``.
**autocreate_post** (was: ``createonpost``)
Controls whether or not to create a folder when a message is first
posted to it (by LTMP).
**autocreate_quota** (was: ``autocreatequota``)
When creating a user mailbox, set the quota for that mailbox to the
value of this configuration option.
**autocreate_quota_messages** (not available)
When creating a user mailbox, set the message quota (maximum number
of messages allowed in the folder hierarchy) to the value of this
configuration option.
**autocreate_sieve_folders** (was: ``autosievefolders``)
Limit the folders that can be created automatically by a Sieve
script performing a "fileinto" action, to the folders listed in
this configuration option.
Separate the folder names by ``|``.
**autocreate_sieve_script** (unchanged)
When creating a user mailbox, associate the Sieve script configured
here.
**autocreate_sieve_script_compile** (was: ``generate_compiled_sieve_script``)
Whether or not to compile the Sieve script configured by
``autocreate_sieve_script``.
**autocreate_sieve_script_compiled** (was: ``autocreate_sieve_compiled_script``)
When creating a user mailbox, associate the already compiled Sieve
script configured here.
**autocreate_subscribe_folders** (was: ``autosubscribeinboxfolders``)
List the folder names to which the user for which a mailbox is
being created should be subscribed.
.. NOTE::
All folders listed here are considered to reside in the
personal namespace.
Separate the folder names by ``|``.
**autocreate_subscribe_sharedfolders** (was: ``autosubscribesharedfolders``)
List the folder names of shared folders to which the user for which
a mailbox is being automatically created should be subscribed.
Separate the folder names by ``|``.
**autocreate_subscribe_sharedfolders_all** (was: ``autosubscribe_all_sharedfolders``)
Rather than subscribe the user for which a mailbox is being
automatically created to some shared folders, simply subscribe the
user to all shared folders.
**autocreate_users** (unchanged)
Limit the users for which mailboxes may be created to the list
configured here.
Default Change: ``delete_mode``
-------------------------------
The default for the :cyrusman:`imapd.conf(5)` configuration option
``delete_mode`` has changed from ``immediate`` to ``delayed``.
This causes mail folders that are deleted by a client to not
immediately dissappear from the filesystem. Instead, they are renamed
to a deleted namespace that is visible only to administrators.
A separate job ``cyr_expire -D $x`` is to be included in the master
service configuration file :cyrusman:`cyrus.conf(5)`, specifically in
the EVENTS section. ``$x`` is a number of days to keep already deleted
folders.
**Example section of :cyrusman:`cyrus.conf(5)`**
.. parsed-literal::
EVENTS {
deleteprune cmd="cyr_expire -D 69" at=0430
}
In the aforementioned example, folders are purged from the filesystem
only after 2 times 31 plus 7 days, corresponding with 2 cycles of a
monthly (full, virtual) backup of which one might fail.
Default Change: ``expunge_mode``
--------------------------------
The default for the :cyrusman:`imapd.conf(5)` configuration option
``expunge_mode`` has changed from ``default`` to ``delayed``.
This causes the mail message files associated with messages that are
flagged as \Deleted in a folder that is subsequently expunged, or
individual messages that are expunged, to not be removed from the
filesystem directly.
A separate job ``cyr_expire -X $x`` is to be included in the master
service configuration file :cyrusman:`cyrus.conf(5)`, specifically in
the EVENTS section. ``$x`` is a number of days to keep the message
files on the filesystem.
**Example section of :cyrusman:`cyrus.conf(5)`**
.. parsed-literal::
EVENTS {
expungeprune cmd="cyr_expire -X 69" at=0430
}
In the aforementioned example, message files are purged from the
filesystem only after 2 times 31 plus 7 days, corresponding with 2
cycles of a monthly (full, virtual) backup of which one might fail.
Option Name Changes for ``ldap_tls_*``
--------------------------------------
Configuration option names for LDAP SSL/TLS configuration in
:cyrusman:`imapd.conf(5)` have been changed:
**ldap_ca_dir** (was: ``ldap_tls_cacert_dir``)
**ldap_ca_file** (was: ``ldap_tls_cacert_file``)
**ldap_client_cert** (was: ``ldap_tls_cert``)
**ldap_verify_peer** (was: ``ldap_tls_check_peer``)
**ldap_ciphers** (was: ``ldap_tls_ciphers``)
**ldap_client_key** (was: ``ldap_tls_key``)
Option Name Changes for ``tls_*``
---------------------------------
Configuration option names for SSL/TLS configuration in
:cyrusman:`imapd.conf(5)` have been changed to better reflect how
they are used, as enhancements would otherwise create great confusion.
**tls_client_ca_dir** (was: ``tls_ca_path``)
**tls_client_ca_file** (was: ``tls_ca_file``)
The former ``tls_ca_*`` configuration options specified one or more
SSL Certificate Authority certificates against which SSL
certificates offered by clients could be verified.
In a Cyrus IMAP Murder topology however, Cyrus IMAP servers
themselves become clients of other Cyrus IMAP servers, but may not
have been issued certificates under the same verification chain.
With the (too) generic names for ``tls_ca_*`` configuration options out
of the way, Cyrus IMAP 2.5.0 adds the following configuration options:
**tls_server_cert** (was: ``tls_cert_file``)
**tls_server_key** (was: ``tls_key_file``)
Server SSL certificate and key to use for connections from
clients.
**tls_sessions_db** (was: ``tlscache_db``)
**tls_sessions_db_path** (was: ``tlscache_db_path``)
New Options for ``tls_*``
-------------------------
**tls_client_cert** (<none>)
**tls_client_key** (<none>)
Client SSL certificate and key to use when cyrus-imapd behaves as
a client (to other cyrus-imapd server (instances)).
**tls_client_ca_file** (<none>)
**tls_client_ca_dir** (<none>)
Certificate Authority file or directory used to verify client SSL
certificates.
**tls_client_certs** (``optional``)
Disable (``off``), allow (``optional``) or require (``require``)
clients authenticate with an SSL certificate.
**tls_server_ca_file** (<none>)
**tls_server_ca_dir** (<none>)
Certificate Authority file or directory used to verify SSL
certificates offered by other servers.
**tls_compression** (``0``)
Enable TLS compression. Disabled by default.
**tls_eccurve** (``prime256v1``)
Select the elliptic curve used for ECDHE. See
:command:`openssl ecparams -list_curves` for supported values on
your platform.
**tls_prefer_server_ciphers** (``0``)
Prefer the cipher order configured on the server-side.
**tls_versions** (``ssl2 ssl3 tls1_0 tls1_1 tls1_2``)
Disable SSL/TLS protocols not in this list.
.. _relnotes-2.5.0-development-autoconf:
Development: Switch to ``autoconf`` and ``libtool``
===================================================
With the release of Cyrus IMAP 2.5.0, the Cyrus IMAP project has
switched to using autoconf and libtool.
.. _relnotes-2.5.0-development-phabricator:
Development: Switch to Phabricator
==================================
An instance of Phabricator is going to be replacing our old Bugzilla.
We believe this better facilitates our processes, and will make it
easier to contribute code and collaborate.
Please see https://git.cyrus.foundation/.
.. _relnotes-2.5.0-development-documentation:
Development: Sphinx for Documentation
=====================================
While a work in progress still, you're looking at the new and improved
documentation effort for the Cyrus project as a whole.
This documentation is written in reStructuredText, and rendered by
Sphinx.
The GIT repository for the documentation is at
.. rubric:: Footnotes
.. [#]
http://opera.brong.fastmail.fm.user.fm/talks/twoskip/twoskip-yapc12.pdf
.. _RFC 5464: http://tools.ietf.org/html/rfc5464>
<<attachment: nic.vcf>>
