Cyrus IMAP 3.2.4 released

[ellie timoney]

The Cyrus team is proud to announce the immediate availability of a new version 
of Cyrus IMAP: 3.2.4

Download URLs:


https://github.com/cyrusimap/cyrus-imapd/releases/download/cyrus-imapd-3.2.4/cyrus-imapd-3.2.4.tar.gz

https://github.com/cyrusimap/cyrus-imapd/releases/download/cyrus-imapd-3.2.4/cyrus-imapd-3.2.4.tar.gz.sig


Please consult the release notes and upgrade documentation before upgrading to 
3.2.4:

https://www.cyrusimap.org/imap/download/release-notes/3.2/x/3.2.4.html
https://www.cyrusimap.org/imap/download/upgrade.html

And join us on Github at https://github.com/cyrusimap/cyrus-imapd to report 
issues, join in the deliberations of new features for the next Cyrus IMAP 
release, and to contribute to the documentation.

On behalf of the Cyrus team,

Kind regards,

ellie timoney

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: sync_clients bails out - failed to order folders correctly

[ellie timoney]

Hi Konrad,

> do_folders: failed to order folders correctly

Do you have "improved_mboxlist_sort" enabled in your imapd.conf?  My guess is 
you don't.

> every time on a specific mailbox of a specific user.

It sounds like this user has two mailboxes where one is an exact substring of 
another, and the difference starts with a  character earlier than your 
hierarchy separator in ascii order (space and hyphen are the usual culprits).  
e.g. Something like this can cause a variety of problems:

user.ellie.foo
user.ellie.foo-etc
user.ellie.foo.bar

Note the "-" sorts earlier than the hierarchy separator, and so even though 
"bar" is a child of "foo" and should be processed right after "foo", "foo-etc" 
sorts earlier than it and gets in the way...

I guess if this only started happening in the last days, then one of these 
folders must have been created recently.

If this is the cause, the quick workaround is to find the offending folder pair 
and ask the user to rename one of them.  They could replace the space or hyphen 
with an underscore ("_") and this will be fine.

The proper fix is to turn on "improved_mboxlist_sort", but note that the 
documentation says this SHOULD NOT be turned on on a live system, and provides 
instructions for converting the mailboxes.db and subscription databases with 
the service down.  I believe the most up to date documentation for this process 
is here:
https://www.cyrusimap.org/imap/developer/thoughts/improved_mboxlist_sort.html

Hope this helps,

ellie

On Wed, 30 Sep 2020, at 11:03 PM, Konrad Mauz wrote:
> Hello list,
> 
> I had a working replication between to cyrus 3.0 servers running.
> 
> The sync_client is started once a day.
> 
> In the last days the sync_client bails out with 
> 
> do_folders: failed to order folders correctly
> 
> every time on a  specific mailbox of a specific user.
> 
> I can't see that there is a problem with the mailbox ( cyr_expire and 
> squatter are running correctly ).
> 
> How can I get the sync_client work again?
> 
> thanks in advance and kind regards,
> 
> Konrad
> -- 
> Konrad Mauz - Rechenzentrum HTWG Konstanz
> Tel: +497531206472 - Fax: +497531206153
> E-Mail: km...@htwg-konstanz.de
> Web: http://www.htwg-konstanz.de/rz.html
> 
> 
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
>

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Replication : IMAP_PROTOCOL_ERROR Protocol error

[ellie timoney]

On Thu, 24 Sep 2020, at 1:44 AM, Jean Charles Delépine wrote:
> Is this a known problem corrected after 3.0.9 ?

Off the top of my head I no longer remember, but the current release in the 3.0 
series is 3.0.14.  I'd suggest, if you haven't already, that you look in the 
release notes from 3.0.8-3.0.14 and see if anything looks relevant.  They're 
here: https://www.cyrusimap.org/imap/download/release-notes/3.0/index.html

We don't generally recommend in-place upgrades between series (so, upgrading 
in-place from 3.0.8 to 3.2.3 is not something we'd recommend).  Within-series, 
an in-place upgrade ought to be safe -- but please check the release notes 
carefully for extra steps/considerations you may need to make, depending on 
your deployment.  I think you'll probably want to upgrade your 3.0 systems in 
place as far forward as you can (while staying 3.0), and then use the 
replication strategy to upgrade to 3.2 after that.
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Cyrus IMAP 3.2.3 released

[ellie timoney]

The Cyrus team is proud to announce the immediate availability of a new version 
of Cyrus IMAP: 3.2.3

Download URLs:


https://github.com/cyrusimap/cyrus-imapd/releases/download/cyrus-imapd-3.2.3/cyrus-imapd-3.2.3.tar.gz

https://github.com/cyrusimap/cyrus-imapd/releases/download/cyrus-imapd-3.2.3/cyrus-imapd-3.2.3.tar.gz.sig


Please consult the release notes and upgrade documentation before upgrading to 
3.2.3:

https://www.cyrusimap.org/imap/download/release-notes/3.2/x/3.2.3.html
https://www.cyrusimap.org/imap/download/upgrade.html

And join us on Github at https://github.com/cyrusimap/cyrus-imapd to report 
issues, join in the deliberations of new features for the next Cyrus IMAP 
release, and to contribute to the documentation.

On behalf of the Cyrus team,

Kind regards,

ellie timoney

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: IOERROR: conversations_audit on store

[ellie timoney]

Hi Frederik,

>From the source, it looks like this error is reported when the internal counts 
>in the conversations data are out of sync.

I don't have any insight as to what might cause this, but I think you should be 
able to run "ctl_conversationsdb" with the "-R" argument for the affected 
users, to recalculate their counts.

Cheers,

ellie

On Mon, 24 Aug 2020, at 6:22 PM, Frederik Himpe via Info-cyrus wrote:
> I am using Cyrus 3.2.2 on Debian Buster 10 (package from buster-
> backports).
> 
> I am having lots of errors similar to these in the logs:
> 
> cyrus/imaps[24052]: IOERROR: conversations_audit on store: 
> /var/lib/cyrus/user/e/example.conversations Bff62cd0852db22e1 0 
> (1452110 1 0 0 () ((1 1452109 1 1 0)) () foobar. 0 () 1452109)
> 
> The same problem happend with Cyrus 3.0.x and seems to happen often for
> specific users, usually with big mailboxes (one of them 40GB+). When
> upgrading to 3.2.2, I ran
> 
> reconstruct -V max
> ctl_conversationsdb -b -r
> quota -f
> dav_reconstruct -a
> 
> as per the instructions given in the documentation.
> 
> 
> /var/lib/cyrus and /var/spool/cyrus are on ext4, Linux kernel at this
> moment is 5.4.48 (also happend with different kernels).
> 
> imapd.conf:
> configdirectory: /var/lib/cyrus
> proc_path: /run/cyrus/proc
> mboxname_lockpath: /run/cyrus/lock
> defaultpartition: default
> partition-default: /var/spool/cyrus/mail
> partition-news: /var/spool/cyrus/news
> altnamespace: yes
> unixhierarchysep: yes
> lmtp_downcase_rcpt: yes
> admins: cyradm
> allowanonymouslogin: no
> popminpoll: 1
> autocreate_quota: 500
> umask: 077
> sieveusehomedir: false
> sievedir: /var/spool/sieve
> httpmodules: caldav carddav
> hashimapspool: true
> allowplaintext: yes
> sasl_mech_list: PLAIN
> sasl_saslauthd_path: /var/spool/postfix/var/run/saslauthd/mux
> sasl_minimum_layer: 128
> sasl_pwcheck_method: saslauthd
> sasl_auto_transition: no
> tls_server_cert: /etc/letsencrypt/live/ai.vub.ac.be/fullchain.pem
> tls_server_key:  /etc/letsencrypt/live/ai.vub.ac.be/privkey.pem
> tls_client_ca_dir: /etc/ssl/certs
> tls_session_timeout: 1440
> tls_ciphers: TLSv1.2:+TLSv1:+HIGH:!aNULL:@STRENGTH
> tls_versions: tls1_0 tls1_1 tls1_2
> lmtpsocket: /run/cyrus/socket/lmtp
> idlesocket: /run/cyrus/socket/idle
> notifysocket: /run/cyrus/socket/notify
> syslog_prefix: cyrus
> delete_mode: delayed
> expunge_mode: delayed
> sync_log: on
> sync_log_channels: squatter
> conversations: 1
> 
> cyrus.conf:
> START {
>   recover cmd="/usr/sbin/cyrus ctl_cyrusdb -r"
>   
>   idled   cmd="idled"
> }
> SERVICES {
>   imapcmd="imapd -U 30" listen="imap" prefork=0 maxchild=100
>   imaps   cmd="imapd -s -U 30" listen="imaps" prefork=0 
> maxchild=100
>   pop3cmd="pop3d -U 30" listen="pop3" prefork=0 maxchild=50
>   pop3s   cmd="pop3d -s -U 30" listen="pop3s" prefork=0 
> maxchild=50
>   httpcmd="httpd -U 30 -p 256" listen="localhost:8008" 
> prefork=0 
> maxchild=100
>   lmtpunixcmd="lmtpd" 
> listen="/var/spool/postfix/var/run/cyrus/socket/lmtp" prefork=0 
> maxchild=20
>   sieve   cmd="timsieved" listen="sieve" prefork=0 maxchild=100
>   notify  cmd="notifyd" listen="/run/cyrus/socket/notify" 
> proto="udp" 
> prefork=1
> }
> EVENTS {
>   checkpoint  cmd="/usr/sbin/cyrus ctl_cyrusdb -c" period=30
>   delprunecmd="/usr/sbin/cyrus expire -E 3" at=0401
>   tlsprunecmd="/usr/sbin/cyrus tls_prune" at=0415
>   deleteprune cmd="/usr/sbin/cyrus expire -E 4 -D 28" at=0430
>   expungeprunecmd="/usr/sbin/cyrus expire -E 4 -X 28" at=0445
>   squatter1   cmd="/usr/bin/nice -n 19 /usr/sbin/cyrus squatter -i" 
> period=120
>   squattera   cmd="/usr/bin/nice -n 19 /usr/sbin/cyrus squatter -i" 
> at=0517
> }
> 
> Any idea what is going wrong?
> 
> -- 
> Frederik Himpe 
> 
> 
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
>

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Cyrus IMAP 2.4.21, 2.5.16 and 3.0.14 released

[ellie timoney]

The Cyrus team is proud to announce the immediate availability of new legacy 
versions of Cyrus IMAP.

The primary motivator for these releases is to allow XFER to correctly 
recognise newer Cyrus backends.  If you run a legacy Cyrus installation in a 
murder configuration, and ever wish to upgrade your backends to 3.1, 3.2 or 
3.3, you will need these fixes.  Details and discussion of this issue are 
available at https://github.com/cyrusimap/cyrus-imapd/issues/3123

If your legacy Cyrus installation is not in a murder configuration, please 
check the release notes to see if any of the other changes are things you care 
about.

2.4.21 release notes:
https://www.cyrusimap.org/imap/download/release-notes/2.4/x/2.4.21.html

2.4.21 download URLs:

https://github.com/cyrusimap/cyrus-imapd/releases/download/cyrus-imapd-2.4.21/cyrus-imapd-2.4.21.tar.gz

https://github.com/cyrusimap/cyrus-imapd/releases/download/cyrus-imapd-2.4.21/cyrus-imapd-2.4.21.tar.gz.sig

2.5.16 release notes:
https://www.cyrusimap.org/imap/download/release-notes/2.5/x/2.5.16.html

2.5.16 download URLs:

https://github.com/cyrusimap/cyrus-imapd/releases/download/cyrus-imapd-2.5.16/cyrus-imapd-2.5.16.tar.gz

https://github.com/cyrusimap/cyrus-imapd/releases/download/cyrus-imapd-2.5.16/cyrus-imapd-2.5.16.tar.gz.sig

3.0.14 release notes:
https://www.cyrusimap.org/imap/download/release-notes/3.0/x/3.0.14.html

3.0.14 download URLs:

https://github.com/cyrusimap/cyrus-imapd/releases/download/cyrus-imapd-3.0.14/cyrus-imapd-3.0.14.tar.gz

https://github.com/cyrusimap/cyrus-imapd/releases/download/cyrus-imapd-3.0.14/cyrus-imapd-3.0.14.tar.gz.sig

There should also be a new stable 3.2 release in the next week or so, so keep 
an eye out for the announcement!

On behalf of the Cyrus team,

ellie timoney

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: assertion failed: imap/mboxevent.c: 743: filled_params(type, event)

[ellie timoney]

Hi,

I've seen something like this before, and my gut feel is that this is going to 
turn out to be a bug in Cyrus.

I think what's happening is that, somewhere in Cyrus, an event is being 
generated with a type that's supposed to contain a serverAddress field, but the 
serverAddress field is not being initialised.

Before a generated event actually gets sent out to the notifier, we validate 
that all the required parameters have been filled ("filled_params()"), and the 
fatal assertion is telling us that this one has not been, even though it should 
have been.

Would you like to open a GitHub issue at 
https://github.com/cyrusimap/cyrus-imapd/issues ?  If you don't, I will.  But 
if I do it, then you won't get automatic notifications about updates, so if you 
can, it's better if you do it.  Feel free to just paste your previous email as 
the issue text. :)

Cheers,

ellie

On Thu, Jul 16, 2020, at 11:23 AM, Matthew Schumacher wrote:
> I'm trying to use external notifications on 3.2.2 but it doesn't work.   
> If I define
> 
> event_notifier: external
> notify_external: /usr/cyrus/bin/cyrus_notify
> event_groups: access
> event_extra_params: clientAddress timestamp service
> 
> Then the imapd thread dies with this assertion:
> 
> Jul 15 18:01:54 snow imaps[28108]: Cannot notify event Login: missing 
> parameters: serverAddress clientAddress
> Jul 15 18:01:54 snow imaps[28108]: Fatal error: Internal error: 
> assertion failed: imap/mboxevent.c: 743: filled_params(type, event)
> 
> If I remove event_groups and event_extra_params then notify never calls 
> my external script and notify breaks.
> 
> If I define "event_groups: access" and omit event_extra_params then I'm 
> back to:
> 
> Jul 15 18:14:26 snow imaps[28934]: Cannot notify event Login: missing 
> parameters: serverAddress
> Jul 15 18:14:26 snow imaps[28934]: Fatal error: Internal error: 
> assertion failed: imap/mboxevent.c: 743: filled_params(type, event)
> 
> Anyone know where this serverAddress is coming from and how to fix it?
> 
> schu
> 
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: www.cyrusimap.org is down

[ellie timoney]

CMU report that it's been fixed, and it does indeed seem to be working again 
for me now.  The usual caveats about DNS propagation probably apply.

I don't know why the entry disappeared, and if the cause was something 
automated I suppose it might disappear again... I guess we'll keep an eye on it 
and see what unfolds.

Cheers,

ellie

On Wed, Jul 15, 2020, at 10:39 AM, ellie timoney wrote:
> It's hosted on GitHub Pages.  There's supposed to be a DNS CNAME entry at 
> "www.cyrusimap.org" pointing to "cyrusimap.github.io", but it seems to have 
> disappeared. /sigh
> 
> Without that DNS entry, even going directly to "cyrusimap.github.io" isn't 
> working, because GitHub Pages wants to redirect to our custom domain... which 
> is currently broken.
> 
> Thanks for the alert, we'll try and get hold of someone at CMU to get this 
> fixed (or maybe they're watching the list and see this thread).
> 
> Cheers,
> 
> ellie
> 
> On Wed, Jul 15, 2020, at 12:52 AM, Sebastian Hagedorn wrote:
>> Hi,
>> 
>> currently the hostname www.cyrusimap.org cannot be resolved.
>> cyrusimap.org still works, but it appears to point to Github, from where
>> there is a redirect to www.cyrusimap.org
>> -- 
>>.:.Sebastian Hagedorn - Weyertal 121 (Gebäude 133), Zimmer 2.02.:.
>> .:.Regionales Rechenzentrum (RRZK).:.
>>   .:.Universität zu Köln / Cologne University - ✆ +49-221-470-89578.:.
>> 
>> 
>> 
>> Cyrus Home Page: http://www.cyrusimap.org/
>> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
>> To Unsubscribe:
>> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
>> 
>> *Attachments:*
>>  * smime.p7s
> 
> 
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: www.cyrusimap.org is down

[ellie timoney]

It's hosted on GitHub Pages.  There's supposed to be a DNS CNAME entry at 
"www.cyrusimap.org" pointing to "cyrusimap.github.io", but it seems to have 
disappeared. /sigh

Without that DNS entry, even going directly to "cyrusimap.github.io" isn't 
working, because GitHub Pages wants to redirect to our custom domain... which 
is currently broken.

Thanks for the alert, we'll try and get hold of someone at CMU to get this 
fixed (or maybe they're watching the list and see this thread).

Cheers,

ellie

On Wed, Jul 15, 2020, at 12:52 AM, Sebastian Hagedorn wrote:
> Hi,
> 
> currently the hostname www.cyrusimap.org cannot be resolved.
> cyrusimap.org still works, but it appears to point to Github, from where
> there is a redirect to www.cyrusimap.org
> -- 
>.:.Sebastian Hagedorn - Weyertal 121 (Gebäude 133), Zimmer 2.02.:.
> .:.Regionales Rechenzentrum (RRZK).:.
>   .:.Universität zu Köln / Cologne University - ✆ +49-221-470-89578.:.
> 
> 
> 
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
> 
> *Attachments:*
>  * smime.p7s

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: backupd and sync_client IOERROR

[ellie timoney]

Hi Marco,

On Tue, Jul 7, 2020, at 12:17 AM, Marco wrote:
> I copied the content of the failing mailbox into another mailbox with a 
> name without dots:

Oh that's very curious.  It suggests that something about the mailbox contents 
is causing it to fail under -A but not under -u.  I was hoping it would just be 
the dot in the name, cause something like that should be fairly easy to 
reproduce and debug.  But it's sounding like it'll be more difficult!

It might be worth running "mbexamine" over some of these mailboxes -- both some 
that work correctly, and the ones that are misbehaving -- and see if any 
interesting patterns emerge?  The man page is here: 
https://www.cyrusimap.org/imap/reference/manpages/systemcommands/mbexamine.html

> > Does it fail in the same way replicating to a normal replica, instead of a 
> > backup server?
> 
> At this moment I don't have a normal replica server yet. I'll try...

Thanks. I think this will be important for figuring out whether the problem is 
replication generally, or backupd specific.

Cheers,

ellie

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: backupd and sync_client IOERROR

[ellie timoney]

On Wed, Jul 1, 2020, at 11:57 PM, Marco wrote:
> Uhm...

Wow, that's wierd.

I notice that the users that worked correctly with "sync_client -A" don't have 
dots in their address localparts.  If you create another user that also has a 
dot, does it fail under -A in the same way?

Does it fail in the same way replicating to a normal replica, instead of a 
backup server?

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: UID THREAD REFS US-ASCII ALL slow / stalls forever on one folder.

[ellie timoney]

Hi,

I think I would do something like:

0) set client_timeout to a big value (see below)
1) let the imapd start normally
2) connect to it with a minimal imap client (like imtest or telnet)
3) check logs to see which imapd process id your client is connected to (if 
there's more than one)
4) use the "gdb /path/to/binary pid" invocation to attach gdb to the running 
imap process
5) repeat #4 in a few different ways if it's unable to find symbols
6) set a breakpoint on the cmd_thread function (since that's what handles the 
THREAD command) and then continue
7) back in your imap client, send the UID THREAD REFS US-ASCII ALL" command and 
step through to see what happens

Note that these are not exhaustive steps, more of a "get started, and then 
react accordingly, depending on what you see"

I would also try variations of that THREAD command -- if you add/remove 
options, does it start working?  Can you isolate the problem to a specific 
combination of options?  Does it only happen for some mailboxes?

You probably want to set client_timeout to a big value.  The default is 10 
seconds, so the server will probably throw your client off while you're reading 
output from gdb, and then you'll wind up debugging the "disconnect an 
unresponsive client code" instead.  I usually set this to be 30 minutes or so 
for debugging.  For 3.2 and newer, you can spell this as "30m".  For older 
versions, the value is in seconds, so you want "1800".

Permissions may be awkward.  You will need to be the cyrus user (or root) to 
connect gdb to the running imapd, but you will also need permission to read the 
source that it was built from, which is probably not owned by the cyrus user.  
In my case it's under my user account's home directory, so I add the cyrus user 
to my user account's group, and make sure the path to the source directory is 
g+rx (because I don't like running gdb as root).

Cheers,

ellie

On Fri, Jul 3, 2020, at 5:23 AM, Jesper Schmitz Mouridsen via Info-cyrus wrote:
> Hi.
> 
> I recently upgraded Cyrus to 3.2.2 from 3.0.13.
> 
> Now threading "UID THREAD REFS US-ASCII ALL" on one specific folder with 
>  >50K mails,
> 
> makes imapd process use 100 CPU% without any progress. truss[1] or dtrace
> 
> shows no output. The process seems totally stuck.
> 
> I installed in a FreeBSD 12.1 jail. Any hints beside the online docs of 
> how to use gdb to
> 
> see what  is going one. I could not make imapd run under gdb even with 
> the -D option
> 
> and debug_command setting or reading 
> https://www.cyrusimap.org/imap/developer/developer-testing.html?highlight=debug
> 
> It is fast enough on other folders also with ~50k mails < 3 secs.
> 
> Any hints, highly appreciated :-)
> 
> [1] https://www.freebsd.org/cgi/man.cgi?truss
> 
> Regards
> 
> Jesper
> 
> 
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Cyrus IMAP 3.2.2 (Re: cyrus imap 2.5.5 build in Linux: problem with pcre)

[ellie timoney]

Hi Sergey,

Quoting out of order, because it's a bit easier to explain that way:

> And there is another problem that is not obvious. -lpcreposix is needed
> in perl/imap/Makefile.PL and in perl/sieve/managesieve/Makefile.PL I seems.

This bit sounds a lot like https://github.com/cyrusimap/cyrus-imapd/issues/2629

There's an experimental patch from me in that issue, which seems to have worked 
around the issue for Fedora package maintainers.  But I can't reproduce the 
problem on Debian, so I can't confirm myself that it fixes things correctly.  
More feedback would be great.

> I returned to this question. 3.2.2 still does not find 
> /usr/include/pcre/pcre.h
> 
> $ pkg-config --cflags libpcre
> -I/usr/include/pcre
> 
> Build is successful with CFLAGS="-I/usr/include/pcre" ./configure ...

Cyrus's configure script does not currently use pkg-config to locate libpcre.  
Instead it uses bespoke detection built from AC_CHECK_HEADER, which I guess 
doesn't know to look in /usr/include/pcre.  There might be a generic way for a 
sysadmin to add this path to the paths that AC_CHECK_HEADER checks, but I don't 
know it offhand.  Otherwise, specifying it in CFLAGS like you have done seems 
like the right thing to do.

The issue linked above also touches on the possibility of changing the libpcre 
detection to use pkg-config instead of bespoke detection, in which case this 
wouldn't be necessary, but it has its own complications.

> Except perl/imap: CFLAGS is not used by Makefile.PL.

Aaand this sounds like its own whole separate bug!  I'm not sure offhand what 
the right way to pass this down will be, but it clearly needs doing.  New issue 
here: https://github.com/cyrusimap/cyrus-imapd/issues/3087

Cheers,

ellie

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Cyrus IMAP 3.2.2 released

[ellie timoney]

On Tue, Jun 30, 2020, at 10:10 PM, Sergey wrote:
> On Monday 22 June 2020, ellie timoney wrote:
> 
> > The Cyrus team is proud to announce the immediate availability
> > of a new version of Cyrus IMAP: 3.2.2 
>  
> There was a problem with AC_SYS_LARGEFILE. Most likely it was
> there before, but I was no need to use AC_SYS_LARGEFILE.
> Programs for x32 cannot work with large partitions without
> building with _FILE_OFFSET_BITS 64. AC_SYS_LARGEFILE in configure.ac
> usually helps. This mostly helps when building Cyrus IMAP, but not
> for all components.
> 
> without AC_SYS_LARGEFILE:
> 
> $ cat cyrus-imapd-3.2.2-alt0-SS.log | grep "uses non-LFS functions"| wc -l
> 42
> 
> with AC_SYS_LARGEFILE:
> 
> $ cat cyrus-imapd-3.2.2-alt0-SS.log | grep "uses non-LFS functions"| wc -l
> 7
> 
> verify-elf: WARNING: ./usr/lib/libcyrus_imap.so.0.0.0: uses non-LFS 
> functions: statvfs
> verify-elf: WARNING: ./usr/lib/cyrus/restore: uses non-LFS functions: 
> lseek open
> verify-elf: WARNING: ./usr/lib/cyrus/httpd: uses non-LFS functions: 
> __fxstat __xstat
> verify-elf: WARNING: ./usr/lib/cyrus/cyr_backup: uses non-LFS 
> functions: lseek open
> verify-elf: WARNING: ./usr/lib/cyrus/ctl_backups: uses non-LFS 
> functions: lseek open
> verify-elf: WARNING: ./usr/lib/cyrus/backupd: uses non-LFS functions: 
> lseek open
> verify-elf: WARNING: ./usr/lib/libcyrus.so.0.0.0: uses non-LFS 
> functions: fopen
> 
> Put it in a bugtracker on github?

Yes, please.  If you don't, I will, but then you won't get automatic 
notifications of updates.

Where do you see these warnings?  Are they appearing in a log somewhere, or is 
this output from an analysis tool you're running?  I tried searching for 
"verify-elf" but most of the results seem to be about signing binaries, which I 
don't think is related.

Cheers,

ellie

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Cyrus IMAP 3.2.2 released

[ellie timoney]

I have no idea what this refers to or where it comes from.  Any further 
information you could provide would be greatly appreciated!

Thanks

On Tue, Jun 30, 2020, at 5:14 AM, Sergey wrote:
> On Monday 22 June 2020, ellie timoney wrote:
> 
> > The Cyrus team is proud to announce the immediate availability of a new 
> > version of Cyrus IMAP: 3.2.2
> 
> Tests have issued a new warning compared to 3.0.x (building in Linux):
> 
> verify-elf: WARNING: ./usr/lib64/libcyrus.so.0.0.0: found executable 
> STACK entry:
>   GNU_STACK  0x00 0x   0x 0x00 
> 0x00 RWE 0x10
> 
> Is executable STACK really needed?
> 
> -- 
> Regards,
> Sergey
> 
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
>

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Newly arrived mail is marked as "read"

[ellie timoney]

Hi,

I'm not sure, but it kind of sounds like your mailbox's index version is too 
old?

Around 2.4, the storage of the mailbox owner's seen state was moved from the 
seen databases to the cyrus.index.  (i.e. nowadays the seen database only 
stores the seen state for _other_ users who have been given access to a mailbox 
-- which is often nobody).

If your mailbox indexes have not been updated yet, then they won't have a field 
to record the owner's seen state, which might result in the behaviour you're 
seeing?

To update your mailbox indexes to the latest version, you can use the "-V max" 
option with the reconstruct utility.  The upgrade documentation for 3.0 
(https://www.cyrusimap.org/3.0/imap/download/upgrade.html#reconstruct-databases-and-cache)
 mentions that this may take a long time, so maybe you already saw this but 
haven't gotten to that stage yet.

You should definitely take note of the warning on that page though, and 
consider upgrading to at least 3.0.11 rather than 3.0.7.

Hope this is helpful!

Cheers,

ellie

On Fri, Jun 26, 2020, at 10:43 PM, Cheng-Jih Chen wrote:
> I'm in the process of testing the upgrade from a CentOS 6 box running 
> 2.3.16 to a CentOS 8 box running 3.0.7.
> 
> The upgrade has been less troublesome than expected, but I'm seeing a 
> strange problem with newly received mail being marked as "read" even 
> though they have not been opened by the client.  This is the case for 
> both mail sent locally as well as mail received externally (Gmail).
> 
> Can anyone point me in the right direction on where to look to fix this?
> 
> Thanks.
> 
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
>

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: cyrus-imap buildin: sse extention

[ellie timoney]

Hi Sergey,

> Hardware support:
>SSE4.2: yes

This is detected for a hardware implementation of the CRC32c algorithm.  Cyrus 
doesn't actually use it though, because it's not compatible with the existing 
CRC32 algorithm: i.e. for the same input, it produces a different checksum, 
which would break everything on a system with pre-existing data.

If you _want_ to use the hardware CRC32c algorithm on a brand new deployment 
(and know what you are doing), I believe at this stage you would need to patch 
Cyrus to use it -- the code is there, but it is never called.

> Can Cyrus-IMAP be running on systems without SSE4 at this case?

Yep, it'll work just fine.  The hardware CRC32c code will simply not be 
compiled, which, since it isn't used anyway, will have no effect.

> If no, can I set limit to SSE2 ?

There's currently no configurability for this at all.  I don't know if it's 
even possible to implement the same algorithm with SSE2.  At the moment I 
assume that, since it's looking for SSE42 specifically, then that means that 
the hardware feature it needs is probably only available in SSE42.

Cheers,

ellie

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: backupd and sync_client IOERROR

[ellie timoney]

> I think there isn't a all-in-one command for this use case: a user 
> expunged some messages and deleted some folders somewhere. I want to 
> recover all expunged messages and all the deleted folders which are no 
> more present in the original IMAP server (because they were expired from 
> cyr_expire).
> 
> I have to set "-x" to avoid duplication of messages.
> With "-a -x" I recover all expunged messages and all deleted mailboxes. 
> But messages inside deleted mailboxes are not marked as expunged, so 
> these mailboxes are recovered empty in the IMAP server.

I would run restore twice in this case:  once without -x, specifying just the 
deleted mailboxes (you can use "cyr_backup list mailboxes ..." to get a list of 
the mailboxes in the user's backup).  And once with -x to get all the expunged 
stuff.  For the -x invocation only, I would probably also use the -M option to 
dump all the recovered stuff into a new folder, so they can easily tell it 
apart from any new mail that might have arrived coincidentally at the same time.

> Another idea is to recover all in another empty IMAP server, without 
> "-x" at all, and the user can look at the mailbox recovered there...

We kinda had a similar idea!  Putting it all into a folder with -M is much 
easier than setting up a separate server, but it'll lose the folder structure.  
Recovering to a separate server means the folder structure can be preserved.  I 
guess it depends on the specific recovery situation, and how hard it is to spin 
up a server in your environment.

Cheers,

ellie

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: reconstructing mailboxes from backup

[ellie timoney]

Hi Tim,

It's worth observing that, in Cyrus, the user "george"'s IMAP inbox is the 
"user/george" folder.  Which means, on disk, this user has another folder 
called "INBOX" within their inbox.  Depending on the Cyrus version, and maybe 
depending on your server's value of "altnamespace", this is invalid -- and it 
looks like your reconstruct has skipped it and everything under it, 
unsurprisingly.

It's also worth observing that there is both a "Deleted Messages" folder as a 
subdirectory of the bad "INBOX", and an "INBOX^Deleted Messages" directory that 
looks like maybe the result of unixhierarchysep being changed out from under 
the client, or something like that.  Looks like reconstruct has pulled the 
latter one in, cause it's not technically bad (but it will be very 
weird/confusing for the user to have a folder called "INBOX.Deleted Messages" 
in their client that is neither their inbox, nor their Deleted Messages folder, 
nor a directory hierarchy of the two).

So, it looks like reconstruct has found all the valid folders, and skipped the 
invalid INBOX and everything in it.  That seems coherent.

If you can start again from scratch: then I'd suggest renaming, on disk, that 
"INBOX" folder to something like "old inbox", and optionally renaming the 
"INBOX^Deleted Messages" folder to something like "old deleted messages", 
before you run the reconstruct.  Then the reconstruct will be able to find 
everything, and the user can then move the messages from the "old..." folders 
back into wherever they want them to be just over IMAP.

If you can't start again from scratch: then you should only rename the bad 
"INBOX" folder on disk, and then reconstruct. The previous reconstruct already 
found and repaired the "INBOX^Deleted Messages" folder, so renaming it on disk 
now might make a new mess.  But it can be renamed over IMAP, either by an admin 
session or the user.

Hope this helps :)

ellie

On Tue, Jun 23, 2020, at 11:42 PM, Tim Coote wrote:
> Hullo
> 
> I have a cyrus implementation on Fedora for a small (~10) users that’s 
> been migrated through many versions of the various components, 
> including several different of IMAP clients.
> 
> Realising the fragility of the setup, I thought I’d restore from a 
> backup. However, I’m finding that several of the mailboxes are not 
> being recovered. I feel that I am missing somethign obvious, but I 
> cannot spot it. 
> 
> The restoring version of cyrus-imap is: cyrus-imapd-3.0.13-2.fc32.x86_64,
> 
> The restored filesystem layout can be summarised thus:
> 
> `sudo find /var/spool/imap/g | grep cyrus.header`:
> 
> /var/spool/imap/g/user/george/Notes/cyrus.header
> /var/spool/imap/g/user/george/cyrus.header
> /var/spool/imap/g/user/george/Sent Messages/cyrus.header
> /var/spool/imap/g/user/george/Deleted Messages/cyrus.header
> /var/spool/imap/g/user/george/Sent/cyrus.header
> /var/spool/imap/g/user/george/Trash/cyrus.header
> /var/spool/imap/g/user/george/INBOX/Sent Messages/cyrus.header
> /var/spool/imap/g/user/george/INBOX/Deleted Messages/cyrus.header
> /var/spool/imap/g/user/george/INBOX/Drafts/cyrus.header
> /var/spool/imap/g/user/george/INBOX^Deleted Messages/cyrus.header
> /var/spool/imap/g/user/george/Drafts/cyrus.header
> 
> [so I would expect all of the subdirectories to be reconstructed as mailboxes]
> 
> however, using:
> `sudo -u cyrus reconstruct -r -f user/george`
> 
> I only get:
> user/george
> user/george/Deleted Messages
> user/george/Drafts
> user/george/INBOX.Deleted Messages
> user/george/Notes
> user/george/Sent
> user/george/Sent Messages
> user/george/Trash
> 
> ie no subdirectories below the top level, but excluding those 
> directories below INBOX.
> Should there be a file: 
> `/var/spool/imap/g/user/george/INBOX/cyrus.header`? 
> 
> Is there anything that I should be doing/how can I recover the other 
> mailboxes?
> 
> 
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Cyrus IMAP 3.2.2 released

[ellie timoney]

The Cyrus team is proud to announce the immediate availability of a new version 
of Cyrus IMAP: 3.2.2

Download URLs:


https://github.com/cyrusimap/cyrus-imapd/releases/download/cyrus-imapd-3.2.2/cyrus-imapd-3.2.2.tar.gz

https://github.com/cyrusimap/cyrus-imapd/releases/download/cyrus-imapd-3.2.2/cyrus-imapd-3.2.2.tar.gz.sig


Please consult the release notes and upgrade documentation before upgrading to 
3.2.2:

https://www.cyrusimap.org/imap/download/release-notes/3.2/x/3.2.2.html
https://www.cyrusimap.org/imap/download/upgrade.html

And join us on Github at https://github.com/cyrusimap/cyrus-imapd to report 
issues, join in the deliberations of new features for the next Cyrus IMAP 
release, and to contribute to the documentation.

On behalf of the Cyrus team,

Kind regards,

ellie timoney

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: backupd and sync_client IOERROR

[ellie timoney]

Hi Marco,

On Fri, Jun 19, 2020, at 6:44 PM, Marco wrote:
> wow, yes, it works. With this config, after the Cyrus restart the 
> mailboxes.db and the skipstamps dbs are created and the error disappears 
> from syslog.

Great! I've updated the documentation, and the website should update shortly.

> I suggested the possibility to choose the recovery date (ie: all 
> messages and mailboxes deleted or expunged from  to ). I 
> hope you could add this feature.

Yep, I saw that, thanks for the request :)

> I now tried a restore, but I still have some problems. I very appreciate 
> if you could read the following issue.
> 
> My goal is to recover a message that was removed (expunged) from the 
> IMAP server, ignoring already existing messages.
> 
> It seems the "-x" flag could help. It seems it allows to restore and 
> unexpunge a previously expunged messages in the IMAP server.

No, you've misunderstood.  The "-x" flag is to ONLY restore expunged messages.  
If you've specified a list of mailboxes or messages on the command line, and 
some of them are not expunged, the "-x" option will cause it to ignore the ones 
that are not expunged, and only restore the expunged ones.  The "-X" option 
does the inverse -- they're filters.  Without one of these filters, it will 
restore everything you asked for, regardless of its expunged status.

So like, maybe a user has deleted some stuff, and you don't want to mess around 
figuring out which individual messages they need restored, so you just want to 
restore everything, and let the user figure it out.  This is what -x is for -- 
so you can say "restore the contents of mailbox foo, but only the expunged 
stuff" or "restore every mailbox (-a), but only the expunged stuff".

Think about it: if the messages _weren't_ expunged, there would usually be no 
need to restore them from backup!  You would simply remove the \Deleted flag 
over IMAP.  So, you are almost always using restore to bring back expunged 
messages, thus there is no special flag needed to enable this functionality.

(But, if you had lost an entire server to some disaster, and restoring to its 
replacement, then you would need to also restore the unexpunged stuff.)
 
> Yesterday I expunged the message 16ceead9802286784d7a54c5bc782891f76f2f2e:
> 
> 2020-06-18T15:44:37.374556+02:00 tst-msg03 cyrus/imap[9461]: auditlog: 
> expunge sessionid= 
> mailbox= 
> uniqueid= uid=<32092> modseq=<46828> 
> sysflags= guid=<16ceead9802286784d7a54c5bc782891f76f2f2e>
> 
> 
> I tried today to restore it:
> cyr_restore -v -U -x tst-msg03.example.com -u 
> utente.archivi...@example.com 16ceead9802286784d7a54c5bc782891f76f2f2e
> example.com!user.utente^archivista: trying to keep 
> uidvalidity(1550228105), uniqueid(nvf96uirjgfs6xaekwwc7g44), 
> highestmodseq(46828)
> 
> and I see:
> 2020-06-19T09:29:24.030624+02:00 tst-msg03 cyrus/imap[32742]: login: 
> tst-msg03-bck.example.com [10.102.42.169] cyr_restore LOGIN User logged 
> in SESSIONID=
> 
> 2020-06-19T09:29:24.162909+02:00 tst-msg03 cyrus/imap[32742]: auditlog: 
> append sessionid= 
> mailbox= 
> uniqueid= uid=<32096> modseq=<46829> 
> sysflags= guid=<16ceead9802286784d7a54c5bc782891f76f2f2e> 
> messageid=
> 
> 2020-06-19T09:29:24.163058+02:00 tst-msg03 cyrus/imap[32742]: auditlog: 
> expunge sessionid= 
> mailbox= 
> uniqueid= uid=<32096> modseq=<46829> 
> sysflags= guid=<16ceead9802286784d7a54c5bc782891f76f2f2e>
> 
> 2020-06-19T09:29:24.167561+02:00 tst-msg03 cyrus/imap[32742]: auditlog: 
> modseq sessionid= 
> mailbox= 
> uniqueid= highestmodseq=<46829>
> 
> 2020-06-19T09:29:24.215036+02:00 tst-msg03 cyrus/imap[32742]: USAGE 
> cyr_restore user: 0.042492 sys: 0.015435
> 
> 2020-06-19T09:29:24.215191+02:00 tst-msg03 cyrus/imap[32742]: auditlog: 
> traffic sessionid= 
> bytes_in=<2412> bytes_out=<1039>
> 
> 2020-06-19T09:29:32.855379+02:00 tst-msg03 cyrus/sync_client[32566]: 
> IOERROR: zero length response to MAILBOXES (end of file reached)
> 
> 2020-06-19T09:29:32.855682+02:00 tst-msg03 cyrus/sync_client[32566]: 
> IOERROR: zero length response to RESTART (end of file reached)
> 
> 2020-06-19T09:29:32.855713+02:00 tst-msg03 cyrus/sync_client[32566]: 
> Error in do_sync(): bailing out! Bad protocol
> 
> 2020-06-19T09:29:32.855738+02:00 tst-msg03 cyrus/sync_client[32566]: 
> Processing sync log file /var/lib/imap/sync/bck/log-run failed: Bad protocol
> 
> I tried again and I see the same result, but without the IOERROR:
> 
> 2020-06-19T09:36:51.153939+02:00 tst-msg03 cyrus/imap[32762]: login: 
> tst-msg03-bck.example.com [10.102.42.169] cyr_restore LOGIN User logged 
> in SESSIONID=
> 
> 2020-06-19T09:36:51.199742+02:00 tst-msg03 cyrus/imap[32762]: 
> example.com!user.utente^archivista: same message appears twice 32096 32097
> 
> (why twice? It expunges 32096 just above)
> 
> 2020-06-19T09:36:51.225525+02:00 tst-msg03 cyrus/imap[32762]: auditlog: 
> append sessionid= 
> mailbox= 
> uniqueid= uid=<32097> modseq=<46834> 
> sysflags= 

Re: backupd and sync_client IOERROR

[ellie timoney]

Hi Marco,

On Thu, Jun 18, 2020, at 10:19 PM, Marco wrote:
> Hello,
> 
>   I'm trying to configure backupd in rolling mode as a final setup. 
> Running a first backup on few users
> 
> sync_client -A -n bck -z -v -v
> 
> after a while the process die with:
> 
> cyrus/sync_client[9540]: MESSAGE received NO response: 
> IMAP_PROTOCOL_ERROR Protocol error
> cyrus/sync_client[9540]: do_folders(): update failed: 
> example.com!user.utente^archivista 'Bad protocol'
> cyrus/sync_client[9540]: IOERROR: do_user_main: Bad protocol for 
> utente.archivi...@example.com to [no channel] (tst-msg03-bck.example.com)
> Error from do_user(utente.archivi...@example.com): bailing out!
> cyrus/sync_client[9540]: Error in 
> do_user(utente.archivi...@example.com): bailing out!
> 
> The backupd host doesn't complain:
> 
> 2020-06-18T11:50:30.528584+02:00 tst-msg03-bck backupd[1308172]: 
> creating sql_db 
> /var/spool/cyr_backup/u/utente.archivista@example.com_cxlsmX.index
> 2020-06-18T11:56:44.971042+02:00 tst-msg03-bck backupd[1308172]: login: 
> tst-msg03.example.com [10.102.42.168] cyr_backup LOGIN User logged in
> 
> So I repeated the same command, but with less verbosity:
> 
> sync_client -A -n bck -z -v
> 
> and it worked well, without errors. Uhm...
> 
> Maybe, before to run the first initial backup (sync_client -A), do I 
> have to stop the sync_client already working in rolling mode?

Rolling mode only makes incremental updates, so if you're starting from a 
server that already has existing data, you should do the first manual initial 
backups before enabling the rolling mode.

It's generally safe to run a manual sync_client side-by-side with a rolling one 
(they will lock things properly and keep out of each other's way).  But the 
exception to this is that a rolling mode replication can't coherently update an 
uninitialised replica (since it only sends new changes recorded in the sync 
log, but not the pre-existing data).  You need to do a full manual replication 
first, to ensure both sides are the same, before rolling mode can work 
sensibly.  This is the same regardless of whether your replica is a normal 
replica or a backupd one.

As you saw, replicating again manually will probably fix you up, at least. :)

> http://www.cyrusimap.org/imap/reference/admin/backups.html says "Update 
> cyrus.conf(5) to add a sync_client(8) invocation to the SERVICES section 
> specifying (at least) the -r and -n channel options.". Really maybe you 
> intend START. In SERVICE I need to specify a listen too, this is not 
> very clear to me.

This is wrong.  It should be the DAEMON section, not the SERVICES section.  I'm 
fixing this now, so the website should be updated shortly.

You can also run it from the START section, but note that if the sync_client 
exits early for some reason, master will not restart it.  If you run it from 
the START section, you will need to monitor/restart it yourself.

> Around of backupd I would ask two other questions:
> 
> 1) The backupd host always claims this:
> 
> backupd[1309970]: DBERROR: reading /var/lib/imap/db/skipstamp, assuming 
> the worst: No such file or directory
> 
> Could you tell me how to fix that DBERROR too?

I think you might need to add the usual recover entry to the START section:

recover   cmd="ctl_cyrusdb -r"

I notice this is missing from the backups documentation -- please let me know 
if it this sorts it out, and I'll fix it up!
 
> 2) Is there a way to delete a user from backupd host?

That's an interesting question...

Under normal use, if a user is deleted from their imap server, then, after the 
deletion has replicated to the backup server, and after the "backup_retention" 
period has elapsed, and after a subsequent "ctl_backups compact ..." has been 
run, then the backup file should be approximately empty.  But I think it will 
still exist; nothing actually deletes it yet.

To get rid of it manually, you can use "ctl_backups list ..." to find the 
filename where it exists on disk.  You can also find it by using cyr_dbtool to 
look up the user's key in backups.db.  You will then need to:

a) use cyr_dbtool to delete that user's key from backups.db
b) delete the named file (it will be like "/some/path/userid_XX")
c) also delete the corresponding "/some/path/userid_XX.index" file

Note that if the user still really exists, and a rolling sync_client replicates 
them to the same backup server again, then the backup file will be recreated 
(with a new XX) -- with the same problems as above wrt no-initial-sync.

Cheers,

ellie

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Cyrus IMAP 3.2.1 released

[ellie timoney]

Hi Marco,

On Thu, Jun 4, 2020, at 11:04 PM, Marco wrote:
> On 29/05/2020 06:20, ellie timoney has written:
> > The Cyrus team is proud to announce the immediate availability of a new 
> > version of Cyrus IMAP: 3.2.1
> 
> Hello,
> 
>   in Redhat EL8 I still fail these tests:
> 
> ERRORS:
> Rename.rename_inbox
>   Perl exception: Errors found in syslog
>   at Cassandane/Instance.pm line 1322,  line 91.

You probably saw on github -- I've just pushed a fix to Cassandane which I 
think should fix up the remaining syslog-related failures...

Though, Rename.rename_inbox is a special exception.  This one will probably 
always error on 3.2.  There's a real (niche) bug, which this test is detecting. 
 It's existed for a very long time, but we're not sure how to fix it yet, so 
you might as well just ignore it.

Though... it's very interesting that it FOUND errors in syslog, when the usual 
RedHat problem is not being able to find the syslog output.  Curious?

> FAILS:
> 
> 1) Admin.imap_admins
>   expected 'ok', got 'no' at Cassandane/Cyrus/Admin.pm line 90.

This is interesting.  I'd like to see the full output please!

> 2) Caldav.supports_event
>   Boolean assertion failed at 
> /usr/share/perl5/vendor_perl/Test/Unit/Exception.pm line 13.

I think we discussed Caldav.supports_event before -- it passes here, still no 
idea what the difference is.  Maybe it's a libical thing?  What version is your 
libical?

> 3) ImapTest.append-binary
>   Boolean assertion failed at 
> /usr/share/perl5/vendor_perl/Test/Unit/Exception.pm line 13.
> 4) ImapTest.fetch-binary-mime
>   Boolean assertion failed at 
> /usr/share/perl5/vendor_perl/Test/Unit/Exception.pm line 13.
> 5) ImapTest.urlauth-binary
>   Boolean assertion failed at 
> /usr/share/perl5/vendor_perl/Test/Unit/Exception.pm line 13.

The "ImapTest.*-binary*" tests were known to be not-quite-right, and I believe 
have been removed from ImapTest-- my build is from 20190509 and doesn't contain 
them.  If you can't update ImapTest, it's reasonable to just suppress these 
three instead.

> 10) Cassandane::Test::Core.core_files_*
>didn't match /(?^:Core files found)/ at Cassandane/Test/Core.pm line 96.

The RedHat environment is probably preventing core files from being created, in 
which case the Test::Core tests will always fail.  That being the case, I'd 
suggest just suppressing them.

> The details are attached, if it could be of interest.

There's no attachment, did the mailing list eat it?  Feel free to send to me 
directly

If you update to the latest Cassandane (for the syslog fixes), then you'll 
start seeing failures on 3.2.1 from these tests:

JMAPEmail.blob_get
JMAPEmail.email_query_guidsearch_mixedfilter

These tests have been updated to detect bugs that will be fixed in 3.2.2, but 
of course they fail for 3.2.1 cause the bugs still exist in that version.

Cheers,

ellie

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Sieve EditHeaders and Logging

[ellie timoney]

Hi David,

> Is it possible to enable the "editheaders" sieve extension? if so, how?

Not in 3.0, but it's available in 3.2

> Are sieve actions logged anywhere, e.g. to aid with debugging?

Generally? I don't know.  Maybe if you increase your syslog log level to 
"debug" and add "debug: yes" to your imapd.conf?

3.2 has an experimental "x-cyrus-log" extension, which adds an action that 
produces a log line.  You probably don't want to allow this for user-supplied 
sieve scripts, but if you have some sort of "sieve builder" system that your 
users use (rather than uploading their own handcrafted sieve scripts), then, if 
you were running 3.2, this extension could be useful.  On the master branch, 
and therefore in the next major release (2021), this extension will be 
formalised as "vnd.cyrus.log".

Cheers,

ellie

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Object Storage and Cyrus IMAP

[ellie timoney]

On Fri, Jun 5, 2020, at 4:48 AM, Albert Shih wrote:
> Le 04/06/2020 à 10:23:12+0200, Marco a écrit
> > Hello,
> >
> >I see that Cyrus IMAP 3 can interface with some Object Storage such as
> > Caringo or OpenIO.
> >
> > Is anyone using these solutions?
> >
> > I would like to know how I can find more details about these deployment,
> > other than the brief description in imapd.conf man page.
> >
> > In particular I would like to know if these interfaces are stable and
> > supported in future releases of Cyrus IMAP.
> >
> > Do you plan to add wider support to object storage, maybe by adopting some
> > standard vendor independent?
> >
> > I thank you very much for every info you could provide.
> 
> No sure if my informations are still correct, but long time ago, I ask
> openio about that. They say the connector between openIO and cyrus are
> maintained by openIO, it's not opensource and you need to pay a licence.
> 
> And when Cyrus make a new release openio would make adjustment to make it
> work.
> 
> I not sure who use that, but as I ear they(openIO) got few customer use
> this solution on a very large scale > 1Po.

Our object storage support was contributed by one of those two vendors (i.e 
Caringo or OpenIO, though I don't remember which).  As I understand it, they 
implemented support in Cyrus for both backends, to ensure that the object 
storage support was generalised, not specific to their own product.  This might 
have been a condition we imposed for accepting their patches?

Anyway, in theory, some third backend (whether closed- or open-source) could be 
similarly integrated, now that the abstract support is there.  I don't know if 
such a third backend even exists.

Looking through commit history, the last commits from the vendor developers to 
the object storage code were just before the release of 3.0.0.  So I would 
expect this works okay for 3.0 deployments.  I'm not sure about 3.2 -- we 
haven't had any specific updates, so maybe it works okay and doesn't need any?  
Or maybe it won't work until it's updated, and their customers are simply 
staying on 3.0 for the time being.

I guess this is kind of a long-winded way of saying, you probably need to speak 
to those vendors about this.  Whether Cyrus supports their backend is almost 
incidental compared to the larger question of whether (and how) they support 
their backend being used from Cyrus.

Cheers,

ellie

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Cyrus IMAP 3.2.1 released

[ellie timoney]

The Cyrus team is proud to announce the immediate availability of a new version 
of Cyrus IMAP: 3.2.1

Download URLs:


https://github.com/cyrusimap/cyrus-imapd/releases/download/cyrus-imapd-3.2.1/cyrus-imapd-3.2.1.tar.gz

https://github.com/cyrusimap/cyrus-imapd/releases/download/cyrus-imapd-3.2.1/cyrus-imapd-3.2.1.tar.gz.sig


Please consult the release notes and upgrade documentation before upgrading to 
3.2.1:

https://www.cyrusimap.org/imap/download/release-notes/3.2/x/3.2.1.html
https://www.cyrusimap.org/imap/download/upgrade.html

And join us on Github at https://github.com/cyrusimap/cyrus-imapd to report 
issues, join in the deliberations of new features for the next Cyrus IMAP 
release, and to contribute to the documentation.

On behalf of the Cyrus team,

Kind regards,

ellie timoney

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Cyrus IMAP 3.2.0 released

[ellie timoney]

On Mon, May 18, 2020, at 4:17 PM, Marco wrote:
> Ouch... I will try
> 
> Thank you for the hint
> Marco

I think I might have figured out a way to work around (not fix) the problem (by 
detecting when the LD_PRELOAD hasn't worked, and ignoring the syslog bits), and 
written it down here so I don't forget: 
https://github.com/cyrusimap/cassandane/issues/85

I still have no ideas or other insight into ~why~ this is happening, and it 
would be good to know, but maybe we can get you moving forward even if we don't 
know.

Cheers,

ellie

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Cyrus IMAP 3.2.0 released

[ellie timoney]

The discussion on github about the lmtpd crash on RHEL7 makes me wonder if 
FORTIFY_SOURCE, or something adjacent to it, is the cause of your Cassandane 
LD_PRELOAD problem as well?

I don't know much about it -- it's not default on my system, and even though 
github reminds me that I've looked at it before, I also haven't made it my own 
default. Whether that was for some good reason, or just that I didn't get 
around to it, I no longer recall!

On Mon, May 11, 2020, at 10:37 PM, Marco wrote:
> Hello Michael,
> 
> On 11/05/2020 10:45, Michael Menge has written:
> > RedHat systems use SELinux by default and SELinux has the habit to
> > block access to files, sockets, ... Especial if software from
> > non-Redhat repos is used.
> 
> thank you for the hint, I think that this is not my issue.
> 
> I disabled SELINUX prior to run Cassandane.
> 
> # sestatus
> SELinux status: disabled
> # getenforce
> Disabled
> 
> Cheers
> Marco
> 
> 
> 
> 
> 
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
>

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Cyrus IMAP 3.2.0 released

[ellie timoney]

Hi Marco,

> But it really seems that the LD_PRELOAD or the syslog.so doesn't 
> work for me.

Thanks for including that detail.  Hopefully someone familiar with RedHat can 
chime in with some insight into the LD_PRELOAD issue!  At a guess, "preventing 
library injection from intercepting syslog calls" might be a security default 
of some sort.

> A list of known failing tests for each stable supported version of Cyrus 
> IMAP could be very appreciated, but I understand that this could be 
> difficult to achieve...

Yeah, this would not really be workable, cause even though an individual 
release of Cyrus doesn't change once it's released, Cassandane changes 
constantly.  We could maybe publish such a list statically at release time, but 
it's potentially out of date days later, and it would be a headache to be 
regularly running Cassandane against all the old builds AND working out which 
new failures are "known failures" and which are "bugs in Cassandane" etc etc.

I think most people don't run Cassandane very often, so it's easier for them to 
just reach out for advice when they do.  And if you run Cassandane half a dozen 
times a day, like I do, you already know which failures can be ignored-for-now 
and which need investigation.

> My currently failing and now ignored tests are based on failing tests in 
> fedora for 3.0.x, reviewed in my env:
> [...]
>  # This fail on 3.2.0
>  # https://github.com/cyrusimap/cyrus-imapd/issues/2332
>  Caldav.supports_event

This one should be passing on 3.2.0, the issue above has been closed

>  # https://github.com/cyrusimap/cyrus-imapd/issues/2087
>  ImapTest.append-binary
>  ImapTest.fetch-binary-mime
>  ImapTest.urlauth-binary
> 
>  # This one seems to fail randomly on fedora, but in my env always 
> seems to be successful.
>  ImapTest.urlauth2

3.2.0 passes all the imaptest-20190504 tests for me, and I don't skip any.  But 
I don't see urlauth2 or any of the -binary ones in the output, so maybe 
upstream imaptest has removed the bad tests.  (The issue quoted was reported 
for imaptest-20170719)

The rest of these look like about what I'd expect, no surprises. Whew!

Cheers,

ellie

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Cyrus IMAP 3.2.0 released

[ellie timoney]

Hi Marco,

On Thu, May 7, 2020, at 7:51 PM, Marco wrote:
> The Instance.pm of Cassandane looks for a log in
> 
> {basedir}/conf/log/syslog
> 
> but the Cyrus IMAP log to system MAIL syslog, in redhat by default is 
> /var/log/maillog.

Did you run "make" in the cassandane directory, like the instructions at 
https://www.cyrusimap.org/imap/developer/developer-testing.html say to?

If you haven't run make, there's probably also an error in syslog telling you 
to do so, like:
> utils/syslog.so not found (do you need to run 'make'?)
> tests will not examine syslog output

Cassandane LD_PRELOADs utils/syslog.so, which intercepts syslog() calls and 
duplicates the lines into [basedir]/conf/log/syslog. That way Cassandane can a) 
read them without needing to know where your system's syslog output resides, 
and b) associate each group of syslog lines to the test that produced them.

Thugh... if it's failing on missing syslog errors, that means it found 
"utils/syslog.so" (otherwise it would be ignoring syslog problems), so you 
probably did already run "make". But if it still didn't find the syslog file, 
maybe that means the syslog intercept isn't working on your system. Off the top 
of my head, I have no particular insight as to why this would be.

Do you know whether Redhat does anything weird with, or needs any additional 
setup for, LD_PRELOAD?

> After patching this

If you patch Cassandane to look at your system's syslog file, then you lose the 
"b" behaviour above. So, tests run in parallel (with the -j option or 
maxworkers cassandane.ini option) will race against each other when they check 
syslog, and will fail randomly.

> 1) The Rename.rename_inbox fails for this syslog error:
> 
> "syslog error: May 7 10:42:36 imap 08292501FB/imap[21159]: IOERROR: 
> conversations_audit on store: 
> /root/rpmbuild/BUILD/cyrus-imapd-3.2.0/cassandane/work/08292501FB/conf/user/c/cassandane.conversations
>  
> B82adf4514b98522d 0 (8 1 0 0 () ((0 5 1 1 0)) () subject1 0 () 5)"

This test is known to fail at the moment. The test is correct (there is a 
subtle problem), and also fails on every previous version of Cyrus (the problem 
has existed for a long time, but has only recently had a test added to observe 
it), but there isn't a fix for it yet. This test doesn't fail against cyrus 
master, ironically because the mechanism Cassandane uses to trigger it no 
longer exists -- but as far as I know the problem still exists on master too.

You can just ignore it, or tell testrunner.pl not to run it by naming it as 
"!Rename.rename_inbox" or "~Rename.rename_inbox" on the command line. ('!' 
might interact with your shell history, so use "~" unless you know that it 
won't!)

I see Robert has already fixed the SearchFuzzy problems (thanks!). I didn't 
read the wall-of-errors in your earlier followup, since you had already 
followed up with the syslog detail. Do you have other tests that are failing 
still?

(If you have configured them to run, then also expect 
Cyrus::Caldav.netcaldavtalktests_fromical to fail, and 
Cyrus::Caldav.netcaldavtalktests_fromje to error. But these two tests won't run 
unless configured to, which they aren't by default. I believe they're are stale 
and need updating somehow, but I haven't looked further.)

Cheers,

ellie
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Cyrus IMAP 3.2.0 released

[ellie timoney]

Hi Marco,

> I work with 
> https://github.com/cyrusimap/cassandane/archive/00bfe0109f80437ed09154aca9fbd53eef8f1b09.tar.gz
> This cassandane release works pretty with 3.0.12. I didn't find build 
> changes in Release notes for 3.2.0...

We don't do releases of Cassandane.  That thing is just a snapshot that github 
generated at some point, not a "release".  I checked the git commit that 
snapshot was generated from, and it's nearly 2 years old? I'm not surprised 
that 3.2.0 fails lots of those tests -- they've never been updated to expect 
3.2.0's behaviour.

Cassandane is designed to be run from a git clone, and if you're using it, you 
should generally keep it up to date.  It is kept generally backwards 
compatible, because I use it for testing new releases from old branches. You 
don't need different versions of it for testing different versions of Cyrus, 
you need the most recent version of it so that you have the most-accurate and 
most-thorough tests currently available.

Cheers,

ellie

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Cyrus IMAP 3.2.0 released

[ellie timoney]

The Cyrus team is proud to announce the first stable release from the new Cyrus 
IMAP 3.2 series: 3.2.0

The main https://www.cyrusimap.org/ website now shows content for the 3.2 
series.

The website for the previous stable 3.0 series is still available, now at 
https://www.cyrusimap.org/3.0/

Download URLs:


https://github.com/cyrusimap/cyrus-imapd/releases/download/cyrus-imapd-3.2.0/cyrus-imapd-3.2.0.tar.gz

https://github.com/cyrusimap/cyrus-imapd/releases/download/cyrus-imapd-3.2.0/cyrus-imapd-3.2.0.tar.gz.sig


Please consult the release notes and upgrade documentation before upgrading to 
3.2:

https://www.cyrusimap.org/3.2/imap/download/release-notes/3.2/x/3.2.0.html
https://www.cyrusimap.org/3.2/imap/download/upgrade.html

And join us on Github at https://github.com/cyrusimap/cyrus-imapd to report 
issues, join in the deliberations of new features for the next Cyrus IMAP 
release, and to contribute to the documentation.

On behalf of the Cyrus team,

Kind regards,

ellie timoney

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: sync_log_chain - is it always needed?

[ellie timoney]

> Just to clarify here - are not IMAP/POP/LMTP actions logged anyway - 
> using sync_log? Won't they grow forever, too?

Yes, but no, because your replica is not taking IMAP/POP/LMTP/etc traffic, 
therefore no such actions are occurring on it.

If it is taking that traffic, it's not a replica, it's a master; if it needs to 
replicate to somewhere else while it's acting as master, then it needs a 
rolling sync_client, which will consume the sync_log, and it won't grow 
forever.  If it doesn't need to replicate anywhere else (i.e. maybe you don't 
have a third server in the set) then it probably wouldn't have sync_log on 
anyway, but if it does, you should turn it off in this situation.

If your replica is accidentally taking that traffic while still being a replica 
for replication purposes, you have a split brain situation, which can be fiddly 
to clean up.  So be careful to avoid this!

> What is a scenario where sync_log and sync_log_chain is used 
> independently? What is the purpose to have sync_log and sync_log_chain 
> as separate options - couldn't we just use sync_log?

sync_log should be enabled on _servers that need to replicate to elsewhere in 
approximately real time_ (let's call this "category A"), and they need to have 
a rolling sync_client active to do this, which consumes the sync logs

sync_log_chain should be enabled on the subset of servers in category A that 
also receive traffic via replication from elsewhere

So, sync_log and sync_log_chain cannot be interchangeable.  sync_log means "i 
am running a rolling sync_client, and it needs a source of events", and 
sync_log_chain means " and that should also include replication events".  A 
replica that is chaining needs _both_.

In an M->R1->R2 setup, M needs "sync_log", R1 needs "sync_log" and 
"sync_log_chain", and R2 needs neither

In an M->R1 + M->R2 setup, M needs "sync_log". R1 and R2 don't need sync 
configuration.  They can safely have sync_log enabled because they're not 
receiving user traffic anyway, but they don't need it.

In an M1->R1 (there is no R2) setup, M needs "sync_log".  R1 doesn't need sync 
configuration (but can safely have sync_log enabled), and R2 doesn't exist.

Presumably, you set up replication so that if something happens to your usual 
master server, you can restore service by carefully promoting one of your 
replicas into the master role.

In an M->R1->R2 role, presumably you're planning to switch to R1 in an M-failed 
scenario, since it will have the most up-to-date data.  It's already set up to 
replicate to R2, and while it is acting as master it won't be receiving 
replication traffic, so leaving sync_log_chain on won't hurt, and it will still 
need sync_log on to keep replicating to R2.  (This also implies that, if you 
have multiple data centres, M and R1 should be in different data centres, R1 
and R2 should be in different data centres, but R2 can be in the same data 
centre as M with the caveat that if you lose that whole dc, then the 
promoted-R1 has nowhere to replicate to.  YMMV depending on the size of your 
deployment, your resiliency requirements, your budget, etc etc.)

In an M->R1 + M->R2 setup, both replicas are reasonably up to date, and either 
would make a suitable failover target.  You probably want to configure 
whichever one you promote to now replicate to the other, where previously it 
didn't replicate to anywhere.

When you get M rebuilt after the disaster, you probably want to initially set 
it up as a replica in the set, so that it can be brought up to date without 
taking user traffic.  Then, if you want to make it the master again, halt 
non-admin traffic to the whole set, perform a final replication to it to make 
sure it's exactly up to date (should be quick, since the current-master will 
only have a few minutes of changes to catch up on, since you already brought it 
up to date), then flip your configurations/dns/proxy/whatever else you need to 
do, and put your sync configuration back to usual.  Make sure everything's 
fine, then re-enable non-admin traffic.

> Are both types of entries - from sync_log and sync_log_chain used only 
> by rolling replication? Or are they used by sync_client -A too?

No, the sync log (both "types") is only used by the rolling sync_client mode.  
The sync log is how a rolling sync_client knows what to replicate.  Every other 
sync_client mode gets a list of "what to replicate" in its command line 
arguments, and only replicates what it's told.

sync_log -A replicates all users.  "-A" constitutes a "list of what to 
replicate".  Thus it ignores the sync log.

> Sorry for the silly questions, the replication is quite scantly documented.

Have you seen this?  
https://www.cyrusimap.org/imap/reference/admin/sop/replication.html

Pull requests appreciated, of course.

> Thank you for all the explanations so far, regards,

No worries, hope this is helpful.

ellie

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: 

Re: sync_log_chain - is it always needed?

[ellie timoney]

On Thu, Apr 30, 2020, at 12:29 AM, Olaf Frączyk wrote:
> However why the sync_log_chain could be not always active? As the server 
> catches all changes from LMTP, IMAP, POP anyway, why to use a special 
> option for the synchronization protocol? Why to treat replication 
> changes differently from LMTP/IMAP? If we want replication - we want to 
> catch all changes - regardless of the source, if we don't do replication 
> - we don't need to catch any changes.

The rolling sync_client takes care of cleaning up each sync_log file as it 
finishes replicating it downstream.

Now consider the case where your replica is an end point, not a link in a 
chain: it does not have a rolling sync_client forwarding replications on, so if 
it automatically logged incoming replications to the sync_log, the sync_log 
would simply grow forever and fill the disk.  You would need to set up a 
special job to delete it Better to simply not write it in the first place!

Thus, the default is to sync_log_chain: off, and if you need the special-case 
chaining behaviour, you turn it on.

Cheers,

ellie

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: sync_log_chain - is it always needed?

[ellie timoney]

On Wed, Apr 29, 2020, at 1:12 AM, Olaf Frączyk wrote:
> I was wondering why do we need to use this option on middle servers in 
> replication chain?

When a user/delivery/admin/etc action happens on a mailbox/user/etc, an entry 
is recorded in the sync log saying that something has happened to that 
mailbox/user/etc, and that it will need to be checked and replicated at some 
point.

sync_client in rolling mode (or log-replay mode) reads that sync log, and for 
each mailbox/user/etc that the sync log reports as having been changed, it 
compares its current state against the replica, and updates the replica to 
match if necessary.

When a change arrives by replication, it is NOT logged to the sync_log -- 
unless sync_log_chain has been enabled.

So if you want a rolling replication arrangement such that Master -> Replica1 
-> Replica2, then Replica1 _must_ have sync_log_chain enabled, otherwise there 
will never be anything in its sync log, and it will therefore never forward the 
replications on to Replica2.  This arrangement has the advantage that Master 
only needs to do one replication (and can focus its energy on servicing clients 
instead), but the disadvantage that the state on Replica2 will not be as fresh 
as the state on Master or Replica1.

But if you want an arrangement such that Master -> Replica1 and Master -> 
Replica2, then you don't need sync_log_chain anywhere.  This arrangement has 
the disadvantage that Master must do more replication work, but the advantage 
that Replica1 and Replica2 are more likely to both be up-to-date at any given 
moment.

> I don't use sync_server. The replication is done using IMAP protocol. Is 
> in this case this setting also necessary?

It makes no difference; replication is replication.  The replication support in 
imapd is a thin wrapper around the exact same implementation as used by 
sync_server.

> Shouldn't the middle server catch all changes that are done via IMAP 
> protocol and forward them to the next server in the replication chain?

The changes are not done "via IMAP protocol", they are done "via replication 
protocol, over an IMAP connection".  For example, you couldn't use Cyrus 
replication to update some other non-Cyrus IMAP server -- it would not 
understand the replication commands.

If you want a chained configuration, then you need sync_log_chain, regardless 
of whether you use imapd or sync_server.

> BTW. Does anyone has some description how sieve scripts replication is 
> done via IMAP protocol?

Can you clarify what you want to know here?  It's replicated the same way as it 
would be to sync_server:

The sync_client queries the replica's state, decides whether it needs updating, 
and if so, then it sends the new state, and checks for an OK response from the 
replica.

Depending on the Cyrus version of your master server, it might (in older 
versions) try to send both the sieve script file and the compiled bytecode 
file; or (in recent versions) just the script file.

Depending on the Cyrus version of your replica servers, it might accept and 
update a bytecode file as-is; or (in recent versions) simply accept and ignore 
it, and recompile the bytecode on demand from the script file that was also 
sent.

If, for some reason, you're replicating from a recent version (that only sends 
the script file) to an old version (that expects both), then your replica will 
not have up to date compiled bytecode.  If your replica needs to be brought 
into service taking real traffic, you will first need to manually (re)compile 
all your scripts using sievec.  This is almost certainly not the only quirk of 
replicating to an older server and then trying to run it with real traffic, so 
try to not need to do that! :)

It's not safe/possible to run bytecode on one Cyrus version that was compiled 
by a different Cyrus version, which is why in recent versions we do not bother 
to replicate bytecode at all.  Recent Cyrus versions instead have a mechanism 
to recompile bytecode on demand if it's missing, out of date, or the wrong 
version, so that you will always be running the correct, up-to-date bytecode 
for the current Cyrus version for the current contents of the script.

Regardless of your Cyrus version, the state of "which script is currently 
activated for the user" is also replicated in the same way.

Hope this helps,

ellie

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Replication failed 3.0.5 -> 3.0.13, now 3.0.13->3.0.13

[ellie timoney]

I think Michael's got this pretty much covered -- you need to disable the 
rolling replication for now, and then use sync_client -u (or if you're brave, 
sync_client -A) to get an initial sync of everything.  These two options work 
entire-user-at-a-time, so they should detect and fix the problems introduced by 
the partial rolling sync.  

If you have mailboxes in a shared namespace (i.e. that are outside the user/ 
namespace), they won't be replicated by -u or -A.  You'll need to initially 
replicate those individually with sync_client -m.

Once you've got a complete initial sync done, you can use rolling replication 
to keep the replica up to date.  You can put the rolling 'sync_client -r' in 
the DAEMON section, so that Cyrus will restart it if it exits.  Or you could 
manage it from outside Cyrus, e.g. via systemd/initd if you prefer.

You cannot put sync_client in the SERVICES section.  The SERVICES section is 
for service processes (i.e. processes that listen on a socket and service 
client requests).  sync_client is a client, not a service.

Cheers,

ellie

On Wed, Apr 22, 2020, at 4:40 AM, Michael Menge wrote:
> 
> Quoting Olaf Frączyk :
> 
> >
> > Yes, at the beginning I was also thinking if initial sync is  
> > necessary, but there was nothing in docs about it, something started  
> > replicating and I simply assumed it does initial resync. I'll try it  
> > this evening. :)
> >
> > Since you use replication - are sieve scripts replicated as well?  
> > There is -s option called sieve mode but it needs to specify which  
> > users' files are to replicate and there is written that it is mostly  
> > for debugging.
> >
> 
> Yes, sieve scripts are replicated.
> 
> The way the rolling replication works is, that every time something is changed
> on the master a "hint" is written in the sync log,
> 
> "MAILBOX user.foo.bar" indicates that the mailbox bar of the user foo  
> has changed
> and the sync_client will sync this (and only this folder)
> There are other "hints" e.g for changed subscription or changed sieve script.
> 
> But if the  sieve script is not changed sync_client in rolling replication
> will not try to sync it. Using the -A or -u Option will sync the all/some
> users, including all mailboxes, folder subscriptions and sieve scripts.
> 
> The -s option is only needed if you change a compiled sieve script so
> that it is not logged in the replication log.
> 
> 
> 
> 
> 
> M.MengeTel.: (49) 7071/29-70316
> Universität Tübingen   Fax.: (49) 7071/29-5912
> Zentrum für Datenverarbeitung  mail:  
> michael.me...@zdv.uni-tuebingen.de
> Wächterstraße 76
> 72074 Tübingen
> 
> 
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Cyrus IMAP 3.2.0-rc1 released

[ellie timoney]

The Cyrus team is proud to announce the first release candidate from the new 
Cyrus IMAP 3.2 series: 3.2.0-rc1

While 3.2 is still in pre-release, the main https://www.cyrusimap.org/ website 
will continue to be focused on the stable 3.0 series.  The 3.2 website is 
available at https://www.cyrusimap.org/3.2/

Download URLs:


https://github.com/cyrusimap/cyrus-imapd/releases/download/cyrus-imapd-3.2.0-rc1/cyrus-imapd-3.2.0-rc1.tar.gz

https://github.com/cyrusimap/cyrus-imapd/releases/download/cyrus-imapd-3.2.0-rc1/cyrus-imapd-3.2.0-rc1.tar.gz.sig


Please consult the release notes and upgrade documentation before upgrading to 
3.2:


https://www.cyrusimap.org/3.2/imap/download/release-notes/3.2/x/3.2.0-rc1.html
https://www.cyrusimap.org/3.2/imap/download/upgrade.html

And join us on Github at https://github.com/cyrusimap/cyrus-imapd to report 
issues, join in the deliberations of new features for the next Cyrus IMAP 
release, and to contribute to the documentation.

On behalf of the Cyrus team,

Kind regards,

ellie timoney

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Cyrus IMAP 3.2.0-beta4 released

[ellie timoney]

Correction:


https://www.cyrusimap.org/3.2/imap/download/release-notes/3.2/x/3.2.0-beta4.html
https://www.cyrusimap.org/3.2/imap/download/upgrade.html

On Mon, Mar 30, 2020, at 3:13 PM, ellie timoney wrote:
> The Cyrus team is proud to announce the fourth beta release from the 
> new Cyrus IMAP 3.2 series: 3.2.0-beta4
> 
> While 3.2 is still in beta, the main https://www.cyrusimap.org/ website 
> will continue to be focused on the stable 3.0 series.  The 3.2 website 
> is available at https://www.cyrusimap.org/3.2/
> 
> Download URLs:
> 
> 
> https://github.com/cyrusimap/cyrus-imapd/releases/download/cyrus-imapd-3.2.0-beta4/cyrus-imapd-3.2.0-beta4.tar.gz
> 
> https://github.com/cyrusimap/cyrus-imapd/releases/download/cyrus-imapd-3.2.0-beta4/cyrus-imapd-3.2.0-beta4.tar.gz.sig
> 
> 
> Please consult the release notes and upgrade documentation before 
> upgrading to 3.2:
> 
> 
> And join us on Github at https://github.com/cyrusimap/cyrus-imapd to 
> report issues, join in the deliberations of new features for the next 
> Cyrus IMAP release, and to contribute to the documentation.
> 
> On behalf of the Cyrus team,
> 
> Kind regards,
> 
> ellie timoney
>

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Cyrus IMAP 3.2.0-beta4 released

[ellie timoney]

The Cyrus team is proud to announce the fourth beta release from the new Cyrus 
IMAP 3.2 series: 3.2.0-beta4

While 3.2 is still in beta, the main https://www.cyrusimap.org/ website will 
continue to be focused on the stable 3.0 series.  The 3.2 website is available 
at https://www.cyrusimap.org/3.2/

Download URLs:


https://github.com/cyrusimap/cyrus-imapd/releases/download/cyrus-imapd-3.2.0-beta4/cyrus-imapd-3.2.0-beta4.tar.gz

https://github.com/cyrusimap/cyrus-imapd/releases/download/cyrus-imapd-3.2.0-beta4/cyrus-imapd-3.2.0-beta4.tar.gz.sig


Please consult the release notes and upgrade documentation before upgrading to 
3.2:

https://cyrusimap.org/3.2/imap/download/release-notes/3.2/x/3.2.0-beta4.html
https://cyrusimap.org/3.2/imap/download/upgrade.html

And join us on Github at https://github.com/cyrusimap/cyrus-imapd to report 
issues, join in the deliberations of new features for the next Cyrus IMAP 
release, and to contribute to the documentation.

On behalf of the Cyrus team,

Kind regards,

ellie timoney

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Cyrus IMAP 3.2.0-beta2 released

[ellie timoney]

The Cyrus team is proud to announce the third beta release from the new Cyrus 
IMAP 3.2 series: 3.2.0-beta3

While 3.2 is still in beta, the main https://www.cyrusimap.org/ website will 
continue to be focused on the stable 3.0 series.  The 3.2 website is available 
at https://www.cyrusimap.org/3.2/

I'm trialling hosting the release files using Github's releases feature.  
Please use the Github download links if possible, and advise if you have any 
problems!  (It may even download faster due to Github's content delivery 
network.)

Download URLs:


https://github.com/cyrusimap/cyrus-imapd/releases/download/cyrus-imapd-3.2.0-beta3/cyrus-imapd-3.2.0-beta3.tar.gz

https://github.com/cyrusimap/cyrus-imapd/releases/download/cyrus-imapd-3.2.0-beta3/cyrus-imapd-3.2.0-beta3.tar.gz.sig


https://www.cyrusimap.org/releases/cyrus-imapd-3.2.0-beta3.tar.gz
https://www.cyrusimap.org/releases/cyrus-imapd-3.2.0-beta3.tar.gz.sig

Please consult the release notes and upgrade documentation before upgrading to 
3.2:

https://cyrusimap.org/3.2/imap/download/release-notes/3.2/x/3.2.0-beta3.html
https://cyrusimap.org/3.2/imap/download/upgrade.html

And join us on Github at https://github.com/cyrusimap/cyrus-imapd to report 
issues, join in the deliberations of new features for the next Cyrus IMAP 
release, and to contribute to the documentation.

On behalf of the Cyrus team,

Kind regards,

ellie timoney

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Xapian index not being used for message search in Roundcube

[ellie timoney]

I don't really understand search in any depth, but it's interesting to observe 
that, in addition to the different command (SEARCH vs THREAD), those two 
searches are also using different search criteria ("BODY linux" vs "TEXT 
linux").

It might be informative to try do the SEARCH search with "TEXT linux" instead 
of "BODY linux", to narrow down whether the difference is due to the use of the 
SEARCH vs THREAD  command, or the use of the "BODY" vs "TEXT" search key?

Looking at the source on master, SEARCH and THREAD both seem to be using the 
same search API, so at a glance it seems like they should both be using Xapian 
if either is.  And looking at the commit dates on those functions, it doesn't 
look like it's changed substantially since 3.0, at least not at a level I can 
easily see.

I had a quick look at the RFC, and "BODY" searches just the message body, 
whereas "TEXT" searches both body and headers.  So I wonder if the difference 
is that TEXT needs to open all the message files to read the headers, whereas 
BODY can just return results straight from the Xapian index?

I'm not sure if there's been changes to header searching (like, maybe we index 
more of the header content?) since 3.0, but this is getting beyond what I know 
off the cuff or can just casually look up.

Anyway, if you could try "UID SEARCH TEXT linux" and see if that's similarly 
slow to the THREAD version, that would give us a definite pointer in the right 
direction.

Cheers,

ellie

On Sun, Feb 23, 2020, at 10:11 PM, Frederik Himpe via Info-cyrus wrote:
> I have configured Cyrus 3.0.13 with the Xapian search engine and
> enabled search_fuzzy_always. This appears to work fine when I search in
> the message body using the Evolution mail client, as I get a response
> quickly:
> 
> <1582453709 >1582453713>* SEARCH 226927
> 226929 226964 226974 226999 227215 227238 [...]
> L03163 OK Completed (643
> msgs in 0.970 secs)
> 
> However when I search messages using the Roundcube webmail client,
> Roundcube does not get a response in time and shows no results. An
> strace of the imapd proceess indicates it is STATing, OPENing and
> MMAPing all files in the mailbox.
> 
> This is the log:
> <1582455581 >1582455723>* THREAD
> (229566)(229570)(229574)(229599)(229618)(229639)[...]
> A0004 OK Completed (157 msgs in 11.340 secs)
> 
> So it appears Roundcube is using a different command to search. Is it
> expected that this command does not use the Xapian search engine? Is
> there a way to make it use it?
> 
> Some relevant snippets from imapd.conf:
> sync_log: on
> sync_log_channels: squatter
> 
> conversations: 1
> search_engine: xapian
> search_index_headers: no
> search_batchsize: 8192
> search_fuzzy_always: 1
> defaultsearchtier: temp
> tempsearchpartition-default: /var/lib/cyrus/search.temp
> datasearchpartition-default: /var/lib/cyrus/search.data
> 
> cyrus.conf:
> 
> EVENTS {
> squatter1   cmd="/usr/bin/nice -n 19 /usr/sbin/cyrus 
> squatter -z data -t temp,data" at=0517
> 
> }
> DAEMON {
>   squatter cmd="squatter -R"
> }
> 
> 
> Regards,
> 
> -- 
> Frederik Himpe 
> 
> 
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
>

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Cyrus IMAP 3.2.0-beta2 released

[ellie timoney]

The Cyrus team is proud to announce the second beta release from the new Cyrus 
IMAP 3.2 series: 3.2.0-beta2

While 3.2 is still in beta, the main https://www.cyrusimap.org/ website will 
continue to be focused on the stable 3.0 series.  The 3.2 website is available 
at https://www.cyrusimap.org/3.2/

I'm trialling hosting the release files using Github's releases feature.  
Please use the Github download links if possible, and advise if you have any 
problems!  (It may even download faster due to Github's content delivery 
network.)

Download URLs:


https://github.com/cyrusimap/cyrus-imapd/releases/download/cyrus-imapd-3.2.0-beta2/cyrus-imapd-3.2.0-beta2.tar.gz

https://github.com/cyrusimap/cyrus-imapd/releases/download/cyrus-imapd-3.2.0-beta2/cyrus-imapd-3.2.0-beta2.tar.gz.sig


https://www.cyrusimap.org/releases/cyrus-imapd-3.2.0-beta2.tar.gz
https://www.cyrusimap.org/releases/cyrus-imapd-3.2.0-beta2.tar.gz.sig

Please consult the release notes and upgrade documentation before upgrading to 
3.2:

https://cyrusimap.org/3.2/imap/download/release-notes/3.2/x/3.2.0-beta2.html
https://cyrusimap.org/3.2/imap/download/upgrade.html

And join us on Github at https://github.com/cyrusimap/cyrus-imapd to report 
issues, join in the deliberations of new features for the next Cyrus IMAP 
release, and to contribute to the documentation.

On behalf of the Cyrus team,

Kind regards,

ellie timoney

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Cyrus IMAP 3.2.0-beta1 released

[ellie timoney]

The Cyrus team is proud to announce the first beta release from the new Cyrus 
IMAP 3.2 series: 3.2.0-beta1

While 3.2 is still in beta, the main https://www.cyrusimap.org/ website will 
continue to be focused on the stable 3.0 series.  The 3.2 website is available 
at https://www.cyrusimap.org/3.2/

I'm trialling hosting the release files using Github's releases feature.  
Please use the Github download links if possible, and advise if you have any 
problems!  (It may even download faster due to Github's content delivery 
network.)

Download URLs:


https://github.com/cyrusimap/cyrus-imapd/releases/download/cyrus-imapd-3.2.0-beta1/cyrus-imapd-3.2.0-beta1.tar.gz

https://github.com/cyrusimap/cyrus-imapd/releases/download/cyrus-imapd-3.2.0-beta1/cyrus-imapd-3.2.0-beta1.tar.gz.sig


https://www.cyrusimap.org/releases/cyrus-imapd-3.2.0-beta1.tar.gz
https://www.cyrusimap.org/releases/cyrus-imapd-3.2.0-beta1.tar.gz.sig

Please consult the release notes and upgrade documentation before upgrading to 
3.2:

https://cyrusimap.org/3.2/imap/download/release-notes/3.2/x/3.2.0-beta1.html
https://cyrusimap.org/3.2/imap/download/upgrade.html

And join us on Github at https://github.com/cyrusimap/cyrus-imapd to report 
issues, join in the deliberations of new features for the next Cyrus IMAP 
release, and to contribute to the documentation.

On behalf of the Cyrus team,

Kind regards,

ellie timoney

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Unable to subscribe to folders.

[ellie timoney]

On Mon, Feb 3, 2020, at 11:02 PM, Sebastian Hagedorn wrote:
> just a guess, but isn't LSUB the command for listing subscribed
> mailboxes? I'm not actually sure it makes a difference, but you should
> give it a try ...
> 

LIST (\Subscribed) is one of the extensions we support, it should work fine 
(don't remember the rfc number off top of head)

I would be looking at whether the ACL for that mailbox is weird (maybe you 
don't have permission to see it?), but right at the moment I don't have a 
reference handy for what you'd need to check for, so just check the docs

Cheers,

ellie
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Database upgrade and Xapian version dependency

[ellie timoney]

>> On Mon, Jan 27, 2020, at 9:51 AM, Egoitz Aurrekoetxea via Info-cyrus wrote:
>>> Just for having it slightly clearer… When you upgrade the Cyrus version and 
>>> the version you are upgrading to is a too close one… for instance from 
>>> 3.0.8 to 3.0.13 and you see the Cyrus version is the same for users mail 
>>> folders, 13 in both… is it needed to launch (or recommended for some 
>>> reason) the final upgrade commands : 
>>> 
>>> reconstruct -V max
>>> ctl_conversationsdb -b -r
>>> quota -f

As long as you're looking at 3.x.x and higher*, then if it's a stable release 
(where only the third number has changed, e.g. 3.0.8->3.0.13) you shouldn't 
need to. We don't do big world-breaking/data-format-changing changes in stable 
releases like this.

That said, we might have fixed a bug between the versions, and your existing 
data might be bad due to the bug that was fixed, and you might need to run 
commands like these after upgrading to the fixed version to repair the bad 
data. If I know this is the case, I'll say so in the release notes (so check 
those, including for the intermediate releases if you're skipping over some). 
But I might not know until someone upgrades and reports that they needed to do 
it.

So, if it doesn't say you need to do it, you probably don't need to do it. But 
if you don't do it, and things like quota or conversations seem weird after the 
upgrade, try these sort of commands. And if it fixes it, let us know (on the 
mailing list or a github issue) so we can retcon the release notes to mention 
it for the next person. :)

* For people looking at 2.5 and earlier, this pattern doesn't necessarily hold, 
and I don't know enough about those older versions to provide general advice. 
If in doubt, ask the mailing list, and if you can run a recent version (that's 
closer to what's in Fastmail's organisational memory), that's your best bet ;)

Cheers,

ellie
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Unable to build Cyrus

[ellie timoney]

Hi Daniel,

On Thu, Jan 23, 2020, at 10:34 PM, Daniel Gultsch wrote:
> /usr/bin/ld: warning: libicui18n.so.57, needed by
> /usr/lib/gcc/x86_64-linux-gnu/6/../../../x86_64-linux-gnu/libxml2.so,
> may conflict with libicui18n.so.64
> /usr/bin/ld: warning: libicuuc.so.57, needed by
> /usr/lib/gcc/x86_64-linux-gnu/6/../../../x86_64-linux-gnu/libxml2.so,
> may conflict with libicuuc.so.64
> /usr/bin/ld: warning: libicudata.so.57, needed by
> /usr/lib/gcc/x86_64-linux-gnu/6/../../../x86_64-linux-gnu/libxml2.so,
> may conflict with libicudata.so.64

For starters, I would ignore these warnings for now, unless it turns out to be 
an actual problem.  

What's happening here is you have libicu v64 installed via cyruslibs, and you 
have libxml installed from a system package.  libxml depends on libicu, which 
means you also have the system package of libicu, which in this case is v57.  
(I guess you're on something akin to Debian Jessie, cause I am, and I also have 
libicu v57 installed for libxml.)  You will see these warnings if the link 
order is not quite right, or if the link order is correct but the linker is 
linking stuff it doesn't need to.

It would be interesting to see what happens if you add "-Wl,--as-needed" to 
your LDFLAGS -- this is very useful for shaking out link ordering problems -- 
but I don't (yet) think that's the cause of the problem below.  Some platforms 
have this linker option enabled by default, but Debian Jessie doesn't.

> imap/ctl_zoneinfo.o: In function `main':
> /root/cyrus-imapd/imap/ctl_zoneinfo.c:232: undefined reference to
> `xmlGetNextNode'
> /root/cyrus-imapd/imap/ctl_zoneinfo.c:234: undefined reference to
> `xmlGetNextNode'
> /root/cyrus-imapd/imap/ctl_zoneinfo.c:240: undefined reference to
> `xmlGetNextNode'
> /root/cyrus-imapd/imap/ctl_zoneinfo.c:251: undefined reference to
> `xmlGetNextNode'
> /root/cyrus-imapd/imap/ctl_zoneinfo.c:253: undefined reference to
> `xmlGetNextNode'
> collect2: error: ld returned 1 exit status
> Makefile:3639: recipe for target 'imap/ctl_zoneinfo' failed

This is where things get interesting.  xmlGetNextNode is a fallback we provide 
for when libxml doesn't provide xmlFirstElementChild (in which case we #define 
xmlFirstElementChild to call our own xmlGetNextNode).  So it sounds like 
configure might be confused about whether your libxml provides 
xmlFirstElementChild or not...  Do you happen to have multiple versions of 
libxml installed, I wonder?

If you remove the following files from your tree:

./imap/ctl_zoneinfo.o
./imap/ctl_zoneinfo
./imap/.deps/ctl_zoneinfo.Po
./imap/.libs/ctl_zoneinfo

(i.e. the generated files for the ctl_zoneinfo program) and then run "make V=1 
imap/ctl_zoneinfo" to rebuild just this program, what commands are being run to 
rebuild this file?  The V=1 ensures the full commands are output, overriding 
--enable-silent-rules.

The contents of your config.h and config.log would also be very helpful, but 
please check/sanitise them carefully for private information before attaching 
them.  Feel free to send them directly to me (ellie at fastmail dot com) if 
you'd rather not share them with the whole list.

Cheers,

ellie

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Ftp server down

[ellie timoney]

Hi,

That's a known issue, it's been down for months, and we don't really expect it 
to come back.

All our releases are available over HTTPS from https://cyrusimap.org/releases/

Cheers,

ellie

On Tue, Jan 14, 2020, at 9:33 AM, Roar Brænden wrote:
> Hi,
> 
> I've tried to install a calendar server from calendarserver.org. That 
> software is based on components from cyrusimap. During the setup there is a 
> call for ftp.cyrusimap.org which fails with a "Connection timed out". It has 
> been like this for a couple of days.
> 
> Is this a known issue?
> 
> Should the address be changed to something else?
> 
> Best regards
> 
> Roar Brænden
> 
> 
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Migration issue with seen/subscription/sieve databases

[ellie timoney]

On Mon, Dec 30, 2019, at 8:59 PM, Gionatan Danti wrote: 
> Are you referring to the problem described here [1]? If so, from the 
> linked page I read:
> "Versions of 3.0 prior to 3.0.11 contained a bug (Issue #2839) that 
> could lead to loss of seen state/flags during reconstruct for some 
> messages that already existed prior to Cyrus 2.3"
> 
> My Cyrus installation was never older than 2.3.x, so I thought the bug 
> should not affect me. Am I wrong?

I think, in that case, you should be fine!  It wasn't clear if your 
installation had only ever been 2.3, or had previously been upgraded to 2.3 
from some earlier version in the past, so I thought I'd better point it out 
just in case :)

Cheers,

ellie

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Migration issue with seen/subscription/sieve databases

[ellie timoney]

On Fri, Dec 27, 2019, at 11:40 PM, Gionatan Danti wrote:
> Il 27-12-2019 04:07 Scott Lambert ha scritto:
> > Didn't the seen status database get moved from it's own file to the
> > message index database for 3.x?  I'm not running 3.x yet but think I
> > remember seeing something about that.
> 
> Hi, from what I read here [1], and from my experience when testing the 
> migration, the seen database has its own file (userid.seen).

You're both right.  Since 2.4ish (??  MAILBOX_MINOR_VERSION=12, anyway), a 
user's seen status on their own mailboxes is stored in the mailbox index.  But 
the separate userid.seen database is still used for storing their seen state on 
mailboxes that they don't own (but have ACL access to see).

On a system that was set up after that change, I think the userid.seen file 
will only be created if the user views messages in a mailbox that they don't 
own.  If no-one has access to view anyone else's mailbox, this will never 
happen, which might explain why Patrick doesn't have any .seen files.

> A very simple workaround seems to soft-link the dot-enabled files with 
> the circumflex ones, ie: "cd ./domain/a/assyoma.it/user/g/; ln -s 
> g^danti.seen g.danti.seen"

It's probably safer to just rename them -- if nothing else, it'll be less 
confusing next time you or someone else looks at them.  If you wanted to move 
cautiously, you could hard link the corrected name, and then unlink the old 
name later once you're sure everything's working correctly.

> I am tasked to migrate an old cyrus 2.3.x CentOS6 installation to a new 
> CentOS8 server with cyrus 3.0.7.

With your old installation being as old as it is, it may contain data 
susceptible to https://github.com/cyrusimap/cyrus-imapd/issues/2839 (I have 
just fixed a typo in the upgrade documentation that linked to the wrong issue 
number, which will update on the web after the next hour.)  This issue wasn't 
fixed until 3.0.11, so I would check carefully whether CentOS has backported 
the patch to their 3.0.7 package.  There have also been security fixes in 
releases since 3.0.7, so you probably also want to check whether CentOS's 3.0.7 
has backported these too.  If the CentOS package hasn't backported these fixes, 
consider running the current version of Cyrus rather than the one CentOS 
provides.

Cheers,

ellie

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Cyrus IMAP 3.0.13 released

[ellie timoney]

The Cyrus team is proud to announce the immediate availability of a new version 
of Cyrus IMAP: 3.0.13

This release contains a fix for CVE-2019-19783, a privilege escalation 
vulnerability that permits creation of arbitrary mailboxes using the 'fileinto' 
directive in user sieve scripts.  If you allow your users to upload custom 
sieve scripts, and if you have the 'mailbox' sieve extension or the 
'anysievefolder' option enabled, you will need this upgrade.

I'm trialling hosting the release files using Github's releases feature.  
Please use the Github download links if possible, and advise if you have any 
problems!  (It may even download faster due to Github's content delivery 
network.)

Download URLs:


https://github.com/cyrusimap/cyrus-imapd/releases/download/cyrus-imapd-3.0.13/cyrus-imapd-3.0.13.tar.gz

https://github.com/cyrusimap/cyrus-imapd/releases/download/cyrus-imapd-3.0.13/cyrus-imapd-3.0.13.tar.gz.sig

https://www.cyrusimap.org/releases/cyrus-imapd-3.0.13.tar.gz
https://www.cyrusimap.org/releases/cyrus-imapd-3.0.13.tar.gz.sig

Please consult the release notes and upgrade documentation before upgrading to 
3.0.13:

https://www.cyrusimap.org/imap/download/release-notes/3.0/x/3.0.13.html
https://www.cyrusimap.org/imap/download/upgrade.html

And join us on Github at https://github.com/cyrusimap/cyrus-imapd to report 
issues, join in the deliberations of new features for the next Cyrus IMAP 
release, and to contribute to the documentation.

On behalf of the Cyrus team,

Kind regards,

ellie timoney

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Cyrus IMAP 2.5.15 released

[ellie timoney]

The Cyrus team is proud to announce the immediate availability of a new version 
of Cyrus IMAP: 2.5.15

This release contains a fix for CVE-2019-19783, a privilege escalation 
vulnerability that permits creation of arbitrary mailboxes using the 'fileinto' 
directive in user sieve scripts.  If you allow your users to upload custom 
sieve scripts, and if you have the 'anysievefolder' option enabled, you will 
need this upgrade.

I'm trialling hosting the release files using Github's releases feature.  
Please use the Github download links if possible, and advise if you have any 
problems!  (It may even download faster due to Github's content delivery 
network.)

Download URLs:


https://github.com/cyrusimap/cyrus-imapd/releases/download/cyrus-imapd-2.5.15/cyrus-imapd-2.5.15.tar.gz

https://github.com/cyrusimap/cyrus-imapd/releases/download/cyrus-imapd-2.5.15/cyrus-imapd-2.5.15.tar.gz.sig

https://www.cyrusimap.org/releases/cyrus-imapd-2.5.15.tar.gz
https://www.cyrusimap.org/releases/cyrus-imapd-2.5.15.tar.gz.sig

Please consult the release notes before upgrading to 2.5.15:

https://www.cyrusimap.org/imap/download/release-notes/2.5/x/2.5.15.html

And join us on Github at https://github.com/cyrusimap/cyrus-imapd to report 
issues, join in the deliberations of new features for the next Cyrus IMAP 
release, and to contribute to the documentation.

On behalf of the Cyrus team,

Kind regards,

ellie timoney

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Just-Enuf Cyrus IMAP for standalone Card-/CalDAV?

[ellie timoney]

I think you might run into problems with calendar notifications like that, 
because Cyrus will think it owns the accounts' email, and try to send 
notifications to its own mailboxes instead of over smtp to the real mail server?

On Mon, Dec 16, 2019, at 3:34 PM, Anatoli wrote:
> You can have a complete Cyrus install but only use the DAV features. You
> just don't open the IMAP ports in firewall and that's it.
> 
> On 16/12/19 01:12, PGNet Dev wrote:
> > I run my own smtp, imap & card/cal-dav services.
> > 
> > Atm, none are using Cyrus.
> > 
> > I'm well aware of Cyrus Imap's Cal/Card-dav support; particularly as the 
> > basis for FastMail's excellent services.
> > 
> > My current card/cal-dav is Radicale.  To my read, the project's crumbling.
> > 
> > I need to implement/deploy an alternative.
> > 
> > But, for **now**, ONLY for the *Dav pieces; I need to leave mail services 
> > as they are, at least 'til mid-2020.
> > 
> > For my domain, 'example.com' -- with smtp/imap @ host 'mx.example.com', I'd 
> > like to deploy a standalone Dav server elsewhere, @ host 'dav.example.com'.
> > 
> > I'd like it to be Cyrus based.
> > 
> > IIUC, Cyrus' *Dav are built on top of its IMAP install.
> > 
> > Can Cyrus be deployed in a standalone *Dav server mode like this?
> > 
> > Is there a doc/how-to/etc that address a Just-Enough-Cyrus-IMAP install  
> > for this kind of setup?
> > 
> > Thx!
> > 
> > Cyrus Home Page: http://www.cyrusimap.org/
> > List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> > To Unsubscribe:
> > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
> > 
> 
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
>

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Public calendars and addressbooks (was RE: Backup compaction optimization in a block-level replication environment)

[ellie timoney]

On Wed, Nov 20, 2019, at 4:41 PM, Deborah Pickett wrote:
> > I'm curious how these are working for you, or what sort of configuration
> > and workflows leads to having #calendars and #addressbooks as top-level
> > shared mailboxes?  I've only very recently started learning how our DAV bits
> > work (they have previously been black-boxes for me), and so far have only
> > seen these existing in user accounts.  Maybe this is a separate thread
> > though.
> 
> We used to use public calendars in Exchange (they call them Public Folders)
> for, among other things, a read-only catalogue of who in the office is on
> leave on any given day.  Some of our branch offices also had shared contact
> lists for phone numbers likely to be needed by all people at the local site
> (the local takeaway, the local hardware store, the local clinic...).
> Exchange public folders are almost an exact analogue to shared-namespace
> mailboxes in Cyrus.
> 
> Once I learned the undocumented magic for creating public calendars and
> address books in Cyrus (@karagian on Github posted it:
> https://github.com/cyrusimap/cyrus-imapd/issues/2373#issuecomment-415738943)
> it's worked very well.  My Outlook users use the free Caldav Synchronizer
> plugin (https://caldavsynchronizer.org/)  to sync selected address books and
> calendars to their clients.  I have a Perl script that queries our Active
> Directory server over LDAP to set ACLs on the folders.

That's interesting, thanks!

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Backup compaction optimization in a block-level replication environment

[ellie timoney]

On Wed, Nov 20, 2019, at 11:06 AM, Deborah Pickett wrote:
> On 2019-11-20 10:03, ellie timoney wrote:
>>> foo also includes "#calendars" and "#addressbooks" on my server so there 
are weird characters to deal with.
>>> 
>> Now that's an interesting detail to consider.

>> 
> I should restate my original message because I'm being fast and loose with 
> the meaning of "contains": two of the values for foo on my server are 
> "#calendars" and "#addressbooks". In other words, there are top-level public 
> mailboxes #calendars and #addressbooks which themselves contain sub-calendars 
> and sub-addressbooks. It never occurred to me to have calendar or contacts 
> folders deeper in the normal shared folder namespace, though it has evidently 
> occurred to you.


Oh, I see how I misread that! And... that also complicates things for me, I 
think (well, it's a possibility I hadn't even considered).

I'm curious how these are working for you, or what sort of configuration and 
workflows leads to having #calendars and #addressbooks as top-level shared 
mailboxes? I've only very recently started learning how our DAV bits work (they 
have previously been black-boxes for me), and so far have only seen these 
existing in user accounts. Maybe this is a separate thread though.

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Backup compaction optimization in a block-level replication environment

[ellie timoney]

On Tue, Nov 19, 2019, at 9:38 AM, Deborah Pickett wrote:
> > Food for thought.  Maybe instead of having one "%SHARED" backup, having one 
> > "%SHARED.foo" backup per top-level shared folder would be a better 
> > implementation?  I haven't seen shared folders used much in practice, so 
> > it's interesting to hear about it.
> >
> > Looking at your own data, if you had one "%SHARED.foo" backup per top level 
> > shared folder, would they be roughly user-sized pieces, or still too big?  
> > If too big, how deep would you need to go down the tree until the worst 
> > offenders are a manageable size?  (If I make it split shared folders like 
> > this, maybe "how-deep-to-split-shared-folders" needs to be a configuration 
> > parameter, because I guess it'll vary from installation to installation.)
> >
> For my data, %SHARED.foo would be the perfect granularity level. Each 
> foo is a shared email address like "sales" or "accounts" and it gets 
> about as much traffic as a user account does.  (Two months ago when we 
> were on Exchange, they _were_ user accounts.)

Ah yep, that makes sense!
 
> foo also includes "#calendars" and "#addressbooks" on my server so there 
> are weird characters to deal with.

Now that's an interesting detail to consider.  I think, with a hypothetical 
depth setting, I would treat any level that contains '#directories' as being 
"too deep" for splitting, regardless of the depth setting, because at that 
point we're looking at things that I guess we expect to belong together.  Like, 
if hypothetical_depth is 3, but foo.#calendars exists, then I think we'd want 
to treat the entirety of foo as a single backup (as if hypothetical_depth were 
1), regardless of what else is deeper in there.  I need to think about this 
more.

I'm gonna have a go at implementing this (I've opened 
https://github.com/cyrusimap/cyrus-imapd/issues/2915) but I'll step through it 
one level of complication at a time.

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Backup compaction optimization in a block-level replication environment

[ellie timoney]

> Related: I had to apply the patch described in
> (https://www.mail-archive.com/info-cyrus@lists.andrew.cmu.edu/msg47320.html),
>  "backupd IOERROR reading backup files larger than 2GB", because during
> initial population of my backup, chunks tended to by multiple GB in size
> (my %SHARED user backup is 23 GB, compressed).  Will this patch be
> merged to a main line?

Those were on master but I'm not sure why I didn't cherry-pick them back to 
3.0 Anyway, I've done that now, they'll be in the next release.

> Progress report: I started with very large chunks (minimum 64 MB,
> maximum 1024 MB) and a threshold of 8 chunks but I found that compaction
> was running every time, even on a backup file that hardly changed.  Not
> certain why this would be; my current theory is that in chunks that size
> there is almost always some benefit to compacting, so the threshold is
> passed easily.  There were 41 chunks in my %SHARED backup.

Hmm.  Yeah, the threshold is "number of chunks that would benefit from 
compaction", so the larger the chunks, the more likely any given chunk is to 
benefit from compaction, and the more likely you are to trip that threshold.

On Sat, Nov 16, 2019, at 12:10 PM, Deborah Pickett wrote:
> Further progress report: with small chunks, compaction takes about 15 
> times longer.  It's almost as if there is an O(n^2) complexity 
> somewhere, looking at the rate that the disk file grows.  (Running perf 
> on a compaction suggests that 90% of the time ctl_backups is doing 
> compression, decompression, or calculating SHA1 hashes.) So I'm going 
> back to large-ish chunks again.  Current values:
> 
> backup_compact_minsize: 1024
> backup_compact_maxsize: 65536
> backup_compact_work_threshold: 10
> 
> The compression ratio was hardly any different (less than 1%) with many 
> small chunks compared with huge chunks.

That's really interesting to hear.  It sounds like maybe the startup and 
cleanup of a gzip stream are more expensive than the compression/decompression 
parts, so it's cheaper to aim for fewer larger chunks than many smaller ones.  

zlib provides a range of 0-9 (default: 6) for whether to prioritise speed (0) 
or size (9) in its compression algorithm, but the backup system isn't using it 
in a way that exposes this as a tunable option (it's just letting it use the 
default by default).  With enough data it might be interesting to make it 
tunable and see what impact it has, but I don't think we're at a stage of 
needing to care this much yet.

> Setting the work threshold to a number greater than 1 is only helping a 
> bit.  I think that the huge disparity between my smaller and larger user 
> backups is hurting me here.  Whatever I set the threshold to, it is 
> going to be simultaneously too large for most users, and too small for 
> the huge %SHARED user.

Food for thought.  Maybe instead of having one "%SHARED" backup, having one 
"%SHARED.foo" backup per top-level shared folder would be a better 
implementation?  I haven't seen shared folders used much in practice, so it's 
interesting to hear about it.

Looking at your own data, if you had one "%SHARED.foo" backup per top level 
shared folder, would they be roughly user-sized pieces, or still too big?  If 
too big, how deep would you need to go down the tree until the worst offenders 
are a manageable size?  (If I make it split shared folders like this, maybe 
"how-deep-to-split-shared-folders" needs to be a configuration parameter, 
because I guess it'll vary from installation to installation.)

> Confession time: having inspected the source of ctl_backups, I admit to 
> misunderstanding what happens to chunks when compaction is triggered.  I 
> thought that each chunk was examined, and either the chunk is compacted, 
> or it is not (and the bytes in the chunk are copied from old to new 
> unchanged).  But compaction happens to the entire file: every chunk in 
> turn is inflated to /tmp and then deflated again from /tmp, minus any 
> messages that may have expired, so the likelihood of the compressed byte 
> stream being the same is slim.  That will confound the rsync rolling 
> checksum algorithm and the entire backup file will likely have to be 
> transmitted again.

Yeah, these files are append-only even within the backup system's own tooling.  
Compacting a backup file to be smaller is literally re-streaming it to a new 
file, minus bits that aren't needed anymore, and then (if all goes well) 
renaming it back over the original.  It's meant to be atomic -- either it 
works, and you get the updated file, or something goes wrong, and the file is 
unchanged.  It's never modified in place!  (There's a note about this somewhere 
in the documentation, with regard to needing enough free disk space to write 
the second file in order to compact the first.)
 
> With that in mind I've decided that I'll make compaction a weekend-only 
> task, take it out of cyrus.conf EVENTS and put a weekly cron/systemd job 
> in place.  

Re: Cyrus backups and deleted (users, mailboxes)

[ellie timoney]

On Fri, Nov 15, 2019, at 5:12 PM, Deborah Pickett wrote:
> Semi-related questions about how Cyrus backup servers deal with deleted 
> stuff.
> 
> I have the following settings on the main mail server:
> 
> deletedprefix: DELETED
> delete_mode: delayed
> expunge_mode: delayed
> 
> My first question is about deleted folders.  Scenario: I delete a folder 
> on the server, and it is renamed into the DELETED namespace as 
> expected.  I cannot back these folders up to the backup server.  They 
> are not enumerated during "XBACKUP * channel", and if I name the DELETED 
> namespace explicitly I get the following output:
> 
> x xbackup DELETED/* rsync
> * NO MAILBOX polyfoam.com.au!DELETED.rawmaterials.Trash.Downer 
> EDI.5DAD45A0 (Mailbox is locked)
> * NO MAILBOX polyfoam.com.au!DELETED.support.Has Spaces.5DCA22F4 
> (Mailbox is locked)
> x NO Mailbox is locked
> 
> (Which is not nearly the whole list either: it should have listed 
> hundreds of deleted folders.)
> 
> Is this expected behaviour?  I suppose that my backup has a copy of this 
> folder in its original location before it was deleted, so no one will 
> miss these folders, (though I can imagine scenarios where a message 
> arrives and is quickly put into a folder which is then deleted, and this 
> message will never touch the backup server).

Hmm.  With rolling replication to the backup server, these will be replicated 
as RENAMEs from "original" to "DELETED.original.hextimestamp", while remaining 
in the original user's backup.  After "cyr_expire -D" deletes these folders 
from the mail server, then they will replicate as UNMAILBOXes, and the backup 
server will mark them in the index as having been deleted at the time the 
UNMAILBOX is replicated.  Eventually, after a further backup_duration_days has 
passed, the messages in them will become eligible for compacting out of the 
backup (providing they're not also in some other folder where they are still 
valid), which will take effect according to the usual compact rules when 
ctl_backups compact runs.  If it was a quick arrive-file-delete, it would 
eventually replicate as a new mailbox (already called DELETED.*) with all the 
messages, so nothing would be missed.

But without rolling replication, or with mailboxes that were already DELETED.* 
before the backup service was configured, I'm not sure what to expect.  I'm not 
sure if this is a case that I considered and did something for, but have since 
forgotten about, or if it's a case that I hadn't considered yet.

What happens if you xbackup the user(s) they belong(ed) to, rather than the 
DELETED.* mailboxes directly?  Does sync_client correctly detect them as 
renames, and replicate accordingly?  If not, what if you sync_client -u them 
manually?  If that doesn't work, what about sync_client -m with the DELETED.* 
names?

If none of these work, I probably need to implement something...

> My second question is about deleted users.  Scenario: A user "alice" 
> departs the company, and I delete their root mailbox on the main 
> server.  It is no longer enumerated during "XBACKUP * channel".
> 
> What will happen to the backup file a/alice_XX on the backup 
> server?  Will it stay there forever?  Do I need to manually delete it?  
> Do I need to manually remove alice from /var/lib/cyrus/backups.db?  I 
> suppose I can use "ctl_backups list -t" to detect such files.

With rolling replication, the deleted user would eventually replicate as an 
"UNUSER", to which backupd would report success and ignore...

Basically, the implementation hasn't gotten as far as removing backups -- and I 
want to see things working correctly and safely before I even think about 
starting to implement it.  In the meantime, "ctl_backups list -t" is a pretty 
good way of identifying candidates to (thoroughly check and then) delete 
manually.  You're right, you'd also want to remove their entry from backups.db, 
which you can do safely using "cyr_dbtool delete".

>  Any other  things I should be aware of?

I guess at this point you're deeper into the woods on this than anyone else has 
been (which is why it's called "experimental"!).  For me, it's great that 
you're trying stuff and shaking out the issues, but you're probably gonna find 
more, not less, as we go along

Cheers,

ellie

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Cyrus IMAP 3.0.12 released

[ellie timoney]

The Cyrus team is proud to announce the immediate availability of a new version 
of Cyrus IMAP: 3.0.12

I'm trialling hosting the release files using Github's releases feature.  
Please use the Github download links if possible, and advise if you have any 
problems!  (It may even download faster due to Github's content delivery 
network.)

This release contains a fix for CVE-2019-18928, a session hijacking 
vulnerability in the httpd daemon.  If you compile cyrus with HTTP support 
enabled, your cyrus.conf contains SERVICES entries that run the httpd daemon, 
and you provide a proxy frontend service that reuses connections to the 
backend, you will need this upgrade.

Download URLs:


https://github.com/cyrusimap/cyrus-imapd/releases/download/cyrus-imapd-3.0.12/cyrus-imapd-3.0.12.tar.gz

https://github.com/cyrusimap/cyrus-imapd/releases/download/cyrus-imapd-3.0.12/cyrus-imapd-3.0.12.tar.gz.sig

https://www.cyrusimap.org/releases/cyrus-imapd-3.0.12.tar.gz
https://www.cyrusimap.org/releases/cyrus-imapd-3.0.12.tar.gz.sig

Please consult the release notes and upgrade documentation before upgrading to 
3.0.12:

https://www.cyrusimap.org/imap/download/release-notes/3.0/x/3.0.12.html
https://www.cyrusimap.org/imap/download/upgrade.html

And join us on Github at https://github.com/cyrusimap/cyrus-imapd to report 
issues, join in the deliberations of new features for the next Cyrus IMAP 
release, and to contribute to the documentation.

On behalf of the Cyrus team,

Kind regards,

ellie timoney

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Cyrus IMAP 2.5.14 released

[ellie timoney]

The Cyrus team is proud to announce the immediate availability of a new version 
of Cyrus IMAP: 2.5.14

This release contains a fix for CVE-2019-18928, a session hijacking 
vulnerability in the httpd daemon.  If you compile cyrus with HTTP support 
enabled, your cyrus.conf contains SERVICES entries that run the httpd daemon, 
and you provide a proxy frontend service that reuses connections to the 
backend, you will need this upgrade.

I'm trialling hosting the release files using Github's releases feature.  
Please use the Github download links if possible, and advise if you have any 
problems!  (It may even download faster due to Github's content delivery 
network.)

Download URLs:


https://github.com/cyrusimap/cyrus-imapd/releases/download/cyrus-imapd-2.5.14/cyrus-imapd-2.5.14.tar.gz

https://github.com/cyrusimap/cyrus-imapd/releases/download/cyrus-imapd-2.5.14/cyrus-imapd-2.5.14.tar.gz.sig

https://www.cyrusimap.org/releases/cyrus-imapd-2.5.14.tar.gz
https://www.cyrusimap.org/releases/cyrus-imapd-2.5.14.tar.gz.sig

Please consult the release notes before upgrading to 2.5.14:

https://www.cyrusimap.org/imap/download/release-notes/2.5/x/2.5.14.html

And join us on Github at https://github.com/cyrusimap/cyrus-imapd to report 
issues, join in the deliberations of new features for the next Cyrus IMAP 
release, and to contribute to the documentation.

On behalf of the Cyrus team,

Kind regards,

ellie timoney

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: [Help] Cyrus 2.4.17 segfault

[ellie timoney]

On Tue, Nov 12, 2019, at 2:51 AM, Marco wrote:
> An user user/a has full ACL to another mailbox user/b. When the user/a 
> SELECT a folder on user/b where he has access the imap process crashes.

If you set up a couple of test accounts with the same sharing arrangement, do 
those crash in the same way?  Or is it specific to the two original accounts?

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Backup compaction optimization in a block-level replication environment

[ellie timoney]

On Fri, Nov 8, 2019, at 1:35 PM, Deborah Pickett wrote:
> I didn't know if copying 
> the filesystem of a (paused) Cyrus replica was a supported way of 
> backing up, but now I do.

Yeah, as long as there are no cyrus processes running, the database/index files 
can just be copied about and won't be corrupted along the way.  This kind of 
backup is useful for a full system restore, but the "help I deleted an 
important email and then emptied my trash and then expunged, and now I need it 
back" type of recoveries... I guess you copy the message file back to the 
partition, reconstruct, and it comes back as a new message (unread, no flags, 
etc)?

You would need to be careful of the window between delivery of a message, 
replication to the replica, and deletion of the message (and replication of the 
deletion), to ensure you get a backup of the state where the message existed.  
I *think* delayed_delete and a long cyr_expire -D time takes care of this, but 
I'm not certain, so please test it before you rely on it.  Also (maybe 
obviously) the need to keep point in time snapshots for as long as your 
recovery policy dictates, and not just delete stuff from backup as soon as its 
deleted from source.

I'm not sure what others are doing in this space really.  There's a few threads 
on the list archive about various backup strategies, but my focus has mainly 
been the backupd-based system.

>  Is there a list of which database and index 
> files I need to copy apart from the files inside the partition structure?

This kind of covers it, I think:
https://www.cyrusimap.org/imap/reference/admin/locations/configuration-state.html

It would be quite useful to have a "this is what you need to back up" document, 
but at the moment there's a certain amount of reading between the lines of 
adjacent documentation :(

> > This setting might be helpful:
> 
> Thanks, I saw that setting but didn't really think through how it would 
> help me.  I'll experiment with it and report back.

That would be great, thanks!

Cheers,

ellie

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Backup compaction optimization in a block-level replication environment

[ellie timoney]

I'm not sure if I'm just not understanding, but if the chunk offsets were to 
remain the same, then there's no benefit to compaction? A (say) 2gb file full 
of zeroes between small chunks is still the same 2gb on disk as one that's 
never been compacted at all!

And if you don't use the compaction feature, you might as well skip the backups 
system entirely, and have your backup server just be a normal replica that 
doesn't accept client traffic (maybe with a very long cyr_expire -D time?), and 
then you shut it down on schedule for safe block/file system backups to your 
offsite location.

> Right now I've set backup_compact_minsize and backup_compact_maxsize to 
> zero but I'm not sure if even that is sufficient to prevent chunk 
> offsets moving.  Perhaps I need to disable the compaction event in 
> cyrus.conf entirely.

I don't have this system entirely in my head at the moment so I'm kinda just 
reading documentation here, but these settings are about optimising the gzip 
algorithm.  Each chunk is compressed separately, and the tradeoff here is that 
bigger chunks compress better, but if the file becomes corrupted somehow you 
lose entire chunks, so smaller chunks are safer.

> A compromise would need to be 
> struck between keeping chunk offsets fixed and wasted fragmented space 
> between chunks as they shrink.

This setting might be helpful:

>   backup_compact_work_threshold: 1
>   The  number of chunks that must obviously need compaction 
> before the com‐
>   pact tool will go ahead with the compaction.  If set to  less  
> than  one,
>   the value is treated as being one.

If you set your backup_compact_min/max_sizes to a size that's 
comfortable/practical for your block backup algorithm, but then set a very lax 
backup_compact_work_threshold, you might be able to find a sweet spot where 
you're getting the benefits of compaction eventually, but are not constantly 
changing every block in the file (until you do).  The default (1) is basically 
for compaction to occur as soon as there's something to compact out, just 
because the default had to be something, and without experiential data any 
other value would just be a hat rabbit.  But this sounds like a case where a 
big number would play nicer.

I guess I'd try to target a minimum size of 1 disk block per chunk, and a 
maximum of (fair dice roll) 4 disk blocks? But you'd need some experimentation 
to figure out ballpark numbers, and won't be able to tune it to exact block 
sizes, because the configured thresholds are the uncompressed data size, not 
the compressed chunk size on disk.

On Wed, Nov 6, 2019, at 8:20 PM, Deborah Pickett wrote:
> (Sorry, that's a lot of big words.  I'll try explaining what I want to do.)
> 
> On my LAN I have a Cyrus IMAP server (3.0.11), and a dedicated Cyrus 
> backup server (patched with Ellie's shared-mailbox and 64-bit fseek 
> fixes).  These are connected by a nice fat link so backups happen fast 
> and often.  A scheduled compaction occurs each morning thanks to an 
> event in cyrus.conf.
> 
> I now want to back up the backups to an off-site server over a much 
> slower link.  The off-site server doesn't speak the Cyrus sync 
> protocol.  What it does do well is block-level backups: if only a part 
> of a file has changed, only that part needs to be transferred over the 
> slow link.  [I haven't decided whether my technology will be the rsync 
> --checksum protocol, or Synology NAS XFS replication, or Microsoft 
> Server VFS snapshots.  They all do block-level backups well.]
> 
> Since Cyrus backup files are append-only, they should behave well with 
> block-level backups. But—correct me if I'm wrong—compaction is going to 
> ruin my day because a reduction in the size of chunk (say) 5 moves the 
> start offset of chunk 6 (and so on).  Even if chunk 6 doesn't change 
> it'll have to be retransmitted in its entirety.
> 
> Right now I've set backup_compact_minsize and backup_compact_maxsize to 
> zero but I'm not sure if even that is sufficient to prevent chunk 
> offsets moving.  Perhaps I need to disable the compaction event in 
> cyrus.conf entirely.
> 
> I really want compaction, though, or else my backups are going to get 
> very, very big.
> 
> Which leads me to my idea.  What if compaction could be friendlier 
> towards block-level backups, by deliberately avoiding changing chunk 
> offsets in the backup file, even if that means gaps of unused bytes when 
> (the aforementioned) chunk 5 shrinks?  It won't always work out, for 
> instance when a chunk grows in size. A compromise would need to be 
> struck between keeping chunk offsets fixed and wasted fragmented space 
> between chunks as they shrink.
> 
> I haven't collected enough data to know if I am making the right 
> assumptions about how chunk size evolves over time and how effective 
> compaction is at removing cruft from a backup file.  Has anyone thought 
> about doing something like this with Cyrus 

Re: XBACKUP and backupd not backing up public folders (3.0.8)

[ellie timoney]

On Mon, Oct 14, 2019, at 10:29 AM, ellie timoney wrote:
> I've created a github issue 
> (https://github.com/cyrusimap/cyrus-imapd/issues/2893), and am about to make 
> a test case to reproduce the problem, so I can get on with fixing it. :)

I think this is fixed now, on both master and 3.0 branches. Testing and 
feedback appreciated, of course!

Cheers,

ellie
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: cyradm and TLS 1.2

[ellie timoney]

Thanks for reporting back. For whatever its worth, the equivalent fix on 2.5+ 
uses "TLS_client_method()", not "TLSv1_2_client_method()". I'm not sure what 
difference it makes, but maybe it requires a newer OpenSSL than you have?

Here's the commit to master, fyi: 
https://github.com/cyrusimap/cyrus-imapd/commit/78f79ea53238c8596e2f8602b7b1e29a16863ae9

On Tue, Oct 15, 2019, at 7:43 AM, John Widera wrote:
> Turns out imclient (at least in the latest RHEL7 pkg) is hardcoded to use 
> TLSv1. Since we're building binary RPMs from Source RPMs anyway we modified 
> imclient.c, rebuilt the RPMs, reinstalled the cyrus-imapd-utils package: 
> Here's the patch we used:

> **


> *--- imclient.c.orig 2012-12-01 13:57:54.0 -0600*
> *+++ imclient.c 2019-10-03 14:40:11.254566297 -0500*
> *@@ -1695,7 +1695,7 @@*
> *return -1;*
> *}*


> *- imclient->tls_ctx = SSL_CTX_new(TLSv1_client_method());*
> *+ imclient->tls_ctx = SSL_CTX_new(TLSv1_2_client_method());*
> *if (imclient->tls_ctx == NULL) {*
> *return -1;*
> *};*

> ---

> Maybe this helps someone else.

> Regards,


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: XBACKUP and backupd not backing up public folders (3.0.8)

[ellie timoney]

Hi Deborah,

Thanks, that's all useful! Looks like in both places it's struggling with lack 
of a userid, which makes some sense because it's a shared mailbox, and shared 
mailboxes don't have userids.

I guess this means that in its current state, the backup system can't handle 
shared mailboxes. :( Which is weird, because the docs say it can, and I 
wouldn't have written that if I didn't think it worked, but maybe I didn't 
understand shared mailboxes as well as I thought way back then.

I've created a github issue 
(https://github.com/cyrusimap/cyrus-imapd/issues/2893), and am about to make a 
test case to reproduce the problem, so I can get on with fixing it. :)

Cheers,

ellie

On Fri, Oct 11, 2019, at 6:02 PM, Deborah Pickett wrote:
> Hi Ellie,

> Thanks for helping me look at this.

> On 2019-10-09 16:17, ellie timoney wrote:
>> Does the same problem occur if you use sync_client (on the master server, as 
>> the cyrus user) to replicate the shared mailbox to the backup server (rather 
>> than using XBACKUP over IMAP)?  Something like "sync_client -n rsync  -m 
>> supp...@polyfoam.com.au" I think? 
>> 
> With -m I get this familiar output on the master:


> # /usr/lib/cyrus/bin/sync_client -v -n rsync -m supp...@polyfoam.com.au
>  MAILBOXES polyfoam.com.au!support
>  Error from sync_do_mailboxes(): bailing out!

> and this is seen in the log on the backup server:


> Oct 11 17:39:58 rsync cyrus/backupd[3969]: login: mail-3175-1.polyfoam.com.au 
> [10.3.244.125] rsync-mail-3175-1 DIGEST-MD5 User logged in
>  Oct 11 17:39:58 rsync cyrus/backupd[3969]: created decompress buffer of 4102 
> bytes
>  Oct 11 17:39:58 rsync cyrus/backupd[3969]: created compress buffer of 4073 
> bytes
>  Oct 11 17:39:58 rsync cyrus/backupd[3969]: decompressed 47 -> 41 bytes
>  Oct 11 17:39:58 rsync cyrus/master[458]: process type:SERVICE name:backupd 
> path:/usr/lib/cyrus/bin/backupd age:201.603s pid:3969 signaled to death by 
> signal 11 (Segmentation fault, core dumped)
>  Oct 11 17:39:58 rsync cyrus/master[458]: service backupd/ipv4 pid 3969 in 
> BUSY state: terminated abnormally
>  Oct 11 17:39:58 rsync cyrus/master[458]: service backupd/ipv4 now has 0 
> ready workers

> There, a core dump. Here is what I get from a backtrace:


> # coredumpctl gdb -1 
>  PID: 3969 (backupd)
>  UID: 103 (cyrus)
>  GID: 8 (mail)
>  Signal: 11 (SEGV)
>  Timestamp: Fri 2019-10-11 17:39:58 AEDT (2min 16s ago)
>  Command Line: /usr/lib/cyrus/bin/backupd
>  Executable: /usr/lib/cyrus/bin/backupd
>  Control Group: /system.slice/cyrus-imapd.service
>  Unit: cyrus-imapd.service
>  Slice: system.slice
>  Boot ID: c887b7eb1d734962b8bddb745df21e8f
>  Machine ID: facebc4e2dcd47a68a097acc9077814e
>  Hostname: rsync
>  Storage: 
> /var/lib/systemd/coredump/core.backupd.103.c887b7eb1d734962b8bddb745df21e8f.3969.157077599800.lz4
>  Message: Process 3969 (backupd) of user 103 dumped core.
> 
>  Stack trace of thread 3969:
>  #0 0x7f95c17e3206 __GI___strlen_sse2 (libc.so.6)
>  #1 0x7f95c2080119 xstrdup (libcyrus_min.so.0)
>  #2 0x5557e89fe440 is_mailboxes_single_user (backupd)
>  #3 0x5557e89eec88 main (backupd)
>  #4 0x7f95c176f09b __libc_start_main (libc.so.6)
>  #5 0x5557e89ef34a _start (backupd)
> 
>  GNU gdb (Debian 8.2.1-2+b1) 8.2.1
>  Copyright (C) 2018 Free Software Foundation, Inc.
>  License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
>  This is free software: you are free to change and redistribute it.
>  There is NO WARRANTY, to the extent permitted by law.
>  Type "show copying" and "show warranty" for details.
>  This GDB was configured as "x86_64-linux-gnu".
>  Type "show configuration" for configuration details.
>  For bug reporting instructions, please see:
> <http://www.gnu.org/software/gdb/bugs/>.
>  Find the GDB manual and other documentation resources online at:
> <http://www.gnu.org/software/gdb/documentation/>.
> 
>  For help, type "help".
>  Type "apropos word" to search for commands related to "word"...
>  Reading symbols from /usr/lib/cyrus/bin/backupd...Reading symbols from 
> /usr/lib/debug/.build-id/e3/b2619440ce57c6ae7db282266976a826059cf2.debug...done.
>  done.
>  [New LWP 3969]
>  [Thread debugging using libthread_db enabled]
>  Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
>  Core was generated by `/usr/lib/cyrus/bin/backupd'.
>  Program terminated with signal SIGSEGV, Segmentation fault.
>  #0 __strlen_sse2 () at ../sysdeps/x86_64/multiarch/../strlen.S:120
>  120 ../sysdeps/x86_64/multiarch/../strlen.S: No such file or directory.
>  (gdb) bt
>  #0 __strlen_sse2 () at ../sysd

Re: XBACKUP and backupd not backing up public folders (3.0.8)

[ellie timoney]

Hi Deborah,

Does the same problem occur if you use sync_client (on the master server, as 
the cyrus user) to replicate the shared mailbox to the backup server (rather 
than using XBACKUP over IMAP)?  Something like "sync_client -n rsync  -m 
supp...@polyfoam.com.au" I think?  What about if you use "sync_client -n rsync 
-u supp...@polyfoam.com.au" instead (i.e. with -u treating the shared mailbox 
as a USER rather than as a -m MAILBOX)?

On the backup server, what does the "ctl_backups verify -vvv -m 
polyfoam.com.au!support" command say about the shared mailbox?  There might be 
personally-identifying information in this output, I can't remember -- please 
check/censor carefully before pasting into email.

> I'm seeing what looks like a segfault in the backup server logs.  Don't
> know if this is significant.

It's almost certainly significant.  Are you able to enable core dumps and get a 
backtrace from it?  That'll probably be the fastest path toward a solution.  
Let me know if you need help with this :)

Cheers,

ellie

On Wed, Oct 9, 2019, at 1:46 PM, Deborah Pickett wrote:
> Hi everyone,
> 
> I'm deploying Cyrus 3.0.8 (Debian buster 3.0.8-6) at $dayjob to replace
> an Exchange server.  That part is going well, but I'm hitting a hurdle
> pulling backups of public folders (shared mailboxes, calendars and
> address books, anything outside the user/ hierarchy) using XBACKUP and
> backupd.
> 
> Steps to reproduce:
> 
> 1. On master server (mail-3175-1), run imtest and authenticate as admin.
> 2. Issue XBACKUP to backup normal user.  This succeeds.
> 3. Issue XBACKUP to backup public shared mailbox.  This produces error
> BAD PROTOCOL.
> 
> Expected behaviour is that the backup server backs up this mailbox with
> an OK response.
> 
> I'm seeing what looks like a segfault in the backup server logs.  Don't
> know if this is significant.
> 
> Help?
> 
> Main server (mail-3175-1): Debian buster, cyrus 3.0.8-6
> Backup server (rsync): Debian buster, cyrus 3.0.8-6
> 
> --- imtest session ---
> 
> mail-3175-1$ /usr/lib/cyrus/bin/imtest -a cyrus
> WARNING: no hostname supplied, assuming localhost
> 
> S: * OK [CAPABILITY IMAP4rev1 LITERAL+ ID ENABLE STARTTLS AUTH=PLAIN
> SASL-IR] mail-3175-1 Cyrus IMAP 3.0.8-Debian-3.0.8-6 server ready
> Please enter your password:
> C: A01 AUTHENTICATE PLAIN ***DELETED***
> S: A01 OK [CAPABILITY IMAP4rev1 LITERAL+ ID ENABLE ACL RIGHTS=kxten
> QUOTA MAILBOX-REFERRALS NAMESPACE UIDPLUS NO_ATOMIC_RENAME UNSELECT
> CHILDREN MULTIAPPEND BINARY CATENATE CONDSTORE ESEARCH SEARCH=FUZZY SORT
> SORT=MODSEQ SORT=DISPLAY SORT=UID THREAD=ORDEREDSUBJECT
> THREAD=REFERENCES THREAD=REFS ANNOTATEMORE ANNOTATE-EXPERIMENT-1
> METADATA LIST-EXTENDED LIST-STATUS LIST-MYRIGHTS LIST-METADATA WITHIN
> QRESYNC SCAN XLIST XMOVE MOVE SPECIAL-USE CREATE-SPECIAL-USE DIGEST=SHA1
> X-REPLICATION URLAUTH URLAUTH=BINARY LOGINDISABLED COMPRESS=DEFLATE
> X-QUOTA=STORAGE X-QUOTA=MESSAGE X-QUOTA=X-ANNOTATION-STORAGE
> X-QUOTA=X-NUM-FOLDERS IDLE] Success (no protection)
> SESSIONID=
> Authenticated.
> Security strength factor: 0
> AAA XBACKUP user/debb...@polyfoam.com.au rsync
> * OK USER debb...@polyfoam.com.au
> AAA OK Completed
> BBB XBACKUP supp...@polyfoam.com.au rsync
> * NO MAILBOX polyfoam.com.au!support (Bad protocol)
> BBB NO Bad protocol
> 
> --- log on master ---
> 
> Oct  9 13:31:37 mail-3175-1 cyrus/imap[189353]: login: localhost [::1]
> cyrus PLAIN User logged in
> SESSIONID=
> Oct  9 13:31:55 mail-3175-1 cyrus/imap[189353]: XBACKUP: connecting to
> server 'rsync.polyfoam.com.au' for channel 'rsync'
> Oct  9 13:32:01 mail-3175-1 cyrus/imap[189353]: XBACKUP: replicating
> user debb...@polyfoam.com.au
> Oct  9 13:32:15 mail-3175-1 cyrus/imap[189353]: XBACKUP: connecting to
> server 'rsync.polyfoam.com.au' for channel 'rsync'
> Oct  9 13:32:21 mail-3175-1 cyrus/imap[189353]: XBACKUP: replicating
> mailbox polyfoam.com.au!support
> Oct  9 13:32:21 mail-3175-1 cyrus/imap[189353]: IOERROR: zero length
> response to MAILBOXES (end of file reached)
> Oct  9 13:32:21 mail-3175-1 cyrus/imap[189353]: IOERROR: zero length
> response to RESTART (end of file reached)
> Oct  9 13:32:22 mail-3175-1 cyrus/imap[189353]: USAGE cyrus user:
> 0.059546 sys: 0.024519
> 
> --- log on backup server ---
> 
> ***successful backup of user debbiep not shown***
> Oct  9 13:32:15 rsync cyrus/backupd[21340]: telling master 2
> Oct  9 13:32:15 rsync cyrus/backupd[21340]: accepted connection
> Oct  9 13:32:15 rsync cyrus/backupd[21340]: telling master 3
> Oct  9 13:32:15 rsync cyrus/master[16697]: service backupd/ipv4 pid
> 21340 in READY state: now unavailable and in BUSY state
> Oct  9 13:32:15 rsync cyrus/master[16697]: service backupd/ipv4 now has
> 0 ready workers
> Oct  9 13:32:15 rsync cyrus/master[16697]: service backupd/ipv4 pid
> 21340 in BUSY state: now serving connection
> Oct  9 13:32:15 rsync cyrus/master[16697]: service backupd/ipv4 now has
> 0 ready workers
> Oct  9 13:32:21 rsync cyrus/backupd[21340]: login:
> 

Re: Possible issue when upgrading to cyrus 3.0.8 using replication ?

[ellie timoney]

Hi Adrien,

The replication upgrade path should be okay. In-place upgrades (that would use 
the affected reconstruct to bring mailboxes up to the same version as the 
server) would get bitten. Whereas if you replicate to a newer version server, 
the mailboxes on the replica will be created at the replica's preferred version 
already, so you don't need to reconstruct afterwards.

If you have messages that would theoretically be affected by this bug in 3.0, 
you won't be able to replicate them to 3.0 in the first place, because I think 
replication won't allow the 0 modseq. If this arises, I'm not sure how to 
recover from it and replicate the affected messages, since 2.4 and 2.5 won't 
alter the 0 modseq. If it can't replicate them, it will complain about it, so 
if you plan for the replication needing some handholding/restarting, you'll at 
least be able to identify which messages are broken in the process, and then 
figure out how to handle it once you know the size of the problem?

Another option, if you want to stick with the Debian packages, would be to skip 
3.0.8 and install 3.0.11 from buster-backports 
(https://packages.debian.org/buster-backports/cyrus-imapd), and then you'll be 
immune to the problem. Though you still won't be able to replicate the affected 
messages to the new server, hmm.

Cheers,

ellie

On Thu, Sep 12, 2019, at 6:50 AM, Adrien Remillieux wrote:
> Hello,
> 
> I have a server that I can't update running cyrus 2.5.10 which contain 
> mailboxes that have existed from 2.3 and earlier (around 300Gb total). My 
> plan is to update by enabling replication with a new server running Debian 
> Buster (so cyrus 3.0.8) and then shutting down the old server. There was a 
> problem when upgrading to 3.x.x with mailboxes created with cyrus 2.3 or 
> before and that was fixed in 3.0.11 (see 
> https://www.cyrusimap.org/imap/download/release-notes/3.0/x/3.0.11.html and 
> https://github.com/cyrusimap/cyrus-imapd/issues/2839 for the bug report)
> 
> Does this upgrade path suffer from the same issue ? I am not familiar with 
> the inner-workings of cyrus. It appears that the Debian maintainers have not 
> backported the patch in 3.0.8 (see 
> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=933163 and I looked at the 
> source code)
> 
> Cheers,
> Adrien
> 
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: RLIST leads to Referral after SELECT

[ellie timoney]

On Mon, Sep 9, 2019, at 4:10 AM, Eduardo Chappa wrote: 
> The use of RLIST is mandated by RFC 2193, which states:
> 
>  A MAILBOX-REFERRALS capable client will issue the RLIST and RLSUB
>  commands in lieu of LIST and LSUB.
> 
> Does anyone see a reason why this server is returning a NO [REFERRAL ...] 
> to a SELECT command after a RLIST command, but not without it?

If "a MAILBOX-REFERRALS capable client will issue RLIST and RLSUB", then it's 
reasonable to suppose that:

* a client that issues RLIST or RLSUB supports mailbox referrals (and so the 
server sends a referral)
* a client that issues a regular LIST does not support mailbox referrals (and 
so the server proxies the remote mailbox instead)

In terms of where this happens in code, looks like it gets set up here (and 
similarly for Rlsub a bit further down):
https://github.com/cyrusimap/cyrus-imapd/blob/master/imap/imapd.c#L1873

and then cmd_list saves the "supports_referrals" flag out for other commands to 
examine, if LIST_SEL_REMOTE was requested and configuration permits it, here:
https://github.com/cyrusimap/cyrus-imapd/blob/master/imap/imapd.c#L8112-L8116

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Upgrade report debian cyrus

[ellie timoney]

On Thu, Aug 1, 2019 at 5:13 PM Lars Schimmer  wrote:
> and on second run it did find a lot of mails with "reappending" messages.

Actually, this sounds like this bug: 
https://github.com/cyrusimap/cyrus-imapd/issues/2839 (fixed last week in 3.0.11)
There is a Debian report for it too: 
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=933163

I guess we're going to see a lot of this here, as people start upgrading older 
systems to the new Debian stable and getting Debian's 3.0.8 instead of whatever 
they had before.


On Fri, Aug 2, 2019, at 12:44 AM, Savvas Karagiannidis wrote:
> there was an issue when upgrading to 3.0.x from older versions (2.x). The 
> issue is this:
> https://github.com/cyrusimap/cyrus-imapd/issues/2208
> 
> Ellie explains the exact issues with old mail in the last comment before 
> closing the issue mentioned above and I'm surprised to not see any mention of 
> this (the -G parameter) in the 3.0 release notes and upgrade instructions...

Oh, I'd forgotten about that! I'm going to add a note about this to the 
reconstruct -G documentation as well as the upgrade notes.

It is a different issue though: the impact of #2208 (since the crash was fixed 
in 3.0.8, anyway) is being unable to replicate due to missing GUID's. You need 
reconstruct with -G before upgrading if you plan to build a new server and 
replicate your old store to it (which you will find out when you try to 
replicate). But it won't affect in-place upgrades until you try to replicate 
them somewhere later.

#2839 is rather more annoying because it affects in-place upgrades and 
copy-files-and-reconstruct upgrades!

Cheers,

ellie

On Fri, Aug 2, 2019, at 12:44 AM, Savvas Karagiannidis wrote:
> Hi Lars,
> there was an issue when upgrading to 3.0.x from older versions (2.x). The 
> issue is this: 
> https://github.com/cyrusimap/cyrus-imapd/issues/2208
> 
> From my experience, the only way to safely upgrade to 3.0 was to run 
> reconstruct -G -V max on all mailboxes while still in the older version.
> Ellie explains the exact issues with old mail in the last comment before 
> closing the issue mentioned above and I'm surprised to not see any mention of 
> this (the -G parameter) in the 3.0 release notes and upgrade instructions...
> You can still run reconstruct -G -V max on the new system with the latest 
> version but the result will probably be that these older messages will appear 
> as unread.
> 
> 
> Regards,
> Savvas Karagiannidis
> 
> On Thu, Aug 1, 2019 at 5:13 PM Lars Schimmer  wrote:
>> Hi!
>> 
>>  Just upgraded my old debian cyrus 2.4.16-4+deb7u2 to cyrus 3.0.8-6.
>>  I did use a new hardware system for this, copied over all mailbox,
>>  sieve,.. data and did a recontruct.
>> 
>>  It seems the reconstruct needs a few uns to reconstruct our 100GB mail
>>  archive (in ~120 mailboxes).
>> 
>>  I did a reconctruct -V max and it did not find all mails, I did run it
>>  twice, and on second run it did find a lot of mails with "reappending"
>>  messages.
>> 
>>  Also, after migration, some users did not seem to have a inbox, I needed
>>  to "cm user.xcy" and copy mailbox data into it and recontruct manually.
>>  (seems like those are new mailboxes and are not in the copied data of
>>  main cyrus db).
>> 
>>  But some mailboxes were existent, but empty after the 2 reconstruct (and
>>  quota) run, I did explicit run reconstruct -r -f user.xyyya on those.
>>  To bad they got all mails as unseen.
>> 
>>  On the most accounts all worked fine, but all mails prior to sept 2011
>>  are set to unseen.
>>  Seems like some change of dataformat, as I did upgrade in sept 2011 :-)
>> 
>>  Now I sit here and wait for more error reports from the users.
>> 
>> 
>> 
>>  MfG,
>>  Lars Schimmer
>>  -- 
>>  -
>>  TU Graz, Institut für ComputerGraphik & WissensVisualisierung
>>  Tel: +43 316 873-5405 E-Mail: l.schim...@cgv.tugraz.at
>>  Fax: +43 316 873-5402 PGP-Key-ID: 0x4A9B1723
>> 
>> 
>> 
>>  
>>  Cyrus Home Page: http://www.cyrusimap.org/
>>  List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
>>  To Unsubscribe:
>> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
> 
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Cyrus IMAP 3.0.11 released

[ellie timoney]

The Cyrus team is proud to announce the immediate availability of a new version 
of Cyrus IMAP: 3.0.11

I'm trialling hosting the release files using Github's releases feature.  
Please use the Github download links if possible, and advise if you have any 
problems!  (It may even download faster due to Github's content delivery 
network.)

This release contains a fix for a potential data loss issue when upgrading mail 
servers containing messages that existed prior to Cyrus 2.3.  For details, 
please see the release notes.  If upgrading a service that has been around a 
long time, consider upgrading to 3.0.11 rather than one of the earlier 3.0 
releases.

Download URLs:


https://github.com/cyrusimap/cyrus-imapd/releases/download/cyrus-imapd-3.0.11/cyrus-imapd-3.0.11.tar.gz

https://github.com/cyrusimap/cyrus-imapd/releases/download/cyrus-imapd-3.0.11/cyrus-imapd-3.0.11.tar.gz.sig

https://www.cyrusimap.org/releases/cyrus-imapd-3.0.11.tar.gz
https://www.cyrusimap.org/releases/cyrus-imapd-3.0.11.tar.gz.sig

Please consult the release notes and upgrade documentation before upgrading to 
3.0.11:

https://www.cyrusimap.org/imap/download/release-notes/3.0/x/3.0.11.html
https://www.cyrusimap.org/imap/download/upgrade.html

And join us on Github at https://github.com/cyrusimap/cyrus-imapd to report 
issues, join in the deliberations of new features for the next Cyrus IMAP 
release, and to contribute to the documentation.

On behalf of the Cyrus team,

Kind regards,

ellie timoney

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Folder subscription issue

[ellie timoney]

I think they said 3.0.8, which is not the latest 3.0, but I also don't think 
anything about replication/subscriptions/etc has changed since then, so it 
might as well be

It would also be very interesting to know whether the master and replica 
definitely both have the same value for "unixhierarchysep". I don't think it 
should matter? But maybe there's a bug hiding there that's making it matter.

On Sun, Jul 14, 2019, at 11:03 PM, Bron Gondwana wrote:
> And this is even more exciting and I'd love to know the version!
> 
> Bron.
> 
> On Fri, Jul 12, 2019, at 23:33, Egoitz Aurrekoetxea wrote:
>> By the way I think I found some more…..
>> 
>> When a sync_client in user mode… creates a non existing user in the replica… 
>> if the user has a dot… the quota gets properly created, but seen, sub files 
>> are wrongly created… for instance….
>> 
>> -rw--- 1 cyrus cyrus 2576528 Jul 12 13:03 f.veiga.conversations
>> -rw--- 1 cyrus cyrus 88 Jul 12 13:03 f.veiga.counters
>> -rw--- 1 cyrus cyrus 451 Jul 12 10:43 f.veiga.sub
>> -rw--- 1 cyrus cyrus 16 Jul 12 01:32 f.veiga.xapianactive
>> -rw--- 1 cyrus cyrus 2584 Apr 3 01:48 f^veiga.seen
>> -rw--- 1 cyrus cyrus 316 Apr 3 01:48 f^veiga.sub
>> 
>> The Apr 3 files are from the initial sync when the box was empty… the first 
>> ones (with the dot and not ^) are the actual files being used by Cyrus…. And 
>> updated with the replication and so…
>> 
>> It seems this to be the reason because I didn’t see in the initial sync any 
>> subscribed folders… but later when set them from a mua where properly 
>> replicated…
>> 
>> With Sieve seemed to happen something similar… but in this case with the ‘^’ 
>> and ‘.’ In the directory name….
>> 
>> It’s like the name used for creating the subscribing and seen databases in a 
>> user mode replication was not properly handled by Cyrus… because it does the 
>> same as with quota database with indeed with ‘^’ is properly created….
>> 
>> Thanks! Bye!
>> 
>> 
>> sarenet
>> *Egoitz Aurrekoetxea*
>> Dpto. de sistemas
>> 944 209 470
>> Parque Tecnológico. Edificio 103
>> 48170 Zamudio (Bizkaia)
>> ego...@sarenet.es
>> www.sarenet.es
>> 
>> Antes de imprimir este correo electrónico piense si es necesario hacerlo.
>> 
>>> El 10 jul 2019, a las 11:39, Egoitz Aurrekoetxea  
>>> escribió:
>>> 
>>> Hi!
>>> 
>>> It happens with any folder… in fact Trash is not the folder we would 
>>> announce in special-use as Trash… it’s just a normal folder really here…. 
>>> It’s a generally happening thing with rename of folders inside the own 
>>> hierarchy inside the own user… (don’t really know if a rename mailbox for 
>>> changing partition would have the same issue). Not something related to the 
>>> Trash folder...
>>> 
>>> Cheers!
>>> 
>>> 
>>> sarenet
>>> *Egoitz Aurrekoetxea*
>>> Dpto. de sistemas
>>> 944 209 470
>>> Parque Tecnológico. Edificio 103
>>> 48170 Zamudio (Bizkaia)
>>> ego...@sarenet.es
>>> www.sarenet.es
>>> 
>>> Antes de imprimir este correo electrónico piense si es necesario hacerlo.
>>> 
 El 10 jul 2019, a las 11:34, Sebastian Hagedorn  
 escribió:
 
 Hi,
 
 I'm curious if this only happens for rename to trash, or for all renames
 of subscribed folders. IMHO it makes no sense to automatically subscribe
 to a folder in the trash. So perhaps the bug isn't in the replication
 code but rather in the handling of rename to trash?
 
 
 Am 10.07.19 um 11:11 Uhr schrieb Egoitz Aurrekoetxea:
> About the folder subscription issue, I think I got something, at least a
> close approximation... When a user causes in mua a rename mailbox (a
> rename for a folder caused by a folder move in hierarchy), after the own
> rename, if folders were subscribed “should” (for the "plain user" at
> least) become subscribed in the new path. It seems that after a user
> rename in Cyrus the new folder is automatically subscribed (even if no
> subscribe command is sent by the mua). But this, does not cause in the
> replica (in the slave, if SUB is not sent by the client)
> a sync_apply_changesub() or something like entering in the
> “ move_subscription” condition in mboxlist_renamemailbox(), and then,
> the folder is properly renamed but not subscribed in the slave. I think
> this is what I’m suffering. Obviously, if after a rename the mua sends a
> subscribe too, no issue is seen. I think the problem happens when a
> mailbox rename happens and a SUB is not send later.
> 
> An example : 
> 
> The folder domain.com
> !user.parta^partb.Archives.2019.TRAFICO.CHINA.XINGANG
> is moved (renamed) to trash.
> 
> May 16 16:16:50 mx6c imap[83976]: conversations_rename_folder: renamed
> domain.com
> !user.parta^partb.Archives.2019.TRAFICO.CHINA.XINGANG
> to domain.com !user.parta^partb.Trash.XINGANG
> May 16 16:16:50 mx6c imap[83976]: Rename: 

Re: backupd IOERROR reading backup files larger than 2GB

[ellie timoney]

Hi Carlos,

Sudden overnight intuition, and I think this will fix the issue -- the problem 
isn't that the lseek is failing at >2GB; the problem is that its off_t return 
value is being truncated to an int before being checked to see if it's negative 
for the error case (so any "I succeeded, the offset is > 2GB" response looks 
like "negative, it's an error"). Doh!

I think this will fix the immediate issue:
https://github.com/cyrusimap/cyrus-imapd/commit/63f8c09fa2076c0e2fa55436e071e3e341fe48b6

But then I found and fixed some similar ones:
https://github.com/cyrusimap/cyrus-imapd/commit/0930d3af4ed9bd44d328db8cfa4d9f2e2be7bada

Remains to be seen whether more similar issues pop up -- since no-one's tripped 
over this previously, I guess you're the first person to try this with a >2GB 
backup file :(

Cheers,

ellie

On Fri, Jun 14, 2019, at 3:50 PM, ellie timoney wrote:
> Hi Carlos,
> 
> This is quite weird, I'm not sure why a 64bit platform would have any trouble 
> around the 2GB mark??
> 
> What does the Cyrus ./configure report for your system's integer sizes? e.g. 
> mine shows:
> 
>> checking size of int... 4 
>> checking size of long... 8 
>> checking size of size_t... 8 
>> checking size of off_t... 8 
>> checking size of time_t... 8 
>> checking size of long long int... 8 
>> checking size of unsigned long long int... 8 
>> checking whether byte ordering is bigendian... no
> 
> What's your level of comfort with C debugging? It'd be very helpful to see a 
> core file+binary from the time that lseek error occurs?
> 
> Cheers,
> 
> ellie
> 
> On Fri, Jun 7, 2019, at 1:58 AM, Carlos Larrañaga wrote:
>> Hi Ellie,
>> 
>>  Thanks for answering. We use latest 64bit Oracle Linux (not CentOs like I 
>> said before, sorry) and zlib is also 64bit version:
>>> # uname -a
>>> Linux xxx 3.10.0-957.12.2.el7.x86_64 #1 SMP Tue May 14 17:35:45 PDT 2019 
>>> x86_64 x86_64 x86_64 GNU/Linux
>>> 
>>> # yum list installed zlib
>>> Loaded plugins: langpacks, ulninfo
>>> Installed Packages
>>> zlib.x86_64 1.2.7-18.el7 @ol7_latest
>>> 
>>>  # lsof -p 17005 |grep lib |grep -i z
>>>  backupd 17005 cyrus mem REG 253,0 90248 134400930 /usr/lib64/libz.so.1.2.7
>>> 
>>>  # file /usr/lib64/libz.so.1.2.7
>>>  /usr/lib64/libz.so.1.2.7: ELF 64-bit LSB shared object, x86-64, version 1 
>>> (SYSV), dynamically linked, 
>>> BuildID[sha1]=b9d5f73428bd6ad68c96986b57bea3b7cedb9745, stripped
>>> 
>>>  # rpm -qf /usr/lib64/libz.so.1.2.7
>>>  zlib-1.2.7-18.el7.x86_64
>> I have summarized here some information about the backupd error reading the 
>> file descriptor 15, which is a backup larger than 2GB. The error is logged 
>> 1755 times. Instead, there is no error for fd 12, which is a backup of less 
>> than 2 GB:
>>> *#-- CLIENT SIDE*
>>> *---*
>>> 
>>> 
>>> # time sync_client -v -o -n backup -A
>>> Thu Jun 6 17:08:02 CEST 2019
>>> USER aaa
>>> QUOTA user.aaa
>>> USER ccc
>>> Error from do_user(ccc): bailing out!
>>> 
>>> real 30m16.223s
>>> user 0m0.006s
>>> sys 0m0.006s
>>> **
>>> 
>>> *#-- BACKUP SERVER SIDE *
>>> # LOGFILE
>>> 
>>> # tail -f /var/log/imapd.log
>>> Jun 6 17:09:11 bcrux cyrus/backupd[2584]: IOERROR: gzuc_read: lseek 15: No 
>>> such file or directory
>>> Jun 6 17:09:11 bcrux cyrus/backupd[2584]: IOERROR: gzuc_read: lseek 15: No 
>>> such file or directory
>>> Jun 6 17:09:11 bcrux cyrus/backupd[2584]: IOERROR: gzuc_read: lseek 15: No 
>>> such file or directory
>>> Jun 6 17:09:11 bcrux cyrus/backupd[2584]: IOERROR: gzuc_read: lseek 15: No 
>>> such file or directory
>>> 
>>> 
>>> *# lsof of the backupd PROCESS
*# lsof -P -p 2584 
>>> 
>>> COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
>>> backupd 2584 cyrus cwd DIR 253,0 4096 128 /
>>> backupd 2584 cyrus rtd DIR 253,0 4096 128 /
>>> backupd 2584 cyrus txt REG 253,0 768632 939542214 
>>> /usr/local/cyrus/libexec/backupd
>>> backupd 2584 cyrus mem REG 253,0 37208 135746334 /usr/lib64/libnss_sss.so.2
>>> backupd 2584 cyrus mem REG 253,0 31416 134406707 
>>> /usr/lib64/libnss_dns-2.17.so
>>> backupd 2584 cyrus mem REG 253,0 61632 134406709 
>>> /usr/lib64/libnss_files-2.17.so
>>> backupd 2584 cyrus mem REG 253,0 43336 68814977 
>>> /usr/lib64/sasl2/webmail.so

Re: backupd IOERROR reading backup files larger than 2GB

[ellie timoney]

EG 253,0 88776 135352942 
>> /usr/lib64/libgcc_s-4.8.5-20150702.so.1
>> backupd 2584 cyrus mem REG 253,0 1137032 134400337 /usr/lib64/libm-2.17.so
>> backupd 2584 cyrus mem REG 253,0 991616 135352396 
>> /usr/lib64/libstdc++.so.6.0.19
>> backupd 2584 cyrus mem REG 253,0 19296 134399434 /usr/lib64/libdl-2.17.so
>> backupd 2584 cyrus mem REG 253,0 142008 134722750 
>> /usr/lib64/libpthread-2.17.so
>> backupd 2584 cyrus mem REG 253,0 2151704 134365659 /usr/lib64/libc-2.17.so
>> backupd 2584 cyrus mem REG 253,0 753232 134401161 
>> /usr/lib64/libsqlite3.so.0.8.6
>> backupd 2584 cyrus mem REG 253,0 90248 134400930 /usr/lib64/libz.so.1.2.7
>> backupd 2584 cyrus mem REG 253,0 11128 134400947 
>> /usr/lib64/libpcreposix.so.0.0.1
>> backupd 2584 cyrus mem REG 253,0 402384 134400940 /usr/lib64/libpcre.so.1.2.0
>> backupd 2584 cyrus mem REG 253,0 67104 135269427 
>> /usr/lib64/libkrb5support.so.0.1
>> backupd 2584 cyrus mem REG 253,0 15920 135873508 /usr/lib64/libcom_err.so.2.1
>> backupd 2584 cyrus mem REG 253,0 210832 134781960 
>> /usr/lib64/libk5crypto.so.3.1
>> backupd 2584 cyrus mem REG 253,0 967864 134781967 /usr/lib64/libkrb5.so.3.3
>> backupd 2584 cyrus mem REG 253,0 320400 134781952 
>> /usr/lib64/libgssapi_krb5.so.2.2
>> backupd 2584 cyrus mem REG 253,0 115856 134400341 /usr/lib64/libnsl-2.17.so
>> backupd 2584 cyrus mem REG 253,0 42168 134401419 /usr/lib64/libwrap.so.0.7.6
>> backupd 2584 cyrus mem REG 253,0 2516640 134400708 
>> /usr/lib64/libcrypto.so.1.0.2k
>> backupd 2584 cyrus mem REG 253,0 470360 134400711 /usr/lib64/libssl.so.1.0.2k
>> backupd 2584 cyrus mem REG 253,0 121208 134401258 
>> /usr/lib64/libsasl2.so.3.0.0
>> backupd 2584 cyrus mem REG 253,0 2898664 83996 
>> /usr/local/cyrus/lib/libcyrus_imap.so.0.0.0
>> backupd 2584 cyrus mem REG 253,0 599528 83999 
>> /usr/local/cyrus/lib/libcyrus_sieve.so.0.0.0
>> backupd 2584 cyrus mem REG 253,0 20112 134406704 /usr/lib64/libuuid.so.1.3.0
>> backupd 2584 cyrus mem REG 253,0 20781704 136182553 
>> /usr/lib64/libicudata.so.50.1.2
>> backupd 2584 cyrus mem REG 253,0 1539544 136197152 
>> /usr/lib64/libicuuc.so.50.1.2
>> backupd 2584 cyrus mem REG 253,0 553264 838890012 
>> /usr/local/cyrus/lib/libcyrus_min.so.0.0.0
>> backupd 2584 cyrus mem REG 253,0 1961968 83994 
>> /usr/local/cyrus/lib/libcyrus.so.0.0.0
>> backupd 2584 cyrus mem REG 253,0 163408 134355369 /usr/lib64/ld-2.17.so
>> backupd 2584 cyrus 0u IPv4 2092952 0t0 TCP 
>> hostb1.upv.es:2005->host1.upv.es:27107 (ESTABLISHED)
>> backupd 2584 cyrus 1u IPv4 2092952 0t0 TCP 
>> hostb1.upv.es:2005->host1.upv.es:27107 (ESTABLISHED)
>> backupd 2584 cyrus 2u IPv4 2092952 0t0 TCP 
>> hostb1.upv.es:2005->host1.upv.es:27107 (ESTABLISHED)
>> backupd 2584 cyrus 3w FIFO 0,9 0t0 2141764 pipe
>> backupd 2584 cyrus 4u IPv4 2141763 0t0 TCP hostb1.upv.es:2005 (LISTEN)
>> backupd 2584 cyrus 5u REG 0,20 0 86644 /run/cyrus-b1/socket/backupd-0.lock
>> backupd 2584 cyrus 6r FIFO 0,9 0t0 2141754 pipe
>> backupd 2584 cyrus 7w FIFO 0,9 0t0 2141754 pipe
>> backupd 2584 cyrus 8r FIFO 0,9 0t0 2141755 pipe
>> backupd 2584 cyrus 9w FIFO 0,9 0t0 2141755 pipe
>> backupd 2584 cyrus 10u unix 0x9d9df53d4400 0t0 2093417 socket
>> backupd 2584 cyrus 11w REG 253,5 680 18253611600 
>> /b1/lib/log/admin/backupd-2584
>> backupd 2584 cyrus 12uW REG 253,5 976281587 13958644353 
>> /b1/bck/b1/a/aaa_ouuUx9
>> backupd 2584 cyrus 13ur REG 253,5 3573760 13958644354 
>> /b1/bck/b1/a/aaa_ouuUx9.index
>> backupd 2584 cyrus 14u REG 253,5 2576 13958644355 
>> /b1/bck/b1/a/aaa_ouuUx9.index-journal
>> backupd 2584 cyrus 15uW REG 253,5 3104341527 12884902595 
>> /b1/bck/b1/c/ccc_Mqmymx
>> backupd 2584 cyrus 16u REG 253,5 47927296 12884902596 
>> /b1/bck/b1/c/ccc_Mqmymx.index
>> 
>> *# BACKUP SIZES*
>>  # ls -lh /b1/bck/b1/a/* 
>> /b1/bck/b1/c/* 
>> *-rw--- 1 cyrus mail 932M Jun 6 17:08*
>> /b1/bck/b1/a/*aaa_ouuUx9*
>>  -rw--- 1 cyrus mail 3.5M Jun 6 16:46
>> /b1/bck/b1/a/aaa_ouuUx9.index
>>  -rw--- 1 cyrus mail 2.6K Jun 6 17:08
>> /b1/bck/b1/a/aaa_ouuUx9.index-journal
>> 
>> -rw--- 1 cyrus mail 2.9G Jun 6 17:01 /b1/bck/b1/c/ccc_Mqmymx
>> -rw--- 1 cyrus mail 46M Jun 6 17:01 /b1/bck/b1/c/ccc_Mqmymx.index
>> 
>> *# IMAP SESSION DEBUG*
>> # cat /b1/lib/log/admin/backupd-2584
>> -- admin Thu Jun 6 17:08:02 2019
>> 
>> <1559833682> >1559833682>OK DEFLATE active
>> <1559833682> >1559833698>* MAILBOX %(UNIQUEID acd014aa-7b1d-43c9-b1d3-f97b06540739 
>>

Re: LDAP auth and ptloader

[ellie timoney]

Hi Sven,

On Thu, Jun 13, 2019, at 12:27 AM, Sven Schwedas wrote:
> Is there another way to get ptloader to spit out debug information and
> pinpoint what's not set up correctly?
> 

I remember this thing as being very noisy, let me see...

Okay, in your cyrus.conf SERVICES entry, if you add "-d1" to the ptloader line 
like this,

 ptloader cmd="ptloader -d1" listen="/path/to/some/socket" 

then ptloader will syslog every user that it's asked about...

You need "debug: 1" in imapd.conf, which will tell Cyrus to not swallow 
LOG_DEBUG level log lines, but ALSO: your syslog itself must be configured to 
log these lines (the default is often to not). We have some makeshift 
instructions here but ymmv: 
https://www.cyrusimap.org/imap/installing.html#setting-up-syslog

If you turn on the "ptloader -d1" switch and set debug:1 and *don't* start 
seeing entries in your logs like "ptloader[pid]: user [user]", then you need to 
fiddle with syslog to enable the LOG_DEBUG log level :)

Here's an example of some ptloader log output from running our test suite, for 
example:

> Jun 14 12:08:03 debian 02032301C4/ptloader[29481]: executed 
> Jun 14 12:08:03 debian 02032301C4/ptloader[29481]: starting: ptloader.c 
> 3.1.6-696-gf38559858 
> Jun 14 12:08:03 debian 02032301C4/ptloader[29481]: accepted connection 
> Jun 14 12:08:03 debian 02032301C4/ptloader[29481]: user admin 
> Jun 14 12:08:03 debian 02032301C4/ptloader[29481]: collecting all domains 
> from ou=domains,o=cyrus 
> Jun 14 12:08:03 debian 02032301C4/ptloader[29481]: Domain filter: 
> (&(objectclass=domainrelatedobject)(associateddomain=*))
> Jun 14 12:08:03 debian 02032301C4/ptloader[29481]: we have a domain internal. 
> Jun 14 12:08:03 debian 02032301C4/ptloader[29481]: ptsmodule_standard_root_dn 
> called for domain internal. 
> Jun 14 12:08:03 debian 02032301C4/ptloader[29481]: Root DN now dc= 
> Jun 14 12:08:03 debian 02032301C4/ptloader[29481]: Root DN now dc=internal 
> Jun 14 12:08:03 debian 02032301C4/ptloader[29481]: Root DN now dc=internal 
> Jun 14 12:08:03 debian 02032301C4/ptloader[29481]: Root DN now dc=internal 
> Jun 14 12:08:03 debian 02032301C4/ptloader[29481]: Found admin in dc=internal 
> Jun 14 12:08:03 debian 02032301C4/ptloader[29481]: we have found admin in 
> dc=internal

And part of another run, showing it resolving a group membership (sorry these 
lines got truncated during copy, but you get the idea):

> Jun 14 13:11:47 debian 0311460101/ptloader[16345]: accepted connection 
> Jun 14 13:11:47 debian 0311460101/ptloader[16345]: user group:group co 
> Jun 14 13:11:47 debian 0311460101/ptloader[16345]: (groups) about to search 
> ou=groups,o=cy 
> Jun 14 13:11:47 debian 0311460101/imap[16344]: timeout_select exiting. r = 1; 
> errno = 0 
> Jun 14 13:11:47 debian 0311460101/imap[16344]: timeout_select: sock = 15, rp 
> = 0x7ffca95e3 
> Jun 14 13:11:47 debian 0311460101/imap[16344]: timeout_select exiting. r = 1; 
> errno = 0 
> Jun 14 13:11:47 debian 0311460101/imap[16344]: ptload read data back 
> Jun 14 13:11:47 debian 0311460101/imap[16344]: ptload returning data 
> Jun 14 13:11:47 debian 0311460101/imap[16344]: canonified group:group co -> 
> group:group co

Not sure if this is helpful, but this is the directory structure our tests are 
working with:

https://github.com/cyrusimap/cassandane/blob/master/data/directory.ldif

... ohhh,

> ldap_member_attribute: memberUid 

This kinda sounds like your groups are what I think of as "normal": a group in 
LDAP is an entry that contains a multi-valued attribute listing all the group 
members. Is that a good description of your schema?

As far as I've been able to figure out while building tests, Cyrus seems to 
expect each *user* entry to contain a multi-valued attribute listing the groups 
it is a member of (e.g. see that directory.ldif linked above). This feels 
backwards to me, but maybe it's normal somewhere?? I don't understand the 
rationale for this choice, or whether Cyrus can support a "normal" setup... 
maybe using the "ldap_member_method: filter" configuration (vs the default 
setting of "attribute") somehow??

Hopefully this is enough for you to get some useful logging out of the thing 
anyway,

Cheers,

ellie
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: backupd IOERROR reading backup files larger than 2GB

[ellie timoney]

It kinda sounds like your platform might be 32bit? Or your zlib is compiled to 
use 32bit integer sizes?

On Mon, Jun 3, 2019, at 10:07 PM, Carlos Larrañaga wrote:
> Hi,
> 
> We're testing backup feature un cyrus-imapd 3.0.10. There's no problem when 
> backup is created first time, but when the backup already exists and is 
> larger than 2GB, we get the following error from the backupd: 
>> cyrus/backupd[]: IOERROR: gzuc_read: lseek 12: No such file or directory
> As said, it happens only when then backup file already exists and is larger 
> than 2GB. The backupd process keeps reading de compressed backup file til 
> sync_client exit with "Error from sync_do_user(xxx): bailing out!".
> 
>  We use xfs with latest CentOs. Seems like backupd is unable to read the 
> compressed backup file.
> 
>  Anyone else with this problem? Any idea how to fix it? 
> Thanks in advance for your help.
> 
>  Best regards,
>  Carlos.
> 
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
> 
> *Attachments:*
>  * smime.p7s

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Cyrus IMAP 3.0.10 released

[ellie timoney]

Yep, that's the correct link, we're waiting on the CVE folks to update the 
listing out of reserved status

On Mon, May 27, 2019, at 11:10 PM, Savvas Karagiannidis wrote:
> Hi Ellie,
> is there any link for details about CVE-2019-11356?
> Tried googling it but couldn't find anything related or in cyrus mailing 
> lists or in reported issues, other than this:
> https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2019-11356
> 
> Regards,
> Savvas Karagiannidis
> 
> On Mon, May 27, 2019 at 5:28 AM ellie timoney  wrote:
>> The Cyrus team is proud to announce the immediate availability of a new 
>> version of Cyrus IMAP: 3.0.10
>> 
>>  This release contains a fix for CVE-2019-11356, a buffer overrun 
>> vulnerability in the httpd daemon. If you compile cyrus with HTTP support 
>> enabled and your cyrus.conf contains SERVICES entries that run the httpd 
>> daemon, you will need this upgrade.
>> 
>>  I'm trialling hosting the release files using Github's releases feature. 
>> Please use the Github download links if possible, and advise if you have any 
>> problems! (It may even download faster due to Github's content delivery 
>> network.)
>> 
>>  Download URLs:
>> 
>> https://github.com/cyrusimap/cyrus-imapd/releases/download/cyrus-imapd-3.0.10/cyrus-imapd-3.0.10.tar.gz
>> https://github.com/cyrusimap/cyrus-imapd/releases/download/cyrus-imapd-3.0.10/cyrus-imapd-3.0.10.tar.gz.sig
>> 
>> https://www.cyrusimap.org/releases/cyrus-imapd-3.0.10.tar.gz
>> https://www.cyrusimap.org/releases/cyrus-imapd-3.0.10.tar.gz.sig
>> 
>>  Please consult the release notes and upgrade documentation before upgrading 
>> to 3.0.10:
>> 
>> https://www.cyrusimap.org/imap/download/release-notes/3.0/x/3.0.10.html
>> https://www.cyrusimap.org/imap/download/upgrade.html
>> 
>>  And join us on Github at https://github.com/cyrusimap/cyrus-imapd to report 
>> issues, join in the deliberations of new features for the next Cyrus IMAP 
>> release, and to contribute to the documentation.
>> 
>>  On behalf of the Cyrus team,
>> 
>>  Kind regards,
>> 
>>  ellie timoney
>>  
>>  Cyrus Home Page: http://www.cyrusimap.org/
>>  List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
>>  To Unsubscribe:
>> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Cyrus IMAP 3.0.10 released

[ellie timoney]

The Cyrus team is proud to announce the immediate availability of a new version 
of Cyrus IMAP: 3.0.10

This release contains a fix for CVE-2019-11356, a buffer overrun vulnerability 
in the httpd daemon.  If you compile cyrus with HTTP support enabled and your 
cyrus.conf contains SERVICES entries that run the httpd daemon, you will need 
this upgrade.

I'm trialling hosting the release files using Github's releases feature.  
Please use the Github download links if possible, and advise if you have any 
problems!  (It may even download faster due to Github's content delivery 
network.)

Download URLs:


https://github.com/cyrusimap/cyrus-imapd/releases/download/cyrus-imapd-3.0.10/cyrus-imapd-3.0.10.tar.gz

https://github.com/cyrusimap/cyrus-imapd/releases/download/cyrus-imapd-3.0.10/cyrus-imapd-3.0.10.tar.gz.sig

https://www.cyrusimap.org/releases/cyrus-imapd-3.0.10.tar.gz
https://www.cyrusimap.org/releases/cyrus-imapd-3.0.10.tar.gz.sig

Please consult the release notes and upgrade documentation before upgrading to 
3.0.10:

https://www.cyrusimap.org/imap/download/release-notes/3.0/x/3.0.10.html
https://www.cyrusimap.org/imap/download/upgrade.html

And join us on Github at https://github.com/cyrusimap/cyrus-imapd to report 
issues, join in the deliberations of new features for the next Cyrus IMAP 
release, and to contribute to the documentation.

On behalf of the Cyrus team,

Kind regards,

ellie timoney

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Cyrus IMAP 2.5.13 released

[ellie timoney]

The Cyrus team is proud to announce the immediate availability of a new version 
of Cyrus IMAP: 2.5.13

This release contains a fix for CVE-2019-11356, a buffer overrun vulnerability 
in the httpd daemon.  If you compile cyrus with HTTP support enabled and your 
cyrus.conf contains SERVICES entries that run the httpd daemon, you will need 
this upgrade.

I'm trialling hosting the release files using Github's releases feature.  
Please use the Github download links if possible, and advise if you have any 
problems!  (It may even download faster due to Github's content delivery 
network.)

Download URLs:


https://github.com/cyrusimap/cyrus-imapd/releases/download/cyrus-imapd-2.5.13/cyrus-imapd-2.5.13.tar.gz

https://github.com/cyrusimap/cyrus-imapd/releases/download/cyrus-imapd-2.5.13/cyrus-imapd-2.5.13.tar.gz.sig

https://www.cyrusimap.org/releases/cyrus-imapd-2.5.13.tar.gz
https://www.cyrusimap.org/releases/cyrus-imapd-2.5.13.tar.gz.sig

Please consult the release notes before upgrading to 2.5.13:

https://www.cyrusimap.org/imap/download/release-notes/2.5/x/2.5.13.html

And join us on Github at https://github.com/cyrusimap/cyrus-imapd to report 
issues, join in the deliberations of new features for the next Cyrus IMAP 
release, and to contribute to the documentation.

On behalf of the Cyrus team,

Kind regards,

ellie timoney

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: cyradm | Duplicate specification

[ellie timoney]

Hi Ismaël,

Which version of perl are you running? (`perl --version` will tell you) A 
fairly newish one, I guess?

The cyradm tools were written using a quite old version of perl, which didn't 
produce a lot of warnings. I expect it's working fine, but your newer perl 
version is producing warnings that the older versions did not.

It would be good to fix up a lot of this cruft -- do you want to raise an issue 
on https://github.com/cyrusimap/cyrus-imapd/issues and include the details from 
your email and your perl version? I can't promise it'll get looked at quickly, 
but at least it won't get forgotten. :)

Cheers,

ellie

On Thu, Apr 25, 2019, at 5:49 AM, Ismaël Tanguy wrote:
> Hello,

> I've got this error after connecting to cyrus with cyradm (as root or cyrus 
> user):

> # cyradm -u cyrus localhost
Variable "$cyrref" will not stay shared at 
/usr/lib64/perl5/vendor_perl/Cyrus/IMAP/Shell.pm line 724.
Variable "$lfh" will not stay shared at 
/usr/lib64/perl5/vendor_perl/Cyrus/IMAP/Shell.pm line 726.
Duplicate specification "server|s=s" for option "s"

> 
> I can make operation on mailbox (lam, sam, xfer, ..), everything seems to 
> work fine but I'm not confident to put that in production..

> Cyrus version is 3.08 installed with rpm on Centos7.

> I've build the rpms, so maybe I've made mistake at this step.

> cyrus was build like that :

> # cyr_buildinfo
{
  "component": {
    "event_notification": true,
    "gssapi": true,
    "autocreate": true,
    "idled": true,
    "httpd": true,
    "kerberos_v4": false,
    "murder": true,
    "nntpd": true,
    "replication": true,
    "sieve": true,
    "calalarmd": true,
    "objectstore": false,
    "backup": true
  },
  "dependency": {
    "ldap": true,
    "openssl": true,
    "pcre": true,
    "clamav": true
  },
  "database": {
    "mysql": false,
    "pgsql": false,
    "sqlite": true,
    "lmdb": false
  },
  "search": {
    "squat": true,
    "sphinx": false,
    "xapian": false,
    "xapian_flavor": "none"
  },
  "hardware": {
    "sse42": true
  }
}
> Thank you

> ---

> Ismaël TANGUY

> 
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: LDAP auth and ptloader

[ellie timoney]

Hi Sven,

I don't know much about running it in a production capacity, but our test suite 
sets up the following for LDAP pts:

imapd.conf:
 ...
 ptloader_sock: /path/to/some/socket
 auth_mech: pts
 pts_module: ldap
 ...

cyrus.conf:
 SERVICES {
 ...
 ptloader cmd="ptloader" listen="/path/to/some/socket"
 ... 
 }

Does this get you going?

Cheers,

ellie


On Tue, Apr 23, 2019, at 7:52 PM, Sven Schwedas wrote:
> I'm trying to set up direct LDAP auth via auth_meth=pts, but on start I
> always get "ptload(): can't connect to ptloader server: No such file or
> directory" as error. The directory for ptloader_sock exists and is the
> same as for all other sockets, so there shouldn't be any permission
> problems with the socket.
> 
> I suppose I need to somehow manually start up ptloader via cyrus.conf,
> but there's no documentation and nothing I can find in the mailing list
> archives as to *how*? What am I missing?
> 
> -- 
> Mit freundlichen Grüßen, / Best Regards,
> Sven Schwedas, Systemadministrator
> ✉ sven.schwe...@tao.at | ☎ +43 680 301 7167
> TAO Digital | Teil der TAO Beratungs- & Management GmbH
> Lendplatz 45 | FN 213999f/Klagenfurt, FB-Gericht Villach
> A8020 Graz | https://www.tao-digital.at
> 
> 
> 
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
> 
> *Attachments:*
>  * signature.asc

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Upgrade cyrus 2.4.17 to 2.4.18 on Ubuntu

[ellie timoney]

On Tue, Mar 26, 2019, at 6:50 AM, Patrick Goetz wrote:
> I believe the version number change (incremental change to stable 
> release) indicates you shouldn't have any problems, but of course shut 
> down the service while it's being updated.

Yep, no major changes within a single x.y series.  Of course, read the release 
notes (they're on the website), they will tell you anything special you need to 
be aware of.

> I thought Debian was the one distro the cyrus crew provided up-to-date 
> packages for?  See for example here:

We don't directly provide any distro packages, but the maintainers of the cyrus 
packages for Debian and Fedora are doing a great job.  We don't hear directly 
from other distros much, but I guess that's because most other distros leverage 
Debian and/or Fedora's work?

Most of us cyrus developers are running Debian on our own systems, and 
FastMail's servers run Debian, so even without an official package, a source 
build on Debian will generally just work (but then you have to choose/configure 
features/dependencies yourself, of course...).

Cheers,

ellie

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: cyrus-imapd build dependencies

[ellie timoney]

On Tue, Mar 19, 2019, at 3:39 PM, Anatoli via Info-cyrus wrote:
>  > The Cyrus httpd provides DAV services (which use the HTTP protocol). If 
> you want the Cyrus httpd to support HTTP/2, you will need libnghttp2. 
> Otherwise it will only support HTTP/1.
> 
>  Always wanted to ask what the nghttp2 dependency was for. From what you say 
> I infer that it's only needed for HTTP/2. But what DAV service could benefit 
> from this? Are there DAV clients that know HTTP/2?
> 

No idea, but it's there if you want it! Speculating wildly, it might be useful 
for JMAP?

>  And speaking about the SNMP agent, are there any plans to complete the 
> transfer of its code from the master process to an independent daemon, issue 
> #1765 ? (It needs to be 
> moved out to implement efficient chroot) 

It's more likely to disappear entirely (see 
https://github.com/cyrusimap/cyrus-imapd/pull/2100) in favour of Prometheus 
(which is more powerful, more flexible, more human-readable, and is actually 
used by Fastmail -- and therefore more tested). But it won't disappear from a 
stable branch, so it won't be a surprise when it does.

Cheers,

ellie
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: cyrus-imapd build dependencies

[ellie timoney]

Hi Patrick,

On Mon, Mar 18, 2019, at 11:33 PM, Patrick Goetz wrote:
> This page on compiling cyrus-imapd:
> 
>https://www.cyrusimap.org/imap/developer/compiling.html

This page is in the developer section, so its context is for people who are 
Cyrus developers (especially for new contributors needing to get rolling 
quickly).  Expect a certain amount of detail to be glossed over on the 
assumption that it's already known and/or reasonably documented elsewhere.

> shows a number of build dependencies; however I was just able to compile 
> cyrus-imapd without these installed:
> 
> gperf
> libbsd
> 
> 
> Are these actually necessary?

Probably depends on which features you enable.  If you run './configure' 
without arguments, a number of large features won't be enabled, so any 
libraries they depend on won't be used.  Some of these features are important 
enough that we (developers) kind of think of them as being 
probably-always-included even if they default to not.

> Later in the page, under "Alternate database formats" it shows the 
> configure flags to use in order to use mysql/mariadb as a backend for 
> cyrus databases.  I think this is needed if one plans to use virtual 
> domains, but I couldn't get a confirmation on this.  

These are literally just "alternate database formats" -- maybe you already have 
extensive expertise in some other database and would rather use that than one 
of the builtin ones.  It has nothing to do with virtual domains.  Documentation 
about the databases used by Cyrus are here: 
https://www.cyrusimap.org/imap/concepts/deployment/databases.html

> In any case, the 
> configure options are given as
>   --with-mysql, --with-mysql-incdir, --with-mysql-libdir
> 
> with no clear indication of what each of these does.  For example, is 
> the --with-mysql all inclusive, or does one need to set all 3?

The canonical source of information on configure options is the output from 
'./configure --help'.  It's kind of assumed that a developer will look there to 
find this information.

> Finally a couple of items in the "Other" category are a real head 
> scratcher.  For example, what is the purpose of net-snmp?

You can click on any of those package names to go to the website for that 
package and get a description of what it does.  For example,

> Simple Network Management Protocol (SNMP) is a widely used 
> protocol for monitoring the health and welfare of network equipment
>  (eg. routers), computer equipment and even devices like UPSs. 
> Net-SNMP is a suite of applications used to implement SNMP v1, 
> SNMP v2c and SNMP v3 using both IPv4 and IPv6. 
 
> libnghttp2 is listed as needed for "HTTP/2 support for httpd" -- what's 
> using httpd?  Is this to faciliate CalDAV/CardDAV?

The Cyrus httpd provides DAV services (which use the HTTP protocol).  If you 
want the Cyrus httpd to support HTTP/2, you will need libnghttp2.  Otherwise it 
will only support HTTP/1.

Hope this helps :)

ellie

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Cyrus IMAP 3.0.9 released

[ellie timoney]

The Cyrus team is proud to announce the immediate availability of a new version 
of Cyrus IMAP: 3.0.9

I'm trialling hosting the release files using Github's releases feature.  
Please use the Github download links if possible, and advise if you have any 
problems!  (It may even download faster due to Github's content delivery 
network.)

Download URLs:


https://github.com/cyrusimap/cyrus-imapd/releases/download/cyrus-imapd-3.0.9/cyrus-imapd-3.0.9.tar.gz

https://github.com/cyrusimap/cyrus-imapd/releases/download/cyrus-imapd-3.0.9/cyrus-imapd-3.0.9.tar.gz.sig

https://www.cyrusimap.org/releases/cyrus-imapd-3.0.9.tar.gz
https://www.cyrusimap.org/releases/cyrus-imapd-3.0.9.tar.gz.sig

Please consult the release notes and upgrade documentation before upgrading to 
3.0.9:

https://www.cyrusimap.org/imap/download/release-notes/3.0/x/3.0.9.html
https://www.cyrusimap.org/imap/download/upgrade.html

And join us on Github at https://github.com/cyrusimap/cyrus-imapd to report 
issues, join in the deliberations of new features for the next Cyrus IMAP 
release, and to contribute to the documentation.

On behalf of the Cyrus team,

Kind regards,

ellie timoney

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: faild to rename deleted mailbox, or how conversation db sucks again

[ellie timoney]

I've raised https://github.com/cyrusimap/cyrus-imapd/issues/2659 to get this 
properly fixed
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: faild to rename deleted mailbox, or how conversation db sucks again

[ellie timoney]

On Sat, Mar 2, 2019, at 2:54 AM, Michael Menge wrote: 
> Is there an other way than disabling
> - conversation db,
> - restoring the folder,
> - enabeling conversation db again (so that squat files are used for  
> header searches),
> - rebuilding conversation db for all users (because there will be some  
> new mails between
>disabling and re-enabling conversation db).

If you can arrange for your MX to hold the incoming mail during the period 
where conversations is disabled, that would at least avoid needing to rebuild 
conversationsdb for everyone again afterwards?  You might be able to do this by 
temporarily disabling the lmtpd service, but I'm not sure (it would depend on 
how your MX handles that)

Cheers,

ellie

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Cyrus-imapd 2.4.17: processes stick on mailbox locking resulting in total mailsystem failure

[ellie timoney]

Hi Ivan,

> #7  0x56193ed6382e in idle_update ()

> with cyrus-imapd 2.4.17 (cyrus-imapd-2.4.17-13.el7.x86_64 rpm)

It looks like this version is old enough to still be using signal handlers in 
its IMAP IDLE implementation.  This is known to be a problem, and IDLE was 
rewritten to not use signals for 2.5 and later.

Thomas Jarosch kindly backported the rewrite for 2.4, and it has been included 
in releases since 2.4.19 (the current 2.4 release is 2.4.20)

Cheers,

ellie

On Wed, Feb 20, 2019, at 1:15 AM, Ivan Kuznetsov wrote:
> Hello
> 
> I have gdb'ed the locked process, backtrace is below. It seems that 
> problem occurs when imapd process catch a signal when is inside malloc() 
> call. The signal handler has a malloc() call too, so finally there is 
> interlock between two mallocs
> 
> I have only a few time to investigate because locked processes list 
> grows up dramatically. So I didn't found what the signal was. But it 
> seems that there is a bug in imapd code...
> 
> (gdb) thread apply all bt
> 
> Thread 1 (Thread 0x7ff6ead5f840 (LWP 22980)):
> #0  0x7ff6e83e282c in __lll_lock_wait_private () from /lib64/libc.so.6
> #1  0x7ff6e835e7e5 in _L_lock_16773 () from /lib64/libc.so.6
> #2  0x7ff6e835b833 in malloc () from /lib64/libc.so.6
> #3  0x56193edc9910 in xzmalloc ()
> #4  0x56193edb2664 in seqset_init ()
> #5  0x56193ed821b0 in index_tellchanges ()
> #6  0x56193ed8232b in index_check ()
> #7  0x56193ed6382e in idle_update ()
> #8  
> #9  0x7ff6e83580f0 in _int_malloc () from /lib64/libc.so.6
> #10 0x7ff6e835b7dc in malloc () from /lib64/libc.so.6
> #11 0x7ff6e956f4b8 in CRYPTO_malloc () from /lib64/libcrypto.so.10
> #12 0x7ff6e96397ec in EVP_PKEY_CTX_dup () from /lib64/libcrypto.so.10
> #13 0x7ff6e962b360 in EVP_MD_CTX_copy_ex () from /lib64/libcrypto.so.10
> #14 0x7ff6e999ab3d in tls1_mac () from /lib64/libssl.so.10
> #15 0x7ff6e998db02 in ssl3_read_bytes () from /lib64/libssl.so.10
> #16 0x7ff6e998a644 in ssl3_read_internal () from /lib64/libssl.so.10
> #17 0x56193edb9745 in prot_fill ()
> #18 0x56193eda9ad7 in getword ()
> #19 0x56193ed670e7 in cmd_idle ()
> #20 0x56193ed7848d in cmdloop ()
> #21 0x56193ed79769 in service_main ()
> #22 0x56193ed62875 in main ()
> 
> 
> 13.12.2018 17:50, Ivan Kuznetsov пишет:
> > Jonk, thank you for the idea. Somewhat looks strange as old mail server 
> > worked w/o this problem 5+ years. But the system environment changed 
> > dramatically, may be some filesystem quircks are significant for this 
> > locks...
> > 
> > I will try gdb'ing the process when problem occurs once more
> > 
> > 13.12.2018 17:34, John Wade пишет:
> >> Without running gdb on the process, I have no idea, but your problem 
> >> sounds similar to something we hit a very long time ago:
> >>
> >> See https://www.oakton.edu/user/3/jwade/cyrus/Readme.html
> >>
> >> In our cases, the problem was the imapd process that was holding the 
> >> lock was trying to obtain a second lock on the same file.   What does 
> >> a stack trace look like on the imapd process that is holding the lock? 
> >> It would appear the lock process has changed since I last looked at 
> >> this, so this may not be a help at all.
> >>
> >> Good luck,
> >> John
> >>
> >>
> >>
> >> On 12/13/2018 5:21 AM, Ivan Kuznetsov wrote:
> >>> Hello
> >>>
> >>> We had a company mail server under Oracle Linux 6 (a clone of RHEL6) 
> >>> with cyrus-imapd 2.3.16 working for years w/o problems. There are >9M 
> >>> messages in ~9500 mailboxes in two partitions. Daily mail traffic is 
> >>> 20-50K messages.
> >>>
> >>> Some weeks ago we migrated the server to a new one under Oracle Linux 
> >>> 7 with cyrus-imapd 2.4.17 (cyrus-imapd-2.4.17-13.el7.x86_64 rpm) and 
> >>> now have problem. Some times a week one of imapd processes locks an 
> >>> "INBOX" mailbox with corresponding 
> >>> /var/lib/imap/lock/user/.lock file and does not unlock it 
> >>> anymore. Other imapd processes trying to access this mailbox sticks 
> >>> waiting to obtain the lock. The bigger problem is that lmtpd 
> >>> processes trying to deliver new mail to this mailbox hangs too. The 
> >>> number of lmtpd processes is limited (maxchild=32) to limit the 
> >>> server load, so free lmtpd pool become empty after a time, and mail 
> >>> delivery stopsto all the mailboxes. MTA (postfix) mail queue blows up 
> >>> quickly
> >>>
> >>> Example lsof output:
> >>>
> >>> lmtpd   8182 cyrus   18uR  REG  249,0 0 202944402 
> >>> /var/lib/imap/lock/user/smilovsky.lock
> >>> lmtpd   8183 cyrus   18uR  REG  249,0 0 202944402 
> >>> /var/lib/imap/lock/user/smilovsky.lock
> >>> lmtpd   8187 cyrus   18uR  REG  249,0 0 202944402 
> >>> /var/lib/imap/lock/user/smilovsky.lock
> >>> lmtpd   8771 cyrus   18uR  REG  249,0 0 202944402 
> >>> /var/lib/imap/lock/user/smilovsky.lock
> >>> lmtpd   8834 cyrus   18uR  REG  

Re: prblems rebuilding conversations db

[ellie timoney]

Hi Michael,

On Tue, Jan 22, 2019, at 12:32 AM, Michael Menge wrote:
> Hi Ellie
> 
> Quoting ellie timoney :
> 
> > Hi Michael,
> >
> > On Wed, Jan 16, 2019, at 10:12 PM, Michael Menge wrote:
> >> Hi,
> >>
> >> because conversations db seems to be required for search indexes, I
> >> enabled this option
> >> on our production servers today and tried to rebuild the conversations
> >> db for all users with
> >>
> >> ctl_conversationsdb -v -b UID
> >>
> >> For most users this did take less than a second. But for some users
> >> this process would
> >> not finish. I did kill one process after about 30 Minutes (most others
> >> after 3 Minutes).
> >> The UserID.conversations has grown over 2 GB (the mailbox itself has
> >> only ~700 MB of mails,
> >> and the conversations files from finnished rebuilds are less then 20 MB)
> >
> > So, a 2GB-and-growing conversations db for a 700MB mailbox is a  
> > ratio of ~3:1, which seems obscene...
> >
> 
> thanks for confirming that this is not expected.
> 
> > For comparison, the Conversations.append_reply_200 cassandane test  
> > (two messages with 100 replies each) produces an 87Kb conversations  
> > db for a 1016Kb mailbox (ratio ~1:12, n.b inverted!), which makes  
> > your ~3:1 and growing ratio seem even more obscene...
> >
> > Obviously you can't share the bad conversations db because  
> > conversations db's are full of personally-identifying information.   
> > But I wonder if looking through it with `cyr_dbtool ... show` turns  
> > up any unusual patterns, especially in comparison to one of the good  
> > users?  I wonder if it's somehow repeating itself?
> 
> I was unable to run "cyr_dbtool ... show" or "ctl_conversationsdb -d"
> while ctl_conversationsdb -v -b was still running
> (waiting for a lock on conversations db file)
> 
> After aborting "ctl_conversationsdb -b" "cyr_dbtool show" didn't show any
> duplicates. But the string command showed more occurrences than cyr_dbtool
> did (i used an hash value (d8087b76083ec5e7) I found in the output of  
> cyr_dbtool
> 
> 
> # cyr_dbtool /path-to/userid.conversations skiplist show | grep  
> d8087b76083ec5e7 | wc -l
> 3
> # cyr_dbtool /path-to/userid.conversations skiplist show | grep  
> Bd8087b76083ec5e7 | wc -l
> 1
> # strings /path-to/userid.conversations | grep d8087b76083ec5e7 | wc -l
> 183
> # strings /path-to/userid.conversations | grep Bd8087b76083ec5e7 | wc -l
> 91
> 
> ctl_conversationsdb -d  did a cleanup because I did abort the  
> "ctl_conversationsdb -b" process
> 
> skiplist recovery /path-to/userid.conversations: found partial txn,  
> not replaying
> skiplist: recovered /path-to/userid.conversations (0 records, 144  
> bytes) in 0 seconds
> skiplist: checkpointed /path-to/userid.conversations (0 records, 144  
> bytes) in 0.010 sec
> 
> 
> I did run ctl_conversationsd -b with strace and discovered that the inbox
> (cyrus.index O_RDWR, cyrus.header O_RDONLY, cache  cyrus.O_RDWR ) of that user
> was opened over and over again, no other folder for that user was accessed.
> For other users the inbox was only opened once and then the other  
> folders followed.
> So not stopping "ctl_conversationsd -b", it would have run till my  
> filesystem was
> full.
> 

I guess something about that inbox is confusing it, and making it redo it over 
and over?  Very curious

> > Bron, does this sound like anything you've seen before, that maybe  
> > got fixed on master but not backported to 3.0?
> >
> >> Cyrus Logs show many "ctl_conversationsdb[933]: mailbox: longlock
> >> user.UserID for 1.5 seconds"
> >> every few seconds.
> >
> > This message is logged when the lock is released, if that lock had  
> > been held for >1s.  It's reporting that it held the lock for longer  
> > than it would like to have, but at least you know the lock has been  
> > released.  I think this is telling you that user-visible performance  
> > on the mailbox would have been impacted while this was occurring  
> > (because imapd/lmtpd wouldn't be able to access their mailbox while  
> > ctl_conversationsdb was holding those locks), but also tells you  
> > that ctl_conversationsdb was doing the right thing, and not just  
> > holding a single lock for the entire job.
> >
> 
> I am not sure if any other processes had a chance to acquire a lock.

If they'd tried, they would have succeeded (by blocking until the lock was 
theirs, during the gaps where ctl_conver

Re: prblems rebuilding conversations db

[ellie timoney]

Hi Michael,

On Wed, Jan 16, 2019, at 10:12 PM, Michael Menge wrote:
> Hi,
> 
> because conversations db seems to be required for search indexes, I  
> enabled this option
> on our production servers today and tried to rebuild the conversations  
> db for all users with
> 
> ctl_conversationsdb -v -b UID
> 
> For most users this did take less than a second. But for some users  
> this process would
> not finish. I did kill one process after about 30 Minutes (most others  
> after 3 Minutes).
> The UserID.conversations has grown over 2 GB (the mailbox itself has  
> only ~700 MB of mails,
> and the conversations files from finnished rebuilds are less then 20 MB)

So, a 2GB-and-growing conversations db for a 700MB mailbox is a ratio of ~3:1, 
which seems obscene...

For comparison, the Conversations.append_reply_200 cassandane test (two 
messages with 100 replies each) produces an 87Kb conversations db for a 1016Kb 
mailbox (ratio ~1:12, n.b inverted!), which makes your ~3:1 and growing ratio 
seem even more obscene...

Obviously you can't share the bad conversations db because conversations db's 
are full of personally-identifying information.  But I wonder if looking 
through it with `cyr_dbtool ... show` turns up any unusual patterns, especially 
in comparison to one of the good users?  I wonder if it's somehow repeating 
itself?

Bron, does this sound like anything you've seen before, that maybe got fixed on 
master but not backported to 3.0?
 
> Cyrus Logs show many "ctl_conversationsdb[933]: mailbox: longlock  
> user.UserID for 1.5 seconds"
> every few seconds.

This message is logged when the lock is released, if that lock had been held 
for >1s.  It's reporting that it held the lock for longer than it would like to 
have, but at least you know the lock has been released.  I think this is 
telling you that user-visible performance on the mailbox would have been 
impacted while this was occurring (because imapd/lmtpd wouldn't be able to 
access their mailbox while ctl_conversationsdb was holding those locks), but 
also tells you that ctl_conversationsdb was doing the right thing, and not just 
holding a single lock for the entire job.

Though, that raises the question: was the user accessing the mailbox while the 
rebuild was in progress?  I wonder if their client is doing something funny and 
tripping things up?

Cheers,

ellie

> I did try reconstruct -r -G user/UserID, followed by   
> ctl_conversationsdb -v -z UID
> and tried to rebuild the db again. Reconstruct reported no errors, and  
> the new rebuild
> hat the same problem.
> 
> Has anybody seen a similar problem?
> 
> Any ideas how to fix this?
> 
> 
> We are running cyrus-imapd 3.0.8
> 
> 
> Kind regards
> 
> Michael Menge
> 
> 
> M.MengeTel.: (49) 7071/29-70316
> Universität Tübingen   Fax.: (49) 7071/29-5912
> Zentrum für Datenverarbeitung  mail:  
> michael.me...@zdv.uni-tuebingen.de
> Wächterstraße 76
> 72074 Tübingen

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Mailboxes with slow access

[ellie timoney]

Hi Miguel,

On Sat, Jan 19, 2019, at 8:20 AM, Miguel Mucio Santos Moreira wrote:
> Hi folks!
> 
> I've had a problem with some huge mailboxes, with a lot of folders and
> subfolders. Problably thousand of folders.> The access on these mailboxes is 
> too slow, if you change from INBOX to
> another folder called CYRUS, even if CYRUS folder has just 4 messages
> for example the access is slow.
This sounds like you don't have enough RAM to hold the metadata for
these mailboxes in memory, and so the system is swapping while
accessing them.
You could find the imapd process that is servicing this connection and
look at its virtual memory size, and compare that to the amount of
physical RAM in your system, to see whether this is the case.
Or, if you post up the size of these mailboxes' cyrus.header,
cyrus.index, cyrus.annotations, cyrus.cache (...) files and the size of
their [user].conversations file (if you have conversations enabled),
then maybe someone will be able to suggest a RAM size that would
support this.
You mention being in a murder environment -- I'm not sure if you will
need to increase the RAM on just the backend for these mailboxes, or the
frontends as well.  Maybe someone running a large murder can chime in
with how they provision their servers?
Sorry I don't have specific numbers for you, but hopefully this points
you in the right direction!
> I was trying to find out  anything on internet about this issue and I
> read that a parameter maxword on imapd.conf  could be the solution.> Nowadays 
> in our cyrus-imap murder all backend servers and
> frontend servers have the maxword parameter set to default value
> which is 128kb.> Based on that information, I'd like to know if I increase 
> this
> parameter to a high value like 1Mb for instance it could fix the issue
> about slow access and which considerations about memory, disk and cpu
> resources I must take care if I set up this value in my environment.> Another 
> question is, this value must be setting in all backends and
> frontends servers?
imapd.conf(5) says:
>   maxword: 131072
>   Maximum size of a single word for the parser.  Default
>   128k
This setting won't have any effect on performance at all.  It's just an
upper limit for the IMAP parser -- if some IMAP client tries to send a
single word that is longer than this, the imapd process serving their
connection will exit immediately with a "word too long" error rather
than trying to keep parsing.  Normal traffic won't hit this limit, it's
just protecting itself from (possibly malicious) garbage.  The default
should be never need changing really.
Hope this helps,

ellie

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Running a script with cyradm throwing ReadLine errors

[ellie timoney]

Hi Binarus,

> Then I have replaced the following code in Cyrus::IMAP::Shell

That's very interesting.  Does the same modified code continue to work if you 
uninstall Term::Readline::Gnu again?  That is to say, does the non-gnu version 
break with that addition, or continue to work?

> In other words, I just have made sure that this mysterious *__DATA__
> variable is reasonably defined in every case before _run is called.

I had a look in Shell.pm and found this comment near the top:

> # run(*FH|'FH')
> #   read commands from the filehandle and pass to exec(); defaults to
> #   __DATA__

So maybe that explains where the expectation for __DATA__ is coming from... so:

> # trivial; wrapper for _run with correct setup

I wonder if the "correct setup" is not correct enough!

> I have no idea why the "buggy" command line / argument parsing does not
> strike when Term::ReadLine::Gnu is uninstalled; I haven't grasped yet
> how *__DATA__ is supposed to be assigned a reasonable value to during
> the normal course of execution. I currently can only speculate that
> Term::ReadLine:: does this for us, while
> Term::ReadLine::Gnu doesn't.

I did a bit of reading, and apparently Term::ReadLine is a stub module that 
just loads "an implementation", which in your case wants to be 
Term::ReadLine::Gnu.  My guess is that, when you uninstall Term::ReadLine::Gnu, 
Term::ReadLine no longer successfully compiles because it's missing an 
implementation, and consequently the fallback code I pointed out previously is 
used instead.  So, from this I'm concluding that the "correct setup" from above 
is adequate for the Cyrus::IMAP::DummyReadline interface, but is not sufficient 
for a real ReadLine implementation.  Sounds like we've found our bug!

I'll have a bit of a play with it and see if I can find/fix the discrepancy 
between the interfaces :)

Cheers,

ellie

On Wed, Dec 19, 2018, at 5:00 AM, Binarus wrote:
> Dear ellie,
> 
> On 17.12.2018 23:57, ellie timoney wrote:
> > Hi Binarus,
> > 
> >> Could anybody please tell me what I might do wrong here?
> > 
> > This kind of smells like maybe your system has two versions of perl 
> > installed (or two versions of Term::ReadLine, or maybe even two versions of 
> > Cyrus::IMAP::Shell), and they're getting in each other's way?
> > 
> > I'm having a quick glance at the (2.5.10) source of Cyrus::IMAP::Shell and 
> > this caught my eye:
> > 
> >> # ugh.  ugh.  suck.  aieee.
> >> my $use_rl = 'Cyrus::IMAP::DummyReadline';
> >> {
> >>   if (eval { require Term::ReadLine; }) {
> >> $use_rl = 'Term::ReadLine';
> >>   }
> >> }
> 
> I have done some further investigations (very roughly because I don't
> have the time at the moment). It seems that the code which parses the
> command line and the run parameters in Cyrus::IMAP::Shell is buggy (or
> at least not prepared to handle Term::ReadLine::Gnu).
> 
> As a proof, I have reinstalled Term::ReadLine:Gnu and verified that the
> problem was showing again.
> 
> Then I have replaced the following code in Cyrus::IMAP::Shell
> 
> # trivial; wrapper for _run with correct setup
> sub run {
>   my $cyradm;
>   _run(\$cyradm, [*STDIN, *STDOUT, *STDERR], *__DATA__);
> }
> 
> by the following code:
> 
> # trivial; wrapper for _run with correct setup
> sub run {
>   my $cyradm;
> open(*__DATA__, "./000");
>   _run(\$cyradm, [*STDIN, *STDOUT, *STDERR], *__DATA__);
> }
> 
> In other words, I just have made sure that this mysterious *__DATA__
> variable is reasonably defined in every case before _run is called.
> 
> Now the command
> 
> perl -MCyrus::IMAP::Shell -e 'run("000")'
> 
> executed without any error message.
> 
> To verify that the script worked as intended, I added a few lines to it:
> 
> connect -noauthenticate localhost
> auth cyrus
> lm
> 
> When run as shown above, it did exactly what it was supposed to. It
> asked for the password and then listed all mailboxes and their subfolders.
> 
> So now I have at least a system where I can have Term::ReadLine::Gnu
> installed (and thus can have a history and command editing capabilities
> in cyradm) _and_ can execute a script, although the script's filename is
> hardcoded.
> 
> Probably it would be absolutely trivial for the authors of
> Cyrus::IMAP::Shell to fix this issue. It would be very nice if somebody
> could care about it. Perhaps it's already fixed in the newer versions? I
> am still on 2.5.10.
> 
> I have no idea why the "buggy" command line / argument parsing does not
> strike when Term::ReadLine::Gnu is uninstalled; I haven't grasped yet
> how *__DATA__ is sup

Re: Documentation for Cyrus::IMAP::Admin and friends

[ellie timoney]

Hi Binarus,

Looks like the documentation for the authenticate() function is pretty 
incomplete...

> Well, this man page "documents" the method by exactly one line 
> (identically in two places):
> 
> $client->authenticate;

>From my read of the documentation, this is enough.  It will figure out on its 
>own which SASL options are appropriate, and then prompt you (on the 
>controlling terminal) for the password if/as needed.  This nicely avoids 
>implicitly instructing users to hardcode their credentials in a script, so in 
>this sense it's probably good that it's underdocumented (maybe deliberately 
>so).

Looking in the source of Cyrus::IMAP, the accepted arguments are:

> -mechanism, -service, -authz, -user, -minssf, -maxssf, -password, -tlskey, 
> -notls, -cafile, -capath

If unspecified, "-service" defaults to "imap", "-maxssf" defaults to 1, 
"user" defaults to the current unix user, and the rest default to zero or 
blank.  

Note that "user" is the user to authenticate as (i.e. whose credentials are to 
be checked), whereas "authz" is who to authoriZe as (i.e. which identity to 
assume, once successfully authenticated).  authz only works for SASL mechanisms 
that allow proxy authentication... the imtest(1) man page says "e.g. PLAIN, 
DIGEST-MD5", I'm not sure if there are others.

> The same applies to CPAN: The modules cannot be found there.

The Cyrus::* perl modules are tightly coupled at build time to the installed 
cyrus version, so it doesn't make any sense to distribute them independently 
via CPAN.

> Where can I find reasonable (in the sense: may be short and bad, but 
> must be *complete*) documentation for Cyrus::IMAP::Admin and Cyrus::IMAP 
> so that I can write my own helper scripts in the future? Do I really 
> have to unpack Cyrus imapd's sources and read the module source code to 
> get some ideas?

Sorta, but you don't need the Cyrus imapd sources -- if you have the perl 
module, you have the perl module's source.  Look for the "Cyrus/IMAP.pm" file 
somewhere on your system (something like "sudo find / -name IMAP.pm" should do 
the trick) and there it is.

Cheers,

ellie

On Sun, Dec 16, 2018, at 9:55 PM, Binarus wrote:
> Dear all,
> 
> as described in my previous messages, I recently had a hard time 
> relocating (i.e. moving) mailboxes and the public namespace from 2.4.16 
> to a new server running 2.5.10. I would have happily written a Perl 
> script which surely had solved much of my problems, but I couldn't do 
> that due to the lack of documentation for the respective Perl modules.
> 
> This is best explained by example:
> 
> I have found a Perl script on the 'net which looks trustworthy at the 
> first sight and which contains (among others) the following lines of 
> code:
> 
> use Cyrus::IMAP::Admin;
> 
> # Connect to Cyrus
> $imap = Cyrus::IMAP::Admin->new("my_server") || die "Unable to connect 
> to my_server";
> 
> $imap->authenticate(-user => "foo", 
>   -mechanism => "LOGIN",
>   -password => "password",
>   );
> 
> I wanted to understand exactly what the script does and thus tried to 
> find Cyrus::IMAP::Admin's documentation. So I issued
> 
> man Cyrus::IMAP::Admin
> 
> and found that some methods are documented, but not the authenticate() 
> method which is needed first and one of the most important ones. 
> However, the man page states (in the section about the "new()" method):
> 
> "Instantiates a cyradm object.  This is in fact an Cyrus::IMAP object 
> with a few additional methods, so all Cyrus::IMAP methods are available 
> if needed.  (In particular, you will always want to use the 
> "authenticate" method.)"
> 
> This tricked me into believing that the "authenticate" method would be 
> part of Cyrus::IMAP and would be explained in that module's 
> documentation, so I issued
> 
> man Cyrus::IMAP
> 
> Well, this man page "documents" the method by exactly one line 
> (identically in two places):
> 
> $client->authenticate;
> 
> It does not explain anything about it; notably, it does not mention any 
> parameters. But from the example script mentioned above, I knew that 
> there was more to it. So I read the rest of that man page and found:
> 
> "The Cyrus::IMAP module provides an interface to the Cyrus imclient library."
> 
> So I installed the development files for Cyrus imapd and issued
> 
> man imclient
> 
> Now, finally, this is a man page a C programmer probably can live with. 
> But when looking more thoroughly into it, I saw that there is no 
> "password" parameter to the authenticate() function although the Perl 
> module's authenticate() method has one:
> 
> int imclient_authenticate (struct imclient *imclient, struct sasl_client 
> **availmech, const char *service, const char *user, int protallowed);
> 
> Plus, I could not find any hints regarding the relationship between the 
> parameters of the C functions and those of the Perl module methods.
> 
> I then headed over to cyrusimap.org and tried to find 

Re: Running a script with cyradm throwing ReadLine errors

[ellie timoney]

Hi Binarus,

> Could anybody please tell me what I might do wrong here?

This kind of smells like maybe your system has two versions of perl installed 
(or two versions of Term::ReadLine, or maybe even two versions of 
Cyrus::IMAP::Shell), and they're getting in each other's way?

I'm having a quick glance at the (2.5.10) source of Cyrus::IMAP::Shell and this 
caught my eye:

> # ugh.  ugh.  suck.  aieee.
> my $use_rl = 'Cyrus::IMAP::DummyReadline';
> {
>   if (eval { require Term::ReadLine; }) {
> $use_rl = 'Term::ReadLine';
>   }
> }

Which... fills me with confidence.  Looks like a workaround for missing 
(broken?) Term::Readline but that comment isn't super enlightening.  I wonder 
if it will Just Work if you uninstall Term::Readline?

I haven't really used cyradm at all myself, so take all this with a grain of 
salt.  Hopefully someone who has can chime in!

Cheers,

ellie

On Sun, Dec 16, 2018, at 8:04 PM, Binarus wrote:
> Dear all,
> 
> I was just trying to explore cyradm a little bit further and hence was 
> experimenting with its scripting capabilities. Having cyradm run a 
> script should be pretty easy. man cyradm tells us:
> 
>   perl -MCyrus::IMAP::Shell -e 'run("myscript")'
> 
> So I created the simplest possible script (that means an empty one) and 
> tried to run it:
> 
>   touch 000
>   chmod a+x 000 (just in case ...)
>   perl -MCyrus::IMAP::Shell -e 'run("000")'
> 
> The only thing I got was:
> 
>   Use of uninitialized value within @layers in string eq at /usr/local/
> lib/x86_64-linux-gnu/perl/5.24.1/Term/ReadLine/Gnu.pm line 280.
>   Bad filehandle: __DATA__ at /usr/local/lib/x86_64-linux-gnu/perl/
> 5.24.1/Term/ReadLine/Gnu.pm line 769.
> 
> Putting something meaningful into the script did not change the situation.
> 
> I have googled and read documentation (mainly on cyrusimapd.org) for 
> several hours, but could not find the reason for the problem.
> 
> I even have put allowplaintext=yes into imapd.conf and restarted imapd 
> (knowing that this probably wasn't very smart, but the term "layers" in 
> the error message made me mistrustful because there are authentication 
> "layers", and I don't have any problems with Term::ReadLine::Gnu in 
> general). As expected, this didn't change the situation either.
> 
> This happened with 2.4.16 as well as with 2.5.10.
> 
> Could anybody please tell me what I might do wrong here?
> 
> Thank you very much in advance,
> 
> Binarus
> 
> 
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Why do we need to run reconstruct after having moved mailboxes using cyradm's xfer command?

[ellie timoney]

Hi Binarus,

> Could anybody please shortly explain why? What exactly are the 
> techniques and mechanisms cyradm's xfer uses to do its thing? I had been 
> quite sure that it just uses the IMAP protocol, but there seems to be 
> more to it ...

You're right, but missing a detail.  The XFER command is an IMAP extension, for 
use in cyrus clusters ("murders"), where mailboxes need to be identical between 
servers.  I don't know much about the details, but it makes sense to me that it 
would copy the mailboxes across at a "deeper" level, to make sure internal 
identifiers and such match.

cyradm uses the IMAP protocol – think of it as a command-driven IMAP client, 
except that it knows about cyrus extensions (like XFER) that generic IMAP 
clients (like imapsync) probably don't

Something like imapsync would probably be using the CREATE  and APPEND 
 commands instead, which let the destination server choose its own 
internal identifiers, and which will probably be different from the source 
ones.  But they would be stored in the exact method the destination server 
prefers (such as using the newest version of the mailbox index format...)

> Now that I have used cyradm's xfer command to relocate the mailboxes / 
> messages / folders / subfolders, I surprisingly had to run the "heaviest" 
> form of reconstruct (-V max).

That's not the "heaviest" form of reconstruct, it's just updating the mailbox 
indexes to the "max"imum (-V)ersion supported by the server.  This ends up 
adding a bunch of extra information to the indexes that the newer server 
version can make use of.  Generally if the index version is too old for what 
it's trying to do, it'll fall back to calculating missing values the long way, 
or provide less info, or just refuse the requested operation.  The assertion 
failure you saw is clearly a bug, but this smells familiar, so maybe it's 
already been fixed.  The latest version of the 2.5 release is 2.5.12, and is 
nearly 2 years newer than 2.5.10, for what it's worth.

Hope this helps!

ellie

On Sun, Dec 16, 2018, at 9:06 PM, Binarus wrote:
> Dear all,
> 
> yesterday, I have moved a bunch of user mailboxes and public folders 
> from a server running 2.4.16 to another server running 2.5.10, using 
> cyradm's xfer command (see my previous messages).
> 
> After having finished the migration, I noticed a weird behavior in 
> Thunderbird (which is our standard email client): When trying to move a 
> message from the Inbox to the Junk or Trash folder, the message 
> disappeared from the Inbox for a short time, then reappeared. The log 
> files on the server were showing the dreaded entries:
> 
> Dec 16 01:12:59 morn cyrus/imaps[14914]: Fatal error: Internal error: 
> assertion failed: imap/mailbox.c: 2850: !message_guid_isnull(
> >guid)
> 
> "Fortunately", a lot of other users are affected by such log entries and 
> weird email client behavior as well, so finding the solution on the 'net 
> was not too difficult: Running reconstruct did not lead to anywhere, but 
> running reconstruct -V max was the solution.
> 
> This lets me scratch my head:
> 
> In the past, I have upgraded Cyrus imapd at least three times, each time 
> using imapsync (instead of cyradm's xfer command) to move the mailboxes 
> and the public namespace (including all messages and subfolders) from 
> the old server to the new one. In none of these cases, it was necessary 
> to reconstruct anything afterwards. This would have been illogical 
> anyway: Each time, the new server had been setup from scratch, and no 
> mailboxes / messages / folders / subfolders had been moved directly via 
> file transfer from the old to the new server.
> 
> Now that I have used cyradm's xfer command to relocate the mailboxes / 
> messages / folders / subfolders, I surprisingly had to run the 
> "heaviest" form of reconstruct (-V max).
> 
> Could anybody please shortly explain why? What exactly are the 
> techniques and mechanisms cyradm's xfer uses to do its thing? I had been 
> quite sure that it just uses the IMAP protocol, but there seems to be 
> more to it ...
> 
> Thank you very much for any insight,
> 
> Binarus
> 
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Question for upgrading

[ellie timoney]

> Would be fine if Bron, Ellie or someone at Fastmail could tell
> something about it to us :) :)
Cheeky!  Thing is, at FM our production servers more or less track the
master branch, so we were running "3.0" from the moment 2.5 forked off,
and so our upgrade process only ever really needs to deal with iterative
changes.  Our accumulated experience is what informs the "upgrade notes"
included with the 2.5.0 and 3.0.0 releases (not sure about older
versions, I wasn't around then).
This mailing list is the right place to find people with real experience
doing big-bang upgrades. :)
Cheers,

ellie

On Mon, Dec 17, 2018, at 8:40 PM, Egoitz Aurrekoetxea wrote:
> Hi mate,


> 


> I think (and say think :) )I finally found a method. Although I'm
> testing it deeply... it seems (say seems too :) ) 2.4 is compatible
> with a mail spool in 2.3 (at least with my config). So I'll try to
> upgrade first to 2.4 and later to 3.0 setting up a replication from
> 2.4 to 3.0. Would be fine if Bron, Ellie or someone at Fastmail could
> tell something about it to us :) :)> 


> Cheers!


> ---
> 
> sarenet
> *Egoitz Aurrekoetxea*
> Departamento de sistemas
> 944 209 470
> Parque Tecnológico. Edificio 103
> 48170 Zamudio (Bizkaia)
> ego...@sarenet.es
> www.sarenet.es
> 
> Antes de imprimir este correo electrónico piense si es necesario
> hacerlo.> 


> El 16-12-2018 22:24, John Capo escribió:


>> On Thu, December 13, 2018 13:25, Egoitz Aurrekoetxea wrote: 
>>> Hi again!
>>> 
>>> 
>>>  Else as a simplication way can you replicate any manner a 2.3
>>>  with some>>>  newer version?. At least in manual mode (not rolling)?.
>> 
>> The replication protocol in 2.4 is not compatible wit 2.3.
>> 
>>  There is no easy way.  I'm rsync'ing one account at a time between
>>  2.3 and 2.4 with IMAP/POP and LMTP access disabled, followed by a
>>  reconstruct, and then enabling IMAP/POP and LMTP access.>> 
>>  Many terrabytes to go.
>> 
>>  You could also look at imapsync but its slow.
>> 
>> 
>>> 
>>> Cheers.
>>> 
>>> 
>>>  ---
>>> 
>>> 
>>>  EGOITZ AURREKOETXEA
>>>  Departamento de sistemas
>>>  944 209 470
>>>  Parque Tecnológico. Edificio 103
>>>  48170 Zamudio (Bizkaia)
>>> ego...@sarenet.es www.sarenet.es [1[1]] Antes de imprimir este
>>> correo electrónico piense si es>>>  necesario hacerlo.
>>> 
>>>  El 13-12-2018 16:52, Egoitz Aurrekoetxea escribió:
>>> 
>>> 
>>> 
 Good afternoon,
 
 
  I was trying to upgrade part of our Cyrus imap installation,
  concretely that one consisting in  still 2.3. I was planning to set 
 up Cyrus 3.0. I have seen all
  works properly except for the  unexpunge command because as someone 
 stated here, a reconstruct -V
  max was needed.The problem  is that this reconstruct command, takes 
 ages and I'm not able to
  keep the service offline so  many time. So I have been thinking in 
 the following scenario :
 
  - Cyrus 2.3 master -> Cyrus 2.4 slave
 
 
  Get this 2.4 slave ready and set it as master. But here comes my
  first doubt. Does the 2.4  replication work with the 2.3 
 replication?. Can in this pair, both
  (the 2.3 and the 2.4) be  both master and slave?. I mean to switch 
 roles in the pair. Make
  one become master and the  other slave and  vice versa?.
 
  Let's think now Cyrus 2.4 is ready and working.
 
 
  - Now, I would set up a new 3.0 slave. I know 2.4 could replicate
with 3.0. So I would get the  3. ready and then set 3.0 as master. 
 Can in this pair both the 2.4
 and 3.0 be master and  slave?. Meaning again to the same role 
 switching commented
  before... to make one to be master  and the other slave or vice 
 versa
 
  I'l will end up with 2 3.0 master and slave... but I need to trace
  the path... 
 
  Does anyone see any other way?.
 
 
  Best regards,
 
 
  -
 
 
  --
 
 
  EGOITZ AURREKOETXEA
  Departamento de sistemas
  944 209 470
  Parque Tecnológico. Edificio 103
  48170 Zamudio (Bizkaia)
 ego...@sarenet.es www.sarenet.es [1[2]] Antes de imprimir este
 correo electrónico piense si es  necesario hacerlo. 
  Cyrus Home Page: http://www.cyrusimap.org/
  List Archives/Info:
  http://lists.andrew.cmu.edu/pipermail/info-cyrus/  To Unsubscribe:
 https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
>>> 
>>> 
>>> Links:
>>>  --
>>>  [1] http://www.sarenet.es


Links:

  1. http://www.sarenet.es
  2. http://www.sarenet.es

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: SASL 2.1.27

[ellie timoney]

Thanks Sergey, these have been corrected and should update automatically in the 
next 15 minutes or so :)

On Sun, Nov 25, 2018, at 11:14 PM, Sergey wrote:
> On Tuesday 20 November 2018, Ken Murchison wrote:
> 
> > I'm pleased to announce the release of the long-awaited SASL 2.1.27
> > which can be downloaded from here: 
> 
> Thanks. But I have one question and one note.
> 
> https://github.com/cyrusimap/cyrus-sasl is not updated as I see,
> or the source tree have not release tag. Is it planned?
> 
> https://www.cyrusimap.org/sasl/ contains the string "The latest 
> stable version of Cyrus SASL is 2.1.26".
> 
> -- 
> Regards,
> Sergey
> 
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Debugging A5 BAD Unexpected extra arguments to Search ?

[ellie timoney]

Hi Kristian,

The "A5" and "A7" are just the "tags" associated with the client's search 
commands, they have no semantic value.  The client prefixes each command with 
some tag, and then the server uses the same tag in the response (so that you 
can identify which response applies to which command).  In these two cases, 
your client has chosen "A5" and "A7" as tags.  Tags are described in detail 
here: https://tools.ietf.org/html/rfc3501#section-2.2.1

Your Java/Java Mail application hopefully has a feature to log the commands 
it's sending.  If you turn this on, you'll be able to see which bad search 
commands were resulting in the error (by matching response tags and command 
tags), and then compare the command syntax to the expected syntax from the 
relevant RFCs.

Cheers,

ellie

On Tue, Oct 23, 2018, at 8:58 PM, Kristian Rink wrote:
> Hi all;
> 
> we're running a Java (and Java Mail) based application interacting with 
> our cyrus imapd server. Trying to use complex search terms (searching 
> for multiple criterias, such as message-id and message date), we 
> regularly (all the time?) end up with messages such as
> 
> A5 BAD Unexpected extra arguments to Searchor
> A7 BAD Unexpected extra arguments to Search
> 
> Questions here: What's the difference between A5 and A7? Any way to 
> debug what exactly is wrong about our search terms here?
> 
> TIA and all the best,
> Kristian

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Cyrus IMAP 2.5.12 released

[ellie timoney]

The Cyrus team is proud to announce the immediate availability of a new version 
of Cyrus IMAP: 2.5.12

I'm trialling hosting the release files using Github's releases feature.  
Please use the Github download links if possible, and advise if you have any 
problems!  (It may even download faster due to Github's content delivery 
network.)

Download URLs:


https://github.com/cyrusimap/cyrus-imapd/releases/download/cyrus-imapd-2.5.12/cyrus-imapd-2.5.12.tar.gz

https://github.com/cyrusimap/cyrus-imapd/releases/download/cyrus-imapd-2.5.12/cyrus-imapd-2.5.12.tar.gz.sig

https://www.cyrusimap.org/releases/cyrus-imapd-2.5.12.tar.gz
https://www.cyrusimap.org/releases/cyrus-imapd-2.5.12.tar.gz.sig

ftp://ftp.cyrusimap.org/cyrus-imapd/cyrus-imapd-2.5.12.tar.gz
ftp://ftp.cyrusimap.org/cyrus-imapd/cyrus-imapd-2.5.12.tar.gz.sig

Please consult the release notes before upgrading to 2.5.12:

https://www.cyrusimap.org/imap/download/release-notes/2.5/x/2.5.12.html

And join us on Github at https://github.com/cyrusimap/cyrus-imapd to report 
issues, join in the deliberations of new features for the next Cyrus IMAP 
release, and to contribute to the documentation.

On behalf of the Cyrus team,

Kind regards,

ellie timoney

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Slow start of imap service

[ellie timoney]

On Mon, Oct 1, 2018, at 12:32 PM, ellie timoney wrote:
> You could use the tls_sessions_db_path imapd.conf(5) option to put this 
> database onto faster storage?
> 
> >  tls_sessions_db_path: 
> >  The absolute path to the TLS sessions db file. If not 
> > specified, will  be
> >  configdirectory/tls_sessions.db
> 
> If you have the RAM for it, you should be able to put tls_sessions.db on 
> a tmpfs filesystem.  This database is only a cache, so nothing valuable 
> will be lost if the machine is rebooted; and as a cache, it benefits 
> from being on the fastest storage you have available. :)

Buuut, note that there's a bug in current releases of 2.5 where tls_prune will 
fail if the tls_sessions.db doesn't exist, preventing the server starting up.  
This will occur after ever reboot if you put this database on ephemeral 
storage!  You can work around this by having your service init script touch the 
file before running master.

The real fix for this is already in git, so it will be included in 2.5.12, 
which will hopefully be out this week!

Cheers,

ellie

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Slow start of imap service

[ellie timoney]

On Sat, Sep 29, 2018, at 8:59 AM, Paul van der Vlis wrote:
> Op 28-09-18 om 15:34 schreef Michael Menge:
> > 
> > Quoting Paul van der Vlis :
> > 
> >> Hello,
> >>
> >> I am using Cyrus-imapd from Debian stable (2.5.10-3), and starting up
> >> takes very long. I see processes starting, but no imapd.
> >>
> >> In most cases I restart Cyrus more then ones before it works. Not sure I
> >> have to wait longer, or restarting after some time helps.
> >>
> >> This problem occurs on only one machine, on two other less busy machine
> >> with the same Cyrus I don't have problems.
> >>
> >> Maybe somebody here knows more about what could be wrong? Or how to
> >> debug this?
> >>
> > 
> > 
> > What is cyrus logging to your logfiles when you restart?
> 
> In my crontab I have this line:
> 00 4 * * * root /usr/sbin/service cyrus-imapd restart
> 
> First I see many of this lines in /var/log/mail.log:
> Sep 25 04:00:01 sigmund cyrus/imap[21598]: graceful shutdown
> 
> Then I see this between those lines this:
> -
> Sep 25 04:00:02 sigmund cyrus/idled[5844]: graceful shutdown initiated
> by unexpected process 5838 (/usr/sbin/cyrmaster -l 32 -C /etc/imapd.conf
> -M /etc/cyrus.conf)
> Sep 25 04:00:02 sigmund cyrus/imaps[16434]: IDLE: error sending message
> DONE to idled for mailbox user.nospam.Junk: Connection refused.
> -
> 
> This line:
> Sep 25 04:00:02 sigmund cyrus/master[5838]: process type:SERVICE
> name:notify path:/usr/lib/cyrus/bin/notifyd age:85080.426s pid:6024
> exited, status 75
> 
> Many of these lines:
> Sep 25 04:00:02 sigmund cyrus/master[5838]: process type:SERVICE
> name:imap path:/usr/lib/cyrus/bin/imapd age:85073.234s pid:6027 exited,
> status 75
> 
> Then this:
> 
> Sep 25 04:00:05 sigmund cyrus/ctl_cyrusdb[21829]: skiplist: clean
> shutdown file missing, updating recovery stamp
> Sep 25 04:00:05 sigmund cyrus/ctl_cyrusdb[21829]: recovering cyrus databases
> Sep 25 04:00:05 sigmund cyrus/ctl_cyrusdb[21829]: done recovering cyrus
> databases
> Sep 25 04:00:05 sigmund cyrus/cyr_expire[21834]: skiplist: recovered
> /var/lib/cyrus/deliver.db (9290 records, 1759220 bytes) in 0 seconds
> Sep 25 04:00:05 sigmund cyrus/cyr_expire[21834]: skiplist: checkpointed
> /var/lib/cyrus/deliver.db (9290 records, 1412288 bytes) in 0.227 sec
> Sep 25 04:00:19 sigmund cyrus/cyr_expire[21834]: Expired 0 and expunged
> 0 out of 1312483 messages from 2984 mailboxes
> Sep 25 04:00:19 sigmund cyrus/cyr_expire[21834]: duplicate_prune:
> pruning back 3.00 days
> Sep 25 04:00:30 sigmund cyrus/cyr_expire[21834]: skiplist: longlock
> /var/lib/cyrus/deliver.db for 1.8 seconds
> Sep 25 04:00:33 sigmund cyrus/cyr_expire[21834]: skiplist: longlock
> /var/lib/cyrus/deliver.db for 2.2 seconds
> Sep 25 04:00:39 sigmund cyrus/cyr_expire[21834]: skiplist: longlock
> /var/lib/cyrus/deliver.db for 1.3 seconds
> Sep 25 04:05:36 sigmund cyrus/cyr_expire[21834]: duplicate_prune: purged
> 2217 out of 9290 entries
> Sep 25 04:05:36 sigmund cyrus/tls_prune[21860]: skiplist: recovered
> /var/lib/cyrus/tls_sessions.db (10219 records, 2235748 bytes) in 0 seconds
> Sep 25 04:05:36 sigmund cyrus/tls_prune[21860]: skiplist: checkpointed
> /var/lib/cyrus/tls_sessions.db (10219 records, 2147768 bytes) in 0.308 sec
> Sep 25 04:09:47 sigmund cyrus/tls_prune[21860]: skiplist: longlock
> /var/lib/cyrus/tls_sessions.db for 1.4 seconds
> Sep 25 04:10:23 sigmund cyrus/tls_prune[21860]: skiplist: longlock
> /var/lib/cyrus/tls_sessions.db for 2.2 seconds
> Sep 25 04:10:45 sigmund cyrus/tls_prune[21860]: skiplist: longlock
> /var/lib/cyrus/tls_sessions.db for 2.2 seconds
> Sep 25 04:12:21 sigmund cyrus/tls_prune[21860]: skiplist: longlock
> /var/lib/cyrus/tls_sessions.db for 1.3 seconds
> Sep 25 04:12:47 sigmund cyrus/tls_prune[21860]: skiplist: longlock
> /var/lib/cyrus/tls_sessions.db for 1.0 seconds
> Sep 25 04:12:49 sigmund cyrus/tls_prune[21860]: skiplist: longlock
> /var/lib/cyrus/tls_sessions.db for 1.8 seconds
> Sep 25 04:17:33 sigmund cyrus/tls_prune[21860]: skiplist: longlock
> /var/lib/cyrus/tls_sessions.db for 1.0 seconds
> Sep 25 04:23:11 sigmund cyrus/tls_prune[21860]: skiplist: longlock
> /var/lib/cyrus/tls_sessions.db for 1.0 seconds
> Sep 25 04:25:31 sigmund cyrus/tls_prune[21860]: tls_prune: purged 4463
> out of 10219 entries
> Sep 25 04:25:31 sigmund cyrus/master[21826]: unable to
> setsocketopt(IP_TOS) service lmtpunix/unix: Operation not supported
> Sep 25 04:25:31 sigmund cyrus/master[21826]: unable to
> setsocketopt(IP_TOS) service notify/unix: Operation not supported
> Sep 25 04:25:31 sigmund cyrus/ctl_cyrusdb[22345]: checkpointing cyrus
> databases
> Sep 25 04:25:31 sigmund cyrus/ctl_cyrusdb[22345]: done checkpointing
> cyrus databases
> Sep 25 04:25:32 sigmund cyrus/imaps[22349]: inittls: Loading hard-coded
> DH parameters
> Sep 25 04:25:33 sigmund cyrus/imaps[22349]: starttls: TLSv1.2 with
> cipher ECDHE-RSA-AES128-SHA (128/128 bits new) no authentication
> Sep 25 04:26:20 sigmund cyrus/imap[22362]: inittls: Loading