Re: Did calculating the quota change from 2.3 to 2.5?
On 31/12/16 06:17, Bron Gondwana via Info-cyrus wrote: > If your cyrus.* files are identical then you have pretty weird mailboxes, > but yeah - I guess it could happen if you had two folders with identical > messages in identical order and all the timestamps identical. Especially empty mailbox cyrus.cache/squat and most cyrus.annotation (2.5) files got linked together on my testhost. I use freedup with the patch sent in https://mid.mail-archive.com/info-cyrus@lists.andrew.cmu.edu/msg42850.html on regular basis without harm. Meanwhile the patch is included in the most recent 1.6-3 release of freedup. Greetings, Wolfgang -- Wolfgang Breyha| http://www.blafasel.at/ Vienna University Computer Center | Austria Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: Did calculating the quota change from 2.3 to 2.5?
On 29/11/16 22:37, Jason L Tibbitts III via Info-cyrus wrote: > Fun random question: Does anything blow up if you run hardlink on your > mail spool? (The hardlink program finds identical files and hardlinks > them.) Using "hardlink" is IMO not save on imap spools since it also links cyrus.* files what's definitely not what you want. I recommend using freedup instead. Something like freedup -n -v -a -T -o '-name "*."' -l /var/spool/.. Greetings, Wolfgang -- Wolfgang Breyha| http://www.blafasel.at/ Vienna University Computer Center | Austria Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: mailboxes.db locking problem after updating from 2.4 to 2.5.9
On 18/11/16 01:07, Bron Gondwana via Info-cyrus wrote: > On Fri, 18 Nov 2016, at 10:51, Wolfgang Breyha via Info-cyrus wrote: >> I already filed a bug https://github.com/cyrusimap/cyrus-imapd/issues/43 >> but no response so far. I directly asked Bron, but no response as well. > > Sorry, I really don't have a clue. 2.5 does have a different mailboxes.db > format, so it's a bit more CPU intensive. The real massive win for CPU > usage is going to come with reverse ACLs: Thanks for the response in the first place! I'm sorry to push this topic continuously because I appreciate all your hard work (and Ellies as well) on cyrus very much! We had a quite hard time keeping our infrastructure operational after doing the final step upgrading our frontends to 2.5. Seeing others having the same stress should be quite a warning for people thinking about an upgrade. Since Deniss has these troubles with skiplist as well on 2.5 it looks like the size triggers it. twoskip seems to perform even worse if mailboxes.db reaches a certain size, or it's only the size penalty which hits twoskip earlier. 150-200 MB seems critical for locking issues. > https://blog.fastmail.com/2015/12/05/reverse-acls-making-imap-list-fast/ I read about them some time ago in the 3.0.0-beta changelog. Hopefully we're on a stable cyrus release supporting it before reaching the critical size for skiplist;-) > But to get there, we need to solve reverse ACLs for groups. I did ask about > it here: > > https://lists.andrew.cmu.edu/pipermail/info-cyrus/2015-November/038628.html Sorry, we don't use groups. > But then didn't follow up to add group reverse ACL support in Cyrus, so > reverse ACLs are broken if you're using groups. So, 2.5 wont get them anyway I guess;-) Greetings, Wolfgang -- Wolfgang Breyha <wbre...@gmx.net> | http://www.blafasel.at/ Vienna University Computer Center | Austria Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: mailboxes.db locking problem after updating from 2.4 to 2.5.9
On 17/11/16 14:00, Deniss via Info-cyrus wrote: > Hello, > > I trying to migrate one big cyrus imap server from 2.4 to 2.5.9. > > I updated binaries, fix db backend in imapd.conf and converted > mailboxes.db with ctl_mboxlist -d & -u to twoskip. > > cyrus ran fine until morning when a count of simultanious sessions > started to rise. Sounds familiar;-) > I converted mailboxes.db back to skiplist and got a bunch (10-20 per > sec) of following logs: I did the same. twoskip was and is useless on my frontends. With skiplist my loads dropped at least down to 2-5. After doubling our vCPUs it runs quite well. Do you use roundcube by any chance? If yes update to 2.5.10 and do not use $config['imap_disabled_caps'] = array('LIST-EXTENDED'); Otherwise roundcube sends two 'LIST "" "*"' on each login and triggers two full table scans on your mailboxes.db > 180M /var/imap/mailboxes.db.skiplist.24 > 215M /var/imap/mailboxes.db.skiplist.25 > 262M /var/imap/mailboxes.db.twoskip.25 Wow, larger then ours. We've ~140MB on skiplist.25. > Any ideas or suggestion for investigation ? I already filed a bug https://github.com/cyrusimap/cyrus-imapd/issues/43 but no response so far. I directly asked Bron, but no response as well. Greetings, Wolfgang -- Wolfgang Breyha| http://www.blafasel.at/ Vienna University Computer Center | Austria Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: Bugs in cyrus murder 2.5
Michael Menge via Info-cyrus wrote on 15/11/16 17:08: > Thanks for the info, we are also still running cyrus 2.4 on our > production environment. > But mostly because that was the stable version at the time we migrated > from stand alone > to murder setup and the "never touch a running system" mentality. I > was not aware that > there was decreased stability for murder setup in the cyrus 2.5 version. For 2.5.10 it's primarily not stability. It's functionality. > Do you have some specific bugs in mind? The major ones are: If you use XFERs between backends... they are still completely broken: https://github.com/cyrusimap/cyrus-imapd/issues/46 This one may still hurt if you use shared mailboxes (on different backends): https://github.com/cyrusimap/cyrus-imapd/issues/41 And last but not least .. performance issues on frontends (compared to 2.4): https://github.com/cyrusimap/cyrus-imapd/issues/43 Greetings, Wolfgang -- Wolfgang Breyha| http://www.blafasel.at/ Vienna University Computer Center | Austria Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: command line deletion of files
Vladislav Kurz via Info-cyrus wrote on 29/09/16 17:04: > ipurge is nice but I would really appreciate if it had a --dry-run > option. I'm never sure what it will really delete. It got one in 2.5.9. Greetings, Wolfgang -- Wolfgang Breyha| http://www.blafasel.at/ Vienna University Computer Center | Austria Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Huge performance problems after updating from 2.4 to 2.5.9
Hi! A can add another story of that type, but with different setup: We already migrated to 2.5.7 on our ten backends some month ago step by step and upgraded to 2.5.9 lately. We never had any performance issues on them. All of them have done a full "reconstruct -V max" and special-use metadata is set to step in for static xlist missing in 2.5 frontends. Yesterday I did the final step by updating our mupdate server and the murder frontends. No problems on the mupdate server so far. The configs are mostly the same like for 2.4 (except for the changed names). The only key change done was from skiplist to twoskip for all databases: annotation_db: twoskip duplicate_db: twoskip ptscache_db: twoskip mboxlist_db: twoskip seenstate_db: twoskip statuscache_db: twoskip subscription_db: flat tls_sessions_db: twoskip userdeny_db: flat All of them are placed in /var/spool/imap/config except statuscache and tlscache which reside in /dev/shm. After migrating the frontends yesterday evening I already recognized a higher load and decided to keep an eye on it. Our 3 frontends are: CentOS 6 (vmware based) 16GB RAM 4 CPUs They usually have ~4000-4500 concurrent imap(s) connections at peak time and had a load ~1-3 with cyrus 2.4. swap was not used. Today in the morning we had the same connection count with load ~100-250 with cyrus 2.5.9 and twoskip. no swap used. top shows a huge amount of imap processes in running state. Another thing I recognized was that changes in mailboxes.db on backends reached the mupdate server, but didn't come through to the frontends anymore even after 15 minutes. I changed mailboxes.db and tlscache back to skiplist and load went down to 5-15. So, much better but still 5 times as high as with 2.4 and still causing headaches. Currently I have no idea what causes the load besides twoskip (which I can live without since skiplist never caused us any troubles). I compared "USAGE" loglines from two days (2.4) ago to 2.5 with skiplist and twoskip. 2.4 : user: 0.384322, sys: 0.029056 (23538 USAGE lines, 09:00-10:00) 2.4 : user: 0.353515, sys: 0.030814 (25790 USAGE lines, 13:00-14:00) 2.5 ts: user: 0.903491, sys: 0.028196 (23429 ..., 08:00-09:00) 2.5 ts: user: 1.130391, sys: 0.032894 (27709 ..., 09:00-10:00) 2.5 sl: user: 0.864270, sys: 0.026580 (29567 ..., 13:00-14:00) 2.5 sl: user: 1.077462, sys: 0.032952 (25879 ..., 14:00-15:00) Comparing all three I wonder why skiplist makes such a big difference. lmtpd shows now relevant difference for all three. Most likely I will go for horizontal scaling putting a 4th frontend in the line. Greetings, Wolfgang -- Wolfgang Breyha| http://www.blafasel.at/ Vienna University Computer Center | Austria Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
2.5.9 build issues
After modifying my spec file from 2.5.8 to 2.5.9 I ended up with two new issues: *) jcal.c can't be compiled with jansson < 2.7 (eg. every RHEL) because json_boolean_value() is missing. I fixed it with: - --- cyrus-imapd-2.5.9/imap/jcal.h.orig 2016-08-20 16:20:52.049359274 +0200 +++ cyrus-imapd-2.5.9/imap/jcal.h 2016-08-20 16:21:03.719723892 +0200 @@ -50,6 +50,10 @@ #include "util.h" +#if (JANSSON_MINOR_VERSION < 7) +#define json_boolean_value json_is_true +#endif + extern char *icalcomponent_as_jcal_string(icalcomponent* comp); extern icalcomponent *jcal_string_as_icalcomponent(const char *str); extern const char *begin_jcal(struct buf *buf); *) htmlstrip.c gets installed as source to /usr/lib/cyrus-imapd. It seems it doesn't get compiled at all and the source gets installed instead. Greetings, Wolfgang -- Wolfgang Breyha| http://www.blafasel.at/ Vienna University Computer Center | Austria Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: squat broken after upgrade from 2.4 to 2.5?
Bron Gondwana via Info-cyrus wrote on 21/06/16 01:52: > This could very well be a bug in the upgrade :( It would be great to have > a test case to check this. I can't do that today, but maybe ellie can do > something. > > Sadly our bug tracking situtation for Cyrus is a mess at the moment, but > I'm hoping to have that cleaned up soon. I filed https://bugzilla.cyrusimap.org/show_bug.cgi?id=3938 Greetings, Wolfgang -- Wolfgang Breyha| http://www.blafasel.at/ Vienna University Computer Center | Austria Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
squat broken after upgrade from 2.4 to 2.5?
Hi! I upgraded some backend hosts from 2.4.18 to 2.5.8. Currently they are all still running with index version 12. Some of these mailboxes have squatter activated as annotation and my cyrus.conf has squatter cmd="squatter -asir *" at=0500 I see that it runs at night and logs entries like: Jun 20 05:00:15 lauren squatter[27593]: indexing mailbox user. There are no errors in the log from squatter. But today I recognized on my own mailbox that squatter returns nothing while searching with thunderbird. The resulting imap request looked like: <1466414132<18 UID Search UNDELETED (OR (OR OR TO "search" HEADER CC "search" SUBJECT "search") BODY "search") <1466414132>* SEARCH 18 OK Completed (0 msgs in 0.010 secs) I then moved the current squat db and the same search returned 214 messages. Rebuiling the squat db and searching again resulted in the exact same 214 messages. It seems that something changed within the squat DB and 2.5 can't search in the old 2.4 DBs. That's bad in two ways. *) search doesn't fail with an error and simply returns no result. no fallback to searching without squat db. *) the squat db is not refreshed/rebuilt after upgrade. Is this a bug or is something broken on my side? Greetings, Wolfgang -- Wolfgang Breyha| http://www.blafasel.at/ Vienna University Computer Center | Austria Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: idled multiplying and consuming the server
Eugene M. Zheganin via Info-cyrus wrote on 10/06/16 16:58: > idled cmd="idled" listen="/var/imap/socket/idled" proto="udp" prefork=0 That looks like an entry in the SERVICE section. That's wrong for idled. idled should be added at the end of the START section like START { idled cmd="idled" } Greetings, Wolfgang -- Wolfgang Breyha| http://www.blafasel.at/ Vienna University Computer Center | Austria Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
prefork and IPv6
Hi! I recently wondered why some of my preforked processes on my murder backends never get used. I detected them because some quite old lmtpd's were holding locks on an already deleted deliver.db. After some debugging I recognized that cyrus-master seems to fork the configured amount of "prefork" daemons twice. One half listening on IPv4 and the other half on IPv6. Since IPv6 is practically never used from our frontends they stay forever doing nothing on the backends. Is there some reasonable way to prevent this other than setting prefork=0? I'm only using SERVICE entries like: Bimap cmd="imapd" listen="imap" prefork=5 Only the port is used for listen= without interface/IP. Greetings, Wolfgang -- Wolfgang Breyha| http://www.blafasel.at/ Vienna University Computer Center | Austria Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: Cannot connect with cyradm
On 06/05/16 04:24, Stuart Castergine via Info-cyrus wrote: > C: A01 AUTHENTICATE PLAIN Y3lydXMAY3lydXMAaGVsbC1oYXRoLW5vLWZ1cnk= I recommend changing the password from the "fury" thingy to something else. Maybe you want to strip base64 encoded credentials in the future. Greetings, Wolfgang -- Wolfgang Breyha| http://www.blafasel.at/ Vienna University Computer Center | Austria Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: disable starttls, ssl only
Gabriele Bulfon via Info-cyrus wrote on 26/04/16 12:25: > Hi, > > is it possible to disable completely the starttls option on cyrus imapd? > I want just connections on 993 over ssl, no starttls at all. > I noticed a number of "Fatal error: tls_start_servertls() failed" and noticed > a slow down of the entire machine during these errors. > I wouldn't like to have some kind of ddos attak using starttls on 993 or > something. Remove it from cyrus.conf? Greetings, Wolfgang -- Wolfgang Breyha| http://www.blafasel.at/ Vienna University Computer Center | Austria Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: 2.5.7 caldav interoperability
rsto--- via Info-cyrus wrote on 21/04/16 22:51: > P.S.: I am based in Vienna as well, so feel free to reach out to me :) Thanks for your offlist help. I think you pointed me in the right direction. 2.5.7 announces VPOLL for all except iOS/8 devices. HEAD doesn't announce VPOLL at all HEAD http_caldav.c:4520 /* Apple clients don't like VPOLL */ types &= ~CAL_COMP_VPOLL; vs. 2.5.7 http_caldav.c:3512 /* XXX iOS/8 doesn't like VPOLL */ if (hdr && strstr(hdr[0], "iOS/8")) types &= ~CAL_COMP_VPOLL; May I ask to backport this change to 2.5.x? I patched it locally and can confirm that our iOS/7 and 9 devices are able to access the Default Calendar now. Greetings, Wolfgang -- Wolfgang Breyha| http://www.blafasel.at/ Vienna University Computer Center | Austria Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: mailboxes.db invalid entries
Jan Kowalsky via Info-cyrus wrote on 22/04/16 01:28: > First I tried to dump the mailbox.db with ctl_mboxlist -d /tmp/mailboxes.txt > > After deleting the wrong entry manually I wanted to reload the mailbox > again with ctl_mboxlist -u /tmp/mailboxes.txt. All operation with > stopped cyrus. Have you renamed your mailboxes.db after using -d and before using -u? Otherwise ctl_mboxlist will import your dump into the existing mailboxes.db. And are this exactly the commands you used? I think ctl_mboxlist -d >/tmp/mailboxes.txt and ctl_mboxlist -f /tmp/mailboxes.db.new -u | http://www.blafasel.at/ Vienna University Computer Center | Austria Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
2.5.7 caldav interoperability
Hi! In our efforts to move on to 2.5 we also try to support caldav/carddav for our employees. In our tests we tried to share the Default calendar through as many devices/platforms as possible (not only) including Thundbird, OSX and iOS. One problem we have is that iOS can access the Account and creates a new Kalender on first access, but it does not use/detect the Default calendar. It creates one with long UID name like 22xxx9EA-EFxx-4Exx-BAxx-Ax3D7xxx and gives it a localized "displayname" attribute. In english "Calendar". Calendars added by OSX are detected as well. Only Default is missing. OSX on the other hand detects the Default as well as the iOS "Calendar" and can access both. For a user it is impossible to "guess" that UID mailbox/calendar name to use it in clients without autodetection like lightning. Is this a know issue that iOS can't/wont detect the Default calendar? Maybe fixed in HEAD? Or is it something we did wrong in our configuration? Greetings, Wolfgang PS: carddav Default addressbook on the other hand works with iOS as well. -- Wolfgang Breyha| http://www.blafasel.at/ Vienna University Computer Center | Austria Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: @ in mailboxname: 2.4 vs. 2.5
On 08/04/16 23:28, Wolfgang Breyha via Info-cyrus wrote: > Is it save to remove this limitation from mboxname.c? https://bugzilla.cyrusimap.org/show_bug.cgi?id=3693 This patch was reverted in "some" releases, but obviously not in 2.5. Please do so. Greetings, Wolfgang -- Wolfgang Breyha <wbre...@gmx.net> | http://www.blafasel.at/ Vienna University Computer Center | Austria Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
@ in mailboxname: 2.4 vs. 2.5
Hi! If not using virtual domains it was always allowed to use "@" in mailbox names and our users did. Up to 2.4. 2.5 considers it as "invalid mailbox name" due to a change in mboxname.c This makes 2.5 highly incompatible with existing murder infrastructure. Why was this change introduced and not mentioned in the changelog? Is it save to remove this limitation from mboxname.c? Greetings, Wolfgang -- Wolfgang Breyha| http://www.blafasel.at/ Vienna University Computer Center | Austria Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: drown/SSL issue
On 02/03/16 12:02, Wolfgang Breyha via Info-cyrus wrote: > You do not need to rebuild OpenSSL. I would check the SPEC File of the CentOS > 7 RPM which patches they included. If the TLS changes were not backported I > would try to build one of the newer 2.4.18 SRPMs for Fedora (eg. 23) on > CentOS 7. As of today RHEL/CentOS ships openssl updates with deactivated SSLv2 at build time. It should be enough to update it and restart cyrus. Greetings, Wolfgang -- Wolfgang Breyha <wbre...@gmx.net> | http://www.blafasel.at/ Vienna University Computer Center | Austria Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: drown/SSL issue
Hi! Tony Galecki via Info-cyrus wrote on 02/03/16 03:57: > I’m trying to figure out how to make my Cyrus install to not be susceptible to > the drown issue. > I have tried limiting the ciphers to TLSv1.2 but haven’t had much success. Limiting the cipher list does not deactive protocol support in OpenSSL. I don't know which patches Fedora backported from 2.4.18, but it seems not enough, because 2.4.18 disables SSLv2/v3 by default and you can set tls_versions: ... in your config. Setting these is the only way to get rid of the protocolls themself. On older cyrus versions you can set tlsonly: 1 but this can/will limit your protocoll support to TLSv1, with disabled v1.1 and v1.2, because TLSv1_server_method() was used. You do not need to rebuild OpenSSL. I would check the SPEC File of the CentOS 7 RPM which patches they included. If the TLS changes were not backported I would try to build one of the newer 2.4.18 SRPMs for Fedora (eg. 23) on CentOS 7. Greetings, Wolfgang -- Wolfgang Breyha| http://www.blafasel.at/ Vienna University Computer Center | Austria Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus