Re: Recent (probably MacOS) mail app provoking endless cyrus.index writes on 2.3 server. [WARNING: DKIM validation failed]
> So the exact text after the OK response doesn't matter, but it MUST > be SP followed by at least 1 TEXT-CHAR. > > This is pretty easy to patch in 2.3.x if you're forced to remain > there for reasons. It is, of course, fixed in later versions. > > Bron. > > I'm seeing this issue in my environment.. I think. I'm only *fairly* sure - I see clients that are identifying themselves as 10.11 causing lots of entries showing up in the sync log and a whole lot of index churning on.. usually smaller folders, luckily. I've been trying to track down a test machine to validate, but resources are quite limited where I am. But I'm a bit confused about the current working theory as to what's causing Mail.app to go into its strange loop. My Cyrus (RedHat provided 2.3.16-13.el6_6 with a very minor local patch) responds to EXPUNGE with " OK COMPLETED" as near as I can tell.. which seems spec compliant if I understand the above. * OK [CAPABILITY IMAP4 IMAP4rev1 LITERAL+ ID MUPDATE=mupdate://redacted.example.com/ AUTH=GSSAPI AUTH=PLAIN AUTH=LOGIN SASL-IR COMPRESS=DEFLATE] redacted.example.com Cyrus IMAP Murder v2.3.16-Fedora-RPM-2.3.16-13.el6_6 server ready A0001 login REDACTED PASSWORD A0001 OK [CAPABILITY IMAP4 IMAP4rev1 LITERAL+ ID MUPDATE=mupdate://redacted.example.com/ LOGINDISABLED AUTH=GSSAPI AUTH=PLAIN AUTH=LOGIN COMPRESS=DEFLATE ACL RIGHTS=kxte QUOTA MAILBOX-REFERRALS NAMESPACE UIDPLUS NO_ATOMIC_RENAME UNSELECT CHILDREN MULTIAPPEND BINARY SORT SORT=MODSEQ THREAD=ORDEREDSUBJECT THREAD=REFERENCES ANNOTATEMORE CATENATE CONDSTORE SCAN IDLE LISTEXT LIST-SUBSCRIBED X-NETSCAPE URLAUTH] User logged in A0002 SELECT Trash * FLAGS (\Answered \Flagged \Draft \Deleted \Seen) * OK [PERMANENTFLAGS (\Answered \Flagged \Draft \Deleted \Seen \*)] * 4560 EXISTS * 0 RECENT * OK [UNSEEN 1] * OK [UIDVALIDITY 1347582296] * OK [UIDNEXT 6437] * OK [NOMODSEQ] Sorry, modsequences have not been enabled on this mailbox * OK [URLMECH INTERNAL] A0002 OK [READ-WRITE] Completed A0003 idle + idling done A0003 OK Completed A0004 EXPUNGE * 4560 EXISTS * 0 RECENT A0004 OK Completed In fact a 2.4.18 install I'm building (I'm using this issue as an excuse to prioritize a long time coming upgrade) seems to respond to expunge in the same way. * OK [CAPABILITY IMAP4rev1 LITERAL+ ID ENABLE MUPDATE=mupdate://redacted.example.com/ STARTTLS AUTH=LOGIN AUTH=PLAIN AUTH=DIGEST-MD5 SASL-IR] redacted.example.com Cyrus IMAP Murder v2.4.18-Invoca-RPM-2.4.18-1.el7.centos server ready A0001 login redacted redacted A0001 OK [CAPABILITY IMAP4rev1 LITERAL+ ID ENABLE ACL RIGHTS=kxte QUOTA MAILBOX-REFERRALS NAMESPACE UIDPLUS NO_ATOMIC_RENAME UNSELECT CHILDREN MULTIAPPEND BINARY CATENATE CONDSTORE SORT SORT=MODSEQ SORT=DISPLAY THREAD=ORDEREDSUBJECT THREAD=REFERENCES ANNOTATEMORE WITHIN SCAN XLIST URLAUTH X-NETSCAPE MUPDATE=mupdate://redacted.example.com/ LOGINDISABLED COMPRESS=DEFLATE IDLE] User logged in SESSIONID= A0002 SELECT Trash * 0 EXISTS * 0 RECENT * FLAGS (\Answered \Flagged \Draft \Deleted \Seen) * OK [PERMANENTFLAGS (\Answered \Flagged \Draft \Deleted \Seen \*)] Ok * OK [UIDVALIDITY 144503] Ok * OK [UIDNEXT 1] Ok * OK [HIGHESTMODSEQ 1] Ok * OK [URLMECH INTERNAL] Ok A0002 OK [READ-WRITE] Completed A0003 idle + idling done A0003 OK Completed A0004 EXPUNGE A0004 OK Completed (And fastmail's IMAP responds with a OK Completed, and I assume it represents the most up to date Cyrus :P) Am I misunderstanding the theoretical issue? My IMAP-fu is a little rusty these days and I'm a bit bleary eyed so apologies in advance if I'm missing something obvious - I'm just testing by typing the commands in raw since imaptest is panicking on me and I've not had time to sort that out (though my Thunderbird debug log also shows an OK Completed as a repsonse). It does seem from one of the trace logs from the Apple discussion that there is a version of Cyrus that responds with just OK, but.. it's not all 2.3.x 's at the very least :P. Sorry again if this noise and thanks in advance for any clarification someone can give. Matt Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Impact of mismatched improved_mboxlist_sort in murder environment?
Hey all, I'm in the middle of moving from a 2.2.13p1 Murder environment to a 2.3.16 Murder, moving the backends first. I've been moving people over slowly (thousand or so now, I think) and while running a few checks, I got the following error when I ran ctl_mboxlist -m -w fatal error: Internal error: assertion failed: ctl_mboxlist.c: 297: strcmp(realpart, unused_mbdata-server) Poking around I would get this on certain mailboxes that ctl_mboxlist would hit, so I shuffled a few of the problematic mailboxes to a test server, where they would show the same error. After hitting my head against the wall, I noticed that the new (2.3.16) backends have improved_mboxlist_sort: 1 set, whereas the not-yet-upgraded frontends (2.2.13p1)+mupdate servers, well, don't even have the option ;). I guessed that this mismatch might be the problem, and while I've not done any extensive testing since the system is live, if I shutdown the test server with my problematic mailboxes, dump the mailboxes.db there, turn off improved_mboxlist_sort, and reload the dump, I can issue ctl_mboxlist -m -w till my heart is content. Outside of ctl_mboxlist functions, everything's fine and I haven't noticed anything amiss, but (obviously) that doesn't mean there isn't some problem waiting to bite me. I guess the question is 1) if I'm right that my problems are likely related to improve_mboxlist_sort, and 2) if anyone has any idea what sort of other problems I may encounter due to this mismatch. Due to circumstances beyond my control, getting downtime in my environment (even for what should be a straight forward dump/reload) anytime soon is going to be problematic, but I can push for it if necessary. On the other hand, if the problem is relatively harmless and going to sort itself out once I get the mupdate server to 2.3.16, I'm fine with waiting until then or upgrading the mupdate server sooner. Thanks in advance for any thoughts! Matt, Wishing he could go straight to 2.4.x. Sigh. Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: Move of mailboxes
Is there a recommended procedure to do the move? Any pointers (even to pitfalls) are welcome. I'm actually in the middle of migrating two backends of a Murder (2.2.13p1) to new machines (and Cyrus version, 2.3.16, though it doesn't sound like you're switching Cyrus versions in your case). Full disclosure, no idea if this is the recommended procedure and 2.2.13's probably too old to be that useful, but it's been working for me. The way I've been doing it is writing a small script that: 1) Checks $cyrus_var/proc to see if the user is logged in. 2) If they are not, connects to one of the proxy frontends, changes permissions on the mailbox to (temporarily) stop the user from manipulating it if they were to log in mid transfer. 3) Issue a rename (which I think ultimately just makes an XFER call to the appropriate backend?) of the format: user.melson user.melson mailboxes3.example.com!partition1 (for example). 4) Set the permissions back to normal after success. (I may be changing this permission switching to temporarily deny the user authentication, that method just didn't fit my environment when I started this). The script's just some perl using Cyrus::IMAP::Admin, and it should be fairly easily to replicate in anything with an IMAP library. (It's basically the same process I use to shuffle people between backends in the murder environment normally, which was handy from a laziness standpoint; the process is used for moving from 2.3.16-2.3.16 backends as well) It's been pretty successful so far, no user has noticed anything amiss (that said, most of my users are relatively light users), and delivered email just gets queued up during the transfer as near as I can tell. I've found it to be an ideal way to go about migration/upgrades - I can introduce users into the new environment slowly to catch configuration glitches, make sure I didn't misjudge the resources I would require, etc, etc. Very handy. It's been good about catching problems, even (I have a customization that extends the valid characters in mailboxes I forgot to apply on the new machines; the XFER noticed right away and left the user's mailbox unmoved and unmolested) Oh, one thing I forgot - I'm fairly sure you *do* have to set allowusermoves: 1 in your configuration if you're using this method. The gotchas I've run into are probably more quirks of the environment I'm working with or relics of a relatively old Cyrus at this point, but I'll share them in case they are useful. *) I've had some weirdness if the transfer gets interrupted in the middle (I forget the exact symptom, but the mailbox on the home spool was flagged as REMOTE (or something like that), but it hadn't actually made it over to the new machine - a simple ctl_mboxlist -m on the backend fixed it up since the frontend had the right information). *) This is probably more my environment than anything else, but I've had to throttle the moves and not run too many simultaneous moves in parallel or I basically overwhelm the server I'm migrating from - it seems to actually be the delete of the mailbox from its old home that hits the hardest. I've not looked into it much, because, old crusty server (2.4 kernel, ext3 file system; like I said, probably my environment :P). -- Matt Elson mel...@fastmail.net Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Mixed Cyrus Version Murder?
Hello all, Sorry if this has been answered or is an obvious place like the documentation, but I've not had any luck finding information on this. Is there a way to have a Cyrus murder that has mixed Cyrus versions in it? That is to say, have one backend server running 2.2.13 and another running 2.3.8? We're hoping to upgrade our backend servers by setting up a new server running 2.3.8, joining it to the murder, and then simply using cyradm to move mailbox from the old backends to the new ones (with the process repeated for each machine where applicable) It seems like this would cause problems since the mailboxes.db would be different between the mupdate master and the various backends, but I figured it couldn't hurt to ask. I'd love to test it, but am lacking time and development machines at this particular moment... Any ideas? Anyone else ever try to do this? Thanks in advance, --- Matt Elson Unix Systems Administrator Wesleyan University Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: backup imapd with TSM
Hans Moser wrote: Hi! Does anyone actually backup and restore Cyrus IMAPd with Tivoli Storage Manager (TSM)? We've use TSM coupled with the snapshot capability of our NetApp filer to do backups here. The system was automated for awhile with a simple web frontend, but some changes on the backend have rendered it manual again. We restore the given mailbox( or mailboxes) underneath a heading called restored, subscribe the user to it, set the permissions to be lrs for the user, and let them move the mail out that they want. After a fixed amount of time (two weeks) we then destroy the restored mailbox. The procedure is roughly (using restoring my inbox (only) as an example): 1) Copy the info from tape to the location of the new mailbox dsmc restore -ina /var/spool/imap/P/user/melson/ /var/spool/imap/P/restored/ 2) Reconstruct the mailbox. reconstruct -p default restored.melson (I have no idea why I'm not using -f for this.. wrote the stuff months ago when I was still getting the hang of some of this so I may have simply not known any better) Then you just set permissions as you would like, subscribe the user to the mailbox, and let them know that there's a restored mailbox waiting for them. It's worked fairly well for us. -- Matt Elson Unix Systems Administrator Wesleyan University Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Sendmail cyrusv2 mailer IGNOREQUOTA argument?
Hey all, Not sure if this is the right place (perhaps a sendmail list would be better?) and apologies if this is documented somewhere (I've looked and look, but have had no luck), but is there an argument or flag I can pass to the cyrusv2 mailer to have it use the IGNOREQUOTA extension of LMTP? As far as why I would want this, the current way email quota is handled is that all communications within the network are not subject to quota restrictions. We currently have a homegrown system that queries the appropriate mailbox store and produces entries in a sendmail access.db on our MX machines to reject users over quota. It works fine for our purposes and it will have to stick around for a variety of reasons, but I'd really like to put the quota information for Cyrus users in Cyrus. There's a number of IMAP clients that can use the QUOTA extension which could be very useful to our users, and well, it seems like a waste to not put this information in Cyrus. It might make my querying of Cyrus for mailbox size go a little bit smoother too, for all I know (now I look at some sort of ANNOTATEMORE flag that gives me the current size). However, in order to do this, I need a way to make sure that all internal mail will flow through to Cyrus users, even when they are over quota. My hope was to have it setup so that the mail server users use to send mail (quite separate from where incoming mail comes from) would just always send IGNOREQUOTA (since anyone using it has to be authenticated) to Cyrus which I'm assuming means that the mail could easily be delivered. Again, apologies if this is documented somewhere, I've dug around for it and I even tried to read the mailer itself, but Sendmail configuration syntax makes my poor newbish brain hurt something fierce. Seems like I could probably put something together using deliver, but I was hoping to avoid that if possible.. As far as our Cyrus setup, it's a cyrus 2.2.13 murder w/ two frontends, two backends, and the mupdate server on one of the frontends. lmtpd listens on the frontends for connections coming from other machines running sendmail. Matt Elson Unix Systems Admin Wesleyan University Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Question about LSUB *% behavior in Cyrus v2.2.13 Murder...
Hi (hope this is the right place), Apologies in advance if this is easily answered; I'm setting up a Cyrus Murder configuration for work, and am admittedly new to such things, so my problem may be pure idiocy on my part. Basically, I'm having a problem with LSUB *% reporting on the front proxy layers when altnamespace is enabled. In short, the query doesn't grab the prefix for users and shared folders (Other Users and Shared Folders) which squirrelmail (why I'm even looking into this) uses to build a list of mailboxes (and appropriate hierarchy); the end result is mostly a cosmetic problem as near as I can tell, but people seem to be tied to such thing so it is unfortunately something I have to look into. Making the same query directly on the backend machines directly results in the answer squirrelmail is expecting. In any case, I was wondering if this is expected behavior or a configuration error on my part, or even if anyone else has run into this and can lend a hand. Again, apologies if my answer is located somewhere, I tried to find information but just haven't had much luck. Details below, more available upon request: Non-authentication related configs are: (frontends) unix_group_enable: true altnamespace: true allowusermoves: true allowallsubscribe: true (backends) altnamespace: true unix_group_enable: true hashimapspool: true allowusermoves: true fulldirhash:true allowallsubscribe: true The query/response: * OK frontend1 Cyrus IMAP4 Murder v2.2.13 server ready a OK User logged in a LSUB *% * LSUB () . Drafts * LSUB () . Sent * LSUB () . Spam * LSUB () . Trash * LSUB (\Noselect \HasChildren) . testfolder * LSUB () . testfolder.testnest * LSUB () . Shared Folders.shared_test a OK Completed * OK backend2 Cyrus IMAP4 v2.2.13 server ready a OK User logged in a LSUB *% * LSUB () . Drafts * LSUB () . Sent * LSUB () . Spam * LSUB () . Trash * LSUB (\Noselect \HasChildren) . testfolder * LSUB () . testfolder.testnest * LSUB (\Noselect \HasChildren) . Shared Folders * LSUB () . Shared Folders.shared_test Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
