Re: Cyrus 2.4 and unexpunge messages.
On 2019-01-02 08:20, chose wrote: Good morning, I've unexpunged messages in the mail box, all is recovered but the flag "deleted" persist, so Roundcube see the email as deleted and the emails are grey. Did I missed some step to full recover emails ? The unexpunge command has a "-d" option: -d Unset the \Deleted flag on any restored messages. -- David Carter Email: dp...@cam.ac.uk University of Cambridge, Phone: (01223) 748408 Information Services,Fax: (01223) 334679 7 J J Thomson Avenue, Cambridge UK. CB3 0RB Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: users filling up disk microsoft office outlook 2013 ?
On Thu, 5 Mar 2015, Rudy Gevaert wrote: Hello, I'm hitting a strange issue. A user had his mailbox grow (multiple GB) without him noticing. I can clean up with cyr_expire. Multiple messages are actually the same. It seems that the client uploads the message and deletes it immediately. Yesterday it filled up a partition, so I couldn't enable telemetry on his mailbox to actually see what happened. The client is outlook 2013. Has anybody seen this issue before? I'm running 2.4.17 Yes, this is the same issue that I descriped in the attached message to info-cyrus last April. At one point we were seeing numerous instances of this every week, to the point that it was a potential denial of service attack against our mail system. Dozens of separate cases. Fortunately things seem to have settled down in the last 6 months. Individual Outlook clients can be fixed using the procedure described at: http://www.ucs.cam.ac.uk/email/hints-and-tips/hermes/outlook-ost The problem appears to be associated with virus scanner plugins to Outlook (at least AVG and Kaspersky: it is possible others are also affected). Applying the most recent patches to Outlook doesn't seem to help. So far the only way that I have been able to reproduce the precise effect including a tight spin of uploads and deletes was when I switched from one AV plugin to another. One of the people affected who I contacted reported that they had made a similar change directly before problems started. A Google search on Outlook 2013 IMAP bandwidth problems strongly suggested that the problem wasn't specific to our mail system, e.g: http://ddkonline.blogspot.co.uk/2013/06/outlook-2013-bug-when-using-imap-beware.html -- David Carter Email: david.car...@uis.cam.ac.uk University of Cambridge, Phone: (01223) 334502 Information Services,Fax: (01223) 334679 7 J J Thomson Avenue, Cambridge UK. CB3 0RB From dp...@cam.ac.uk Tue Apr 29 13:00:27 2014 Date: Tue, 29 Apr 2014 12:56:07 +0100 From: David Carter dp...@cam.ac.uk To: info-cyrus@lists.andrew.cmu.edu Subject: Outlook 2013 broken synchronisation We have had a couple of cases recently where the IMAP synchronisation process in Outlook 2013 seems to go nuts. The client system APPENDs large numbers of messages stored on the user's local hard disk to the Cyrus inbox and then deletes and expunges them (one at a time). It does this repeatedly. X-OlkEid: in the message headers and MIME multipart boundary markers in the body are different in each message, so the additional messages aren't exact copies. (I guess this could be some kind of broken filter rule. At the moment I'm working with third hand information). In one case the client managed to generate a 3.5 GByte cyrus.cache file (with about 1.5 million recently expunged messages), which is dangerously close to the 4 GByte limit on that file. We are running Cyrus 2.4.17. I appreciate that this isn't really a Cyrus question: I was just hoping that someone somewhere had seen the same effect and had worked out how to make Outlook behave. A Google search suggests that Microsoft improved the IMAP support in Outlook 2013, but I haven't found a specific match to the symptoms that I am seeing. -- David Carter Email: david.car...@uis.cam.ac.uk University of Cambridge, Phone: (01223) 334502 Information Services,Fax: (01223) 334679 7 J J Thomson Avenue, Cambridge UK. CB3 0RB Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: users filling up disk microsoft office outlook 2013 ?
On 2015-03-05 23:47, Bron Gondwana wrote: On Thu, Mar 5, 2015, at 08:34 PM, Rudy Gevaert wrote: Quoting Frank Richter frank.rich...@hrz.tu-chemnitz.de, Thu, 05 Mar 2015: Last night ecactly this happened on our mail server the 1st time. One partition was filled up ... The user was noticing some duplicate messages, but not thousands. I'll check up if a virus scanner is involved in our case. This isn't a Cyrus issue or even something that Cyrus can fix I don't think :( One thing which would help is a limit on the amount of expunged data which can be held in a single mailbox before an expire automatically kicks in. At the moment my largest concern is that we can hit the 4 GByte limit on the cyrus.cache file, which causes replication to break (at least in Cyrus 2.4.17). That's how we found out that this was going on in the first place. -- David Carter Email: david.car...@uis.cam.ac.uk University of Cambridge, Phone: (01223) 334502 Information Services,Fax: (01223) 334679 7 J J Thomson Avenue, Cambridge UK. CB3 0RB Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Outlook 2013 broken synchronisation
We have had a couple of cases recently where the IMAP synchronisation process in Outlook 2013 seems to go nuts. The client system APPENDs large numbers of messages stored on the user's local hard disk to the Cyrus inbox and then deletes and expunges them (one at a time). It does this repeatedly. X-OlkEid: in the message headers and MIME multipart boundary markers in the body are different in each message, so the additional messages aren't exact copies. (I guess this could be some kind of broken filter rule. At the moment I'm working with third hand information). In one case the client managed to generate a 3.5 GByte cyrus.cache file (with about 1.5 million recently expunged messages), which is dangerously close to the 4 GByte limit on that file. We are running Cyrus 2.4.17. I appreciate that this isn't really a Cyrus question: I was just hoping that someone somewhere had seen the same effect and had worked out how to make Outlook behave. A Google search suggests that Microsoft improved the IMAP support in Outlook 2013, but I haven't found a specific match to the symptoms that I am seeing. -- David Carter Email: david.car...@uis.cam.ac.uk University of Cambridge, Phone: (01223) 334502 Information Services,Fax: (01223) 334679 7 J J Thomson Avenue, Cambridge UK. CB3 0RB Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: Need guidelines on how to migrate a Cyrus-Imapd server
On Fri, 1 Feb 2013, Thibault Le Meur wrote: I'll go the rsync way then... pity I would have loved to understand what kind of file is to be fed to the sync_client -u -f command, in order to give it a try.. It is just a list of userids, one per line, in a file rather than specified on the command line. -- David Carter Email: david.car...@ucs.cam.ac.uk University Computing Service,Phone: (01223) 334502 New Museums Site, Pembroke Street, Fax: (01223) 334679 Cambridge UK. CB2 3QH. Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: HasChildren flag on two mailboxes with similar name
On Wed, 13 Feb 2013, Tobias Riekeberg wrote:
I have two users, named joe and joe-doe, each with subfolders
drafts and sent.
[...]
So the user.joe has the flag \HasNochildren instead of
\HasChildren. This happens whenever I have two users of which one's
name is the first part of the other users name. Result of this seem to
be, that I cannot access user.joe in Thunderbird.
Do you have improved_mboxlist_sort set?
We hit the same problem a long time back. If I recall correctly the
problem is the order that '-' and '.' occur with a simple ASCII sort.
('-' is the only commonly used character which sorts before '.').
--
David Carter Email: david.car...@ucs.cam.ac.uk
University Computing Service,Phone: (01223) 334502
New Museums Site, Pembroke Street, Fax: (01223) 334679
Cambridge UK. CB2 3QH.
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
expunge_days option in Cyrus 2.4
We moved from Cyrus 2.3 to 2.4 earlier in the year. We run with delayed
expunge, with a nightly job to expire old expunged messages after 28 days:
cyr_expire -E 3 -D 28 -X 28
This worked nicely under Cyrus 2.3.
A few days back we had our first request to restore email which has been
expunged more than seven days previously, and I made the unpleasant
discovery that we were only actually keeping expunged messages for 7 days.
A bit of digging through source code uncovered:
expunge_days: 7
Number of days to retain expunged messages before cleaning up
their index records. The default is 7. This is a necessary for
QRESYNC to work correctly. If combined with delayed expunge (above)
you will also be able to unexpunge messages during this time.
which seems to expire messages as soon as a mailbox is closed after 7
days, regardless of what cyr_expire might be up to. I believe increasing
expunge_days to 30 days will make things work the way that I expected.
I put this down to a bad experience and failing to trawl through
imap.conf.5 carefully enough looking for new options (it isn't mentioned
in changes.html, but a lot changed between 2.3 and 2.4). I have since
found a short thread on this list from January discussing the new option.
However I had a second bad experience with expunge_days over the weekend.
We use the replication engine to seed new mailstores and move users around
the cluster. This worked nicely using my own code in Cyrus 2.1 and 2.3.
In 2.4 the following innocent looking test:
sync_client.c :
/* check that replication stands a chance of succeeding */
if (remote !is_repeat) {
if (mailbox-i.deletedmodseq remote-highestmodseq) {
syslog(LOG_NOTICE, inefficient replication (
MODSEQ_FMTMODSEQ_FMT ) %s,
mailbox-i.deletedmodseq, remote-highestmodseq,
local-name);
r = IMAP_AGAIN;
goto done;
}
appears to kick in if you run sync_client -u between a master and
replica or backup mailstore with a gap of more than 7 days between the two
runs. mailbox_full_update() gets rather excited about messages on the
replica which have been deleted, expunged and unlinked from the master.
Again expunge_days: 30 should help, but I thought that others might want
to be warned before they try to use sync_client to move users around.
--
David Carter Email: david.car...@ucs.cam.ac.uk
University Computing Service,Phone: (01223) 334502
New Museums Site, Pembroke Street, Fax: (01223) 334679
Cambridge UK. CB2 3QH.
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: expunge_days option in Cyrus 2.4
On Tue, 17 Jul 2012, Bron Gondwana wrote: The sync_client thing is a separate matter... it should consider those to be bogus and clean them up I think... hmm. It definitely cleaned up many cases. I found a couple of examples where messages seemed to get pulled back to the master, fortunately not on one of the live systems. The problem of course is that you only get to see the after state. The logs do indicate a lot of UID and MODSEQ ping pong took place as the two ends worked out their differences. Noone has come to ask me what happened, which I take as a good sign :). -- David Carter Email: david.car...@ucs.cam.ac.uk University Computing Service,Phone: (01223) 334502 New Museums Site, Pembroke Street, Fax: (01223) 334679 Cambridge UK. CB2 3QH. Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Cyrus 2.4 IMAP SEARCH
We have just finished migrating from Cyrus 2.3 to Cyrus 2.4. One of my users (almost the last to be moved!) spotted a small change in behaviour: Cyrus 2.3: . SEARCH SUBJECT * SEARCH 1 2 3 ... 3873 -- long list of messages . OK Completed (3869 msgs in 0.000 secs) Cyrus 2.4: . SEARCH SUBJECT * SEARCH -- empty list of messages . OK Completed (0 msgs in 0.030 secs) The user was just trying to cancel an existing search. I have suggested a more efficient approach in their mail client (Mulberry 4.0.8). RFC 3501, section 6.4.4 SEARCH Command says: In all search keys that use strings, a message matches the key if the string is a substring of the field. The matching is case-insensitive. and I guess is strictly speaking a substring of all other strings. Was this a deliberate change or just a consequence of the search engine being reworked in 2.4? I don't imagine that it is a very common problem. -- David Carter Email: david.car...@ucs.cam.ac.uk University Computing Service,Phone: (01223) 334502 New Museums Site, Pembroke Street, Fax: (01223) 334679 Cambridge UK. CB2 3QH. Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: Cyrus 2.4.13 memory use
On Wed, 8 Feb 2012, David Carter wrote: One small curiosity is that the memory use per IMAP session seems to have increased dramatically. I'm looking at the output of the Linux free command after buffer cache has been subtracted: 2.3.14: 2572296 KBytes with 2909 IMAP sessions: 884 KBytes/session 2.4.13: 3426388 KBytes with 1247 IMAP sessions: 2747 KBytes/session This is moving from 32-bit SLES 10 to 64-bit SLES 11. I was expecting a modest increase as pointers and unsigned long double in size. A 3.1 times increase is rather more than I was expecting. (I'm going to try running 32bit binaries on a 64 bit system to see if that makes a significant difference). Two quick experiments reveal: 2.4.13 (32bit binaries, imapd -U 250): 1969752 KBytes with 1484 IMAP sessions: 1327 KBytes/session: So it looks like x86_64 binaries really are quite expensive. 2.4.13 (64bit binaries, imapd -U 1): 1743176 KBytes with 1182 IMAP sessions: 1474 KBytes/session imapd -U 1 stops master from recycling existing imap processes for new logins. (thanks to Wes Craig for the suggestion here). The problem with this is that Linux doesn't pass memory which has been free()ed back to the operating system: it just goes into the pool used by malloc() in the current process. So once an imapd process opens a large mailbox, it hangs onto all of the memory which has been allocated there. All slightly academic as my boss said yes, just spend the money. But I thought that the 32bit/64bit experiment might be of interest to others. -- David Carter Email: david.car...@ucs.cam.ac.uk University Computing Service,Phone: (01223) 334502 New Museums Site, Pembroke Street, Fax: (01223) 334679 Cambridge UK. CB2 3QH. Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Cyrus 2.4.13 memory use
We are currently in the process of migrating from 2.3.14 (with lots of local patches) to a fairly vanilla 2.4.13. One small curiosity is that the memory use per IMAP session seems to have increased dramatically. I'm looking at the output of the Linux free command after buffer cache has been subtracted: total used free shared buffers cached Mem: 823943672065201032916 0 4262764207948 -/+ buffers/cache:25722965667140 ^^^ 2.3.14: 2572296 KBytes with 2909 IMAP sessions: 884 KBytes/session 2.4.13: 3426388 KBytes with 1247 IMAP sessions: 2747 KBytes/session This is moving from 32-bit SLES 10 to 64-bit SLES 11. I was expecting a modest increase as pointers and unsigned long double in size. A 3.1 times increase is rather more than I was expecting. (I'm going to try running 32bit binaries on a 64 bit system to see if that makes a significant difference). pmap and /proc/[pid]/smaps suggests that most of the increase is in the heap segment which is used by malloc() and the brk() system call. It looks like 3000 IMAP sessions are going to take around 8 GBytes of RAM just to run, and we will need to buy additional RAM for buffer cache. This isn't the end of the world: memory is cheap. I'm just curious if anyone else saw a similar increase when upgrading from 2.3 to 2.4. -- David Carter Email: david.car...@ucs.cam.ac.uk University Computing Service,Phone: (01223) 334502 New Museums Site, Pembroke Street, Fax: (01223) 334679 Cambridge UK. CB2 3QH. Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: Replication and INBOX rename in 2.4.12
On Wed, 18 Jan 2012, Janne Peltonen wrote: I recently upgraded Cyrus from version 2.3.16 to 2.4.12 (using the wonderful RPMs from INVOCA). Now, I ran into a bit of a problem with replication. From time to time, a user thinks it a great idea to rename their INBOX in order to save time in archiving old messages. Just to add: it looks like this is broken altogether in 2.4.13: 2.4.12: . rename INBOX foo . OK Completed . logout 2.4.13: . rename INBOX bar Connection closed by foreign host. My logs report: IOERROR: opening .../user/dpc99/bar/cyrus.header.NEW: No such file or directory IOERROR: failed to commit mailbox user.dpc99.bar, probably need to reconstruct -- David Carter Email: david.car...@ucs.cam.ac.uk University Computing Service,Phone: (01223) 334502 New Museums Site, Pembroke Street, Fax: (01223) 334679 Cambridge UK. CB2 3QH. Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: Cyrus 2.4.X delete mailbox oddities
On Tue, 28 Jun 2011, Bron Gondwana wrote: The separate IMAP session should clean it up when it disconnects. It's basically last one out turn off the lights. Some further testing suggests that this happens if I try: C1: . SELECT foo C2: . DELETE foo . OK Completed . LOGOUT C1: . NOOPThis is new . LOGOUT C1 doesn't clean up without a NOOP (or some other command which synchronises state) before the LOGOUT. -- David Carter Email: david.car...@ucs.cam.ac.uk University Computing Service,Phone: (01223) 334502 New Museums Site, Pembroke Street, Fax: (01223) 334679 Cambridge UK. CB2 3QH. Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
RE: Does anyone allow unlimited or extremely large quotas?
On Fri, 19 Nov 2010, Michel Sébastien wrote: On a 32 bit architecture: we had one folder with over a million messages which was causing processes to run out of virtual memory trying to map the cache file in. This wouldn't be a problem with a 64 bit userland. very impressive to have so much messages in one folder therefor in one partition!. We have hit this limit once, and so far only once, as well. A user was sorting their email archive (thousands of messages) generating copies to the Trash mailbox. They repeated this exercise multiple times. Each time that the user reached hit their quota limit (several GBytes), they emptied the Trash folder. Consequently the live mailbox itself never contained huge numbers of messages. However delayed expunge means that a lot of wreckage was left behind: hundreds of thousands of messages. Easily fixed by a reconstruct which discarded the obsolete information. Is mmap still efficient ? map a gigabit file should cost a lot of I/O and a relatively long reponse time to just access the records of the most recent emails. The mmap() itself has very little cost. It would only become a problem if something actually tried to read all of the cache entries, causing the data to be paged in from disk. -- David Carter Email: david.car...@ucs.cam.ac.uk University Computing Service,Phone: (01223) 334502 New Museums Site, Pembroke Street, Fax: (01223) 334679 Cambridge UK. CB2 3QH. Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: painful mupdate syncs between front-ends and database server
On Fri, 30 Oct 2009, Michael Bacon wrote: On all systems in the murder, we'll see instances where the mupdate process goes into a spin where, in truss, it's an endless repeat of fcntl, stat, fstat, fcntl, thousands of times over. These execute extremely quickly, but I do wonder if we're assuming that something that takes very little time takes an insignificant amount of time, when the time involved becomes significant on an 800k mailboxes database. I agree that latency is probably your problem here. I'm wondering if fsync() latency on the frontends might be a factor given that you report little disk I/O on the mupdate master (IOPS are much more important than Kps, but I'm sure that you already know that). The update process will only be as fast as its weakest link, and you stated earlier: When we spec'ed out our servers, we didn't put much I/O capacity into the front-end servers -- just a pair of mirrored 10k disks doing the OS, the logging, the mailboxes.db, and all the webmail action going on in another solaris zone on the same hardware. No mention of battery backed write cache there, which tends to be fairly critical for anything involving fsync(). There is an easy way to find out: skiplist_unsafe: 0 If enabled, this option forces the skiplist cyrusdb backend to not sync writes to the disk. Enabling this option is NOT RECOMMENDED. You can ignore the scary warning (at least for test purposes) on murder frontends, given that it is just a readonly replica of the mupdate master. I hope that this isn't a complete red herring. It just struck me that it would be a really easy test to make. -- David Carter Email: david.car...@ucs.cam.ac.uk University Computing Service,Phone: (01223) 334502 New Museums Site, Pembroke Street, Fax: (01223) 334679 Cambridge UK. CB2 3QH. Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: cyrus replication : master to replica and replica to master
On Fri, 23 Oct 2009, Bron Gondwana wrote: I've seen heartbeat get split brain before. We gave up on it. We do all our fencing via humans now! Check the KVM, kick the box, manually run the failover script. Some of my colleagues have had a lot of grief with Heartbeat going split brain. It seems to really be designed for a pair of machines sitting next to each other in a rack with a serial link for the heartbeat, rather servers installed in a pair of machine rooms three miles apart. We do manual failover with our Cyrus mailstores: I would rather 1/8th of my users had an outage of a couple of hours (and typically just a few minutes) than end up with a split brain. On the one occasion in five years that we did end up with a Cyrus split brain (replication failed because of a memory DIMM error and then the entire master failed a few minutes later) it was easy enough to fish missing messages out of the dead system the following day and reinject them using LMTP. Certainly easier than reengineering the entire Cyrus mailstore to allow active/active replication. On Wed, Oct 21, 2009 at 08:45:11PM +0200, David Touzeau wrote: I would like to know if it is possible to SET the replica has the master too in order to replicate new mail saved on the replica to the master and vis versa In this case it should be turn to active/active.. We do this to a limited degree: the set of active users on a pair of mailstores can be partitioned and bounced back and forth between the two servers in a pair. This is mostly useful for load balancing between our two machine rooms, or migrating all the users off a master so that we can patch and reboot without any user visible downtime. However this is using my own replication code rather than the branch which was rewritten into Cyrus by Ken. I have additional safeguards to stop sync_client from overwriting the master data in a pair (which has only ever happened because of stupidity on my part when testing). I've never used the standard replication code in Cyrus other than to backport (sideport?) additional features such as CONDSTORE and GUID support. Given the grief Fastmail had with the early Cyrus replication code I think that I'm rather glad about this. Every once in a while I think about moving to standard Cyrus replication. Unfortunately there are a lot of warts that I really don't like. It is much easier to just drop my own replication code onto new versions of Cyrus (typically 5 minutes work each time). That was one of my original design objectives. -- David Carter Email: david.car...@ucs.cam.ac.uk University Computing Service,Phone: (01223) 334502 New Museums Site, Pembroke Street, Fax: (01223) 334679 Cambridge UK. CB2 3QH. Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: cyr_expire signaled to death by 11
On Thu, 25 Sep 2008, Per olof Ljungmark wrote: No idea why, comments anyone? A limitation of some sort when expunging a lot of messages? cyr_expire [96599]: Expunged 3005 messages from user.myuser.Trash cyr_expire[96599]: expunged 4907 out of 109443 messages from 143 mailboxes master[833]: process 96599 exited, signaled to death by 11 Probably a corrupt cyrus.cache file (at least that's the cause when I see these). Try reconstruct on the mailbox. -- David Carter Email: [EMAIL PROTECTED] University Computing Service,Phone: (01223) 334502 New Museums Site, Pembroke Street, Fax: (01223) 334679 Cambridge UK. CB2 3QH. Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Very annoying IMAP problem (cyrus + Outlook)
On Fri, 22 Aug 2008, Denis BUCHER wrote: As far as I understand, the cause of the problem is : * Outlook is sending either a token or quoted string that is longer than 8K bytes. Your problem is a mailbox which contains several thousand messages. Possibly several thousand messages which Outlook hasn't seen previously. Outlook tries to construct a single IMAP command of the form: UID FETCH uid,uid,uid,... some set of fields where the list of UIDs is larger than 8 KBytes in size. But how could I correct this problem either in Outlook or cyrus ??? Split the problem mailbox into smaller mailboxes using something other than Outlook. Alternatively you could increase the word limit in Cyrus. MAXWORD is 32k in recent Cyrus 2.3 versions. -- David Carter Email: [EMAIL PROTECTED] University Computing Service,Phone: (01223) 334502 New Museums Site, Pembroke Street, Fax: (01223) 334679 Cambridge UK. CB2 3QH. Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Very annoying IMAP problem (cyrus + Outlook)
On Fri, 22 Aug 2008, Ciprian Marius Vizitiu wrote: Any other possibility? I mean other than several thousand messages? Several thousand messages is the obvious cause. We've seen this once (with Cyrus 2.3) after running unexpunge on a very large mailbox. The mailbox got a new UIDvalidity, so Outlook wanted to resychronise the whole thing. But /var/imap/log will give a definitive answer. -- David Carter Email: [EMAIL PROTECTED] University Computing Service,Phone: (01223) 334502 New Museums Site, Pembroke Street, Fax: (01223) 334679 Cambridge UK. CB2 3QH. Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Replication verification
On Fri, 27 Jun 2008, Vladimir Klejch wrote: I searchig for a posibility to use both server's in production as master-master. Afraid that replication in Cyrus doesn't support full master-master, only master/slave. UIDs in IMAP make full master-master rather involved. It is possible to run a mix of master and replica mailstores on a single system. There are tools like nake_md5 and make_sha1, but the manpages document only howto config them, but not how to realy use them for replication check. I download the md5 files to a single location and run a 50 line Perl script to spot mismatches. You are welcome to a copy of that script. To make sure that the replica is up to date I run sync_client in an extra verbose mode (-v -v) and check for unexpected updates. Unfortunately that code didn't make it it into the vanilla Cyrus tree because of the reorganisation required to run sync_server from master using prot streams for communication. It wouldn't take a huge amount of effort to add -v -v into standard Cyrus. I believe that Fastmail have an external test suite which does spot checks on the master and replica versions of each account. This is the opposite approach, and makes sense if you have a convenient IMAP client library. -- David Carter Email: [EMAIL PROTECTED] University Computing Service,Phone: (01223) 334502 New Museums Site, Pembroke Street, Fax: (01223) 334679 Cambridge UK. CB2 3QH. Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Hyphens in folder names break LIST
On Tue, 20 May 2008, Matthew Hodgson wrote: If I create a hierarchy of folders such as: test test.SPAM test-foo and try to list the folder hierarchy with something like: 11 LIST test% I get broken output, where test is listed twice - the second time with a \Noselect flag: The problem is that '-' sorts before '.' in ASCII. Try: improved_mboxlist_sort: 1 (You will need to dump and then restore the mboxlist). -- David Carter Email: [EMAIL PROTECTED] University Computing Service,Phone: (01223) 334502 New Museums Site, Pembroke Street, Fax: (01223) 334679 Cambridge UK. CB2 3QH. Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: cyr_expire -E ?
On Fri, 18 Apr 2008, David R Bosso wrote: I don't specify a -X, I just want to prune the duplicate db. What am I doing wrong? -X expunge-days Expunge previously deleted messages older than expunge-days (when using the delayed expunge mode). The default is 0 (zero) days, which will expunge all previously deleted messages. Try -X large number. cyr_expire is a bit overloaded. -- David Carter Email: [EMAIL PROTECTED] University Computing Service,Phone: (01223) 334502 New Museums Site, Pembroke Street, Fax: (01223) 334679 Cambridge UK. CB2 3QH. Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: mbexamine blocks mailbox
On Mon, 21 Jan 2008, Michael Menge wrote: i used mbexamine user/testuser | less to check the check the mailbox of testuser. I forgot to quit less over the weekend and today i discoverd two problems. 1. The mailbox was blocked. No new mails were deliverd to this mailbox. The mailbox was left in a locked state. (fcntl() or flock() locks on both cyrus.header and cyrus.index). 2. After quitting less, the new mails had been delivered multiple times depending on the retries of lmtpd. I imagine that you had a whole stack of lmtp processes waiting to acquire the lock on the mailbox. I'm a little suprised that something didn't time out, but I guess that the entire message would be sitting in the staging directory before lmtpd attempts to lock the target mailbox. is this a (known) bug? Are there other cyrus tools which will have the same effect? unexpunge -l ? Any command line tool which locks a mailbox and then generates output will have the same behaviour. unexpunge would be the obvious example. Whether this is a bug is debatable. Neither mbexamine or unexpunge actually needs to lock the mailbox when they are just dumping mailbox state, but locking is normally the safest course of action. They aren't supposed to be long running processes. -- David Carter Email: [EMAIL PROTECTED] University Computing Service,Phone: (01223) 334502 New Museums Site, Pembroke Street, Fax: (01223) 334679 Cambridge UK. CB2 3QH. Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: cyr_expire and messages where Date: is in the future?
On Wed, 9 Jan 2008, Mike Eggleston wrote: I tried and experiment that worked using 'ipurge -d 1 -X user.$user.spam'. The delete flags were set properly. I guess I still need to run a cyr_expire to expunge the messages? I want to \Delete all messages in user.*.spam and user.*.backup. Can I use those patterns with ipurge? ipurge -f -d 1 -X user/*/spam works for me (2.3cvs). The messages get expunged and expired immediately, which is what I would expect from: mailbox_expunge(the_box, purge_check, stats, EXPUNGE_FORCE); cyr_expire is normally used to expire expunged messages if you are running with delayed expunge. ipurge appears to bypass this. -- David Carter Email: [EMAIL PROTECTED] University Computing Service,Phone: (01223) 334502 New Museums Site, Pembroke Street, Fax: (01223) 334679 Cambridge UK. CB2 3QH. Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: cyr_expire and messages where Date: is in the future?
On Wed, 9 Jan 2008, David Carter wrote: ipurge -f -d 1 -X user/*/spam works for me. user/%/spam if I didn't want to match user/dpc22/foo/bar/spam -- David Carter Email: [EMAIL PROTECTED] University Computing Service,Phone: (01223) 334502 New Museums Site, Pembroke Street, Fax: (01223) 334679 Cambridge UK. CB2 3QH. Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: cyr_expire and messages where Date: is in the future?
On Wed, 9 Jan 2008, Mike Eggleston wrote: Using spam assassin and sieve I have messages flagged as spam filed into user.*.spam folders. I also have a 'cyr_expire -E 1 -X 1' job running each morning and a 'cyradm mboxcfg expire user.*.spam expire 2' set on all spam folders. One of the oddities I'm seeing is where a spam message has a header of 'Date: Fri, 28 Mar 2008 00:16:17 -0800'. These messages are not expiring even though the messages may physically be months or a year old. I believe that ipurge is the problem here, not cyr_expire. ipurge uses the Date: header unless you use the -X flag. Consequently a message with: Date: Fri, 28 Mar 2008 00:16:17 -0800 wouldn't be expunged until March. -- David Carter Email: [EMAIL PROTECTED] University Computing Service,Phone: (01223) 334502 New Museums Site, Pembroke Street, Fax: (01223) 334679 Cambridge UK. CB2 3QH. Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: cyr_expire and messages where Date: is in the future?
On Wed, 9 Jan 2008, Mike Eggleston wrote: Ok so 'user/%/spam' only matches user/$user/spam. Using 'user/*/spam' matches user/$user/.../spam, right? Must I use '/' in the pattern or can/do I use '.'? I use unixhiersep, hence '/'. If you don't, use '.' Is there some undocumented feature like a '-n' to ipurge so I can test what this command does without erasing all my user's messages in the 'test'? Try it on a test system first :). ipurge gives a fairly decent running commentary about the mailboxes it is processing. You could always take the source code and comment out the mailbox_expunge() if you want to test on a live system. -- David Carter Email: [EMAIL PROTECTED] University Computing Service,Phone: (01223) 334502 New Museums Site, Pembroke Street, Fax: (01223) 334679 Cambridge UK. CB2 3QH. Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: skiplist_unsafe
On Fri, 7 Dec 2007, Janne Peltonen wrote: If you feel that your filesystem/buffercache will do a good job at writing things out to disk, and you've got battery-backed cache on your storage, you should be relatively well off. But if I were to turn skiplist_unsafe on, and the OS crashed - or, say, the cluster system forcibly unmounted my Cyrus spool and config filesystems - wouldn't that result in horribly unrecoverable databases all over the place? (I have everything in skiplist, except quota and subscriptions.) It is easy enough to find out. Take an fsync() test rig such as Brad Fitzpatrick's diskchecker.pl and comment out the fsync()s. If the disk checker moans, then updates have been lost in buffer cache. Under Linux this is only safe if the filesystem is mounted with the sync option, even with data=journal. Part of the point of fsync() is to make sure updates hit nonvolatile storage in the correct order. A specific example: skiplist commit records are written after an fsync(), immediately followed by another fsync() before the write lock is released. If writes get reordered before they hit disk, then there is a good chance that the database will become corrupt. -- David Carter Email: [EMAIL PROTECTED] University Computing Service,Phone: (01223) 334502 New Museums Site, Pembroke Street, Fax: (01223) 334679 Cambridge UK. CB2 3QH. Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Replication error
On Wed, 5 Dec 2007, Gabor Gombas wrote: Hmm, can a regular user rename his own INBOX? I'm pretty sure no admin actions were performed. RFC 3501, section 6.3.5: Renaming INBOX is permitted, and has special behavior. It moves all messages in INBOX to a new mailbox with the given name, leaving INBOX empty. If the server implementation supports inferior hierarchical names of INBOX, these are unaffected by a rename of INBOX. -- David Carter Email: [EMAIL PROTECTED] University Computing Service,Phone: (01223) 334502 New Museums Site, Pembroke Street, Fax: (01223) 334679 Cambridge UK. CB2 3QH. Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Replication error
On Tue, 4 Dec 2007, Wesley Craig wrote: The internal Cyrus mailbox ID ought to be unique, but it's not. On the sub folder, remove the cyrus.header file and reconstruct. This will assign a new, unique mailbox ID. Any ideal how they ended up with the same IDs? Given that a user inbox is involved, my guess would be: Changes to the Cyrus IMAP Server since 2.3.9 [...] * Fixed the special case of RENAMEing an Inbox, so that it doesn't keep the same mailbox uniqueid, thus allowing it to replicate properly (seen state is still preserved). -- David Carter Email: [EMAIL PROTECTED] University Computing Service,Phone: (01223) 334502 New Museums Site, Pembroke Street, Fax: (01223) 334679 Cambridge UK. CB2 3QH. Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: LARGE single-system Cyrus installs?
On Sun, 11 Nov 2007, Bron Gondwana wrote: 250,000 mailboxes, 1,000 concurrent users, 60 million emails, 500k deliveries/day. For us, backups are the worst thing, followed by reiserfs's use of BLK, followed by the need to use a ton of disks to keep up with the i/o. For us backups are hardly a blip on the radar :) The joy of writing your own custom backup system that knows more about Cyrus internals than just about anything else. It starts with some stat calls, and if any of the cyrus.header, cyrus.index or cyrus.expunge files have changed then it will lock them all then stream them all to the backup server. Cyrus is pretty ideal for fast incremental updates to a backup system: hence replication. You shouldn't need to lock anything with delayed expunge, delayed delete and fast rename in place. -- David Carter Email: [EMAIL PROTECTED] University Computing Service,Phone: (01223) 334502 New Museums Site, Pembroke Street, Fax: (01223) 334679 Cambridge UK. CB2 3QH. Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Replication: sync_client -r dies
On Mon, 12 Nov 2007, Bron Gondwana wrote: It seems to me that the replication code ought to be a bit more robust than this when a replica goes down or loses network connectivity. Is the 2.3.10 code any better than 2.3.9 in the way this kind of situation is handled? I believe David Carter has been working on some stuff for this which is lined up to go in soon. The autorestart stuff is already in 2.3.10. It was Ken's work, based on a suggestion on my part. -- David Carter Email: [EMAIL PROTECTED] University Computing Service,Phone: (01223) 334502 New Museums Site, Pembroke Street, Fax: (01223) 334679 Cambridge UK. CB2 3QH. Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Replication: does it work in both directions?
On Sun, 11 Nov 2007, Rich Wales wrote: So, I would have replication set up going both directions between my two servers, but the sets of users handled in each direction would be disjoint. Each user would be assigned to one IMAP server (the master for their mailbox collection), and the other server would be their replica and act as their backup. We do this. It is quite useful to be able to bounce users back and forth between the two machines in a pair so that servers can be maintained (patches, O/S upgrades, whatever) without any user visible downtime. Three caveats: 1) It won't work with shared mailboxes. 2) I'm not running the same replication code as the rest of you (though replication in 2.3 is based on an old version of my code). I seem to remember Ken raising an objection when this last discussed a year or two back now. The objection may just have just been (1). 3) Sanity checks are good: USER dpc22 NO IMAP_INVALID_USER Attempt to update master for dpc22 -- David Carter Email: [EMAIL PROTECTED] University Computing Service,Phone: (01223) 334502 New Museums Site, Pembroke Street, Fax: (01223) 334679 Cambridge UK. CB2 3QH. Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Multiple skiplist bugs found, patches attached
On Tue, 13 Nov 2007, Simon Matter wrote: I didn't have much troubles with skiplist over the years and it has been a blessing since moving away from BDB. But I did have a few issues with broken skiplist files so your patches are very welcome. I have included the patches in my private rpm packages to try how they work. Do you recommend both for general consumption? It is certainly very easy to break mailboxes.db using cyr_dbtool. Kudos to Bron for tracking down the problems. -- David Carter Email: [EMAIL PROTECTED] University Computing Service,Phone: (01223) 334502 New Museums Site, Pembroke Street, Fax: (01223) 334679 Cambridge UK. CB2 3QH. Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Just in case it is of general interest: ZFS mirroring was the culprit in our case
On Tue, 13 Nov 2007, Pascal Gienger wrote: Our latency problems went away like a miracle when we detached one half of the mirror (so it is no more a mirror). Read-Rates are doubled (not per device, the total read rate!), latency is cut off. No more latency problems. When attaching the volume again, resilvering puts the system to a halt - reads and writes do block for seconds (!). Definitely of interest to those of us keeping one eye on ZFS. Thanks. Can someone else running ZFS confirm this behaviour? -- David Carter Email: [EMAIL PROTECTED] University Computing Service,Phone: (01223) 334502 New Museums Site, Pembroke Street, Fax: (01223) 334679 Cambridge UK. CB2 3QH. Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: LARGE single-system Cyrus installs?
On Tue, 13 Nov 2007, Bron Gondwana wrote: If you're planning to lift a consistent copy of a .index file, you need to lock it for the duration of reading it (read lock at least). mailbox_lock_index() blocks flag updates (but this doesn't seem to be something that imapd worries about when FETCHing data). You don't need to worry about expunge or append events once the mailbox is open. But since I would like a consistent snapshot of the mailbox state, I lock the cyrus.header and then the cyrus.index and then (if it's there) the cyrus.expunge. That means no sneaky process could (for example) delete the mailbox and create another one with the same name while I was busy downloading the last file - giving me totally bogus data. chdir() into the mailbox data directory: with delayed delete and fast rename it shouldn't matter if the mailbox is replaced under your feet. That's the way replication worked on my 2.1 systems, prior to split-meta. (Locking isn't a big deal, but safe concurrent access is always nice). -- David Carter Email: [EMAIL PROTECTED] University Computing Service,Phone: (01223) 334502 New Museums Site, Pembroke Street, Fax: (01223) 334679 Cambridge UK. CB2 3QH. Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Deleting top-level mailbox with 'delete_mode: delayed'
On Tue, 13 Nov 2007, Bron Gondwana wrote: I have delete_mode: immediate on the replica and delete_mode: delayed on the master. sync_server doesn't pay any attention to delete_mode, so the option shouldn't have any effect on the replica. -- David Carter Email: [EMAIL PROTECTED] University Computing Service,Phone: (01223) 334502 New Museums Site, Pembroke Street, Fax: (01223) 334679 Cambridge UK. CB2 3QH. Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: IOERROR: reading message: unexpected end of file (message_copy_strict)
On Tue, 23 Oct 2007, Ken Murchison wrote: Your problem is most likely related to using NFS. NFS has never been recommended for Cyrus because is doesn't play nice with mmap() and flock(), both of which are critical to the operation of Cyrus. While I agree entirely with don't use Cyrus over NFS, I see these errors using a local filesystem. A quick grep pins the likely cause down to message_copy_strict(), which is called by append_fromstream(). I don't think that this is anything more sinister than TCP connections dropping out partway through a large IMAP APPEND operation. Entirely safe. -- David Carter Email: [EMAIL PROTECTED] University Computing Service,Phone: (01223) 334502 New Museums Site, Pembroke Street, Fax: (01223) 334679 Cambridge UK. CB2 3QH. Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: squatter running longer than 24 hours
On Sun, 21 Oct 2007, Vincent Fox wrote: I have seen squatter run more than 24 hours. This is on a large mail filesystem. I've seen it start up a second one while the first is still running. Should I: 1) Forget about squatter 2) Remove from cyrus.conf, run from cron every other day 3) Find some option to cyrus.conf for same effect as #2? I squat a fraction of mailboxes each night using: http://www-uxsup.csx.cam.ac.uk/~dpc22/cyrus/patches/2.3.8/squatter.patch For example: squatter -s -m 0 -M 7 would update the squat indexes for 1 in 7 mailboxes, based on modulo arithmetic on the mailbox UniqueID. squatter would really benefit from incremental updates. At the moment a single new message in a mailbox containing 20k messages causes it to read in all the existing messages in order to regenerate the index. Unfortunately, the code is rather impenetrable. I infer that it is collecting information about adjacent characters in the message body. Presumably a 5 character search term provides 4 required pairing as a prefilter from the squat engine before message by message search kicks in. -- David Carter Email: [EMAIL PROTECTED] University Computing Service,Phone: (01223) 334502 New Museums Site, Pembroke Street, Fax: (01223) 334679 Cambridge UK. CB2 3QH. Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: LARGE single-system Cyrus installs?
On Sat, 6 Oct 2007, Rob Mueller wrote: Are you comparing an old reiserfs partition with a new ext3 one where you've just copied the email over to? If so, that's not a fair comparison. No, a newly created partitions in both cases. Fragmented partitions are slower still of course. Give it a month or two of active use though (delivering new emails, deleting old ones, etc), and everything starts getting fragmented again. Then ext3 really started going to crap on us. Machines that had been absolutely fine under reiserfs, the load just blew out to unuseable under ext3. We've only been using ext3 for about 3 months now, so I may still have this to look forward to :). Talking with Chris Mason about this, data=journal is faster in certain scenarios with lots of small files + fsyncs from different processes, exactly the type of workload cyrus generates! I can't see much difference on our Cyrus systems, but battery backed write cache on our RAID controllers probably masks a lot of the change. I agree that it theory it should make a very substantial difference. As it turns out, the memory leaks weren't critical, because the the pages do seem to be reclaimed when needed, though it was annoying not knowing exactly how much memory was really free/used. Okay, I think that we had a different kernel memory bug. We were running out of memory after 24 hours, and a 20 line test program could exhaust memory in seconds. This bug was in SLES four years back, and it was still there the last time that I looked (some months back now). -- David Carter Email: [EMAIL PROTECTED] University Computing Service,Phone: (01223) 334502 New Museums Site, Pembroke Street, Fax: (01223) 334679 Cambridge UK. CB2 3QH. Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: LARGE single-system Cyrus installs?
On Sat, 6 Oct 2007, Rob Mueller wrote:
That's strange. What mount options are/were you using? We use/used:
reiserfs - rw,noatime,nodiratime,notail,data=journal
ext3 - noatime,nodiratime,data=journal
Same, but data=ordered in both cases
If you weren't using notail on reiserfs, that would definitely have a
performance impact.
Definitely using notail.
Wow weird, must be something different. What kernel was it? Do you know
where the memory leak was occuring?
Standard SLES kernels for SLES9.
The memory leak could be show by mmap() on a single file (see attachment).
Kernel memory explodes, and nothing is released when the program exits.
--
David Carter Email: [EMAIL PROTECTED]
University Computing Service,Phone: (01223) 334502
New Museums Site, Pembroke Street, Fax: (01223) 334679
Cambridge UK. CB2 3QH.
#include stdio.h
#include sys/types.h
#include sys/stat.h
#include fcntl.h
#include unistd.h
#include stdlib.h
#include string.h
#include sys/mman.h
#define BLOCK_SIZE (4096)
static void fatal(char *err)
{
fprintf(stderr, %s\n, err);
exit(1);
}
int main(int argc, char *argv[])
{
int i;
int fd, fd2;
char *s;
if (argc != 3)
fatal(Args: source file dest file);
for (i=0; i 10 ; i++) {
if ((fd = open(argv[1], O_RDONLY)) 0)
fatal(open);
if ((s=mmap(NULL, BLOCK_SIZE, PROT_READ, MAP_SHARED, fd, 0)) == NULL)
fatal(mmap);
if ((fd2 = open(argv[2], O_WRONLY|O_CREAT|O_TRUNC, 0666)) 0)
fatal(open);
if (write(fd2, s, BLOCK_SIZE) BLOCK_SIZE)
fatal(write);
if (close(fd2) 0)
fatal(close);
if (munmap(s, BLOCK_SIZE) 0)
fatal(munmap);
if (close(fd) 0)
fatal(close);
}
return(0);
}
Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: IOERROR: writing cache file
On Fri, 28 Sep 2007, Bron Gondwana wrote: I can tell you I sleep a lot easier at night knowing that there are replication systems in place so we can lose an entire server and at worst a few emails will be lost. Indeed. It sounds like we might be getting a second machine room Real Soon Now, which was kind of the whole point when I started back in 2002. Hurrah. -- David Carter Email: [EMAIL PROTECTED] University Computing Service,Phone: (01223) 334502 New Museums Site, Pembroke Street, Fax: (01223) 334679 Cambridge UK. CB2 3QH. Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Unexpected response MessageID 000000000000000000000000
On Thu, 27 Sep 2007, Rudy Gevaert wrote:
I'm seeing this error.
/sync_client[28862]: RESERVE: Unexpected response MessageID
in
Does anybody have an idea what it is triggered by.
Bug fixed in 2.3 CVS earlier this week. (the missing (i count) in
cmd_reserve). I attach the message that I sent to cyrus-devel.
sync_client will be ignoring the spurious responses.
--
David Carter Email: [EMAIL PROTECTED]
University Computing Service,Phone: (01223) 334502
New Museums Site, Pembroke Street, Fax: (01223) 334679
Cambridge UK. CB2 3QH.
/* === */
From [EMAIL PROTECTED] Tue Sep 25 11:00:38 2007
Date: Tue, 25 Sep 2007 10:53:56 +0100 (BST)
From: David Carter [EMAIL PROTECTED]
To: [EMAIL PROTECTED]
Subject: sync_server RESERVE command
Two separate bugs fixes, against current 2.3 CVS.
The missing i++ means that we would have failed to RESERVE some messages
where a given GUID appears multiple times in a mailbox. Not a huge deal.
The missing (i count) was potentially serious: we fall off the end of the
ids array where RESERVE doesn't involve the last message in a mailbox.
I think that we got away with it. My reasoning follows:
If you are really unlucky 12/20 bytes of allocated but uninitiated memory
(ids[count]) will match one of the GUIDs later in the mailbox. This will
cause cmd_reserve() to reserve that message and issue a spurious response.
Fortunately sync_client will moan and then ignore the response if it wasn't
expecting the GUID. If it _was_ expecting the GUID then sync_server has
reserved a legitimate copy. In fact this should be impossible. If
sync_client wanted a message with that GUID, it would have asked for that
copy in that mailbox, so the ids array would have been longer.
ids[count] is always allocated.
There is a one in hundred chance that ids[count+1] is not, which would lead
to a segmentation fault if ids[count] matched a GUID.
--
David Carter Email: [EMAIL PROTECTED]
University Computing Service,Phone: (01223) 334502
New Museums Site, Pembroke Street, Fax: (01223) 334679
Cambridge UK. CB2 3QH.
Index: imap/sync_server.c
===
RCS file: /cvs/src/cyrus/imap/sync_server.c,v
retrieving revision 1.11
diff -u -d -r1.11 sync_server.c
--- imap/sync_server.c 24 Sep 2007 12:48:32 - 1.11
+++ imap/sync_server.c 25 Sep 2007 09:28:09 -
@@ -1465,14 +1465,16 @@
goto cleanup;
}
-for (i = 0, msgno = 1 ; msgno = m.exists; msgno++) {
+for (i = 0, msgno = 1 ; (msgno = m.exists) (i count); msgno++) {
mailbox_read_index_record(m, msgno, record);
if (!message_guid_compare(record.guid, ids[i]))
continue;
-if (sync_message_find(message_list, record.guid))
+if (sync_message_find(message_list, record.guid)) {
+i++;
continue; /* Duplicate GUID on RESERVE list */
+}
/* Attempt to reserve this message */
snprintf(mailbox_msg_path, sizeof(mailbox_msg_path),
Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Replication and single instance store
On Tue, 4 Sep 2007, Bron Gondwana wrote: for all the users across which the single instance store needs to apply, then run 'sync_client -r -f $file'. I typically use -u -f to do this. However: Creating files like this and passing them with the -f flag forces sync_client to consider them in the same run, so it finds the matching message on the replica. sync_server maintains a fairly modest UUID cache on the server side: 1000 messages in 2.3. A restart is negotiated after each UPLOAD command. -- David Carter Email: [EMAIL PROTECTED] University Computing Service,Phone: (01223) 334502 New Museums Site, Pembroke Street, Fax: (01223) 334679 Cambridge UK. CB2 3QH. Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Replication and single instance store
On Tue, 4 Sep 2007, Bron Gondwana wrote: Ah - yeah, that's right. Except that the restart only got negotiated after each folder was processed, and if you're pushing a new folder with 200,000 messages (say, after a user move in our case) then that got a bit memory hungry and all sorts insane. Yes, this was the 5*MAX_MAILBOX_PATH allocation for each message when support for partitions was added. My original code cached 300k messages on the server between restarts (without any substantial memory leak). Better, but not perfect. Does this mean there is no way to get single-instance-store on a replica if you're rebuilding it from scratch? No. You would need a database which maintained a persistent mapping between UUID and a list of files on each partitition which are that UUID. I'm open to suggestions. -- David Carter Email: [EMAIL PROTECTED] University Computing Service,Phone: (01223) 334502 New Museums Site, Pembroke Street, Fax: (01223) 334679 Cambridge UK. CB2 3QH. Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: storing mail across several cyrus partitions
On Sun, 2 Sep 2007, Bokhan Artem wrote: Sorry, I didn't understand you clearly... Did you mean, that subfolders of single user may be moved across partitions? Yes. -- David Carter Email: [EMAIL PROTECTED] University Computing Service,Phone: (01223) 334502 New Museums Site, Pembroke Street, Fax: (01223) 334679 Cambridge UK. CB2 3QH. Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: storing mail across several cyrus partitions
On Fri, 31 Aug 2007, Artem Bokhan wrote: Partition or sub-partition -- no matter. The purpose -- to store the mail of a single user accross different storages. Yes, you can do this. /etc/imapd.conf: partition-default: /spool/cyrus/mailstore partition-test: /spool/cyrus/test/data metapartition-test: /spool/cyrus/test/meta partition-test2: /spool/cyrus/test2/data metapartition-test2: /spool/cyrus/test2/meta partition-test3: /spool/cyrus/test3/data metapartition-test3: /spool/cyrus/test3/meta Raw IMAP as a cyrus admin user (simply because it gives more feedback than the cyradm rename command), using the optional third argument to rename. Start out with three mailboxes on default partition: . LIST user/dpc22 * * LIST (\HasChildren) / user/dpc22 * LIST (\HasNoChildren) / user/dpc22/bar * LIST (\HasNoChildren) / user/dpc22/foo . OK Completed (0.000 secs 4 calls) Move user/dpc22/foo and user/dpc22/bar to test2 and test3: . RENAME user/dpc22/foo user/dpc22/foo test2 * OK rename user/dpc22/foo user/dpc22/foo . OK Completed . RENAME user/dpc22/bar user/dpc22/bar test3 * OK rename user/dpc22/bar user/dpc22/bar . OK Completed The only gotcha is that each rename moves all subsidiary mailboxes: . RENAME user/dpc22 user/dpc22 default * OK rename user/dpc22 user/dpc22 * OK rename user/dpc22/bar user/dpc22/bar * OK rename user/dpc22/foo user/dpc22/foo . OK Completed -- David Carter Email: [EMAIL PROTECTED] University Computing Service,Phone: (01223) 334502 New Museums Site, Pembroke Street, Fax: (01223) 334679 Cambridge UK. CB2 3QH. Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Replication to more than one replica?
On Fri, 10 Aug 2007, Per olof Ljungmark wrote: It would be a way to keep a second offline replica for backing up to a tape archive, which is what I plan to do. This is certainly what we do, and it seems to work nicely. -- David Carter Email: [EMAIL PROTECTED] University Computing Service,Phone: (01223) 334502 New Museums Site, Pembroke Street, Fax: (01223) 334679 Cambridge UK. CB2 3QH. Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: 8G RAM in 32bit platform
On Fri, 13 Jul 2007, Patrick T. Tsang wrote: We will start up the mail server with 4G RAM. As I know the 32bits cannot handle RAM more than 3.2G. The client plans to upgrade the RAM to 8G in coming years. Can the 64bits platform is the only solution to it? You don't say which CPU or operating system you are using. The Linux bigsmp kernel supports PAE extensions on IA32 platforms: all 8 GBytes will be available as buffer cache, which is what matters to Cyrus. 64 bit pointers don't really do anything: no single process in Cyrus needs 2 GBytes of address space. 64 bit integer arithmetic would be a slight benefit for quota arithmetic (unsigned long long). However my systems spend about 2% of their time in user CPU state according to vmstat. You really aren't going to notice on any modern Intel/AMD CPU. -- David Carter Email: [EMAIL PROTECTED] University Computing Service,Phone: (01223) 334502 New Museums Site, Pembroke Street, Fax: (01223) 334679 Cambridge UK. CB2 3QH. Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: LTMPD rejecting large messages, maxmessagesize is _not_ set
On Tue, 10 Jul 2007, Chris St. Pierre wrote: LMTPD is rejecting large messages; I've been unable to figure out the exact threshold, but I am seeing messages like this in my Postfix logs: Jun 28 20:23:12 vostok postfix/qmgr[9323]: 22F5373D6F6: from=[EMAIL PROTECTED], size=16243464, nrcpt=1 (queue active) Jun 28 20:23:22 vostok postfix/lmtp[13405]: warning: non-LMTP response from imap.nebrwesleyan.edu[10.1.1.31]: sendmail: fatal: [EMAIL PROTECTED](76): Message file too big I don't think that Cyrus generated that error messages. Try strings on the lmtpd binary. Errors from Cyrus should be all variants on: ec IMAP_MESSAGE_TOO_LARGE, Message size exceeds fixed limit Is sendmail/postfix using a staging partition which has run out space? -- David Carter Email: [EMAIL PROTECTED] University Computing Service,Phone: (01223) 334502 New Museums Site, Pembroke Street, Fax: (01223) 334679 Cambridge UK. CB2 3QH. Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: cyrus 2.3.8, internal dates not stored in message file time ?
On Fri, 22 Jun 2007, Nicolas KOWALSKI wrote: The source and target mailboxes are on the same user account, same partition, but strace shows that link is not used in 2.3.8. The following traces are obtained with strace, when copying messages from testbox to testbox3 with cyrus 2.2.12 and to testbox4 with cyrus 2.3.8. In both case, I used the same configuration/mailstore/client: [...] There must be something really wrong in my configuration... Is singleinstancestore disabled in your imapd.conf? While mailbox_copyfile() hasn't changed, the parent routine index_copy() appears to have gained an extra argument, and the top level cmd_copy handler in imapd.c has: r = index_copy(imapd_mailbox, sequence, usinguid, mailboxname, copyuid, !config_getswitch(IMAPOPT_SINGLEINSTANCESTORE)); -- David Carter Email: [EMAIL PROTECTED] University Computing Service,Phone: (01223) 334502 New Museums Site, Pembroke Street, Fax: (01223) 334679 Cambridge UK. CB2 3QH. Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Replication question - cross replication?
On Thu, 14 Jun 2007, Nels Lindquist wrote: I'm setting up a high-availability mail server setup with two boxes that will essentially be mirrors of each other. If both are configured for local delivery, can I have them replicate each other if I utilize UUIDs? No. IMAP is not well suited to active-active replication. Replication in Cyrus is strictly active-passive. -- David Carter Email: [EMAIL PROTECTED] University Computing Service,Phone: (01223) 334502 New Museums Site, Pembroke Street, Fax: (01223) 334679 Cambridge UK. CB2 3QH. Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Cyrus with a NFS storage. random DBERROR
On Sun, 10 Jun 2007, Rob Mueller wrote: I can try and keep an eye on bailouts some more, and see if I can get some more details. It would be nice if there was some more logging about why the bail out code path was actually called! It typically means that something deep in libimap has thrown an error. sync_client logs the only information that it has (the return code r). It probably wouldn't hurt to try and log the current mailbox/user in some consistent fashion. -- David Carter Email: [EMAIL PROTECTED] University Computing Service,Phone: (01223) 334502 New Museums Site, Pembroke Street, Fax: (01223) 334679 Cambridge UK. CB2 3QH. Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Cyrus with a NFS storage. random DBERROR
On Sat, 9 Jun 2007, Rob Mueller wrote:
I run it directly, outside of master. That way when it crashes, it
can be easily restarted. I have a script that checks that it's
running, that the log file isn't too big, and that there are no log-
PID files that are too old. If anything like that happens, it pages
someone.
Ditto, we do almost exactly the same thing.
And for that matter, so I do.
I think there's certain race conditions that still need ironing out,
because rerunning sync_client on the same log file that caused a bail
out usually succeeds the second time.
I suspect that the problem is with mailbox renames, which are not atomic
and can take some time to complete with very large mailboxes.
sync_client retries a number of times and then bails out.
if (folder_list-count) {
int n = 0;
do {
sleep(n*2); /* XXX should this be longer? */
...
} while (r (++n SYNC_MAILBOX_RETRIES));
if (r) goto bail;
}
This was one of the most significant compromises that Ken had to make when
integrating my code into 2.3.
My original code cheats, courtesy of two other patches:
HERMES_FAST_RENAME:
Translates mailbox rename into filesystem rename() where possible.
Useful because sync_client chdir()s into the working directory.
Would be less useful in 2.3 with split metadata.
HERMES_SYNC_SNAPSHOT:
If mailbox action fails, promote to user action (no shared mailboxes)
If user action fails then lock user out of the mboxlist and try again.
Together with my version of delayed expunge this pretty much guarantees
that things aren't moving around under sync_client's feet. Its been an
awful long time (about a year?) since I last had a sync_client bail out.
We are moving to 2.3 over the summer (initially using my own original
replication code), so this is something that I would like to sort out.
Any suggestions?
--
David Carter Email: [EMAIL PROTECTED]
University Computing Service,Phone: (01223) 334502
New Museums Site, Pembroke Street, Fax: (01223) 334679
Cambridge UK. CB2 3QH.
Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: sync_client doesn't sync sieve scripts
On Mon, 26 Mar 2007, Dmitriy Kirhlarov wrote: I have properly configured sync between two cyrus-imapd 2.3.8 servers. Mailboxes rolling synchronization work good. This also updates sieve scripts. Now I want to synchronized sieve scripts too. sync_client -v -s sync_client -v -s $username -s is new in 2.3, but it looks like it was only there for testing. The manual page says: Principally used for debugging purposes: not exposed to sync_client -u should replicate an entire user including the Sieve files. -- David Carter Email: [EMAIL PROTECTED] University Computing Service,Phone: (01223) 334502 New Museums Site, Pembroke Street, Fax: (01223) 334679 Cambridge UK. CB2 3QH. Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: FastMail.FM patchset - new patches
On Thu, 15 Mar 2007, John Capo wrote:
masterconf_getsection() calls add_service() which inits the service
structures with have_uuid but have_uuid is not set till after the
services are initialized.
Ah, have_uuid is new in 2.3. That line definitely needs to move, but I
think that the message_uuid_master_init() call should stay where it is.
--
David Carter Email: [EMAIL PROTECTED]
University Computing Service,Phone: (01223) 334502
New Museums Site, Pembroke Street, Fax: (01223) 334679
Cambridge UK. CB2 3QH.
--- master/master.c-DIST2007-03-16 08:45:56.0 +
+++ master/master.c 2007-03-16 08:46:13.0 +
@@ -1938,6 +1938,8 @@
/* set signal handlers */
sighandler_setup();
+have_uuid = (config_getint(IMAPOPT_SYNC_MACHINEID) = 0);
+
/* initialize services */
for (i = 0; i nservices; i++) {
service_create(Services[i]);
@@ -1954,7 +1956,6 @@
}
}
-have_uuid = (config_getint(IMAPOPT_SYNC_MACHINEID) = 0);
if (have_uuid !message_uuid_master_init()) {
syslog(LOG_ERR, Couldn't initialise UUID subsystem);
exit(EX_OSERR);
Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: FastMail.FM patchset - new patches
On Fri, 16 Mar 2007, David Carter wrote:
Ah, have_uuid is new in 2.3. That line definitely needs to move, but I
think that the message_uuid_master_init() call should stay where it is.
Or even (as per the fastmail patch).
--
David Carter Email: [EMAIL PROTECTED]
University Computing Service,Phone: (01223) 334502
New Museums Site, Pembroke Street, Fax: (01223) 334679
Cambridge UK. CB2 3QH.
--- master/master.c-DIST2007-03-16 08:45:56.0 +
+++ master/master.c 2007-03-16 09:49:01.0 +
@@ -1931,6 +1931,8 @@
init_snmp(cyrusMaster);
#endif
+have_uuid = (config_getint(IMAPOPT_SYNC_MACHINEID) = 0);
+
masterconf_getsection(START, add_start, NULL);
masterconf_getsection(SERVICES, add_service, NULL);
masterconf_getsection(EVENTS, add_event, NULL);
@@ -1954,7 +1956,6 @@
}
}
-have_uuid = (config_getint(IMAPOPT_SYNC_MACHINEID) = 0);
if (have_uuid !message_uuid_master_init()) {
syslog(LOG_ERR, Couldn't initialise UUID subsystem);
exit(EX_OSERR);
Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: FastMail.FM patchset - new patches
On Thu, 15 Mar 2007, Bron Gondwana wrote: * Make UUIDs work at all. The initialisation order of the UUID subsystem was wrong, so we had very few messages with a non-zero UUID. message_uuid_master_init() should only be called after become_cyrus(). Every time that message_uuid_master_next_child() overflows, master needs to update MASTER_UUID_FILE. Is MASTER_UUID_FILE owned by a user other than cyrus on your systems? I can't see any other obvious reason that moving message_uuid_master_init() up a few lines would help. service_create() is binding ports, which obviously has to happen as root, while masterconf_getsection() is just parsing master.conf. I am probably missing something obvious here, but the current ordering works for me. * MD5 UUIDs - we've created a new scheme for UUID generation, of the format: 02[first 11 bytes of message file md5]. This allows some basic integrity checking of the file on disk, and is still plenty random. Also adds the non standard IMAP FETCHable items UUID, RFC822.MD5 (calculated on the fly), RFC822.FILESIZE (does a stat or looks at the MMAP result if something else needs it) I don't think that this is safe. It is important that the UUIDs really are unique, which is the reason for the paranoia in message_uuid_master_init. The assertion: Is it safe? - we calulated that with one billion messages you have a one in 1 billion chance of a birthday collision (two random messages with the same UUID). They then have to get in the same MAILBOXES collection to sync_client to affect each other anyway. Isn't the case: UUIDs span all MAILBOXES and APPEND event until a restart. If a UUID appears in one event and then is referenced by a second event some minutes later then the first message seen will be reused. At the moment sync_client in 2.3 tracks the last few thousand messages by UUIDs. My original code tracked the last few hundred thousand messages (diminishing returns, but useful when seeding accounts). There is a much greater chance of collisions there than just comparing two messages. -- David Carter Email: [EMAIL PROTECTED] University Computing Service,Phone: (01223) 334502 New Museums Site, Pembroke Street, Fax: (01223) 334679 Cambridge UK. CB2 3QH. Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: FastMail.FM patchset - new patches
On Thu, 15 Mar 2007, Rob Mueller wrote: May not be true, but: Is it safe? - we calulated that with one billion messages you have a one in 1 billion chance of a birthday collision (two random messages with the same UUID). Is true. Fair enough. With hindsight I should probably have defined message UUIDs to be the full MD5 hash: 128 bits isn't that much worse than 96 bits per message. What is the CPU overhead like for calculating MD5 sums for everything on the fly? UUIDs started out life as Mailbox UniqueID (64 bits) plus Message UID (32 bits), hence the size and rather unfortunate name. The hash algorithmn used to generate mailbox uniqueIDs is a bit basic, which is why I switched to generating them on the fly from master. -- David Carter Email: [EMAIL PROTECTED] University Computing Service,Phone: (01223) 334502 New Museums Site, Pembroke Street, Fax: (01223) 334679 Cambridge UK. CB2 3QH. Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: single instance store and replication
On Thu, 1 Mar 2007, Jerome Nenert wrote: I'm using Cyrus replication. After several tests, it seems that the single instance store facility is not replicated. I mean, the same message sent to several recipient is stored once on the master, but stored several times on the slave. Is there a special thing to do to activate single instance store replication or it just doesn't exist yet? Message UUIDs are used to replicate the single instance store (see docs/text/install-replication). This won't have much effect when you first replicate a mailstore as sync_server in 2.3 only tracks the last few thousand messages that have been uploaded. It becomes much more effective when a replica has been seeded and you switch to rolling replication. -- David Carter Email: [EMAIL PROTECTED] University Computing Service,Phone: (01223) 334502 New Museums Site, Pembroke Street, Fax: (01223) 334679 Cambridge UK. CB2 3QH. Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Potential replica message file corruption/replacement
On Fri, 16 Feb 2007, Bron Gondwana wrote: Looks innocent, doesn't it... Mea culpa (and a definite Argh, how did I miss _that_ when it was pointed out to me yesterday). I would advise anyone who has been using replication for any length of time to undertake an audit of the files on their replicas to ensure that none of them have been replaced by this, because if you need to fail over you could present users with emails that are not their own. A simple size check will find almost all cases, compare what the imapd returns for rfc822.size with the size of the file on disk. If you want to get fancy - compute the sha1 or similar of the file at each end and compare that. This incident underlines the need for automated sanity checks. People shouldn't just blindly trust the replication system. I generate (and constantly regenerate) checksums for message bodies and cache entries. On four occasions this has picked up oddities which in hindsight were obviously this bug. -- David Carter Email: [EMAIL PROTECTED] University Computing Service,Phone: (01223) 334502 New Museums Site, Pembroke Street, Fax: (01223) 334679 Cambridge UK. CB2 3QH. Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: load balancing at fastmail.fm
On Mon, 12 Feb 2007, Marten Lehmann wrote: what do you think about moving the mailspool to a central SAN storage shared via NFS and having several blades to manage the mmapped files like seen state, quota etc.? Why do you need NFS? The whole point of a SAN is distributed access to storage after all :). So still only one server is responsible for a certain set of mailboxes, but these SAN boxes have nice backup and redundancy features which are hard to get with common servers It depends how much you trust your SAN. Some of my colleagues who run a SAN have had no end of grief. At which point you are dependant on the abilities of the vendor to diagnose and fix problems. It was this experience that encouraged me to try application level replication with lots of small servers in the first place. At least that way I can keep a close eye on what the various copies are up to. A SAN doesn't protect you if your filesystem decides to explode: I believe that Fastmail have direct experience of this. Two independent copies of the data allows you to keep running a service for the hours that an fsck typically takes to complete with file per msg stores on large modern disks. It also means rather less stress if the fsck fails to complete. I've heard horror stories about all the common Linux filesystems and I've personally watched fsck.ext3 (supposedly the safest option) unravel a filesystem, with thousands of entries left in lost+found. ZFS looks nice. -- David Carter Email: [EMAIL PROTECTED] University Computing Service,Phone: (01223) 334502 New Museums Site, Pembroke Street, Fax: (01223) 334679 Cambridge UK. CB2 3QH. Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: load balancing at fastmail.fm
On Mon, 12 Feb 2007, Marten Lehmann wrote: because NFS is the only standard network file protocol. I don't want to load a proprietary driver into the kernel to access a SAN device. Fair enough, although NFS is likely to be really rather slow compared to a block device which just happens to be accessed via a fibre channel link. I would be surprised if NFS worked given that it is only a approximation to a real Unix filesystem. Cyrus really hammers the filesystem. I've heard horror stories about all the common Linux filesystems and I've personally watched fsck.ext3 (supposedly the safest option) unravel a filesystem, with thousands of entries left in lost+found. ext3 with journal? I have never experienced this. It was in a RAID set which had had a dodgy disk, but there was a definite urk moment when I saw what fsck had done. Fortunately not critical data. ZFS looks nice. Well, but you are on your own because this project for linux is pretty young. I don't have any problem with OpenSolaris, though it would be a little amusing given that we moved from Solaris to Linux about 4 years back. -- David Carter Email: [EMAIL PROTECTED] University Computing Service,Phone: (01223) 334502 New Museums Site, Pembroke Street, Fax: (01223) 334679 Cambridge UK. CB2 3QH. Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: namespace question
On Thu, 9 Nov 2006, Ivo Petrov wrote: I use alternate namespace in order to get representation of the folders to be as they are not subfolders of INBOX but instead to be on the same level as INBOX. Altnamespace doesn't allow mailboxes under INBOX. The problem is the mapping between altnamespace and the internal Cyrus namespace. Consider: INBOX -- user.dpc22 INBOX.foo -- user.dpc22.foo foo-- user.dpc22.foo -- David Carter Email: [EMAIL PROTECTED] University Computing Service,Phone: (01223) 334502 New Museums Site, Pembroke Street, Fax: (01223) 334679 Cambridge UK. CB2 3QH. Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: How to tell POP3 daemon to mark messages as read?
On Fri, 3 Nov 2006, Georgy Goshin wrote: Is it possible to mark all messages downloaded via POP3 daemon as read so that next time user connected via IMAP could see wich messages are new? No, the POP daemon doesn't open the seen message database. There would be a fairly substantial performance hit if it did. I believe (its been some time now) that the UW server works the way that you want. The POP protocol doesn't have any concept of \Seen messages, so there isn't really a right or wrong way to do this. -- David Carter Email: [EMAIL PROTECTED] University Computing Service,Phone: (01223) 334502 New Museums Site, Pembroke Street, Fax: (01223) 334679 Cambridge UK. CB2 3QH. Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: performance on large inboxes
On Thu, 9 Nov 2006, Marten Lehmann wrote: That was merged a long time back. doc/text/changes: is it enabled by default? Or do I have to specify which headers in particular shall be cached? It is enabled by default, but only applies to messages delivered after you started to run 2.2.1 or later (unless you reconstruct). Only a subset of X-Whatever headers are cached, but just about everything else is cached apart from the Received headers. I should point out that the code in Cyrus is a significant improvement on my original version, courtesy of Rob Siemborski. The version in Cyrus makes it easy to add headers to the list which is cached. -- David Carter Email: [EMAIL PROTECTED] University Computing Service,Phone: (01223) 334502 New Museums Site, Pembroke Street, Fax: (01223) 334679 Cambridge UK. CB2 3QH. Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Does the quota include deleted but not yet expunged mails in v2.3 with delayed expunge?
On Thu, 9 Nov 2006, Farzad FARID wrote: I'm running Cyrus Imapd 2.3.7 with the delayed expunge mode. Do the messages deleted by the user, but not yet expunged by the system, count in the user's quota? I'd say yes but I'd like a confirmation. Yes. \Deleted is just another flag on messages. -- David Carter Email: [EMAIL PROTECTED] University Computing Service,Phone: (01223) 334502 New Museums Site, Pembroke Street, Fax: (01223) 334679 Cambridge UK. CB2 3QH. Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Does the quota include deleted but not yet expunged mails in v2.3 with delayed expunge?
On Thu, 9 Nov 2006, David Carter wrote: On Thu, 9 Nov 2006, Farzad FARID wrote: I'm running Cyrus Imapd 2.3.7 with the delayed expunge mode. Do the messages deleted by the user, but not yet expunged by the system, count in the user's quota? I'd say yes but I'd like a confirmation. Yes. \Deleted is just another flag on messages. Ah. If you meant expunged by the user but not yet expired by the system, then these don't count again the quota, just as Paul Engle suggests. Sorry about any confusion. -- David Carter Email: [EMAIL PROTECTED] University Computing Service,Phone: (01223) 334502 New Museums Site, Pembroke Street, Fax: (01223) 334679 Cambridge UK. CB2 3QH. Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: performance on large inboxes
On Wed, 8 Nov 2006, Phil Pennock wrote: The relevant stuff is HERMES_CACHE_MOST in mailbox.c; I've really no idea whether or not these changes are roughly independent and if they can be pulled out. That was merged a long time back. doc/text/changes: Changes to the Cyrus IMAP Server since 2.2.1 * Significantly improved message header caching (based in large part on code supplied by David Carter [EMAIL PROTECTED] from the University of Cambridge) -- David Carter Email: [EMAIL PROTECTED] University Computing Service,Phone: (01223) 334502 New Museums Site, Pembroke Street, Fax: (01223) 334679 Cambridge UK. CB2 3QH. Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: replication, how to see if it 'up to date'
On Thu, 28 Sep 2006, Rudy Gevaert wrote: Does anybody know how to see if the sync replication is up to date? All suggestions are welcome. I replicate all users from all masters to replicas about once a month and check the output for any inconsistancies. Something like: cyrus-23[cyrus:cyrus]$ replicate -s cyrus-24 -v -v -u dpc22 USER dpc22 USER_ALL dpc22 ENDUSER where replicate is just a little wrapper around sync_client. We also maintain databases of MD5 checksums for messages and cache entries, generated by make_md5. -- David Carter Email: [EMAIL PROTECTED] University Computing Service,Phone: (01223) 334502 New Museums Site, Pembroke Street, Fax: (01223) 334679 Cambridge UK. CB2 3QH. Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: sync_client bails out after 3 MAILBOXES need upgrading to USER in one run
On Tue, 12 Sep 2006, Wesley Craig wrote: On a related note, what was the problem with accepting the Cambridge patches for delayed folder deletion? I'm interested in working on getting that or similar code accepted. Now that we have delayed expunge for messages, we continue to run tape backups only for the case where users inadvertently delete folders. Replication and delayed expunge were added by Ken (working as a contractor) back before he joined CMU. Replication was sponsored by Columbia. Delayed expunge was sponsored by Fastmail, but mostly as a performance enhancement. Unexpunge was just a nice side effect. I believe that Ken implemented the delayed expunge from scratch. My original two expunge expunge code is rather more involved: 1) Users can access expunged mail and deleted mailboxes using magic mailbox hierarchies (.EXPUNGED/ and .DELETED/). 2) It hooks into the quota system to record the amount of expunged space in each quota root. Messages are automatically expired when global or per quota root limits are reached. With hindsight (1) was a daft idea on my part. Our users struggle with the idea of multiple mailboxes in their account, let alone magic mailbox hierarchies. (2) is arguably useful if you don't have infinite storage on your IMAP backends. There are however lots of other spool partitions which can fill up under a determined denial of service attack. Without (1) or (2), delayed mailbox deletion is really nothing more exciting than a RENAME operation to some part of the mailbox hierarchy without a quota root that only the system administrator can access. -- David Carter Email: [EMAIL PROTECTED] University Computing Service,Phone: (01223) 334502 New Museums Site, Pembroke Street, Fax: (01223) 334679 Cambridge UK. CB2 3QH. Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: sync client bailing out
On Tue, 12 Sep 2006, F. Rossetti wrote: I am running two cyrus 2.3.7 servers with replication. The replica server crashed, and I fsck'd the /var/imap/* and /var/spool/imap/* partition finding some errors. Now everything seems to have come back up but the replication still has problems. The client process on the master bails out frequently with these errors in the log: It looks like you have some corruption in cyrus.index files on the replica following the crash. Try running reconstruct on the mailboxes in question. -- David Carter Email: [EMAIL PROTECTED] University Computing Service,Phone: (01223) 334502 New Museums Site, Pembroke Street, Fax: (01223) 334679 Cambridge UK. CB2 3QH. Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: sync_server memory leak with giant new mailbox first sync
On Sun, 10 Sep 2006, Wesley Craig wrote: My solution (such as it is) was to reduce the wasteful amount of space sync_server was allocating per message: [...] The times-5 is completely gratuitous. In fact the pre-allocation of any memory for paths is wasteful, but I was not up for reengineering the memory scheme in sync_server at the time. This started out life as: sprintf(tmp, %lu., l-count); result-msg_path = xmalloc(l-stage_dir_len+strlen(tmp)+2); At around 35 bytes a shot, you get rather more of these to the MByte. If I recall correctly, the 5 * (MAX_MAILBOX_PATH+1) was put in to support partitions. A lookup table for partitions and two integers (one for the partition number, one of the message number on that partition) should be all that is needed to reconstruct the paths at a later date. -- David Carter Email: [EMAIL PROTECTED] University Computing Service,Phone: (01223) 334502 New Museums Site, Pembroke Street, Fax: (01223) 334679 Cambridge UK. CB2 3QH. Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Another sync-client issue
On Sun, 27 Aug 2006, Bron Gondwana wrote: This is just a copy-and-pasteo that I noticed while looking for the other issue in the sync code tonight. I think the fix is pretty self evident since exactly the same comment exists elsewhere with the correct error code after it and the value that's there now duplicates a test just above and is an unreachable path. Agreed. -- David Carter Email: [EMAIL PROTECTED] University Computing Service,Phone: (01223) 334502 New Museums Site, Pembroke Street, Fax: (01223) 334679 Cambridge UK. CB2 3QH. Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: sync_client bails out after 3 MAILBOXES need upgrading to USER in one run
On Sun, 27 Aug 2006, Bron Gondwana wrote: I've attached my trivial solution (against CVS of last week some time), but I'm thinking a better (as in, less wasteful) solution might be to not return an error at all for a failed mailbox, but instead keep walking the entire tree, and then generate a USER event for every mailbox that hasn't been marked yet. My original code (which we are still running: I'm not in any hurry to upgrade to 2.3) sorts mailbox actions by user. If a single mailbox action associated with a user fails the rest are discarded and a USER event is generated. If the USER event fails it locks the given user out of the mboxlist and tries again. This is close to what you describe above. From memory the 3 retries thing was introduced to cope with transient problems on shared mailboxes, caused by mailboxes moving around under the replication engines feet. No promotion is possible in this case. Ken and David - is there a reason why you chose to pass a single MAILBOXES command with multiple mailboxes to the backend rather than single mailbox commands? The little birdy in my head is whispering (it does that at 1am after many hours of debugging) that it has something to do with supporting renames. Rename and copying messages between mailboxes. With single mailbox commands RENAME becomes DELETE + CREATE/UPLOAD (which would work, but would be a pain if a GByte mailbox was involved). COPY would upload new messages rather than reusing the single instance store on the replica. -- David Carter Email: [EMAIL PROTECTED] University Computing Service,Phone: (01223) 334502 New Museums Site, Pembroke Street, Fax: (01223) 334679 Cambridge UK. CB2 3QH. Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: sync_client stalls the rest of cyrus while 'no route to host'
On Sun, 27 Aug 2006, Bron Gondwana wrote: To tell you the truth, I'm seriously considering writing a replacement to sync_client that does a bunch of different things including multiple replicas, maintaining log files, etc. All of this drops out pretty easily from a pattern which produces a single log file per day and calls the sync_client fork children with a byte-range on the log file to run rather than moving the file and then running the copy. I think that you would be better off with multiple log files and multiple sync_client processes, one for each replica. That way each replication stream is independent and can progress at its own best speed. Particularly important if a replica dies (or is shut down for routine maintenance) and needs to catch up from a big backlog of transactions. -- David Carter Email: [EMAIL PROTECTED] University Computing Service,Phone: (01223) 334502 New Museums Site, Pembroke Street, Fax: (01223) 334679 Cambridge UK. CB2 3QH. Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Replication woes with a specific mailbox...
On Fri, 28 Jul 2006, Pascal Gienger wrote: sync_client seems to try to RENAME the Inbox to the subfolder Uni which is complete nonsense. Do the mailboxes have the same UniqueID (see cyrus.header files)? The replication engine expects UniqueID to be unique. Cyrus makes a bit of a hash of renaming user inboxes (user.XXX - user.XXX.Uni). Removing the cyrus.header file and running reconstruct should fix the problem. -- David Carter Email: [EMAIL PROTECTED] University Computing Service,Phone: (01223) 334502 New Museums Site, Pembroke Street, Fax: (01223) 334679 Cambridge UK. CB2 3QH. Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: machine mismatch UUID
On Thu, 27 Jul 2006, Fabio Rossetti wrote: The old master used UUIDs ( sync_machineid:1) but using the same configuration on the ex-replica gave this in syslog: Jul 27 00:00:00 exreplica master[23136]: Machine mismatch: 1 != 2 Jul 27 00:00:00 exreplica master[23136]: Couldn't initialise UUID subsystem This means that the sync_machineid setting didn't match the machine=X line in /var/imap-hermes/master_uuid. This file should be created automatically the first time that you run Cyrus with sync_machineid set. Message UUIDs are seeded from sync_machineid and the time at which master starts. These values are recorded in /var/imap-hermes/master_uuid as a sanity check in case the clock on a system is behaving erratically. There is also a timestamp_generation field which can be used to sort things out if a system clock has been playing up, although I've never had to use it. -- David Carter Email: [EMAIL PROTECTED] University Computing Service,Phone: (01223) 334502 New Museums Site, Pembroke Street, Fax: (01223) 334679 Cambridge UK. CB2 3QH. Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Cyrus 2.3.7 Replication Question
On Thu, 13 Jul 2006, Robert Mueller wrote: I think that should do it. There might be another option as well to make this easier. From the top: 1. Server A is master (sync_client) replicating to Server B (sync_server) 2. Server A dies/is stopped 3. Restart Server B after adding this to the imapd.conf sync_log: 1 4. All IMAP/POP/LMTP connections are directed to Server B Now Server B should be logging all changes to it's sync log, so you don't have to sync_client -u all users. Then to change back, follow from step 6-11 above. It should be possible to leave the sync_log: 1 enabled on all master and replica systems. The IMAP/POP/LMTP processes will only start to log events when a system becomes a master. My original code allows a mixture of master and replica mailboxes on each Cyrus backend system (with a Perdition like proxy sitting in front to direct logins to specific backend servers). This makes it possible to push inactive accounts back and forth without any downtime. It doesn't work with shared mailboxes, which is why Cyrus 2.3 only supports simple master-replica pairs. -- David Carter Email: [EMAIL PROTECTED] University Computing Service,Phone: (01223) 334502 New Museums Site, Pembroke Street, Fax: (01223) 334679 Cambridge UK. CB2 3QH. Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Migration form UW with replication
On Thu, 13 Jul 2006, Michael Menge wrote: http://www-uxsup.csx.cam.ac.uk/~dpc22/cyrus/replication.html says that replication can used to migrate from UW-Imapd to Cyrus, is there any Documentation/Howto howto use this feature? That part wasn't merged into Cyrus 2.3. It was very specific to our environment. -- David Carter Email: [EMAIL PROTECTED] University Computing Service,Phone: (01223) 334502 New Museums Site, Pembroke Street, Fax: (01223) 334679 Cambridge UK. CB2 3QH. Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: purge sync. replication folder
On Wed, 12 Jul 2006, Patrice wrote: I have seen that the folder sync. on my replica contains more than 1Go of files. can I delete all the content of it or I must keep the current day with its cache file ? I can do it with or without the replica stopped ? This is equivalent to the stage. area for mail delivery, with a pid subdirectory for each active sync_server process. Active sync./pid directories will clear out automatically when the cache file reaches a certain size. Recent 2.3.x releases clear the directory much more aggressively because of a nasty memory leak which appeared when support for partitions was added. Old sync./pid directories which don't correspond to any running sync_server can be cleared out safely by hand on a running system. -- David Carter Email: [EMAIL PROTECTED] University Computing Service,Phone: (01223) 334502 New Museums Site, Pembroke Street, Fax: (01223) 334679 Cambridge UK. CB2 3QH. Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Errors while doing Replication
On Tue, 13 Jun 2006, David Korpiewski wrote: I'm just about to do some major testing with cyrus and I've been having a terrible time with replication throwing up these types of errors randomly. I don't understand what this promoting and DELETE are trying to do. I definitely don't want to delete the user.cyrus under any condition, so why is it trying to delete it?! How can I fix these errors oh great gurus of replication? The reason that the user.cyrus is even showing is that in postfix, I tell it to always_bcc the cyrus user. This way I have a running queue of all of the messages that I have received on this server. This is so if we failover and I have to use the replica, I can bounce the email that may not have been processed by the sync process when I bring the master back up. Jun 13 16:33:02 lmc1 sync_client[16188]: SEEN davidk user.davidk Jun 13 16:33:02 lmc1 sync_client[16188]: MAILBOX user.cyrus Jun 13 16:33:02 lmc1 sync_client[16188]: DELETE received NO response: This suggests that the Mailbox UniqueID on the replica does not match the mailbox UniqueID on the master. The replication system wants to delete the old version so that it can update the new mailbox. Check the cyrus.header files. Failed to delete user.cyrus: Operation is not supported on mailbox Looking at mboxlist_deletemailbox() there is a specific test to stop the currently logged in user from deleting their own inbox. I infer that you are running replication as user cyrus. I would suggest that you always_bcc to some other mailbox, preferably not an admin user. Jun 13 16:33:02 lmc1 sync_client[16188]: Promoting: MAILBOX user.cyrus - USER cyrus This is just a consequence of the delete operation failing. The replication engine retries before giving up completely. -- David Carter Email: [EMAIL PROTECTED] University Computing Service,Phone: (01223) 334502 New Museums Site, Pembroke Street, Fax: (01223) 334679 Cambridge UK. CB2 3QH. Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Replication and Virtual Domains
On Mon, 15 May 2006, Ken Murchison wrote: replication and virtdomains should work together, but currently don't. IIRC, David's code was written against Cyrus 2.1,which was before virtdomains. Yes, that's correct. I never even thought about virtdomains (or altnamespace) when I ported the code. Its on my TODO list, but I can'tgive you a timetable. altnamespace shouldn't be a problem (we use it). The replication code works entirely in the internal namespace. -- David Carter Email: [EMAIL PROTECTED] University Computing Service,Phone: (01223) 334502 New Museums Site, Pembroke Street, Fax: (01223) 334679 Cambridge UK. CB2 3QH. Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: 2.3 Replication and lost Hardlinks
On Fri, 31 Mar 2006, Roland Pope wrote: It would appear from my testing of the new 2.3 Replication code, that you lose any 'SingleInstanceStore' benefits on the replica as hardlinks on the master cannot be reproduced on the replica. This is what message UUIDs are for. I've been replicating my single instance message stores quite happily for about 3 years now. I don't use Cyrus 2.3. but here is the relevant section from the install-replication document that Ken wrote: Universally Unique Identifiers (UUIDs) An optional, but recommended step is to enable UUIDs for messages. Use of UUIDs improves efficiency by eliminating the synchronization of messages which the replica has already received from the master. Note that UUIDs can be safely enabled and disabled at any time. 1. Define the sync_machineid option in imapd.conf. This option specifies the numeric identifier (1 - 255) of the master machine which is used in constructing the UUID for each message on the server. This identifier MUST be unique across all active backend servers in a Murder. Example: sync_machineid: 1 2. For each IMAP, NNTP and LMTP service in cyrus.conf, enable the provide_uuid argument. Example: imapcmd=imapd listen=imap prefork=5 provide_uuid=1 -- David Carter Email: [EMAIL PROTECTED] University Computing Service,Phone: (01223) 334502 New Museums Site, Pembroke Street, Fax: (01223) 334679 Cambridge UK. CB2 3QH. Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: sync_client stop working suddenly
On Wed, 29 Mar 2006, Patrice wrote: what is exactly the 7 signal ? That will depend on the platform that you are using. Signal 7 is SIGBUS (a bus error) on Linux: http://en.wikipedia.org/wiki/SIGBUS. If both sync_client and imapd are affected on a single system, then this would suggest a problem with a library shared between the two or (just conceivably) a subtle hardware problem. -- David Carter Email: [EMAIL PROTECTED] University Computing Service,Phone: (01223) 334502 New Museums Site, Pembroke Street, Fax: (01223) 334679 Cambridge UK. CB2 3QH. Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Restored mailboxes causes replication to bail
On Thu, 23 Mar 2006, Roland Pope wrote: When I try to replicate a user which has such a 'Recovered' mailbox, the replication client bails out because it seems to treat the 'user.xxx.Recovered' mailbox as the user.xxx INBOX and trys to do a rename (which fails). It sounds you have ended up with two mailboxes with the same UniqueID. This will confuse the replication code which tracks mailboxes by UniqueID rather than by name in order to implement rename. If you delete the cyrus.header file and run reconstruct it should generate a new UniqueID. -- David Carter Email: [EMAIL PROTECTED] University Computing Service,Phone: (01223) 334502 New Museums Site, Pembroke Street, Fax: (01223) 334679 Cambridge UK. CB2 3QH. Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: 2.3.1 replication and deliver problem
On Tue, 31 Jan 2006, Dmitry Melekhov wrote: This is what I see. Promoting: MAILBOX user.dm - USER dm Error in do_sync(): bailing out! Not too informational message... syslog should tell you why it decided to bail out. -- David Carter Email: [EMAIL PROTECTED] University Computing Service,Phone: (01223) 334502 New Museums Site, Pembroke Street, Fax: (01223) 334679 Cambridge UK. CB2 3QH. Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: defining multiple replica hosts
On Thu, 26 Jan 2006, Khalid Mehmood wrote: Can someone tell me how to setup more then one replica hosts in imapd.conf. If we are talking about mailbox rolling replication here then I'm afraid that this isn't possible at the moment. It wouldn't be hard to add this. The issue is that imapd, pop3d and lmtpd all log to a single file which is picked up by the replication engine. You would need to create multiple log files (one for each replication engine), or find some other way of piping the log data to lots of different processes. The imapd.conf manual says: The replicated config is one in which multiple backend servers all share the same mailspool, but each have their own replicated copy of mailboxes.db. That reference is to replicated Murder configurations which is something else altogether. -- David Carter Email: [EMAIL PROTECTED] University Computing Service,Phone: (01223) 334502 New Museums Site, Pembroke Street, Fax: (01223) 334679 Cambridge UK. CB2 3QH. Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: signaled to death by 25
On Tue, 17 Jan 2006, Adam Tworkowski wrote: Various posting suggest that signalled to death by 11 is a Berkley DB issue but signaled to death by 25 seems to be an anomaly on the internet. Well signalled to death by 11 is just a process throwing a SIGSEGV (segmentation fault). Signal 25 is SIGXFSZ File size limit exceeded on my Linux boxes. However, signal 25 might mean something else on a different platform. -- David Carter Email: [EMAIL PROTECTED] University Computing Service,Phone: (01223) 334502 New Museums Site, Pembroke Street, Fax: (01223) 334679 Cambridge UK. CB2 3QH. Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Seen flag reverts for some users
On Tue, 10 Jan 2006, Daniel O'Connor wrote: Some of my users are seeing their email revert to unread. Sometimes it happens after a search, but othertimes it Just Happens. Outlook Express has the same problem. I suspect the flushseenstate option in 2.2 or 2.3 will fix this problem for you. -- David Carter Email: [EMAIL PROTECTED] University Computing Service,Phone: (01223) 334502 New Museums Site, Pembroke Street, Fax: (01223) 334679 Cambridge UK. CB2 3QH. Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: duplicate suppression and syncronizing?
On Mon, 14 Nov 2005, Bill Kearney wrote: Somewhat tangentally, is there a way to use sieve or something else within cyrus to investigate duplicates in a given folder? I do know some mail is duplicated. Thunderbird has an extension available to dig through the messages to find duplicates. But is there something clever within sieve that can do it? Each message delivery or sieve action (fileinto, redirect etc) causes a duplicate_check() to see if the combination of Message-ID plus action plus (mailbox name or redirect address) has been seen before, followed by a duplicate_mark() if it is new. This isn't visible to the end user. -- David Carter Email: [EMAIL PROTECTED] University Computing Service,Phone: (01223) 334502 New Museums Site, Pembroke Street, Fax: (01223) 334679 Cambridge UK. CB2 3QH. Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: duplicate suppression and syncronizing?
On Sun, 13 Nov 2005, Bill Kearney wrote: What's the story with duplicate suppression and it's affect on syncronizing with tools like imapsync or mailsync? The duplicate supression database is used by lmtpd when delivering mail. It is not used at all by imapd. If it was amusing things would happen when people tried to copy messages around. -- David Carter Email: [EMAIL PROTECTED] University Computing Service,Phone: (01223) 334502 New Museums Site, Pembroke Street, Fax: (01223) 334679 Cambridge UK. CB2 3QH. Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: SQUATTER script and non English folders
On Thu, 27 Oct 2005, [EMAIL PROTECTED] wrote: Since you've looked at the code, what about: squatter -r 'user.a*' That will look for mailboxes matching user.a*.* which would match user.abc123.foo2 but not user.abc123. The code just adds .* to the end of any string that you give it, hence the slightly odd wording in the manual page: -r: Recursively create indexes for all submailboxes of the mailboxes or mailbox prefixes given as arguments. I'm worried that if I try to squat the entire server every night, the I/O from that combined with backups will mean that the squatting never finishes in time for active business hours. I had the same concern. I made a small change to my Cyrus install so that it only squats a fraction of the mailboxes each night using some simple modulo arithmetic (e.g: squatter -m0 -M5 squats 1/5th of the mailboxes). -- David Carter Email: [EMAIL PROTECTED] University Computing Service,Phone: (01223) 334502 New Museums Site, Pembroke Street, Fax: (01223) 334679 Cambridge UK. CB2 3QH. Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
RE: SQUATTER script and non English folders
On Fri, 28 Oct 2005, [EMAIL PROTECTED] wrote:
Interesting, when I use webinterface (IMP or Squirremail) after I've finished
to run squatter I got this message:
imap[15835]: SQUAT returned 45 messages
But after a while, when I got new messages in this account, I've got
this message: imap[16924]: SQUAT returned 46 messages
squat is just a first approximation to narrow down the search. After imapd
gets a list of messages from squat it has to search the messages to find
out which ones are genuine matches. Any messages appearing after the squat
index was generated are automatically included as potential matches:
imap/search_engines.c:
/* Add in any unindexed messages. They must be searched manually. */
for (i = 0; i vlen; i++) {
msg_vector[i] |= unindexed_vector[i];
}
Does it mean that cyrus.squat file has been updated automatically, even
before I've run squatter -r user again? If so, why should I run
squatter -r user on regular basis?
The squat indexes are just a snapshot of the state of a given mailbox. As
that mailbox changes imapd will have to work harder to search messages not
covered by the index.
--
David Carter Email: [EMAIL PROTECTED]
University Computing Service,Phone: (01223) 334502
New Museums Site, Pembroke Street, Fax: (01223) 334679
Cambridge UK. CB2 3QH.
Cyrus Home Page: http://asg.web.cmu.edu/cyrus
Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: SQUATTER script and non English folders
On Thu, 27 Oct 2005, [EMAIL PROTECTED] wrote: 1) I'm running the following script as a cronjob every night: #!/bin/bash su - cyrus -c /usr/lib/cyrus/bin/squatter -r user/* But, it creates cyrus.squat file only in subfolders of the users INBOX, i.e. /var/spool/imap/user/username/ doesn't include cyrus.squat but /var/spool/imap/user/username/Drafts/ does include cyrus.squat after running the above script. Looking at the code, that will generate squat indexes for all mailboxes matching user/*/*, which will match user/dpc22/foo, but not user/dpc22. I suggest that you try: /usr/lib/cyrus/bin/squatter -r user or just /usr/lib/cyrus/bin/squatter. 2) When I create Hebrew named folders via OE or web client, it's look like this in the FS : BdcF0wXpBdUF6g- but if I want to cd to this folder I get an error: # cd BdcF0wXpBdUF6g- [1] 14268 -bash: BdcF0wXpBdUF6g-: command not found [1]+ Donecd That's two commands: cd run as a background task without any arguments, followed by BdcF0wXpBdUF6g-, which isn't a valid command on your system. You need to quote the character. Something like: cd BdcF0wXpBdUF6g- or cd \BdcF0wXpBdUF6g- -- David Carter Email: [EMAIL PROTECTED] University Computing Service,Phone: (01223) 334502 New Museums Site, Pembroke Street, Fax: (01223) 334679 Cambridge UK. CB2 3QH. Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: High-Availability IMAP server
On Tue, 27 Sep 2005, Patrick Radtke wrote: We made great use of it Monday morning when one of our backend machines failed. Switching to the replica was quite simple and relatively fast (maybe 5 to 10 minutes from deciding to switch to the replica before replica was fully in action) We use the replication engine all the time to move users back and forth between systems so that we can patch and upgrade operating systems and/or Cyrus without any user visible downtime. There have also been a number of forced failovers because of hardware problems, specifically some dodgy RAID controller firmware that we were running for a few months until we got a fix. Its worked very nicely for us, but it is important that people don't just trust the software blindly. We maintain and constantly regenerate a database of MD5 checksums for all of the messages and cache entries across the cluster. Its been a long time now since this has turned up errors, but I still check it religiously. I consider the code to stable, though on occasion strange things happen Which is not really my definition of stable :). (e.g. when user renames user.INBOX to user.saved.INBOX) and you have to restart the replication process (no downtime to Cyrus involved). This one is odd behaviour on the part of mboxlist_renamemailbox(): it does special magic when running as a non-admin user. There's actually a more serious underlying bug in Cyrus here which I believe Ken is working on. Again we don't see this one. Partly because our replication engine doesn't run as an admin user (afraid you don't have that option), partly because of overenthusiastic hacking on my part in other parts of the Cyrus code. -- David Carter Email: [EMAIL PROTECTED] University Computing Service,Phone: (01223) 334502 New Museums Site, Pembroke Street, Fax: (01223) 334679 Cambridge UK. CB2 3QH. Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Problems installing ssl certificate for cyrus imap
On Mon, 26 Sep 2005, Nicole Skyrca wrote: The certificate that we purchased has an intermediate certificate. I'm afraid that this is likely to be a problem. Cyrus (imap/tls.c) uses SSL_CTX_use_certificate_file() rather than the more advanced SSL_CTX_use_certificate_chain_file() to set up its certificate. My experience with other applications is that you need to use the _chain_ version in order for chained certificates to work. Given that the two functions can be used interchangably, Cyrus should probably be using SSL_CTX_use_certificate_chain_file(). The SSL manual page for the two functions certainly recommends this. -- David Carter Email: [EMAIL PROTECTED] University Computing Service,Phone: (01223) 334502 New Museums Site, Pembroke Street, Fax: (01223) 334679 Cambridge UK. CB2 3QH. Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: High-Availability IMAP server
On Mon, 26 Sep 2005, Aaron Glenn wrote: There is replication code in the 2.3 branch; though from what I can tell it hasn't been touched in a few months and makes me wonder if it's being actively developed still. Nevertheless, in my exhaustive search for any and all information on IMAP replication, I came across a few list posts detailing the 2.3 replication code in production, without many issues, for over a year. I wrote the code which was eventually merged into the 2.3 branch back in Autumn 2002. We've been using it on our production systems for a little over two years now, and all of our users have been replicated (rolling replication to hot spare system, plus nightly replication to tape spooling array) for about 18 months. The last significant change to my code base was November last year. That's not a sign of neglect, the code just does everything we need right now. Ken merged the code into 2.3 at the start of this year. He put in quite a lot of work to merge the code properly into Cyrus (I had deliberately left it to one side to make updates easier) and add support for features such as shared mailboxes and annotation that we just don't need right now. Its still conceptually the same code and the same design. Invariably working on code introduces new bugs (including a particularly exciting one caused by a stray semicolon). People are also pushing the code in new and interesting ways. Ken fixed a bug involving account renaming (user.xxx - user.yyy) a couple of weeks back: that's something our nightly useradmin scripts just never try to do. Looking back at features which were introduced into earlier versions of Cyrus, I imagine that people will start to test the new code seriously when Cyrus 2.3 is released, and that there will be a period of fixing bugs which will tail off as 2.3 stabilises. The complication is that there doesn't appear to be anyone left at CMU to release new versions of Cyrus at the moment. Poor Jeffrey Eaton seems to be the last man standing there. My own experience of running things single handed is that it doesn't leave much time for development work. -- David Carter Email: [EMAIL PROTECTED] University Computing Service,Phone: (01223) 334502 New Museums Site, Pembroke Street, Fax: (01223) 334679 Cambridge UK. CB2 3QH. Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
RE: cyrus imapd authentication problem
On Wed, 17 Aug 2005, Thor Vik wrote: Hi, I am tearing my hair out over this one. When I run an imtest -m login localhost, I keep getting an L01 NO Login failed error and a generic failure. What could this mean? I have configured imapd.conf's sasl_pwcheck_method: saslauthd. I have saslauthd set to pam and when I run a saslauthdtest, it works fine. My syslog give me an error like this: badlogin: localhost.localdomain [127.0.0.1] plaintext cyrus SASL(-1): generic failure: checkpass failed. In my auth.log, I get cannot connect to saslauthd server: Permission denied. Any ideas would be met with appreciation. saslauthd creates a Unix domain socket for clients (imapd) on connect to. The standard location is /var/state/saslauthd/mux, but it can be changed using the --with_saslauthd option to cyrus-sasl's configure script. This socket must be accessible by the user or group that imapd is running as. The other possible problem is that saslauthd only supports simple LOGIN authentication. If the CAPABILITY response that you get from imtest includes: AUTH=OTP AUTH=DIGEST-MD5 AUTH=CRAM-MD5, then you need to add the following to your imapd.conf file: sasl_mech_list: plain -- David Carter Email: [EMAIL PROTECTED] University Computing Service,Phone: (01223) 334502 New Museums Site, Pembroke Street, Fax: (01223) 334679 Cambridge UK. CB2 3QH. Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: imapd LIST bug?
On Wed, 17 Aug 2005, Jukka Salmi wrote: a0021 LIST INBOX.% [...] * LIST (\HasNoChildren) . INBOX.Spam * LIST (\HasNoChildren) . INBOX.Test * LIST (\HasNoChildren) . INBOX.Test-1 * LIST (\Noselect \HasChildren) . INBOX.Test * LIST (\HasNoChildren) . INBOX.Trash a0021 OK Completed (0.000 secs 36 calls) So, why is INBOX.Test listed twice? And why the \Noselect flag? It is a long standing bug in the LIST command. I imagine that you have three mailboxes named: INBOX.Test INBOX.Test.something INBOX.Test-1 The problem is that - sorts before . in the mailbox list, so the two INBOX.Test entries become separated. The mstringdata() function which generates each line of output for the LIST and LSUB commands only compares adjacent entries, so it gets confused by this ordering. One solution would be to change the sort order (as it happens the flat backend already sorts '.' before '-'). A small group of us had a discussion about this last October. I think that we were all rather wary about messing with the mailbox list sort order: there's a fairly high potential for breaking things in the process. -- David Carter Email: [EMAIL PROTECTED] University Computing Service,Phone: (01223) 334502 New Museums Site, Pembroke Street, Fax: (01223) 334679 Cambridge UK. CB2 3QH. Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
