We are looking for a Cyrus contractor
If you are interested in maintaining our Cyrus e-mail system on a contract basis, please let me know. I don't know the details of responsibilities or requirements, but I'll be happy to put you in touch with my manager for discussion on these matters. -- -Gary Mills--Unix Group--Computer and Network Services- Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: Vacation Not Working...
On Tue, May 10, 2011 at 10:33:28AM -0500, Gary Mills wrote: On Tue, May 10, 2011 at 08:59:48AM -0600, Nathanael D. Noblet wrote: On 05/10/2011 06:17 AM, Gary Mills wrote: On Tue, May 10, 2011 at 07:39:48AM +0200, Simon Matter wrote: To start, you may want to post your imapd.conf here and an example sieve script used for vacations. Find a message that should have triggered a vacation response. Check the envelope recipient and header recipient of this message. They must match to trigger vacation. Sent from a google account. I sent the sieve script in a previous message. Return-Path: That's the envelope sender, used for error returns like non-delivery reports. Nothing will be sent to that one. I don't know if sieve vacation uses it, but it wouldn't work if it did. It may also be an indication of a message that should not receive a response. Look what I found in the sieve vacation document (draft-ietf-sieve-vacation-07): Vacation is used to respond to a message with another message. Vacation's messages are always addressed to the Return-Path address (that is, the envelope from address) of the message being responded to. That's your problem. -- -Gary Mills--Unix Group--Computer and Network Services- Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: Vacation Not Working...
On Tue, May 10, 2011 at 07:39:48AM +0200, Simon Matter wrote: To start, you may want to post your imapd.conf here and an example sieve script used for vacations. Find a message that should have triggered a vacation response. Check the envelope recipient and header recipient of this message. They must match to trigger vacation. -- -Gary Mills--Unix Group--Computer and Network Services- Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: Vacation Not Working...
On Tue, May 10, 2011 at 08:59:48AM -0600, Nathanael D. Noblet wrote: On 05/10/2011 06:17 AM, Gary Mills wrote: On Tue, May 10, 2011 at 07:39:48AM +0200, Simon Matter wrote: To start, you may want to post your imapd.conf here and an example sieve script used for vacations. Find a message that should have triggered a vacation response. Check the envelope recipient and header recipient of this message. They must match to trigger vacation. Sent from a google account. I sent the sieve script in a previous message. Return-Path: That's the envelope sender, used for error returns like non-delivery reports. Nothing will be sent to that one. I don't know if sieve vacation uses it, but it wouldn't work if it did. It may also be an indication of a message that should not receive a response. Received: from postman ([unix socket]) by titanium.nobletdesign.com (Cyrus v2.3.7-Invoca-RPM-2.3.7-7.el5_4.3) with LMTPA; Tue, 10 May 2011 09:57:50 -0500 X-Sieve: CMU Sieve 2.3 Received: from titanium.nobletdesign.com (localhost.localdomain [127.0.0.1]) by titanium.nobletdesign.com (Postfix) with ESMTP id 93332F78069 for t...@greatexcursions.com; Tue, 10 May 2011 09:57:50 -0500 (CDT) That will be the envelope recipient. [...] Subject: TEsting auto responder From: Nathanael Noblet nathanaelnob...@gmail.com To: t...@greatexcursions.com That's the header recipient. They do match. -- -Gary Mills--Unix Group--Computer and Network Services- Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Local delivery seems to limit IMAP performance
We run cyrus-imapd-2.3.8 in a murder configuration with one front end and one back end server. One thing I've noticed is that local message delivery has a pronounced effect on IMAP client performance. We've had to limit the number of lmtpd processes to 64 on both servers to maintain an adequate level of IMAP performance. At peak times, we will have over 6000 IMAP sessions and over 500 SMTP sessions on the front end. The result is that local deliveries are often delayed during peak times. What it is about lmtpd that causes it to affect IMAP client performance so profoundly? Can anything be done about this? When I have an IMAP session running with mutt, it displays one line for each message, and displays the contents quite quickly as I select each message. However, I've noticed that when I get a new delivery (for which I get a separate notification), there's a long delay when I select the next message. It must be rebuilding some indexes at that time, and fetching the new copies. Is that correct? What could we do to improve this procedure? -- -Gary Mills--Unix Group--Computer and Network Services- Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Expiry limit for the duplicate delivery database
We are running cyrus-imapd-2.3.8 with this entry in /etc/cyrus.conf: delprune cmd=cyr_expire -E 3 at=0400 I believe this is the default setting. Is there any reason to keep message IDs around for three days? I'm wondering if reducing it to one day will improve performance for local deliveries simply by making the database smaller. I assume that it would function equally well in detecting mail loops with a one-day expiry. -- -Gary Mills--Unix Group--Computer and Network Services- Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: Expiry limit for the duplicate delivery database
On Tue, Mar 08, 2011 at 04:30:45PM +0100, Eric Luyten wrote: On Tue, March 8, 2011 4:24 pm, Gary Mills wrote: We are running cyrus-imapd-2.3.8 with this entry in /etc/cyrus.conf: delprune cmd=cyr_expire -E 3 at=0400 I believe this is the default setting. Is there any reason to keep message IDs around for three days? I'm wondering if reducing it to one day will improve performance for local deliveries simply by making the database smaller. I assume that it would function equally well in detecting mail loops with a one-day expiry. Correct, but bear in mind the delivery db is also used for limiting the number of vacation replies per time unit per mail address, so you'll be sending out quite a bit more of those, I assume. That might be a problem. My understanding is that duplicate suppression records the message ID and recipient. Sieve vacation must record the sender and a time interval of some sort. These are different. Does `cyr_expire' treat them both the same? We have people now who set vacation responder intervals longer than three days. Is that not working? -- -Gary Mills--Unix Group--Computer and Network Services- Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: imapd and pop3d processes accumulate when clients disappear
On Mon, Jan 10, 2011 at 11:39:31PM +0100, Sebastian Hagedorn wrote: That was fixed quite a while ago ... we had the same problem, so I worked with one of the developers to debug and fix it. Thanks for the information. So, a Cyrus or SASL upgrade some time in the future should fix this problem. I'm pleased that I won't need to carry my local fix forward to a new version. -- -Gary Mills--Unix Group--Computer and Network Services- Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
imapd and pop3d processes accumulate when clients disappear
I've noticed on our murder front end that imapd and pop3d processes
gradually accumulate. Some of these can be several months old. In
both cases, the reason seems to be that the process is listening on
standard input, but the client has disappeared. Here's a typical
stack trace:
# pstack 5802
5802: imapd -s
feb1a8f5 read (0, 822dd48, 5)
fec2dfaf sock_read () + 3f
This only seems to happen when the client is using SSL or STARTTLS.
The read() never times out. Of course, restarting the Cyrus service
does clean up these abandoned processes, but there should be a better
way. I've found that a simple modification to the daemons that
enables TCP keepalives solves the problem. We also shorten the
keepalive interval with a global setting, but that shouldn't affect
the results once the client has disappeared.
I'll attach the two patches. They are for cyrus-imapd-2.3.8. It
would be better to have a Cyrus master option to enable these socket
options, but these certainly work.
--
-Gary Mills--Unix Group--Computer and Network Services-
--- pop3d.c-nokeep Wed Apr 11 10:49:59 2007
+++ pop3d.c Mon May 17 18:17:22 2010
@@ -494,6 +494,12 @@
if (getsockname(0, (struct sockaddr *)popd_localaddr, salen) == 0) {
popd_haveaddr = 1;
}
+ /* Set keepalive option */
+ {
+ int oval = 1;
+ (void)setsockopt(0, SOL_SOCKET, SO_KEEPALIVE, (const void *)oval,
+sizeof(oval));
+ }
}
/* other params should be filled in */
--- imapd.c-nokeep Sun May 13 08:41:16 2007
+++ imapd.c Tue Jan 4 08:03:05 2011
@@ -786,6 +786,12 @@
imapd_haveaddr = 1;
}
}
+ /* Set keepalive option */
+ {
+ int oval = 1;
+ (void)setsockopt(0, SOL_SOCKET, SO_KEEPALIVE, (const void *)oval,
+sizeof(oval));
+ }
#ifdef DRAC_AUTH
if (((struct sockaddr *)imapd_remoteaddr)-sa_family == AF_INET)
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: reconstruct caused mailboxes (skiplist) corruption?
On Fri, Nov 12, 2010 at 10:33:15AM +1100, Bron Gondwana wrote: Sorry - I've been busy working on the specific problem rather than the overview, and I realised I kind of glossed over this bit: On Thu, Nov 11, 2010 at 02:24:47PM -0200, Henrique de Moraes Holschuh wrote: This probably needs a redesign of master/service fd-passing protocol, and of prot streams to be fixed for good. While at it, we should switch the master/service interaction to a modern design, since the operating system worth bothering with nowadays deal sanely with the thundering herd effect, and all of them have proper socket event support (epoll-like. Would require one of the event abstraction libraries, though, so as to support linux/bsd/solaris with minimum fuss). Certainly worth considering. I won't have the time to work on it for while since what we have now works fine for us. I'll be focussing my work on new features pretty soon, once 2.4.x is stable enough that I can trust that it will be reliable for people! But if you want to look at it and come up with something better for 2.5 or even further ahead, that would be fantastic. There's certainly plenty of parts of Cyrus that could do with some modernising! Isn't the modern design multiple threads, rather than multiple processes? That seems to me to be the right direction for Cyrus. It might even make for a simpler design. -- -Gary Mills--Unix Group--Computer and Network Services- Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: Got cyrus to compile but now it's not working.
On Thu, Oct 28, 2010 at 01:43:37PM -0500, Frank Pittel wrote: One last question. Does anyone have any smf methods for master and saslauthd? Attached are the ones I'm using. They install in /var/svc/manifest/site and /lib/svc/method/site . -- -Gary Mills--Unix Group--Computer and Network Services- ?xml version='1.0'? !DOCTYPE service_bundle SYSTEM '/usr/share/lib/xml/dtd/service_bundle.dtd.1' service_bundle type='manifest' name='sasl-authd' service name='site/sasl-authd' type='service' version='0' create_default_instance enabled='true'/ single_instance/ dependency name='config-file' grouping='require_all' restart_on='refresh' type='path' service_fmri value='file://localhost/etc/sasl.conf'/ /dependency dependency name='local' grouping='require_all' restart_on='none' type='service' service_fmri value='svc:/system/filesystem/local'/ /dependency dependency name='log' grouping='require_all' restart_on='none' type='service' service_fmri value='svc:/system/system-log'/ /dependency dependent name='sasl_multi-user' restart_on='none' grouping='optional_all' service_fmri value='svc:/milestone/multi-user'/ /dependent exec_method name='start' type='method' exec='/lib/svc/method/site/sasl-authd' timeout_seconds='60' method_context method_credential user='root' group='root'/ /method_context /exec_method exec_method name='stop' type='method' exec=':kill' timeout_seconds='60' method_context method_credential user='root' group='root'/ /method_context /exec_method property_group name='startd' type='framework' propval name='ignore_error' type='astring' value='core,signal'/ /property_group stability value='Unstable'/ template common_name loctext xml:lang='C'SASL Authentication Server/loctext /common_name /template /service /service_bundle #!/sbin/sh # # Start method script for the sasl authentication server # PATH=/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin; export PATH SASLAUTHDDIR=/var/run/saslauthd . /lib/svc/share/smf_include.sh if /usr/bin/pgrep -x -U 0 -z `/sbin/zonename` saslauthd /dev/null 21; then echo $0: saslauthd is already running exit $SMF_EXIT_ERR_NOSMF fi if [ ! -d $SASLAUTHDDIR ]; then mkdir $SASLAUTHDDIR chown cyrus $SASLAUTHDDIR chmod 700 $SASLAUTHDDIR fi echo starting saslauthd saslauthd -a pam -c -n 40 -t 1800; echo started saslauthd sleep 2 exit $SMF_EXIT_OK #!/end ?xml version='1.0'? !DOCTYPE service_bundle SYSTEM '/usr/share/lib/xml/dtd/service_bundle.dtd.1' service_bundle type='manifest' name='cyrus-master' service name='site/cyrus-master' type='service' version='0' create_default_instance enabled='true'/ single_instance/ dependency name='config-file' grouping='require_all' restart_on='refresh' type='path' service_fmri value='file://localhost/etc/cyrus.conf'/ /dependency dependency name='stat-file' grouping='require_all' restart_on='none' type='path' service_fmri value='file://localhost/imap/conf/.stat'/ /dependency dependency name='local' grouping='require_all' restart_on='none' type='service' service_fmri value='svc:/system/filesystem/local'/ /dependency dependency name='log' grouping='require_all' restart_on='none' type='service' service_fmri value='svc:/system/system-log'/ /dependency dependent name='cyrus_multi-user' restart_on='none' grouping='optional_all' service_fmri value='svc:/milestone/multi-user'/ /dependent exec_method name='start' type='method' exec='/lib/svc/method/site/cyrus-master' timeout_seconds='60' method_context method_credential user='root' group='root'/ /method_context /exec_method exec_method name='stop' type='method' exec=':kill' timeout_seconds='60' method_context method_credential user='root' group='root'/ /method_context /exec_method property_group name='startd' type='framework' propval name='ignore_error' type='astring' value='core,signal'/ /property_group stability value='Unstable'/ template common_name loctext xml:lang='C'Cyrus Master Server/loctext /common_name /template /service /service_bundle #!/sbin/sh # # Start method script for the master Cyrus process # PATH=/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin; export PATH LMTPDIR=/var/run/imap WASPROC=/imap/conf/proc PROCDIR=$LMTPDIR/proc MAS_PID=`cat /var/run/cyrus-master.pid 2/dev/null` . /lib/svc/share/smf_include.sh if [ $MAS_PID -gt 0 ] ps -p $MAS_PID /dev/null 21; then echo $0: master is already running exit $SMF_EXIT_ERR_NOSMF fi if [ ! -d $LMTPDIR ]; then mkdir $LMTPDIR chown cyrus:mail $LMTPDIR chmod 755 $LMTPDIR fi rm -rf $PROCDIR mkdir $PROCDIR chown cyrus:mail $PROCDIR chmod 755 $PROCDIR if [ ! -h $WASPROC ]; then rm -rf $WASPROC ln
Re: Reducing ZFS blocksize to improve Cyrus write performance ?
On Mon, Aug 09, 2010 at 09:03:44PM +0200, Pascal Gienger wrote: Am 09.08.10 19:46, schrieb Vincent Fox: * Turn off ZFS cache flushing set zfs:zfs_nocacheflush = 1 For hardware (fiberchannel, iSCSI, SSA, ...) arrays with their own Cache this is a must. Only if the SAN device handles cache flush requests incorrectly. It should consider a write to battery-backed memory as a write to permanent storage, and manage its own writes to disk from there. * Increase DNLC (Directory Name Lookup Cache) set ncsize = 50 vmstat -s | grep 'total name lookups' 135562914356 total name lookups (cache hits 96%) :-) Unless the percent ratio is not below 90% increasing the DNLC is not so useful. According to: http://docs.sun.com/app/docs/doc/817-0404/chapter2-35?a=view the proper statistics to determine if the cache is too small are provided by `kstat -n dnlcstats'. Beware also that the cache will always overflow during backups because they typically read all of the directories once, running the cache. It's the cache activity during normal IMAP access that's important. -- -Gary Mills--Unix Group--Computer and Network Services- Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Cyrus backend crashing (Solaris)
On Mon, Jul 19, 2010 at 05:04:18PM +0100, David Mayo wrote: When trying to diagnose the issue, on any attempt to run ps, prstat or to HUP the syslogd process (to set the log level for imapd to debug) the command hangs and cannot be exited with Ctrl+C. Similarly, attempts to kill the master process or shut down the system (even bypassing the shutdown scripts by using reboot) do not have any effect other than hanging the shell in which the commands were issued. New shells can be opened and certain commands run, but we aren't much closer to knowing precisely what is wrong. The only way to bring the system back is to reset it via the on board console. This happened to me about a year ago with Cyrus on a Solaris 10 server. The cause was a deadlock in one of the kernel ZFS modules. The problem is fixed in the current Solaris patches. -- -Gary Mills--Unix Group--Computer and Network Services- Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Experiment to test TCP keepalive for pop3d proxies
On Tue, Jun 01, 2010 at 04:49:52PM -0400, Wesley Craig wrote: On 01 Jun 2010, at 14:05, Gary Mills wrote: # pstack 12708 12708: pop3d -s feb1a5c5 read (0, 817faf0, b) fec2dfaf sock_read () + 3f I don't know why the stack trace is so short with these. Thinking about this a little more... sock_read() is probably from openssl's BIO layer, and the stack trace is probably truncated because pstack can't follow the backtrace through the function pointer that SSL_read() is using to access sock_read(). This is an SSL connection, right? Take a look at the SSL_read() man page, for an idea of what can go wrong. Having written something very much like the prot suite for another project, I can say that SSL is really much happier with non-blocking IO, if you're hoping for control to return to the call at some point. Yes, it's an SSL connection. It's possible that only POP3 connections with SSL had the problem. I do notice some changes between cyrus-imapd-2.3.8 and cyrus-imapd-2.3.16, in pop3d.c, that may be relevant: @@ -517,9 +541,10 @@ proc_register(pop3d, popd_clienthost, NULL, NULL); /* Set inactivity timer */ -timeout = config_getint(IMAPOPT_POPTIMEOUT); -if (timeout 10) timeout = 10; -prot_settimeout(popd_in, timeout*60); +popd_timeout = config_getint(IMAPOPT_POPTIMEOUT); +if (popd_timeout 10) popd_timeout = 10; +popd_timeout *= 60; +prot_settimeout(popd_in, popd_timeout); prot_setflushonread(popd_in, popd_out); if (kflag) kpop(); @@ -1075,6 +1143,7 @@ result=tls_start_servertls(0, /* read */ 1, /* write */ + pop3s ? 180 : popd_timeout, layerp, auth_id, tls_conn); I wonder if an upgrade will solve this problem? -- -Gary Mills--Unix Group--Computer and Network Services- Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Experiment to test TCP keepalive for pop3d proxies
On Fri, May 28, 2010 at 03:49:41PM -0400, Wesley Craig wrote: On 28 May 2010, at 12:42, Gary Mills wrote: 0805e4ee proxy_check_input (815d168, 81a7228, 819e520, 81a3d60, 81a7700, 0) + 5e That last argument to proxy_check_input()? It's the timeout. Setting it to 0 means don't time out. I'm sure the theory is that the underlying select() will return when the backend's poptimeout happens, and the connection is closed. It would be good to know why that's not happening as expected. Of course, the fact that bitpipe() isn't checking the return value of prot_flush() is also bug. Yes, the timeout is set to zero in the pop3d.c file. However, the idle timeout actually works when I test it. In one window, I do this: $ telnet setup01 pop3 Trying 130.179.16.64... Connected to setup01.cc.umanitoba.ca. Escape character is '^]'. +OK testing.umanitoba.ca Cyrus POP3 Murder v2.3.8 server ready user gmills +OK Name is a valid mailbox pass XX +OK Mailbox locked and ready /* wait for the timeout */ -ERR [SYS/PERM] Fatal error: Lost connection to input stream Connection to setup01.cc.umanitoba.ca closed by foreign host. Sure enough, on the server the new pop3d pop3d process exits after 20 minutes. While it's waiting, the stack trace looks like this: # pstack 13804 13804: pop3d feb1a465 pollsys (8042da0, 2, 8042e60, 0) feac3b8a pselect (d, 8042eb4, feb90318, feb90318, 8042e60, 0) + 18e feac3e80 select (d, 8042eb4, 0, 0, 8042ea8, 0) + 82 0808981b prot_select (8189548, , 8043f94, 0, 8042ea8, 0) + 44b 0805e4ee proxy_check_input (8189548, 8145a30, 8145aa8, 814d718, 814d308, 0) + 5e 0805dd74 bitpipe (8145c38, 0, feb921ec, 0, 8044fed, 8044fed) + c4 0805acb7 cmdloop (8135594, 8138980, 14, 2, 31203133, 312e3033) + 27 0805aa53 service_main (1, 8142a50, 8047db8) + 473 08062c13 main (1, 8047db0, 8047db8, feffb818) + a83 08059bbd _start (1, 8047e58, 0, 8047e5e, 8047e69, 8047e7c) + 7d It stays in the pollsys system call the entire time but finally returns with a zero return code. The process then writes that error message to FD 1, has a little dialogue with the back end, and then terminates. The ones I saw before were not stuck in pollsys() however. They were stuck in a read() from FD 0. The timeout didn't work on those, but the TCP keepalive does get them. They had a very short stack trace, like this: # pstack 12708 12708: pop3d -s feb1a5c5 read (0, 817faf0, b) fec2dfaf sock_read () + 3f I don't know why the stack trace is so short with these. -- -Gary Mills--Unix Group--Computer and Network Services- Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Experiment to test TCP keepalive for pop3d proxies
On Thu, May 27, 2010 at 08:52:18PM -0400, Wesley Craig wrote: For your problem, pop3d calls: prot_settimeout(popd_in, popd_timeout); just below where you've inserted the KEEPALIVE. What do you have poptimeout set to? It's set to 20 minutes. I wouldn't be surprised by a bug in prot, BTW. I'm pretty sure I've seen a case where select() is used to implement the timeout but once there's *some* input, read() is called with blocking (wrong!). In any case, if you can get a traceback with gdb for some hung pop3d's, I'm sure we can pinpoint the issue. Of course, we don't have those anymore. TCP keepalive cleans them up. I did find one that's been present for some time and seems idle. It does have an established TCP connection to a remote client. Here's a stack trace on it: # pstack 5432 5432: pop3d -s feb1a465 pollsys (8042da0, 2, 8042e60, 0) feac3b8a pselect (d, 8042eb4, feb90318, feb90318, 8042e60, 0) + 18e feac3e80 select (d, 8042eb4, 0, 0, 8042ea8, 0) + 82 0808981b prot_select (815d168, , 8043f94, 0, 8042ea8, 0) + 44b 0805e4ee proxy_check_input (815d168, 81a7228, 819e520, 81a3d60, 81a7700, 0) + 5e 0805dd74 bitpipe (8145c08, 0, feb921ec, 0, 8044fed, 8044fed) + c4 0805acb7 cmdloop (8135594, 8138980, 14, 2, 32203832, 31312e34) + 27 0805aa53 service_main (2, 8142a50, 8047db4) + 473 08062c13 main (2, 8047da8, 8047db4, 8047d9c) + a83 08059bbd _start (2, 8047e54, 8047e5a, 0, 8047e5d, 8047e68) + 7d It just vanished. TCP keepalive must have gotten it. -- -Gary Mills--Unix Group--Computer and Network Services- Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Experiment to test TCP keepalive for pop3d proxies
Ever since I can remember, our Cyrus installation had a problem with
pop3d processes accumulating on the murder front end server. This
didn't happen with imapd processes or with pop3d on the back end. A
couple of weeks ago, I counted 423 pop3d processes on the front end
but only 37 on the back end. Some of them were months old. All had
an established TCP connection from a client. Here's a typical stack
trace:
# pstack 12708
12708: pop3d -s
feb1a5c5 read (0, 817faf0, b)
fec2dfaf sock_read () + 3f
POP3 timeouts were enabled on both front and back ends, but it seemed
not to work on the front end. We're still running cyrus-imapd-2.3.8.
It's possible that this problem is fixed in the current version,
cyrus-imapd-2.3.16.
In any case, I wanted to try enabling TCP keepalive to see if it had
any effect on the problem. This only required a few lines of code:
--- pop3d.c-nokeep Wed Apr 11 10:49:59 2007
+++ pop3d.c Mon May 17 18:17:22 2010
@@ -494,6 +494,12 @@
if (getsockname(0, (struct sockaddr *)popd_localaddr, salen) ==
0) {
popd_haveaddr = 1;
}
+ /* Set keepalive option */
+ {
+ int oval = 1;
+ (void)setsockopt(0, SOL_SOCKET, SO_KEEPALIVE, (const void *)oval,
+sizeof(oval));
+ }
}
/* other params should be filled in */
A complete installation would include a configuration setting to
enable or disable TCP keepalive, along with ways to set keepalive
values that exist in many operating systems. This was just a test,
but it was quite impressive. `pop3d' processes no longer accumulated
on the front end, but were similar in number to the ones on the back
end. The cause must have been clients that disappeared without
closing their TCP connections. The TCP keepalive mechanism now does
this for them, after about half an hour of idleness.
Does anyone know if this problem has been solved by a timeout in
later Cyrus versions? That's actually a better solution. It does
only seem to happen when pop3d runs on a murder front end, relaying
connections to a back end. If it hasn't been solved, I'll proceed
with the keepalive solution. Otherwise, I'll plan for an upgrade.
--
-Gary Mills--Unix Group--Computer and Network Services-
Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Setting TCP keepalive for Cyrus daemons
On Sun, Feb 14, 2010 at 09:30:49AM -0600, Gary Mills wrote: I just noticed something else when I went to apply the patch. I would have added the options to cyrus.conf so a typical entry would change from: imap cmd=imapd listen=imap proto=tcp4 prefork=0 maxchild=6000 to: imap cmd=imapd listen=imap proto=tcp4 tcp_keepalive prefork=0 maxchild=6000 That way you could have a different keepalive setting for each service. You've designed it so these settings go into imapd.conf . Is that going to work the same way? I haven't seen a response on this question. Is it better to set TCP options for Cyrus daemons in /etc/imapd.conf or /etc/cyrus.conf? I'm willing to test this facility, but I'd like to know where the settings should be made first. -- -Gary Mills--Unix Group--Computer and Network Services- Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Setting TCP keepalive for Cyrus daemons
On Sun, Feb 14, 2010 at 08:56:29AM +1100, Bron Gondwana wrote: On Sun, Feb 14, 2010 at 08:38:34AM +1100, Bron Gondwana wrote: One thing to watch is that only SO_KEEPALIVE is standard. The other three symbols: TCP_KEEPCNT, TCP_KEEPIDLE, and TCP_KEEPINTVL only exist in some operating systems. They have global settings but don't have per-socket options. For these, the setsockopt() function calls need to be conditional on the symbols. Hmm - yeah, OK. I'll protect them with #ifdefs. Here's the patch redone with those... I just noticed something else when I went to apply the patch. I would have added the options to cyrus.conf so a typical entry would change from: imap cmd=imapd listen=imap proto=tcp4 prefork=0 maxchild=6000 to: imap cmd=imapd listen=imap proto=tcp4 tcp_keepalive prefork=0 maxchild=6000 That way you could have a different keepalive setting for each service. You've designed it so these settings go into imapd.conf . Is that going to work the same way? -- -Gary Mills--Unix Group--Computer and Network Services- Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Setting TCP keepalive for Cyrus daemons
On Sat, Feb 13, 2010 at 09:09:05PM +1100, Bron Gondwana wrote:
On Fri, Feb 12, 2010 at 09:45:02AM -0600, Gary Mills wrote:
I'm willing to add a `keepalive' option to Cyrus master along with the
setsockopt() system call to enable that setting. This option could be
added to the cyrus.conf file for any services that could benefit from
it. Would this be a reasonable addition to Cyrus?
How does this look?
Wow, you beat me to it! You even covered all of the settings.
+{ tcp_keepalive, 0, SWITCH }
+/* Enable keepalive on TCP connections */
+
+{ tcp_keepalive_cnt, 0, INT }
+/* Number of TCP keepalive probes to send before declaring the
+ connection dead (0 == system default) */
+
+{ tcp_keepalive_idle, 0, INT }
+/* Number of seconds a connection must be idle before keepalive
+ probes are sent (0 == system default) */
+
+{ tcp_keepalive_intvl, 0, INT }
+/* Number of seconds between keepalive probes (0 == system default) */
A switch to enable keepalive, plus options to edit each of the
tunables. The full patch is attached - not tested except for
a compile yet.
I can't comment on imap/sync_client.c because I don't use that
technique. In master.c, I would have put my changes into
spawn_service(), just before master exec'ed the daemon. However,
putting them where the connection was first accepted should be fine
too.
One thing to watch is that only SO_KEEPALIVE is standard. The other
three symbols: TCP_KEEPCNT, TCP_KEEPIDLE, and TCP_KEEPINTVL only exist
in some operating systems. They have global settings but don't have
per-socket options. For these, the setsockopt() function calls need
to be conditional on the symbols.
For which Cyrus version is your patch intended. I'm still running
cyrus-imapd-2.3.8 .
--
-Gary Mills--Unix Group--Computer and Network Services-
Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Setting TCP keepalive for Cyrus daemons
I've been noticing idle pop3d processes on our Cyrus front end server for some time. These should be transient. One that was several days old had an established TCP connection to a wireless client that had disappeared. Presumably the client never closed the connection. Setting TCP keepalive on the file descriptor should permit the kernel to close the connection in this situation. Does this sound reasonable? Perhaps it's already been addressed in a later Cyrus version. We're running cyrus-imapd-2.3.8. I'm willing to add a `keepalive' option to Cyrus master along with the setsockopt() system call to enable that setting. This option could be added to the cyrus.conf file for any services that could benefit from it. Would this be a reasonable addition to Cyrus? -- -Gary Mills--Unix Group--Computer and Network Services- Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Setting TCP keepalive for Cyrus daemons
On Fri, Feb 12, 2010 at 03:59:31PM -0600, Paul M Fleming wrote: Shouldn't these client connections already be handled by the poptimeout timeout options? unless you have it set to zero... They don't seem to be. We're using the default timeout setting. It seems to have no effect on front end daemons. Here's a stack trace on one that's several days old: # pstack 12708 12708: pop3d -s feb1a5c5 read (0, 817faf0, b) fec2dfaf sock_read () + 3f We have had problems within the murder (old code had several spots where murder front - back communications could deadlock).. -- -Gary Mills--Unix Group--Computer and Network Services- Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Front end and back end idle timeout settings?
We're running a murder configuration with one front end and one back end, using cyrus-imapd-2.3.8. Should the idle timeout setting for POP3 and IMAP sessions be different between the front end and back end? Which of the two should be higher? Does it matter at all? -- -Gary Mills--Unix Group--Computer and Network Services- Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Building cyrus sasl on solaris 10
On Tue, Jan 05, 2010 at 08:37:57AM +0100, Egoitz Aurrekoetxea wrote: I'm doing the following configure : 670 export CXX=/opt/sunstudio12.1/bin/CC 671 export CC=/opt/sunstudio12.1/bin/cc 672 export CPPFLAGS=-I/usr/include 673 export LDFLAGS=-L/lib 674 ./configure --prefix=/export/binarioscompilados/correo/cyrussasl --with-saslauthd=/var/run --with-pam=/export/binarioscompilados/correo/pammysql/modulos --with-configdir=/export/binarioscompilados/correo/cyrussasl/etc --disable-anon --enable-plain --disable-cram --disable-digest --enable-login --disable-otp --disable-gssapi --without-krb4 --without-des --without-authdaemond And it creates libplain.a but not libplain.so Check in config.log . There should be a test at some point to see if the compiler can create shared libraries. It's possible that configure made a mistake at that point, so that the test failed for some other reason. Any known reason... I have tried too you're configure... but I continue seeing in make the warnings told before about libraries and dlopen... That may be normal. Any ideas please?... have tried too with --enable-static and --enable-shared... but no way I can only speak for cyrus-sasl-2.1.22 under Solaris 10. If you are building a later version, somebody else will have to comment. -- -Gary Mills--Unix Group--Computer and Network Services- Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Building cyrus sasl on solaris 10
On Mon, Jan 04, 2010 at 11:25:01PM +0100, Egoitz Aurrekoetxea wrote: So is there any procedure for building this on Solaris 10?? I know solaris comes with some parts of cyrus sasl library... but it's only one part and it doesn't come with binaries like saslauthd... so I'd rather to build whose cyrus sasl. Is it any documented way for building this properly?... or is there any documented way of running saslauthd with Postfix and cyrus IMAP I mean... any solaris way for making this work please?. Which version are you building? This is what I used to configure cyrus-sasl-2.1.22. It certainly does create dynamic libraries. env LDFLAGS=-R/usr/local/lib \ CC=cc \ INSTALL=/usr/ucb/install \ ./configure \ --localstatedir=/var/run \ --with-dblib=none \ --with-saslauthd=/var/run/saslauthd \ --with-ipctype=doors \ --with-openssl=/usr/sfw \ --with-mysql=/usr/local/src/mysql/mysql-4.0.18 \ --disable-checkapop \ --disable-otp \ --enable-login \ --enable-ntlm \ --enable-sql \ --disable-krb4 \ --disable-gssapi -- -Gary Mills--Unix Group--Computer and Network Services- Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Murder confusion -- two mupdate slaves, lmtpproxyd's always connecting to master
On Tue, Nov 10, 2009 at 11:51:42AM -0500, Michael Bacon wrote: I have two questions regarding the Murder architecture. The first is just a general annoyance at the way master starts up the mupdate processes. In order to get master to fire off anything, you have to set prefork=1. However, the result of this is that you generate one mupdate, it reports as unavailable to the master, and so the master fires off another one. These two seem to fight with each other over connections to the mupdate master, over locks to the mailboxes database, and over who gets to serve connections. I don't think more than one is needed from what I can tell, but you get two just out of the master architecture. Do I really need two, and if not, is there some way to keep more than one from starting up? (maxchild=1 does not do the trick for whatever reason. I'm still deciphering code on this one.) I finally fixed that one, but it took a long time to find the reason. I always had two copies of the mupdate master running, but one of them did almost nothing... # ps -fp $(pgrep mupdate) UID PID PPID CSTIME TTY TIME CMD cyrus 3024 700 0 Apr 03 ? 0:01 mupdate -C /etc/mupdate/imapd.conf -m cyrus 3026 700 0 Apr 03 ? 49:02 mupdate -C /etc/mupdate/imapd.conf -m It turned out that one was listening on an IPv4 port and the other on an IPv6 port. Changing cyrus.conf from `listen=3905 prefork=1' to `listen=3905 proto=tcp4 prefork=1' solved the problem for me. I did the same thing for other Cyrus services as well. -- -Gary Mills--Unix Group--Computer and Network Services- Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
IOERROR: opening quota file: File name too long
We're running cyrus-imapd-2.3.8 on Solaris 10. Recently, the `quota' command failed with this error in syslog: Jun 29 19:58:49 castor quota[27067]: [ID 240394 local6.error] IOERROR: opening quota file /imap/conf/quota/N/user.___.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.Deleted Messages: File name too long Running it by hand resulted in this error: # /usr/local/cyrus/bin/quota /var/tmp/1.quota failed building quota list for '*': System I/O error: %m The deeply-nested folder seems to be created by Apple Mail. I could delete it with `cyradm', but the user kept recreating it. Is there a fix for this problem, perhaps in later Cyrus versions? We use the `quota' command to generate over-quota warnings automatically. -- -Gary Mills--Unix Support--U of M Academic Computing and Networking- Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Automatically moving marked mails?
On Fri, Jul 03, 2009 at 01:02:35AM -0400, Greg A. Woods wrote: I really don't know anyone, neither amongst home-based users nor corporate e-mail users, who truly believe they're better off with an MS-Exchange server handling their e-mail, especially if they've previously used a decent IMAP client connected to a Cyrus server. Most folks put up with it because they don't have any choice and that's because their IT guy got a good free game of golf or similar from the sales guy who sold him up the creek on using Exchange. There's pressure here too to move from Cyrus to Microsoft Exchange. It seems to be coming from administrators rather than students. Is there someplace an unbiased comparison of the two? I see lots of negative reports about Exchange, but they mostly come from people who are using another product based on open standards. -- -Gary Mills--Unix Support--U of M Academic Computing and Networking- Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Multiple copies of cyr_expire running
I notice that there are two of these running today: $ ps -fp $(pgrep cyr_expire) UID PID PPID CSTIME TTY TIME CMD cyrus 2510 986 3 04:00:01 ? 219:28 cyr_expire -E 3 cyrus 18280 986 3 Apr 27 ?1580:15 cyr_expire -E 3 There are also lots of errors like this. They refer to the same message over and over again: Apr 28 08:07:56 castor cyr_expire[18280]: [ID 264569 local6.error] DBERROR: mydelete: error deleting 200904201356.n3kdujes008...@taygeta.cc.umanitoba.ca: DB_NOTFOUND: No matching key/data pair found Should I kill one of the cyr_expire processes? Is there a safe way to do this? Is the duplicate delivery database broken? Is there a way to fix it? -- -Gary Mills--Unix Support--U of M Academic Computing and Networking- Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Multiple copies of cyr_expire running
On Tue, Apr 28, 2009 at 02:10:02PM -0400, Adam Tauno Williams wrote: On Tue, 2009-04-28 at 08:13 -0500, Gary Mills wrote: I notice that there are two of these running today: $ ps -fp $(pgrep cyr_expire) UID PID PPID CSTIME TTY TIME CMD cyrus 2510 986 3 04:00:01 ? 219:28 cyr_expire -E 3 cyrus 18280 986 3 Apr 27 ?1580:15 cyr_expire -E 3 There are also lots of errors like this. They refer to the same message over and over again: Apr 28 08:07:56 castor cyr_expire[18280]: [ID 264569 local6.error] DBERROR: mydelete: error deleting 200904201356.n3kdujes008...@taygeta.cc.umanitoba.ca: DB_NOTFOUND: No matching key/data pair found Should I kill one of the cyr_expire processes? Is there a safe way to do this? I'd kill -15 both of them. Then watch to see if they get stuck again. I did that last time around, with bad results. POP3 stopped working. I had to restart master to fix that. Is the duplicate delivery database broken? Is there a way to fix it? There is no reason to fix it; I'd just delete it. You maybe will be a couple duplicates but no big deal. I thought that some information need by the sieve vacation responder was stored in that database. I don't want to break that feature for thousands of people. -- -Gary Mills--Unix Support--U of M Academic Computing and Networking- Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Multiple copies of cyr_expire running
On Wed, Apr 29, 2009 at 10:12:03AM +1000, Bron Gondwana wrote: On Tue, Apr 28, 2009 at 01:55:01PM -0500, Gary Mills wrote: On Tue, Apr 28, 2009 at 02:10:02PM -0400, Adam Tauno Williams wrote: On Tue, 2009-04-28 at 08:13 -0500, Gary Mills wrote: I notice that there are two of these running today: $ ps -fp $(pgrep cyr_expire) UID PID PPID CSTIME TTY TIME CMD cyrus 2510 986 3 04:00:01 ? 219:28 cyr_expire -E 3 cyrus 18280 986 3 Apr 27 ?1580:15 cyr_expire -E 3 There are also lots of errors like this. They refer to the same message over and over again: Apr 28 08:07:56 castor cyr_expire[18280]: [ID 264569 local6.error] DBERROR: mydelete: error deleting 200904201356.n3kdujes008...@taygeta.cc.umanitoba.ca: DB_NOTFOUND: No matching key/data pair found Bloody BDB. I wish I understood it better. Lots of people use it, so it seems it must be something odd Cyrus does that causes it to be relatively unreliable... Yes, I hate that one too! It's the only one. The others are all skiplist or flat. Should I kill one of the cyr_expire processes? Is there a safe way to do this? I'd kill -15 both of them. Then watch to see if they get stuck again. I did that last time around, with bad results. POP3 stopped working. I had to restart master to fix that. Odd - it shouldn't. I have killed cyr_expire without problems before. I didn't expect a problem either. Then again, we only run it once per week, so it never wraps! Is the duplicate delivery database broken? Is there a way to fix it? There is no reason to fix it; I'd just delete it. You maybe will be a couple duplicates but no big deal. I thought that some information need by the sieve vacation responder was stored in that database. I don't want to break that feature for thousands of people. It may send a vacation response again. All it stores is the vacation already sent data. Okay, that's likely what I'll do. I'll try your cyr_dbtool first to see if it can delete that index entry. I would restart the master while deleting it though. Bron ( yes, that does kick off all your users... ) Yep. Most e-mail clients seem to reconnect quickly, so it shouldn't be too bad. -- -Gary Mills--Unix Support--U of M Academic Computing and Networking- Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Multiple IMAP connections from new IMAP clients
We've had a problem recently with the number of imapd processes on our Cyrus front-end increasing steadily until it filled the process table. It seems that some recent IMAP clients will normally open a number of IMAP connections to their server, and will open more based on user activity. Each of these causes a new imapd process to be spawned on the front-end. As far as I know, the server treats each connection independantly, even though the client may consider one to be permanent and the others to be transient. What are people doing to protect their Cyrus servers from this increasing number of connections, each of which consumes resources on the server? This problem is going to get worse as more sophisticated clients become popular. Is many small front-ends the solution? -- -Gary Mills--Unix Support--U of M Academic Computing and Networking- Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Multiple IMAP connections from new IMAP clients
On Thu, Apr 23, 2009 at 02:23:10PM -0500, Nic Bernstein wrote: On 04/23/2009 01:57 PM, Gary Mills wrote: We've had a problem recently with the number of imapd processes on our Cyrus front-end increasing steadily until it filled the process table. It seems that some recent IMAP clients will normally open a number of IMAP connections to their server, and will open more based on user activity. Each of these causes a new imapd process to be spawned on the front-end. As far as I know, the server treats each connection independantly, even though the client may consider one to be permanent and the others to be transient. What are people doing to protect their Cyrus servers from this increasing number of connections, each of which consumes resources on the server? This problem is going to get worse as more sophisticated clients become popular. Is many small front-ends the solution? We've been using imapproxyd to help solve just this kind of problem. Haven't used it with a murder, but expect it could still be useful. Does it actually combine separate connections from a single client into one connection to the server? I don't know how it could do that without violating the protocol. -- -Gary Mills--Unix Support--U of M Academic Computing and Networking- Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Why do lmtpd processes accumulate?
We have a fairly conventional Cyrus server with one front-end and one back-end. Recently, I've noticed that when the number of lmtpd processes on the back-end server increases to the 400 range, performance drops to a crawl, including local deliveries. When I put an upper limit of 128 or 64 to these processes on the front-end, which requires a Cyrus restart, all of the local deliveries succeed in a short time. Performance also comes back to normal. I can't tell if it's the restart that fixes the problem or if it's the limit on lmtpd children. I'm wondering, though, if the lmtpd processes are all waiting on some Cyrus database, so that more of them just makes it worse. These are the databases, from imapd.conf: annotation_db: skiplist duplicate_db: berkeley-nosync mboxkey_db: skiplist mboxlist_db:skiplist quota_db: quotalegacy seenstate_db: skiplist subscription_db:flat tlscache_db:berkeley-nosync I believe those are current recommendations. Which ones might be causing the problem? Is there tuning that can be done on them? -- -Gary Mills--Unix Support--U of M Academic Computing and Networking- Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Pause while selecting INBOX
On Fri, Sep 26, 2008 at 09:19:25AM -0700, Rob Banz wrote: On Sep 26, 2008, at 06:44, Gary Mills wrote: We have a moderately sized Cyrus installation with 2 TB of storage and a few thousand simultaneous IMAP sessions. When one of the backup processes is running during the day, there's a noticable slowdown in IMAP client performance. When I start my `mutt' mail reader, it pauses for several seconds at `Selecting INBOX'. That behavior disappears when the backup finishes. As always, the answer is probably complicated. Of course. I just wanted to know what the usual suspects might be. What's your storage backend look like? Some sort of RAID with cache? The backup is very probably blowing out your cache while running, which can manifest itself in ways that will make it look like the performance profile of your storage has taken on a different personality. Especially with respect to writes, since you might usually be blessed with an abundance of write-back cache that's now fighting with all the reads for the backup. Do you have your cyrus partitions mounted with noatime? Your backups could be causing fs write operations that you really don't want. Our storage backend is four Iscsi LUNs from our Netapp filer. It will indeed be RAID with cache. Yes, the backup certainly would run the cache, and that may well be the problem. There will also be a large memory cache on the IMAP server. That could be upset as well. I'll see if I can find some cache statistics for both places. `noatime' is also certainly a good idea. I haven't yet tried it. -- -Gary Mills--Unix Support--U of M Academic Computing and Networking- Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Pause while selecting INBOX
On Fri, Sep 26, 2008 at 01:37:30PM -0400, Wesley Craig wrote: I'd probably use imtest to connect, get the PID of the server process that I'm connected to, and then attach to that process with ktrace (or whatever) with timestamps enabled. Then I'd select the mailbox -- this is assuming that mutt is only issuing a select when it says Selecting INBOX. Obviously it could be doing any number of things. You can get positive confirmation of which command is taking a long time by enabling telemetry, of course. Thanks for the suggestions. I believe I'll start with the telemetry, to get an idea what mutt is doing when it pauses. -- -Gary Mills--Unix Support--U of M Academic Computing and Networking- Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Pause while selecting INBOX
On Sat, Sep 27, 2008 at 01:46:39PM +1000, Rob Mueller wrote: Where's the first place to look for this problem? I/O statistics show a higher read bandwidth while the backup is running, but writes still dominate. The backup would typically read all of the files in a single Cyrus partition. Some more information about your setup would be helpful. I deliberately omitted that information, hoping that somebody would know what happens on the server side when mutt says `selecting INBOX'. I don't want to redesign the server at this stage, at least not until I've isolated the problem. 1. Make sure you have noatime (and nodiratime if applicable) set on your FS 2. If on linux, use the deadline IO scheduler 3. If you can, split your data and meta data onto separate volumes (probably a lot of work) 4. Install more RAM and use a 64-bit OS. x64_64 linux can cache many, many more inodes and dentries in memory than x86 linux. 5. Upgrade to the latest cyrus and enable the status cache Those are all good suggestions. I'll investigate some of them. -- -Gary Mills--Unix Support--U of M Academic Computing and Networking- Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Pause while selecting INBOX
On Sun, Sep 28, 2008 at 09:42:27AM +1000, Bron Gondwana wrote: On Sat, 27 Sep 2008 13:46:39 +1000, Rob Mueller [EMAIL PROTECTED] said: Some things to try if you haven't: 6. Don't fetch every single file from your cyrus server every backup run. One nice property of cyrus's on-disk format is that all the data files are immutable once created: b/user/brong/798. is always going to be the same file, no matter what happens to the mailbox. Only the cyrus.* files and new data file actually need to be fetched. (our system also stats the cyrus.* files, and uses some other funky tricks to ensure consistent backups with minimum IO) Most of our restores are done from ZFS snapshots. We keep two weeks of them. That's worked out extremely well. We use Networker with tapes for longer-term backups. It's that that seems to cause the problem. Certainly our Networker backup system has no knowledge of the structure and properties of the Cyrus IMAP spool. It would just read every file for a full backup, or look at timestamps for an incremental backup. I don't know if much can be changed in the way it operates. However, we may be able to change the way it takes backups if contention is really the problem. I'm not convinced that it is just now. -- -Gary Mills--Unix Support--U of M Academic Computing and Networking- Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Pause while selecting INBOX
We have a moderately sized Cyrus installation with 2 TB of storage and a few thousand simultaneous IMAP sessions. When one of the backup processes is running during the day, there's a noticable slowdown in IMAP client performance. When I start my `mutt' mail reader, it pauses for several seconds at `Selecting INBOX'. That behavior disappears when the backup finishes. Where's the first place to look for this problem? I/O statistics show a higher read bandwidth while the backup is running, but writes still dominate. The backup would typically read all of the files in a single Cyrus partition. -- -Gary Mills--Unix Support--U of M Academic Computing and Networking- Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: How to repair a broken seen state file
On Mon, Jul 07, 2008 at 03:37:56PM +0200, Konrad Mauz wrote: On Mon, Jul 07, 2008 at 08:31:59AM -0500, Gary Mills wrote: I'm seeing errors like this regularly in our messages log: Jul 4 11:43:37 castor imap[16398]: [ID 514311 local6.error] DBERROR: skiplist recovery: 058C should be INORDER Jul 4 11:43:37 castor imap[16398]: [ID 729713 local6.error] DBERROR: opening /imap/conf/user/O/inqarts.seen: cyrusdb error It's always for this one file. All the others are fine. It's a skiplist database. The ownership is correct. A reconstruct on the mailbox has no effect on the file. Can I just remove the file, with no ill effects? Can I fix it somehow? We're running cyrus-imapd-2.3.8. perheps the skiplist.py tool from http://oss.netfarm.it/python-cyrus.php can help you to restore the corrupted seen file. Yes, that worked! skiplist.py extracted the data to a text file. Then cvt_cyrusdb converted from flat to skiplist again. There are no more errors in the log. -- -Gary Mills--Unix Support--U of M Academic Computing and Networking- Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
How to repair a broken seen state file
I'm seeing errors like this regularly in our messages log: Jul 4 11:43:37 castor imap[16398]: [ID 514311 local6.error] DBERROR: skiplist recovery: 058C should be INORDER Jul 4 11:43:37 castor imap[16398]: [ID 729713 local6.error] DBERROR: opening /imap/conf/user/O/inqarts.seen: cyrusdb error It's always for this one file. All the others are fine. It's a skiplist database. The ownership is correct. A reconstruct on the mailbox has no effect on the file. Can I just remove the file, with no ill effects? Can I fix it somehow? We're running cyrus-imapd-2.3.8. -- -Gary Mills--Unix Support--U of M Academic Computing and Networking- Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: How to repair a broken seen state file
On Mon, Jul 07, 2008 at 10:26:33AM -0400, Wesley Craig wrote: 058C is an offset in hex. If you truncate the file at this point, you should be able to proceed. Obviously, everything after that will be gone, but in my experience, these error typically occur at the end of skiplist files, so you probably won't lose much if anything. Also, this user's mail probably looks quite odd, e.g., all new mail is always unseen. Anyway, something like: dd if=/imap/conf/user/O/inqarts.seen of=/imap/conf/user/O/ inqarts.seen.fixed bs=1420 count=1 and mv /imap/conf/user/O/inqarts.seen.fixed /imap/conf/user/O/inqarts.seen Make sure /imap/conf/user/O/inqarts.seen is still permitted correctly. I neglected to mention that I'd already tried that... # dd if=inqarts.seen of=inqarts.seen.new bs=1 count=1420 1420+0 records in 1420+0 records out # chmod 600 inqarts.seen.new # mv inqarts.seen inqarts.seen.old mv inqarts.seen.new inqarts.seen # ll -t inqarts.* -rw--- 1 cyrusmail1420 Mar 28 14:11 inqarts.seen -rw--- 1 cyrusmail 10052 Mar 28 11:37 inqarts.seen.old -rw--- 1 cyrusmail 320 Feb 21 10:53 inqarts.sub It still logged the same error afterwards. I guess I'll have to try something more drastic now. -- -Gary Mills--Unix Support--U of M Academic Computing and Networking- Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: pam pop issue
On Tue, Jun 17, 2008 at 02:32:46PM +0530, Ashay Chitnis wrote: On Tue, Jun 17, 2008 at 12:09 AM, Gary Mills [EMAIL PROTECTED] wrote: Gary, thanks for your help. I have had one sleepless night trying to read out the sasl manuals from SUN :). the pam_acct_mgmt() call must be removed from saslauthd/auth_pam.c and added to lib/server.c instead. can you elaborate more on how you have acheived it? By modifying the SASL source and recompiling it. I can post my patches if anyone else is interested. first the item passed by sasl is the service name (pop) and not the remote network ip and this is compared with the actual IP address. pam_get_item should be getting the IP address and passing it to pam NOT the service name.. As others have mentioned, the information stored in the PAM handle depends on the application. Many different types are possible, but the application has to store the data to make it available to the PAM module. PAM_RHOST, the remote host name, would be the one that interests you. That information is not always present. For cyrus and sasl, it appears not to be present. I haven't confirmed this. I have checked its works beautifully in sshd. Now i need to find a way to work it in sasl for pop imap service. -- -Gary Mills--Unix Support--U of M Academic Computing and Networking- Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: pam pop issue
On Mon, Jun 16, 2008 at 10:49:11PM +0530, Ashay Chitnis wrote: I need to access pop and imap on user based IP level restrictions. I found pam to be best suited for this service level restriction. The restriction will be as below. User pqr should be allowed POP from IPADDR-1 User B should be allowed IMAP from IPADDR-2 User C should be allowed POP and IMAP from IPADDR-3 and so on. To achieve this below settings are done in /etc/pam.d/pop cat /etc/pam.d/pop authrequired/lib/security/pam_ldap.so account required /lib/security/pam_access.so debug accessfile=/usr/local/etc/popaccess.conf account required/lib/security/pam_ldap.so cat /usr/local/etc/popaccess.conf +:pqr:[1]192.168.2.66/32 OR -:pqr:ALL EXCEPT [2]192.168.2.66/32 But this does not see to be working as it is not yielding desired effect even after restarting saslauthd and cyrus.. We use a similar restriction in the account management section of PAM, except that the checks are for account status and service class. To make this work properly, it's necessary to modify SASL. Specifically, the pam_acct_mgmt() call must be removed from saslauthd/auth_pam.c and added to lib/server.c instead. -- -Gary Mills--Unix Support--U of M Academic Computing and Networking- Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Is skiplist dependant on byte order?
On Mon, Jun 16, 2008 at 10:18:47PM +1000, Bron Gondwana wrote: On Mon, 16 Jun 2008 03:29:25 -0700, Scott Likens [EMAIL PROTECTED] said: I'm going to take a shot in the dark, BIG Endian vs. Little Endian? Skiplist has had quite a lot of care taken to use network order for all values. I don't _believe_ there are any issues. Perhaps I had an older version, or I didn't do it quite correctly. Unfortunately I do believe bdb databases do care if it was big or little... and going from Sparc (BIG) to x86 (little)... Would not work very well :( Yeah - they are pretty version and system specific. I would always dump BDB databases for a transfer. Certainly crossing architectures. Yes, I omitted those from my original message because I also did a version upgrade on BDB; I expected problems there. Fortunately, they were all empty on my murder front end, so I just deleted them after my test on mailboxes.db. I am going to guess that a reconstruct may not be a bad idea, your seen databases may or may not work, and pretty much guess that any and all databases related to Cyrus will need to be re-worked... I'm sure someone like Bron (fastmail.fm) might have something already whipped up for this. Seen should be in /var/imap/user ... I would check on sieve (it is compiled also) ... (/var/imap/sieve?) Yeah, recompiling your sieve files doesn't sound like a bad idea at all. There are no mailboxes on my murder front end, so these shouldn't exist either. I'm not upgrading the back end this time around. then the /var/spool/imap/a/user/.. and you can more then likely just do a reconstruct -rf and be fine... That's a serious amount of IO. All the index and cache files are supposed to be endian-clean as well. It's all htonl and ntohl everywhere. Again, these shouldn't exist on the front end. On Jun 15, 2008, at 6:38 PM, Gary Mills wrote: (there's more below) I recently upgraded a murder front end server from Solaris 9 SPARC to Solaris 10 x86 by copying the /imap directory. I did dump the mailboxes database before the copy. It's a skiplist database. I'm running cyrus-imapd-2.3.8 on both systems. As a test, I first checked on the mailboxes database like this: # su cyrus -c ksh # /usr/local/cyrus/bin/ctl_mboxlist -d | wc -l 0 Do you have cyr_dbtool in that version? I can't remember when it got take upstream. dbtool is nice because it dumps all the cyrus databases, not just the mailboxes.db. I don't believe so. This message appeared in the log: Jun 11 16:24:43 setup01 ctl_mboxlist[14082]: [ID 864961 local6.crit] DBERROR: critical database situation That sounds like BDB to me. Are you running BDB mailboxes.db? That would certainly explain it. The mailboxes database certainly is skiplist, but perhaps there was some other involved as well. I actually got two messages. They do sound like BDB errors: Jun 11 16:24:43 setup01 ctl_mboxlist[14082]: [ID 866726 local6.warning] DBERROR db4: PANIC: fatal region error detected; run recovery Jun 11 16:24:43 setup01 ctl_mboxlist[14082]: [ID 864961 local6.crit] DBERROR: critical database situation After I reloaded it, I got the correct output: # /usr/local/cyrus/bin/ctl_mboxlist -u mailboxes.txt # /usr/local/cyrus/bin/ctl_mboxlist -d | wc -l 77 These commands generated four log messages. I renamed and recreated the `db' directory before running them, and of course renamed `mailboxes.db'. Jun 11 16:29:34 setup01 ctl_mboxlist[14091]: [ID 143423 local6.error] DBERROR: reading /imap/conf/db/skipstamp, assuming the worst: No such file or directory Jun 11 16:29:35 setup01 ctl_mboxlist[14091]: [ID 275131 local6.notice] skiplist: recovered /imap/conf/mailboxes.db (0 records, 144 bytes) in 0 seconds Jun 11 16:29:57 setup01 ctl_mboxlist[14093]: [ID 143423 local6.error] DBERROR: reading /imap/conf/db/skipstamp, assuming the worst: No such file or directory Jun 11 16:29:57 setup01 ctl_mboxlist[14093]: [ID 275131 local6.notice] skiplist: recovered /imap/conf/mailboxes.db (77 records, 8460 bytes) in 0 seconds I'm assuming that skiplist is dependant on the machine's byte order, and that a dump and reload is necessary in this case. No, it really shouldn't matter. One of the good things about the skiplist design. There are other bits that aren't so good - but the byte order part is nice. I'm not clear which parts of the `db' directory are associated with skiplist databases and which with BDB databases. Are there any other databases that I should also dump and reload? As far as I can tell, the annotation_db, duplicate_db, and tlscache_db are empty and can simply be removed. Are there any others on a murder front end that I've missed? Where do they reside? Yeah, we nuke all those on restart. duplicate_db is the most interesting of that lot - but not a giant concern
Is skiplist dependant on byte order?
I recently upgraded a murder front end server from Solaris 9 SPARC to Solaris 10 x86 by copying the /imap directory. I did dump the mailboxes database before the copy. It's a skiplist database. I'm running cyrus-imapd-2.3.8 on both systems. As a test, I first checked on the mailboxes database like this: # su cyrus -c ksh # /usr/local/cyrus/bin/ctl_mboxlist -d | wc -l 0 This message appeared in the log: Jun 11 16:24:43 setup01 ctl_mboxlist[14082]: [ID 864961 local6.crit] DBERROR: critical database situation After I reloaded it, I got the correct output: # /usr/local/cyrus/bin/ctl_mboxlist -u mailboxes.txt # /usr/local/cyrus/bin/ctl_mboxlist -d | wc -l 77 This is a test server with only a few mailboxes. I'll upgrade the production server later. I'm assuming that skiplist is dependant on the machine's byte order, and that a dump and reload is necessary in this case. Are there any other databases that I should also dump and reload? As far as I can tell, the annotation_db, duplicate_db, and tlscache_db are empty and can simply be removed. Are there any others on a murder front end that I've missed? Where do they reside? -- -Gary Mills--Unix Support--U of M Academic Computing and Networking- Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Sieve forwarding loop destroys e-mail
On Mon, Mar 31, 2008 at 02:04:29PM +0200, Alain Spineux wrote: On Mon, Mar 31, 2008 at 5:12 AM, Gary Mills [EMAIL PROTECTED] wrote: On Sun, Mar 30, 2008 at 02:27:29PM +0100, Alain Spineux wrote: On Mon, Mar 17, 2008 at 5:39 PM, Gary Mills [EMAIL PROTECTED] wrote: Once again, we had somebody use the sieve facility to redirect e-mail back to the same mailbox and then go on vacation. This sets up a forwarding loop which cyrus breaks by discarding the e-mail. During this vacation, all of the person's e-mail disappeared. If you force a keep in your sieve script, the mail will be delivered at least once in the mailbox It's perfectly valid to have nothing but a `forward' in a sieve script. People do this all the time when they don't want to keep a copy for themselves. Unfortunately, some also forward e-mail to themselves, expecting that to work. sieve script is only a language. The language nor its interpreter nor its compiler dont need to be smart, because the script writer is supposed to be smart enough. If the user in unable to write such script, it must use a sieve script manager, (application written by a smarter developer) that will help him generating well suited script. There's no general solution to this problem in the script language, even with a generator like we use. Again, that's because even the generator can't tell what forwarding will cause a loop. I could, for example, forward e-mail to my @fastmail.fm address, from where it would come back to me. The generator could never know about my remote forwarding. In the simplest case, writing a sieve script to forward to yourself causes the e-mail to disappear. The script writer could be considered `smart' in expecting this to work. Calling the writer `stupid' doesn't solve the problem. -- -Gary Mills--Unix Support--U of M Academic Computing and Networking- Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Sieve forwarding loop destroys e-mail
On Mon, Mar 31, 2008 at 11:52:10AM -0400, Ken Murchison wrote: Gary Mills wrote: Once again, we had somebody use the sieve facility to redirect e-mail back to the same mailbox and then go on vacation. This sets up a forwarding loop which cyrus breaks by discarding the e-mail. During this vacation, all of the person's e-mail disappeared. Shouldn't we have a better solution to this problem? Some people expect that forwarding e-mail to yourself should work; nobody expects the messages to vanish without a trace. I'm all for trying fix this if someone can come up with some logic to do so. IMO, the code is correctly processing the script as written. Here is the current code logic: - original message is sent to lmtpd - message is forwarded and a record is put in deliver.db stating as much - forwarded message comes back to lmtpd - lmtpd executes the script which tells it to forward to another address - lmtpd sees that it has already forwarded the message, so doesn't forward it again At what point should we decide to deliver the message? The user hasn't asked us to do that, even though they think that they have. How can lmtpd be intelligent enough to know that the forwarded address will cause the message to come back? Yes, it's difficult to do this correctly and still maintain a separation between the delivery process, with duplicate suppression, and the sieve facility. To begin with, the forwarding mechanism of sieve would have to mark a forwarded message in some manner so that it can be identified later. This could be something in the message header, or something associated with the message in a database. It would be a guarantee that it wasn't going to forward it again. Next, the delivery mechanism would have to bypass duplicate suppression for such a marked message. That way, it would be delivered again, and the sieve facility would be responsible for breaking the loop. It would have to do something other than forwarding at that point. `keep' is the simplest option. Perhaps there could be others. -- -Gary Mills--Unix Support--U of M Academic Computing and Networking- Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Sieve forwarding loop destroys e-mail
On Sun, Mar 30, 2008 at 02:27:29PM +0100, Alain Spineux wrote: On Mon, Mar 17, 2008 at 5:39 PM, Gary Mills [EMAIL PROTECTED] wrote: Once again, we had somebody use the sieve facility to redirect e-mail back to the same mailbox and then go on vacation. This sets up a forwarding loop which cyrus breaks by discarding the e-mail. During this vacation, all of the person's e-mail disappeared. If you force a keep in your sieve script, the mail will be delivered at least once in the mailbox It's perfectly valid to have nothing but a `forward' in a sieve script. People do this all the time when they don't want to keep a copy for themselves. Unfortunately, some also forward e-mail to themselves, expecting that to work. Once the message has been `seen' and is about to be forwarded again, it would be better to deliver the message into the mailbox, rather than deleting it. Is this possible with the current design of the duplicate delivery database? How can a message be 'seen' if it is not in the mailbox, and how can the mailbox owner read this message if he is in vacation ? Or you mean 'seen', already in the delivered db ? But then this is the job of the delivered db to drop the email to avoid loop. I suppose I mean `delivered'. Yes, it will be discarded the next time it's delivered. That's the problem. There needs to be some connection between delivery and the forwarding process. Of course, it's impossible to distinguish between a forwarding loop and a real duplicate unless another `Received' header is added to the message header. -- -Gary Mills--Unix Support--U of M Academic Computing and Networking- Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Sieve forwarding loop destroys e-mail
Once again, we had somebody use the sieve facility to redirect e-mail back to the same mailbox and then go on vacation. This sets up a forwarding loop which cyrus breaks by discarding the e-mail. During this vacation, all of the person's e-mail disappeared. Shouldn't we have a better solution to this problem? Some people expect that forwarding e-mail to yourself should work; nobody expects the messages to vanish without a trace. Once the message has been `seen' and is about to be forwarded again, it would be better to deliver the message into the mailbox, rather than deleting it. Is this possible with the current design of the duplicate delivery database? -- -Gary Mills--Unix Support--U of M Academic Computing and Networking- Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: cyrus on iscsi
On Wed, Feb 27, 2008 at 09:20:39PM -0500, Jeffrey T Eaton wrote: Our system consists of 10 backend servers. Each is a Sun v240 running Solaris 8, with a QLogic iSCSI initiator with two gigabit ethernet links to our SAN network. Each system mounts 4x250 GB paritions for mailbox storage, and one 50GB partition for Cyrus databases, logs, and sieve scripts. If you upgrade to Solaris 10, you can omit the QLogic cards by using the native Iscsi initiator instead. It may actually work better. -- -Gary Mills--Unix Support--U of M Academic Computing and Networking- Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: cyrus on iscsi
On Thu, Feb 28, 2008 at 09:48:00AM -0500, Jeffrey Eaton wrote: We will probably be upgrading to Solaris 10 for our backend mailservers in the very near future, because Sun doesn't appear to be selling any more v240's, and the v245's won't boot Solaris 8. We may also seriously consider dropping the Sun hardware entirely, and moving toward Linux. The rest of our mail infrastructure already runs on Linux (frontends, mupdate, and smtp/mx layer), so it seems likely that we will at least consider it. The Sun T2000 servers make wonderful Cyrus IMAP server backends. These are SPARC and run Solaris 10. Anything built for Solaris 8 will run without modification on them. I recommend them. -- -Gary Mills--Unix Support--U of M Academic Computing and Networking- Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: cyrus on iscsi
On Fri, Feb 22, 2008 at 08:15:38AM +0100, Rudy Gevaert wrote: Is someone running a cyrus instance on iscsi? And is your instance big/ heavy loaded? Yes, we've been doing this for about a year now. Our server is a Sun T2000 with two ethernet connections to the Iscsi SAN. It's using the native Solaris load balancing and failover. It mounts two LUNs exported from our Netapp file server. These are concatenated into a single ZFS pool. So far, we are only using about 600 gigs of the 1 TB of storage. We will be adding two more LUNs when we need more space. We use ZFS snapshots for most mailbox restores, but we also do full backups to tape. $ zpool list NAMESIZEUSED AVAILCAP HEALTH ALTROOT space 1016G645G371G63% ONLINE - The server is lightly loaded. We do have about 35,000 users, but only about 2000 simultaneous users at peaks. -- -Gary Mills--Unix Support--U of M Academic Computing and Networking- Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: dracauth/RPC problem
On Tue, Jan 29, 2008 at 12:01:47PM +0100, Anders Norrbring wrote: I get a lot of these in my cyrus log; Jan 29 11:53:54 svea pop3[2703]: accepted connection Jan 29 11:53:54 svea pop3[2703]: login: someone.cust.bredbandsbolaget.se [213.112.58.xxx] mpb0xxx plaintext User logged in Jan 29 11:53:54 svea pop3[2703]: dracauth: localhost: RPC: Port mapper failure - RPC: Unable to receive Jan 29 11:53:54 svea pop3[2703]: DRAC notifications disabled What does it mean, and how do I correct it? Is rpcbind/portmap running? Is rpc.dracd running? Try `rpcinfo -p'. It should show program number 900101 for the DRAC daemon. -- -Gary Mills--Unix Support--U of M Academic Computing and Networking- Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: prefork mupdate slaves
On Wed, Nov 21, 2007 at 02:58:11PM -0800, Andrew Morgan wrote: With the following service entry on a traditional murder frontend (v2.3.10): mupdate cmd=/usr/local/cyrus/bin/mupdate listen=3905 prefork=1 I notice that 2 mupdate processes are spawned at startup: Nov 21 14:43:43 cyrus-fe3 mupdate[6935]: successful mupdate connection to cyrus-mm.onid.oregonstate.edu Nov 21 14:43:43 cyrus-fe3 mupdate[6935]: unready for connections Nov 21 14:43:43 cyrus-fe3 mupdate[6935]: synchronizing mailbox list with master mupdate server Nov 21 14:43:43 cyrus-fe3 mupdate[6931]: successful mupdate connection to cyrus-mm.onid.oregonstate.edu Nov 21 14:43:43 cyrus-fe3 mupdate[6931]: unready for connections Nov 21 14:43:43 cyrus-fe3 mupdate[6931]: synchronizing mailbox list with master mupdate server Nov 21 14:43:48 cyrus-fe3 mupdate[6935]: mailbox list synchronization complete Nov 21 14:43:52 cyrus-fe3 mupdate[6931]: mailbox list synchronization complete And those 2 mupdate processes remain running. If I set prefork=0, then no mupdate processes are ever spawned, as far as I can tell. On my old v2.2.13 murder frontends, prefork=1 only causes 1 mupdate process to be spawned. Is this a bug? Should I be concerned that 2 mupdate slaves are running? I have two on the murder front end as well: UID PID PPID CSTIME TTY TIME CMD cyrus 765 708 0 Sep 24 ? 332:49 mupdate cyrus 768 708 0 Sep 24 ?1:35 mupdate On the murder back end, I actually have three, although one terminates almost immediately with this error: bind: /imap/mupdate/conf/socket/mupdate.target: Address already in use bind failed service mupdate pid 1595 in READY state: terminated abnormally I also have `prefork=1' in both places. This is with cyrus-imapd-2.3.8. -- -Gary Mills--Unix Support--U of M Academic Computing and Networking- Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: One more attempt: stuck processes
On Mon, Nov 19, 2007 at 12:35:46PM -0500, Ken Murchison wrote:
Sebastian Hagedorn wrote:
-- Ken Murchison [EMAIL PROTECTED] is rumored to have mumbled on
17. November 2007 11:21:38 -0500 regarding Re: One more attempt: stuck
processes:
Here's a patch that seems to fix the problem. I did some basic testing
(Linux only) to make sure that it doesn't break anything else, but its
always possible that it has some unforseen side effects. Keep an eye on
it and let me know if you see anything unusual.
Thanks, it seems to be working fine so far! Enjoy your weekend now ...
Gary, have you tried the patch?
On Solaris 9, SO_RCVTIMEO is not mentioned in the setsockopt man page.
My reading tells me that it is defined in the header file, but if it's
actually used, setsockopt() will return an error. I understand that
this is the case for other operating systems too. This seems to be a
known problem with setsockopt(). I just checked the Opensolaris source;
it does function there. This is the code:
case SO_RCVTIMEO:
if (optlen == sizeof (uint32_t))
sockets[i].in_timeout = *(uint32_t *)optval;
else {
errno = EINVAL;
}
break;
Note that the option has to be an int, and the length has to be
that of an int as well. Linux wants a `struct timeval'. I wonder
if there's a standard in this area?
--
-Gary Mills--Unix Support--U of M Academic Computing and Networking-
Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Timed Actions in Sieve
On Tue, Nov 13, 2007 at 11:24:48AM +, Ian G Batten wrote: We've been having a chat about how useful it would be to have timed actions in sieve: so that a vacation message could be set up for a duration which would automatically revert, so that a forwarding could be set up for the duration of a short-term project, etc, etc. The naive way is to add support to the sieve interface of choice (the squirrelmail plugin in our case) to handle deferred actions, but I can think of all sorts of security problems with that. Another would be a means to auto-generate regexps to match on Date: headers, but that's really tacky. The full solution would be to have the current time available in sieve scripts, to then match on. Has anyone else thought about this area? We've had occasional complaints from people who set up a vacation message and then forgot to remove it later. They would like to be able to put a time limit on such things, so that they would stop working when that limit expires. More generally, I suppose they could specify start and stop times, so that they could set up the sieve script in advance of their vacation. -- -Gary Mills--Unix Support--U of M Academic Computing and Networking- Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: One more attempt: stuck processes
On Fri, Nov 16, 2007 at 03:20:57PM +0100, Sebastian Hagedorn wrote: --On 16. November 2007 08:00:07 -0600 Gary Mills [EMAIL PROTECTED] wrote: This timeout doesn't work in some cases. We have lots of POP sessions that never terminate. That's interesting to hear! Especially since you are using Solaris. About 30 out of 40 are in that state now. Here's an example: cyrus 13075 708 0 Oct 14 ?0:05 pop3d -s cyrus 20023 708 0 Oct 29 ?0:00 pop3d cyrus 24560 708 1 07:38:03 ?0:03 pop3d cyrus 631 708 0 Oct 03 ?0:10 pop3d -s cyrus 6786 708 0 Oct 20 ?0:00 pop3d -s cyrus 29777 708 0 07:45:03 ?0:00 pop3d cyrus 19175 708 0 Oct 04 ?0:04 pop3d -s One I just checked is stuck in a read(): # truss -p 19175 read(0, 0x002316F0, 5) (sleeping...) ^?# pfiles 19175 19175: pop3d -s Current rlimit: 256 file descriptors 0: S_IFSOCK mode:0666 dev:271,0 ino:25813 uid:0 gid:0 size:0 O_RDWR sockname: AF_INET 130.179.16.23 port: 995 peername: AF_INET 130.179.188.184 port: 51771 Could you get a stack trace? If you have gdb you just call it with gdb -p 19175. Then you can do bt at the prompt. I forget how to do it with Sun's debugger. Easy: # pstack 19175 19175: pop3d -s fef9f810 read (0, 2316f0, 5) fee1d2d0 read (0, 2316f0, 5, 0, 0, 0) + 5c ff06bb38 sock_read (1f0860, 2316f0, 5, 5, 0, 0) + 24 ff068af0 BIO_read (1f0860, 2316f0, 5, fef98b84, 0, 0) + 110 ff278488 ssl3_read_n (212798, 5, 8805, 0, 0, 203958) + 174 ff2785fc ssl3_get_record (204ce0, 8000, 8400, 4400, f1, f0) + d0 ff279424 ssl3_read_bytes (212798, 1000, 2000, 4, 0, ffbfe731) + 228 ff27a99c ssl3_get_message (ff2a259c, 2070a0, 0, , 19000, ffbfe7a0) + d0 ff27042c ssl3_accept (2150, 2160, 2180, 21e0, 2110, 2122) + 904 ff27bd2c ssl23_get_client_hello (2316fb, 6c, 6c, 4, fe79, 0) + 828 ff27b4b4 ssl23_accept (4000, 2000, 0, 0, 0, 0) + 2a4 00032d00 tls_start_servertls (0, 1, ffbfee24, ffbfee20, 1849a8, ff00) + 198 0002c504 cmd_starttls (1, 1fd8b8, 0, 0, 0, 0) + 184 0002a638 service_main (2, 192198, ffbffce0, 1aec4, 3508c, 1) + 488 00035250 main (2, ffbffcd4, ffbffce0, 17c400, 0, 0) + e18 00029298 _start (0, 0, 0, 0, 0, 0) + 108 I've confirmed that the client has gone away a long time ago. -- -Gary Mills--Unix Support--U of M Academic Computing and Networking- Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: One more attempt: stuck processes
On Fri, Nov 16, 2007 at 01:54:24PM +0100, Alain Spineux wrote: On Nov 16, 2007 12:36 PM, Sebastian Hagedorn [EMAIL PROTECTED] wrote: --On 16. November 2007 11:27:09 +0100 Sebastian Hagedorn [EMAIL PROTECTED] wrote: 1. In the absence of the SO_KEEPALIVE option it is entirely possible that a TCP connection remains ESTABLISHED even when the other side has gone. I said that socket should timeout, but this is true only when the protocol (TCP here) require a response (usualy AK here) or at connection establishement. On the contrary it should stay open indefinitely util something happens. Router doing NAT can drop a too old connection, because it has to maintains a NAT table and make some cleanup time to time, this where KEEPALIVE become usefull. This may not be a solution to this particular problem, but it made me wonder why Cyrus does *not* use SO_KEEPALIVE. Is there a downside to it? Cyrus has already a built-in time out, it seems a lite conflicting to actively maintains the connection until it drop it itself ! This is the works of the client to actively maintains the connection, if it want it ! This timeout doesn't work in some cases. We have lots of POP sessions that never terminate. About 30 out of 40 are in that state now. Here's an example: cyrus 13075 708 0 Oct 14 ?0:05 pop3d -s cyrus 20023 708 0 Oct 29 ?0:00 pop3d cyrus 24560 708 1 07:38:03 ?0:03 pop3d cyrus 631 708 0 Oct 03 ?0:10 pop3d -s cyrus 6786 708 0 Oct 20 ?0:00 pop3d -s cyrus 29777 708 0 07:45:03 ?0:00 pop3d cyrus 19175 708 0 Oct 04 ?0:04 pop3d -s One I just checked is stuck in a read(): # truss -p 19175 read(0, 0x002316F0, 5) (sleeping...) ^?# pfiles 19175 19175: pop3d -s Current rlimit: 256 file descriptors 0: S_IFSOCK mode:0666 dev:271,0 ino:25813 uid:0 gid:0 size:0 O_RDWR sockname: AF_INET 130.179.16.23 port: 995 peername: AF_INET 130.179.188.184 port: 51771 -- -Gary Mills--Unix Support--U of M Academic Computing and Networking- Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: One more attempt: stuck processes
On Fri, Nov 16, 2007 at 05:13:13PM +0100, Sebastian Hagedorn wrote: --On 16. November 2007 14:23:17 +0100 Simon Matter [EMAIL PROTECTED] wrote: Did you ever see non SSL connections get stuck? No. Most of mine are `pop3d -s', but I have seen a few without the `-s'. When I did a stack trace on one, it also turned out to be for an SSL session. So, I have to agree. -- -Gary Mills--Unix Support--U of M Academic Computing and Networking- Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: One more attempt: stuck processes
On Fri, Nov 16, 2007 at 03:54:50PM -0500, Ken Murchison wrote: That's exactly what Gary is seeing. Its blocking in SSL_accept(). Apparently the client connects to port 995, and then either sends nothing, or goes away and leaves the socket open. I've reproduced the former by telneting to port 995 and doing nothing. I have been unable to reproduce the latter because as soon as I QUIT the telnet session or kill() the telnet process, pop3d exits gracefully. You probably have to reboot the client at that point, or just disconnect the cable and take it home. -- -Gary Mills--Unix Support--U of M Academic Computing and Networking- Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: One more attempt: stuck processes
On Wed, Nov 14, 2007 at 04:15:13PM +0100, Sebastian Hagedorn wrote: I've brought up this topic before. We've been running cyrus-imapd very happily for several years. Yet there's one issue that none of the updates have resolved. The last time I reported it we were running 2.2.12. Now we're running 2.3.8, but the issues is the same: POP and IMAP processes that use TLS/SSL get stuck. My observations point towards dropped dial-up connections as the cause. Here's an example: Have you tried setting `tls_session_timeout' to zero in imapd.conf to disable caching of TLS sessions? I believe that that solved the problem for me, although it happened rarely. I only use `berkeley-nosync' for `duplicate_db' and `tlscache_db', but I suspected that the problem was the result of a database deadlock. -- -Gary Mills--Unix Support--U of M Academic Computing and Networking- Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: How many people to admin a Cyrus system?
On Thu, Nov 08, 2007 at 05:05:39PM -0600, Blake Hudson wrote: Gary Mills wrote: We have a moderate-sized Cyrus system for 30,000 students and 3000 employees. ... I know that CMU and other universities want to maintain their own e-mail systems. What's the justification in these cases? From a security perspective, you maintain control and privacy of your and your user's data. This is one of the main reasons why many people run their own services, not limited to mail (www,db, instant messaging, etc). Whether it is the perceived value or because of other obligations with regard to privacy/security this alone is often justification to maintain a mail server in-house. There's also the business perspective of cost, especially cost over time. Thanks everyone for your responses. I don't want to clutter up this technical mailing list with more management issues, although I'd certainly be pleased to receive personal e-mail on this topic. There appear to be two types of outsourcing. The Google example was one where all of the e-mail resided on an external site. In addition to the issues mentioned above, there is authentication and backup/restore to consider. For all of those reasons, I don't think that this type will be suitable here. The Zimbra example, however, was one where a contractor was hired to install a new e-mail system at the university, and to do the migration and management. This one I could see happening here, so that people with programming and development skills would no longer need to be kept on staff. That seems like a bizarre idea to me. It's essentially outsourcing the employees. Since there are no problems whatsoever with the existing Cyrus system, I suppose that contracting with a company to maintain and manage it might be better than just abandoning it. -- -Gary Mills--Unix Support--U of M Academic Computing and Networking- Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
How many people to admin a Cyrus system?
We have a moderate-sized Cyrus system for 30,000 students and 3000 employees. It's a critical service in the sense that thousands of people depend on it. It has excellent performance, lots of capacity, and plans for expansion. I'm the only one familiar enough with Cyrus and sendmail to maintain it, although this is not normally full time. I'm also the one who tracks down hard problems in Unix and does development in a number of other areas. Other than our data management person, who supplies the LUNs for the e-mail store, I'm the one who manages the system. I'm also going to be 65 in two days, although I plan to stay around for a year or so. My director will be replacing me, but I assume this will be with an entry-level person who will not have the ability to maintain the Cyrus system, at least initially. Other people in my group do not have the skills or the available time to administer this system. I'm trying to convince him to hire somebody with development and programming experience, but without much success so far. How many and what sort of people does it take to maintain a system such as this? I need a good argument for hiring a replacement for me. My director seems interested in outsourcing our e-mail system, judging by the number of articles on outsourcing that he sends to me. Google and Zimbra with a commercial contractor are the latest two. Replacing a perfectly functioning e-mail system seems ludicrous to me, as does subjecting our users to a migration for no reason. I assume at least that he wants vendors to quote on a replacement system. Perhaps once he sees the cost, he will change his mind. I suppose it depends on whether the quote includes the real cost. Does anyone here have experience in this area? I know that CMU and other universities want to maintain their own e-mail systems. What's the justification in these cases? -- -Gary Mills--Unix Support--U of M Academic Computing and Networking- Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Cyrus murder scalability
On Thu, Oct 25, 2007 at 10:38:01AM -0400, Ken Murchison wrote: Vladimir Nikolic wrote: I have questions regarding murder scalability. Untill last week we had one imap server with about 5 users and 2TB mailboxes. Server was overloaded and little slow but there was no troubles like connections drop and repeatedly asking users for password. Now we have that old server and three new servers in murder configuration (1 frontend, 1 murder master in 2 backends) and we have full troubles with users (almost all of them with pop3 accounts) complaining that mail clients repeatedly asking for password. Logs on fronted are full of lines: pop3[12914]: couldn't authenticate to backend server pop3[9822]: couldn't authenticate to backend server: authentication failure What version of Cyrus? I see these messages occasionally on the front end too. This is with cyrus 2.3.8. Do you only have this problem with POP3? IMAP and LMTP connections are proxied to the backends without any problems? Ours are usually for IMAP. I assumed that the session timed out for idleness on the back end. There's no corresponding message on the back end. Oct 25 10:01:54 electra imap[12446]: [ID 637875 local6.error] Fatal error: Lost connection to selected backend Oct 25 10:02:11 electra imap[21715]: [ID 637875 local6.error] Fatal error: Lost connection to selected backend Oct 25 10:02:32 electra imap[29415]: [ID 637875 local6.error] Fatal error: Lost connection to selected backend On the back end, these are common: Oct 25 09:36:34 castor pop3[8294]: [ID 390285 local6.error] Unable to lock maildrop user.shaykew: Mailbox is locked by POP server Oct 25 09:50:30 castor pop3[9277]: [ID 390285 local6.error] Unable to lock maildrop user.umpromo: Mailbox is locked by POP server Oct 25 09:52:40 castor pop3[8920]: [ID 390285 local6.error] Unable to lock maildrop user.harropp: Mailbox is locked by POP server -- -Gary Mills--Unix Support--U of M Academic Computing and Networking- Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Murder works wonderfully but alarms users
We have a Cyrus murder configuration with one proxy front-end and one storage back-end. I'm very pleased with it. However, users who happen to look at the full headers of their e-mail are often alarmed by the word `murder' that appears in the first `Received' header. It's even worse when the message is from daemon! Here are some typical headers. Would it be possible to eliminate the offensive word or replace it with something more meaningful? Received: from murder (electra.cc.umanitoba.ca [130.179.16.23]) by mbox.cc.umanitoba.ca (Cyrus v2.3.8) with LMTPA; Sun, 09 Sep 2007 22:14:47 -0500 X-Sieve: CMU Sieve 2.3 Received: from electra.cc.umanitoba.ca ([unix socket]) by cc.umanitoba.ca (Cyrus v2.3.8) with LMTPA; Sun, 09 Sep 2007 22:14:47 -0500 X-Sieve: CMU Sieve 2.3 -- -Gary Mills--Unix Support--U of M Academic Computing and Networking- Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Too many open files
On Fri, Aug 24, 2007 at 04:49:10PM +0100, Rob McMahon wrote:
Cyrus-2.2.13 (yes, I know it's getting on a bit). Solaris 10 SPARC,
around 115,000 mailboxes.
Same thing here with cyrus-imapd-2.3.8 on Solaris 10 SPARC.
I was mopping up a few users today with a script which basically does:
while () {
chomp;
my $mb = 'user.' . $_;
$client-setacl($mb, $user = 'c');
$client-delete($mb);
}
After deleting just a few, I start getting errors like Must login
first, System I/O Error (sorry, I haven't got the exact messages to
cut paste). Looking in the logs I see:
Aug 24 16:20:44 narcissus imap[24338]: [ID 240394 local6.error] IOERROR:
opening quota file /var/imap/quota/V/user.??.Trash: Too many open files
Aug 24 16:21:38 narcissus imap[24338]: [ID 816447 local6.error] warning:
cannot open /etc/hosts.deny: Too many open files
Mine was like this:
Aug 23 09:43:46 castor imap[20105]: [ID 136705 local6.error] IOERROR: opening
/imap/spool/K/user/xx/cyrus.header: Too many open files
The problem is intermittent, only happening with certain imapd
processes. I've noticed it when processing large batches of mailboxes
from a perl script. When I rerun the batch, with a different imapd,
the error doesn't appear. It almost never happens in normal IMAP use.
I don't know how to track it down in these circumstances.
--
-Gary Mills--Unix Support--U of M Academic Computing and Networking-
Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Expiry date for sieve vacation messages
What would be required to add an expiry date feature to sieve vacation messages? The need is pretty clear: people set up a vacation message when they begin their vacation, but forget to remove it when they return. They have no indication in their e-mail reader that their vacation message is still being sent out. -- -Gary Mills--Unix Support--U of M Academic Computing and Networking- Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Odd mailbox xfer result
On Tue, Aug 07, 2007 at 07:57:56PM -0400, Robert Banz wrote: Earlier today I was testing how well transferring my rather massive personal folder hierarchy from one backend server to another -- which sent apparently without a hitch. Quite fast, I might add. However, after the move I noticed my sieve rules weren't working -- and when I connected up with 'sieveshell' to have a look-see at my script, I was surprised to find my sieve script turned out to be an actual mail message!? So, anyone seen anything like this before? I did some looking around on bugzilla, and nothing jumped out at me. Look for bug 2917 `xfer copies the last message instead of sieve scripts to the remote server'. The patch is there too. -- -Gary Mills--Unix Support--U of M Academic Computing and Networking- Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Cyrus ZFS performance
On Thu, Jul 05, 2007 at 09:10:51AM -0700, Vincent Fox wrote: More importantly what does iostat -cxn 5 look like during peak? For us this is 1100-1300 hours hitting about 40%. Of course this is summertime usage so things are a bit slack here with most students being home, not so much daily class-related chitchat. This is on a Sun T2000 with 500 GB of e-mail on a ZFS filesystem. Storage is from a Netapp using iSCSI. There are 1500 IMAP sessions, about half of what we get in the winter. I'm quite pleased with this configuration. cpu us sy wt id 4 3 0 93 extended device statistics r/sw/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device 15.0 192.4 1000.6 1900.3 0.0 0.60.02.9 0 34 c3t0d0 15.2 275.6 941.0 2024.7 0.0 0.70.02.6 0 36 c3t3d0 -- -Gary Mills--Unix Support--U of M Academic Computing and Networking- Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: unified murder gets mailboxes list truncated
On Sat, Jun 30, 2007 at 03:16:30PM +0300, Janne Peltonen wrote: On Thu, Jun 28, 2007 at 01:57:57AM +0300, Janne Peltonen wrote: I'm running a unified murder and have been experiencing a strange problem. I shut down a cluster member cleanly, then start the member again, run ctl_cyrusdb -r and ctl_mboxlist -m, all goes well (even if the ctl_mboxlist -m takes a long time), the services start - the mupdate slaves starts to synchronize with the mupdate master, and, at the beginning, throws away all records of remote mailboxes! Only after it has synchronized with the master does it know how to proxy to other murder members. Now what might be wrong here? Now I wonder. ctl_mboxlist -m truncating the local (backend) mailboxes list to contain only local mailboxes makes perfect sense - in a /traditional/ murder. But mine is a unified murder. This is bug 2915: On a unified server, `ctl_mboxlist -m' deletes remote mailboxes. It's fixed in the CVS version of Cyrus imapd. Look at: http://bugzilla.andrew.cmu.edu/show_bug.cgi?id=2915 -- -Gary Mills--Unix Support--U of M Academic Computing and Networking- Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Recommend how to move 31GB of mail to a new server
On Sat, Jun 23, 2007 at 02:33:46PM -0700, Jose Hales-Garcia wrote: I'm preparing a new iMAP server and have 31GB of user mail on the current server to move. I'm looking for suggestions on doing the move with the minimum of down time as possible. I did this recently, although it was 300 gigabytes. My strategy was to convert the old server into a unified murder and configure the new server as a standard murder node. I was then able to use the XFER function of Cyrus IMAP to move mailboxes transparently from the old server to the new one. There was no downtime other than for occasional reboots. The migration took a couple of weeks. To begin, I had to upgrade the old server to cyrus-imapd-2.3.8. I recommend great caution if you follow this example. I tested all of my procedures on a pair of test servers before using them on the production servers, finding a few bugs and other pitfalls along the way. For only 30 gigabytes, you might be better off just copying the files over, with IMAP down. It could only take a few hours. You can copy a sample from the live system to get an idea of the timing. -- -Gary Mills--Unix Support--U of M Academic Computing and Networking- Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Replication question - cross replication?
On Fri, Jun 15, 2007 at 08:56:53AM +0100, David Carter wrote: On Thu, 14 Jun 2007, Nels Lindquist wrote: I'm setting up a high-availability mail server setup with two boxes that will essentially be mirrors of each other. If both are configured for local delivery, can I have them replicate each other if I utilize UUIDs? No. IMAP is not well suited to active-active replication. Replication in Cyrus is strictly active-passive. The way to do this is with a cluster filesystem mounted on both boxes. It's not an easy way to go. -- -Gary Mills--Unix Support--U of M Academic Computing and Networking- Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Connection throttling POP3.
On Mon, May 21, 2007 at 03:36:34PM -0500, Blake Hudson wrote: Andrew Morgan wrote: On Mon, 21 May 2007, Matthew Schumacher wrote: And this spammer is racking up a zillion processes which is killing my machine. I need a way to throttle this somehow where he is only allowed one connection per IP at a time, or perhaps a way to ignore them after so many invalid passwords. One-connection-per-IP only works when the users have individual IP addresses. If they're on a multi-user server or behind a proxy server, this won't work. You can use tcp-wrappers to block connections from that IP address entirely. I believe there are also some solutions to monitor connections and automatically add IP addresses to the /etc/hosts.deny file, but I've never used them myself. These types of threats are becoming more and more common and in reaction awareness is increasing and more software seems to be implementing mechanisms to cope. I would personally love to see Cyrus implement some sort of connection limit or throttling per IP/network/user. The current process limits do help ensure that one daemon does not make the machine unusable, but does nothing to prevent a DoS attack. One thing that should be done is to report the behavior to the relevant ISP. If they don't know what their customer is doing, they're not likely to take any actions. I realize that sometimes complaining is futile, but sometimes it has the desired effect. -- -Gary Mills--Unix Support--U of M Academic Computing and Networking- Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Trouble with Eudora and cyrus-imapd-2.3.8
On Thu, May 17, 2007 at 06:54:45AM -0500, Amos wrote: On 5/13/07, Gary Mills [EMAIL PROTECTED] wrote: Since our recent upgrade from cyrus-imapd-2.1.14 to cyrus-imapd-2.3.8, I've had one report that Eudora is behaving very badly. It seems to have to do with synchronization between the client and the server. The symptom is that the inbox view is not correct. Deleted messages don't disappear, but when they're deleted again, other messages disappear. As well, some messages stay in the inbox when they should have been filtered to a local folder. Filtering them again results in duplicates in the local folder. Did you enable telemetry logging for this user to see what the communications between the client and server look like? Not sure if that would help, but maybe worth a try I haven't done that yet, but I plan to do so. I wanted to see if it was a known problem before I tried that. The user did make a curious discovery the other day. There's a right-click menu item on his INBOX that says `Show deleted messages'. When he selected that, all his missing messages showed up, but they were marked as unread, not deleted. Other IMAP clients have been showing those unread messages. -- -Gary Mills--Unix Support--U of M Academic Computing and Networking- Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Trouble with Eudora and cyrus-imapd-2.3.8
Since our recent upgrade from cyrus-imapd-2.1.14 to cyrus-imapd-2.3.8, I've had one report that Eudora is behaving very badly. It seems to have to do with synchronization between the client and the server. The symptom is that the inbox view is not correct. Deleted messages don't disappear, but when they're deleted again, other messages disappear. As well, some messages stay in the inbox when they should have been filtered to a local folder. Filtering them again results in duplicates in the local folder. Has anyone else heard of this sort of problem with Eudora? Is there anything that can be done, either on the client or server side, to correct it? -- -Gary Mills--Unix Support--U of M Academic Computing and Networking- Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
How to stop IMAP referrals?
Is there a way to prevent an imapd front-end from advertizing or performing referrals to a back-end server, and just do proxying instead? There are a couple of reasons that I'd prefer this. The main one is that the back-end server doesn't offer the full range of authentication mechanisms. Another is that Pine users complain that they keep getting prompted for a password whenever they send mail, if they have their `sent mail' folder on the IMAP server. I'd be happier if the front-end would just proxy the connections to the back-end. -- -Gary Mills--Unix Support--U of M Academic Computing and Networking- Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: ZFS for Cyrus IMAP storage
On Fri, May 04, 2007 at 06:35:41PM -0500, Gary Mills wrote: There was a question earlier regarding ZFS for Cyrus IMAP storage. We recently converted to that filesystem here. I'm extremely pleased with it. Our server has about 30,000 users with over 200,000 mailboxes. It peaks at about 1900 IMAP sessions. It currently has 1 TB of storage with about 300 GB in use. The server is a Sun T2000 with 16 GB of memory, running Solaris 10. It uses iSCSI to provide two 500 GB devices from our Netapp filer. Disk redundancy is all on the Netapp because it's currently superior to that provided by ZFS. I see that I need to clarify this statement. ZFS' disk redundancy, which is excellent, is not used in this configuration. ZFS' disk management, which also can't be used in this configuration, is currently incomplete. Disk management on the Netapp works very nicely. -- -Gary Mills--Unix Support--U of M Academic Computing and Networking- Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: ZFS for Cyrus IMAP storage
On Sat, May 05, 2007 at 10:14:07AM -0700, Vincent Fox wrote: Gary Mills wrote: There was a question earlier regarding ZFS for Cyrus IMAP storage. We recently converted to that filesystem here. I'm extremely pleased with it. Our server has about 30,000 users with over 200,000 mailboxes. It peaks at about 1900 IMAP sessions. It currently has 1 TB of storage with about 300 GB in use. The server is a Sun T2000 with 16 GB of memory, running Solaris 10. It uses iSCSI to provide two 500 GB devices from our Netapp filer. Disk redundancy is all on the Netapp because it's currently superior to that provided by ZFS. Our T2000 have 8 GB for the cluster we are building here at University of California Davis. Would you expect we will be having to upgrade RAM, or is it just gravy? I really don't know. IMAP sessions need lots of memory. ZFS will use all it can get for its ARC cache. For a one-off server, I find it cheaper to overdesign it than to devote a lot of time to full load testing. -- -Gary Mills--Unix Support--U of M Academic Computing and Networking- Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
ZFS for Cyrus IMAP storage
There was a question earlier regarding ZFS for Cyrus IMAP storage. We recently converted to that filesystem here. I'm extremely pleased with it. Our server has about 30,000 users with over 200,000 mailboxes. It peaks at about 1900 IMAP sessions. It currently has 1 TB of storage with about 300 GB in use. The server is a Sun T2000 with 16 GB of memory, running Solaris 10. It uses iSCSI to provide two 500 GB devices from our Netapp filer. Disk redundancy is all on the Netapp because it's currently superior to that provided by ZFS. ZFS is fully POSIX compliant; Cyrus is quite happy with that. Performance is excellent. The full backups go much faster than they did when we were using UFS on a pair of locally-attached RAID-5 arrays. The ability to expand the storage pool simply by adding more devices from the Netapp is very appealing. No longer will we have to move mailboxes from one partition to another. The snapshots are wonderful. We keep 14 daily snapshots, instead of doing daily incremental backups. I can certainly recommend ZFS for Cyrus. -- -Gary Mills--Unix Support--U of M Academic Computing and Networking- Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
How many copies of the mupdate master should run?
On our mupdate master server, mupdate is defined in cyrus.conf with `prefork=1'. However, two of them are running, both children of master... UID PID PPID CSTIME TTY TIME CMD cyrus 1594 483 0 Apr 15 ? 0:00 mupdate -C /etc/mupdate/imapd.conf -m cyrus 1596 483 0 Apr 15 ? 22:18 mupdate -C /etc/mupdate/imapd.conf -m The second one seems to be doing all the work. What's the first one for? Both have opens on the unix-domain socket. Only the second one has TCP connections. In fact, there were three started, but one terminated... Apr 15 22:09:16 castor mupdate[1595]: [ID 166678 local6.error] bind: /imap/mupdate/conf/socket/mupdate.target: Address already in use Apr 15 22:09:16 castor mupdate[1595]: [ID 702911 local6.error] bind failed Apr 15 22:09:16 castor master[483]: [ID 684980 local6.warning] service mupdate pid 1595 in READY state: terminated abnormally What's going on here? -- -Gary Mills--Unix Support--U of M Academic Computing and Networking- Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: LMTP AUTH with sendmail?
On Fri, Apr 06, 2007 at 02:41:41PM +0200, Andreas Haumer wrote: Andrzej Adam Filip schrieb: You use older MAILER(`cyrus'). Take a look at newer MAILER(`cyrusv2'). I *am* using cyrusv2 mailer already (as I wrote in my first mail) Cyrusv2 supports direct LMTP over unix socket. I don't want to use LMTP over unix domain socket (which of course works and of course gives me access control using unix file permissions) but over INET sockets (i.e. LMTP over TCP), as machines running sendmail and cyrus imapd are distinct and connected through a TCP/IP network. Here's an example of a mailer definition that does LMTP over TCP: Mlocal, P=[IPC], F=lsDFMnqSmXzwWA5@/:|, S=EnvFromL/HdrFromL, R=EnvToL/HdrToL, E=\r\n, T=DNS/RFC822/X-Unix, A=TCP [cyrus-server-host.domain] lmtp I wrote my own m4 feature to generate this definition. -- -Gary Mills--Unix Support--U of M Academic Computing and Networking- Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Example of a perl callback with send()
The Cyrus::IMAP perl library describes send() this way: $client-send(\callback, \cbdata, $format, ...); However, none of the examples I could find uses the callback and callback data arguments. Does anybody have an example of how these are used. I'm designing a perl script that makes several IMAP connections to the server, to send long-running commands in parallel. In this case, the script can't block on a send(). It needs to issue all of them and then poll for results. I'm assuming that a perl callback along with processoneevent() will make this possible. -- -Gary Mills--Unix Support--U of M Academic Computing and Networking- Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Notes on operating a unified murder
For some time now, I've been testing a unified murder configuration of the Cyrus IMAP server. I'm using two servers. Server A is a unified front-end/back-end server, hosting mailboxes and proxying connections to the back-end server. Clients connect only to server A. Server B is a conventional back-end server, only hosting mailboxes. My objective in this exercise is to set up a production server in the same manner, and then move mailboxes from the front-end to the back-end in a transparent manner. I've accomplished this successfully with the test server. Here are some notes I kept along the way: o The mupdate master cannot run on unified server, even with a different configuration directory. o The mupdate master will run on the backend server, with a different configuration directory. o A mupdate slave is required on a unified server, to update the local mailboxes database. This limitation should be fixable. o The mupdate server is multi-threaded but also runs as multiple processes. However, it uses mutex locking that can't work between processes. o E-mail delivery on the back-end server doesn't work if lmtpd has the `-a' option. At least, I couldn't figure out how to make it work. o For proxy delivery to work on the back-end server, the proxy user from the front-end must be defined in `lmtp_admins' on the back-end. o The service name configuration prefix as in `lmtp_admins' appears to work but is not documented anyplace. Where are the names listed? o `cyradm' connected to the unified server honours redirection to the backend, but prompts for a password repeatedly. This could be fixed by having it accept multiple server names on the command line, and opening persistent connections to all of them. o `cyradm' must connect to the backend to create mailboxes there. It doesn't work via proxy from frontend. o `sieveshell' connected to the unified server honours redirection to the backend, but prompts for a password twice. o For the cyradm `xfer' command, the unified server connects to the backend server as the proxy user, even when cyradm authenticates to the front-end as the administrator. o The mupdate master sometimes attempts to connect to itself, producing this error: ``kick_mupdate: can't connect to target: Connection refused''. o `autocreatequota' on the backend server may be necessary for `xfer' to work. o Defining the proxy user in `proxyservers' on the backend server does not work to enable cyradm `xfer' to work. o The proxy user must be defined in `imap_admins' for `xfer' to work. What privileges does this imply? o cyradm `xfer' only works from the unified server to the backend server, when connected to the unified server. Could it be fixed to work in reverse? Will it work when connected to backend server? -- -Gary Mills--Unix Support--U of M Academic Computing and Networking- Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: can not run mupdate master on a unified server
On Wed, Feb 28, 2007 at 07:07:19PM -0600, Gary Mills wrote: On Fri, Feb 23, 2007 at 04:49:28PM -0600, Gary Mills wrote: I just ran into this error: can not run mupdate master on a unified server when I tried to test a Cyrus murder configuration with one unified server and one backend server. Can I assume that the mupdate master won't run on the backend either? Is this because of a database conflict? By reading the source, I determined that the mupdate master maintains the same mailboxes database that's used for local mailboxes. That's why it can't be run on a unified server or on a backend server. Giving the mupdate master its own mailboxes database would solve this problem. In that case, each unified or frontend server would require a mupdate slave to synchronize its mailboxes file to the master copy. I got around this problem by running the mupdate master under a separate instance of Cyrus master, with its own imapd.conf file. I can now run `ctl_mboxlist -m' successfully on either the unified front/back server or a backend server. Mailboxes from both of these appear in the mupdate master's mailboxes.db file. However when I run either `ctl_mboxlist -d' or `cyradm' on the unified server, they show only the mailboxes on that one server. What's missing? Do I also need a mupdate slave on that server? How is mailboxes.db supposed to get updated on a unified server? By reading the source, I determined that only the mupdate slave is able to synchronize a local mailboxes database with the master copy. So, I moved the mupdate master to the backend server, again running it under a separate instance of Cyrus master, with its own imapd.conf file. Now everything works. The mailboxes database on the unified server now contains both local and remote mailboxes. -- -Gary Mills--Unix Support--U of M Academic Computing and Networking- Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Beware the unified Cyrus server
On Sat, Mar 03, 2007 at 06:36:12PM -0600, Gary Mills wrote: If you have: 1) An existing conventional IMAP server, and 2) convert it to a unified murder configuration, with 3) a mupdate slave, and without 4) a mupdatepush entry in cyrus.conf, it will delete all of the local mailboxes from the mailboxes database. This seems like very dangerous behavior to me, even though it can be avoided by adding the mupdatepush entry in cyrus.conf. How can the mupdate slave be modified to make it safer? On reflection, I'm convinced that the mupdate slave should not be doing this. In a conventional murder, with discrete frontend and backend servers, it's reasonable to have mailboxes on the backend that are not known to the mupdate master. This is a temporary condition, of course. The same thing should be permitted in a unified murder. The mupdate slave should be modified to avoid deleting local mailboxes in a unified configuration. There's a similar, but less serious, problem when `ctl_mboxlist -m' is run on a unified server: it deletes all of the remote mailboxes from the local mailboxes database. These are restored when the mupdate slave starts up. Again, `ctl_mboxlist -m' should not be doing this in a unified configuration. It might be reasonable for a backend server, although in that case the remote mailboxes should never be in the local mailboxes database. -- -Gary Mills--Unix Support--U of M Academic Computing and Networking- Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
One mupdate master always dies on startup
Below are the logs of the startup of a mupdate master server. On completion, two copies of `mupdate -m' are running as children of the master. However, three copies are actually started but the second one always terminates with a bind failure. Otherwise, they appear to be running normally. What's going on? Mar 3 09:51:33 castor master[14284]: [ID 965400 local6.notice] process started Mar 3 09:51:34 castor ctl_cyrusdb[14285]: [ID 702911 local6.notice] recovering cyrus databases Mar 3 09:51:34 castor ctl_cyrusdb[14285]: [ID 275131 local6.notice] skiplist: recovered /imap/mupdate/conf/mailboxes.db (2 records, 332 bytes) in 0 seconds Mar 3 09:51:34 castor ctl_cyrusdb[14285]: [ID 275131 local6.notice] skiplist: recovered /imap/mupdate/conf/annotations.db (0 records, 144 bytes) in 0 seconds Mar 3 09:51:34 castor ctl_cyrusdb[14285]: [ID 127214 local6.notice] done recovering cyrus databases Mar 3 09:51:34 castor master[14284]: [ID 139525 local6.notice] ready for work Mar 3 09:51:34 castor ctl_cyrusdb[14287]: [ID 702911 local6.notice] checkpointing cyrus databases Mar 3 09:51:34 castor mupdate[14289]: [ID 166678 local6.error] bind: /imap/mupdate/conf/socket/mupdate.target: Address already in use Mar 3 09:51:34 castor mupdate[14289]: [ID 702911 local6.error] bind failed Mar 3 09:51:34 castor master[14284]: [ID 684980 local6.warning] service mupdate pid 14289 in READY state: terminated abnormally Mar 3 09:51:34 castor ctl_cyrusdb[14287]: [ID 127214 local6.notice] done checkpointing cyrus databases -- -Gary Mills--Unix Support--U of M Academic Computing and Networking- Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Beware the unified Cyrus server
If you have: 1) An existing conventional IMAP server, and 2) convert it to a unified murder configuration, with 3) a mupdate slave, and without 4) a mupdatepush entry in cyrus.conf, it will delete all of the local mailboxes from the mailboxes database. Note that running without the mupdatepush entry is the recommended way to start a murder node the first time. This problem doesn't happen with a murder backend because there's no mupdate slave. It also doesn't happen with a murder frontend because there are no local mailboxes. It only happens with a unified front/backend server. It's particularly annoying because there's no easy way to rebuild the mailboxes database. This seems like very dangerous behavior to me, even though it can be avoided by adding the mupdatepush entry in cyrus.conf. How can the mupdate slave be modified to make it safer? Here are some typical log entries of the server startup. There seem to be two copies of mupdate running. One silently deletes the local mailboxes. The other tries to, but can't because they're already gone. Mar 2 19:10:23 setup01 mupdate[2580]: [ID 974332 local6.error] successful mupdate connection to castor Mar 2 19:10:23 setup01 mupdate[2580]: [ID 697712 local6.notice] unready for connections Mar 2 19:10:23 setup01 mupdate[2577]: [ID 974332 local6.error] successful mupdate connection to castor Mar 2 19:10:23 setup01 mupdate[2577]: [ID 697712 local6.notice] unready for connections Mar 2 19:10:23 setup01 mupdate[2577]: [ID 104270 local6.notice] synchronizing mailbox list with master mupdate server Mar 2 19:10:23 setup01 mupdate[2580]: [ID 104270 local6.notice] synchronizing mailbox list with master mupdate server Mar 2 19:10:23 setup01 mupdate[2577]: [ID 538854 local6.error] MUPDATE: can't delete mailbox entry 'user.cox' ... Mar 2 19:10:24 setup01 mupdate[2577]: [ID 538854 local6.error] MUPDATE: can't delete mailbox entry 'user.pehn' Mar 2 19:10:24 setup01 mupdate[2577]: [ID 622122 local6.notice] mailbox list synchronization complete Mar 2 19:10:24 setup01 mupdate[2580]: [ID 538854 local6.error] MUPDATE: can't delete mailbox entry 'user.wyatt' Mar 2 19:10:24 setup01 mupdate[2580]: [ID 622122 local6.notice] mailbox list synchronization complete -- -Gary Mills--Unix Support--U of M Academic Computing and Networking- Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: can not run mupdate master on a unified server
On Fri, Feb 23, 2007 at 04:49:28PM -0600, Gary Mills wrote: I just ran into this error: can not run mupdate master on a unified server when I tried to test a Cyrus murder configuration with one unified server and one backend server. Can I assume that the mupdate master won't run on the backend either? Is this because of a database conflict? I got around this problem by running the mupdate master under a separate instance of Cyrus master, with its own imapd.conf file. I can now run `ctl_mboxlist -m' successfully on either the unified front/back server or a backend server. Mailboxes from both of these appear in the mupdate master's mailboxes.db file. However when I run either `ctl_mboxlist -d' or `cyradm' on the unified server, they show only the mailboxes on that one server. What's missing? Do I also need a mupdate slave on that server? How is mailboxes.db supposed to get updated on a unified server? -- -Gary Mills--Unix Support--U of M Academic Computing and Networking- Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
can not run mupdate master on a unified server
I just ran into this error: can not run mupdate master on a unified server when I tried to test a Cyrus murder configuration with one unified server and one backend server. Can I assume that the mupdate master won't run on the backend either? Is this because of a database conflict? How much CPU and memory resources does the mupdate master require, relative to a frontend or backend server? Does it really require dedicated hardware? If not, I could run it in a container on one of the other machines. Is there a better way to do this? -- -Gary Mills--Unix Support--U of M Academic Computing and Networking- Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: The annoyance of repeating Makefiles
On Tue, Jan 23, 2007 at 09:50:32AM -0500, Ken Murchison wrote:
Gary Mills wrote:
This behavior is annoying because I build the Cyrus software on a
development server but then install it on other servers where there
is no compiler and the source tree is mounted read-only. This breaks
`make install', which should only install things, not recompile them.
Can this be fixed, or am I condemned to hack Makefiles myself?
A 'make install' shouldn't compile anything if all of the generated
files already exist. I would consider this a bug and would gladly
accept a patch which fixes this behavior.
I've attached a patch, against cyrus-imapd-2.3.8, that does just that.
--
-Gary Mills--Unix Support--U of M Academic Computing and Networking-
--- imap/Makefile.Oin Tue Jan 9 11:41:35 2007
+++ imap/Makefile.inSat Feb 10 09:50:27 2007
@@ -123,7 +123,7 @@
lmtpstats.c lmtpstats.h xversion.h mupdate_err.c mupdate_err.h \
nntp_err.c nntp_err.h
-all: $(BUILTSOURCES) $(PROGS) $(SUIDPROGS)
+all: xversion $(BUILTSOURCES) $(PROGS) $(SUIDPROGS)
pure: imapd.pure lmtpd.pure mupdate.pure
@@ -151,11 +151,10 @@
### Built Source Files
-xversion:
- rm -f version.o
+xversion xversion.h:
AWK=$(AWK) $(srcdir)/xversion.sh
-xversion.h: xversion
+version.o: xversion.h
pushstats.c: pushstats.snmp $(srcdir)/../snmp/snmpgen
$(srcdir)/../snmp/snmpgen $(srcdir)/pushstats.snmp
@@ -199,7 +198,7 @@
$(SERVICE) lmtpd.o proxy.o $(LMTPOBJS) $(SIEVE_OBJS) \
mutex_fake.o libimap.a $(SIEVE_LIBS) $(DEPLIBS) $(LIBS) $(LIB_WRAP)
-imapd: xversion $(IMAPDOBJS) mutex_fake.o libimap.a $(DEPLIBS) $(SERVICE)
+imapd: $(IMAPDOBJS) mutex_fake.o libimap.a $(DEPLIBS) $(SERVICE)
$(CC) $(LDFLAGS) -o imapd \
$(SERVICE) $(IMAPDOBJS) mutex_fake.o \
libimap.a $(DEPLIBS) $(LIBS) $(LIB_WRAP)
--- imap/xversion.Osh Wed Oct 22 13:03:00 2003
+++ imap/xversion.shSat Feb 10 08:04:31 2007
@@ -9,12 +9,13 @@
AWK=awk
fi
+TMPF=/tmp/xversion.$$
DATEPAT=[1-2][0-9][0-9][0-9]/[0-1][0-9]/[0-3][0-9]
TIMEPAT=[0-2][0-9]:[0-5][0-9]:[0-5][0-9]
-printf /* Generated automatically by xversion.sh */\n\n xversion.h
+printf /* Generated automatically by xversion.sh */\n\n $TMPF
-printf #define CYRUS_CVSDATE xversion.h
+printf #define CYRUS_CVSDATE $TMPF
find .. -name '*.[chly]' -print | \
xargs egrep '\$Id: ' | \
@@ -22,4 +23,12 @@
match ($0, pattern) {
printf \%s\\n, substr($0, RSTART, RLENGTH)
}' pattern=$DATEPAT $TIMEPAT | \
- sort | tail -1 xversion.h
+ sort | tail -1 $TMPF
+
+if [ -f xversion.h ] cmp -s $TMPF xversion.h
+then
+rm $TMPF
+else
+mv $TMPF xversion.h
+fi
+
--- perl/Makefile.Oin Wed Oct 22 13:50:17 2003
+++ perl/Makefile.inFri Feb 9 17:54:37 2007
@@ -77,7 +77,7 @@
@for d in $(SUBDIRS); \
do \
(cd $$d; echo ### Making all in `pwd`; \
- if [ -f Makefile.PL ]; then \
+ if [ -f Makefile.PL -a ! -f Makefile ]; then \
LIB_RT=$(LIB_RT) \
BDB_LIB=$(BDB_LIB) BDB_INC=$(BDB_INC) \
OPENSSL_LIB=$(OPENSSL_LIB)
OPENSSL_INC=$(OPENSSL_INC) \
--- perl/sieve/Makefile.Oin Wed Oct 22 13:50:24 2003
+++ perl/sieve/Makefile.in Fri Feb 9 18:02:16 2007
@@ -79,7 +79,7 @@
@for d in $(SUBDIRS); \
do \
(cd $$d; echo ### Making all in `pwd`; \
- if [ -f Makefile.PL ]; then \
+ if [ -f Makefile.PL -a ! -f Makefile ]; then \
LIB_RT=$(LIB_RT) \
BDB_LIB=$(BDB_LIB) BDB_INC=$(BDB_INC) \
OPENSSL_LIB=$(OPENSSL_LIB)
OPENSSL_INC=$(OPENSSL_INC) \
Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: The annoyance of repeating Makefiles
On Tue, Jan 23, 2007 at 09:50:32AM -0500, Ken Murchison wrote: Gary Mills wrote: I've noticed for some time that whenever I type `make' in the Cyrus source tree, it always recompiles something. A repeated make should evenually stop compiling but this one never does. The worst offenders are the various perl Makefiles which rebuild Makefile from Makefile.PL every time, instead of only doing it when it's out of date. This I'm not a Perl guy, so I don't know if this is necessary or not. The problem is that Makefile.PL is not really a Makefile. It's run by perl to create the Makefile. I suppose that that should be done by the configure step, not by the compile step. causes the perl modules to be recompiled. As well, imap/Makefile recreates xversion.h each time, resulting in more recompiles. We recreate xversion.h (and imapd as a result) so we get an accurate CVS timestamp in imapd for version reporting. I'm not a CVS guy, but there must be a better way to accomplish this without causing a recompile. This behavior is annoying because I build the Cyrus software on a development server but then install it on other servers where there is no compiler and the source tree is mounted read-only. This breaks `make install', which should only install things, not recompile them. Can this be fixed, or am I condemned to hack Makefiles myself? A 'make install' shouldn't compile anything if all of the generated files already exist. I would consider this a bug and would gladly accept a patch which fixes this behavior. Is there a release coming soon, perhaps to fix the core dump that we heard about? If so, I'll use that distribution to work on a patch. -- -Gary Mills--Unix Support--U of M Academic Computing and Networking- Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Can murder be used for IMAP server migration?
On Thu, Jan 04, 2007 at 07:47:54AM -0500, Ken Murchison wrote: Gary Mills wrote: On Wed, Jan 03, 2007 at 08:18:15AM -0500, Ken Murchison wrote: Gary Mills wrote: Can I use the old server as both a front end and one of the back ends for a murder configuration, with the new server as the second back end? Will that allow me to migrate mailboxes at my convenience? How do I prevent a port conflict between the IMAP server and the proxy on the old server? You'll have to run the frontend + mupdate master on a separate machine. so the frontend has to run there. Of course, the old IMAP server has to run there too. Is there not a way to have the old IMAP server listen on different ports, so that only the frontend connects to it? If not, could it listen on a secondary IP address only? It *might* make sense to upgrade the existing server to 2.3.x, since imapd can both proxy and serve local mailboxes (a unified Murder). Is there any documentation on how to configure and manage a unified Murder? I couldn't find anything in the 2.3.7 source that described it. In fact, the examples in the documentation still show the proxy daemons, which no longer exist. -- -Gary Mills--Unix Support--U of M Academic Computing and Networking- Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
The annoyance of repeating Makefiles
I've noticed for some time that whenever I type `make' in the Cyrus source tree, it always recompiles something. A repeated make should evenually stop compiling but this one never does. The worst offenders are the various perl Makefiles which rebuild Makefile from Makefile.PL every time, instead of only doing it when it's out of date. This causes the perl modules to be recompiled. As well, imap/Makefile recreates xversion.h each time, resulting in more recompiles. This behavior is annoying because I build the Cyrus software on a development server but then install it on other servers where there is no compiler and the source tree is mounted read-only. This breaks `make install', which should only install things, not recompile them. Can this be fixed, or am I condemned to hack Makefiles myself? -- -Gary Mills--Unix Support--U of M Academic Computing and Networking- Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Can murder be used for IMAP server migration?
On Thu, Jan 04, 2007 at 09:22:24AM +0100, Miros?aw Jaworski wrote: On Wed, 2007-01-03 at 20:08 -0600, Gary Mills wrote: On Wed, Jan 03, 2007 at 08:18:15AM -0500, Ken Murchison wrote: Gary Mills wrote: Can I use the old server as both a front end and one of the back ends for a murder configuration, with the new server as the second back end? Will that allow me to migrate mailboxes at my convenience? How do I prevent a port conflict between the IMAP server and the proxy on the old server? You'll have to run the frontend + mupdate master on a separate machine. Unfortunately, all the clients know the IP address of the old server, so the frontend has to run there. Yes. But not the way you think - assign old server ip to the new machine with frontend. That won't work either. There are many different services linked to that IP address. I don't want to move all of them. Of course, the old IMAP server has to run there too. Wrong. Assign new ip to the old server. Frankly there are simplier ways to do the migration than playing with frontend and mupdate - perdition. You will use same scheme though - assign the ip users are used to use to the perdition and give new ip to the old server. I've investigated perdition, but I don't think it supports all of the SASL mechanisms that our clients use. Some use NTLM, for example. So many things work nicely with Cyrus that I'd like to stick with it. -- -Gary Mills--Unix Support--U of M Academic Computing and Networking- Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Can murder be used for IMAP server migration?
On Thu, Jan 04, 2007 at 07:47:54AM -0500, Ken Murchison wrote: Gary Mills wrote: On Wed, Jan 03, 2007 at 08:18:15AM -0500, Ken Murchison wrote: Gary Mills wrote: Can I use the old server as both a front end and one of the back ends for a murder configuration, with the new server as the second back end? Will that allow me to migrate mailboxes at my convenience? How do I prevent a port conflict between the IMAP server and the proxy on the old server? You'll have to run the frontend + mupdate master on a separate machine. Unfortunately, all the clients know the IP address of the old server, They actually use the IP address instead of a DNS name? They actually use a CNAME, but the same one for SMTP and IMAP, with the same SSL certificate for both. It would be painful to split them up now. so the frontend has to run there. Of course, the old IMAP server has to run there too. Is there not a way to have the old IMAP server listen on different ports, so that only the frontend connects to it? If not, could it listen on a secondary IP address only? AFAIK, the proxy is hardcoded to talk to the backend(s) on port 143. Perhaps you could force imapd to only listen on localhost, but then you have to find a way to get the mailboxes.db used by the proxy to use localhost as the servername for the existing mailboxes. It *might* make sense to upgrade the existing server to 2.3.x, since imapd can both proxy and serve local mailboxes (a unified Murder). That sounds like the way to go. I'll investigate and try it on my test servers. Thanks for the suggestion. -- -Gary Mills--Unix Support--U of M Academic Computing and Networking- Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Can murder be used for IMAP server migration?
On Wed, Jan 03, 2007 at 08:18:15AM -0500, Ken Murchison wrote: Gary Mills wrote: Can I use the old server as both a front end and one of the back ends for a murder configuration, with the new server as the second back end? Will that allow me to migrate mailboxes at my convenience? How do I prevent a port conflict between the IMAP server and the proxy on the old server? You'll have to run the frontend + mupdate master on a separate machine. Unfortunately, all the clients know the IP address of the old server, so the frontend has to run there. Of course, the old IMAP server has to run there too. Is there not a way to have the old IMAP server listen on different ports, so that only the frontend connects to it? If not, could it listen on a secondary IP address only? -- -Gary Mills--Unix Support--U of M Academic Computing and Networking- Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Can murder be used for IMAP server migration?
My current e-mail server has both SMTP and IMAP running on the same machine. I'm building a new IMAP server on a different machine that will ultimately have ten times the capacity. I'm looking for a way to migrate mailboxes from the old server to the new one without shutting down the whole service for an extended period of time. Can I use the old server as both a front end and one of the back ends for a murder configuration, with the new server as the second back end? Will that allow me to migrate mailboxes at my convenience? How do I prevent a port conflict between the IMAP server and the proxy on the old server? Do I have to run the same Cyrus version on both machines? Currently, the old server runs cyrus-imapd-2.1.14 and the new one runs cyrus-imapd-2.3.6. Is there anything that won't work in this configuration? -- -Gary Mills--Unix Support--U of M Academic Computing and Networking- Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Beyond rtcyrus2 (sendmail integration)
On Mon, Dec 04, 2006 at 12:35:23PM -0800, Jo Rhett wrote: On Dec 4, 2006, at 6:09 AM, Gary Mills wrote: In our case there's no export involved. All of the information comes from the same SQL database. Somebody reinventing this scheme today would probably use LDAP instead. Really? Sendmail queries the SQL for each delivery recipient? Potentially, but nscd caches the results. Sendmail also does authenticated SMTP via SASL, just like Cyrus IMAP does it. They are nicely integrated. Clearly, there is an export mechanism. Nothing periodic, if that's what you mean. There's a replication feed from the SQL master to an instance on the e-mail server. -- -Gary Mills--Unix Support--U of M Academic Computing and Networking- Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Beyond rtcyrus2 (sendmail integration)
On Sun, Dec 03, 2006 at 11:29:08PM +0100, Andrzej Adam Filip wrote: Andrzej Adam Filip [EMAIL PROTECTED] writes: I would like to know who is interested in helping to develop next versions of advanced Sendmail and Cyrus IMAP integration methods. I would like to further improve methods available at links below: http://anfi.homeunix.net/sendmail/rtcyrus2.html http://anfi.homeunix.net/sendmail/#cyrus We've had excellent sendmail/cyrus integration for years, with 35,000 users. It's done by having all users in the NIS map on the mail server. No modification to sendmail is necessary because getpwnam() returns the passwd entry for the user. Users can't log in to the mail server, of course, because PAM rules prevent that. The same thing could be done with other user databases, such as LDAP. Why would you ever need a different form of integration? I have uploaded alpha one files. -- -Gary Mills--Unix Support--U of M Academic Computing and Networking- Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
