Re: cpu and cyrus
John Madden wrote: It's well worth your time to maintain your own compiles and even packages of Cyrus because the package maintainers can't keep up. Stating as if it were fact packagers are not able to keep up is somewhat of a faux pas, and seriously frowned upon by yours sincerely. Kind regards, Jeroen van Meeuwen -- Senior Engineer, Kolab Systems AG e: vanmeeuwen at kolabsys.com t: +44 144 340 9500 m: +44 74 2516 3817 w: http://www.kolabsys.com pgp: 9342 BF08 Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: cpu and cyrus
On 09/06/2011 02:55 PM, Jeroen van Meeuwen (Kolab Systems) wrote: John Madden wrote: It's well worth your time to maintain your own compiles and even packages of Cyrus because the package maintainers can't keep up. Stating as if it were fact packagers are not able to keep up is somewhat of a faux pas, and seriously frowned upon by yours sincerely. These ideas don't grow in a vacuum - they grow partially because traditionally, packages HAVE been a long way behind with Cyrus - and even now, there's no centrally agreed and published resource with here is where you obtain up-to-date packages for your distribution/operating system. Besides, the world isn't Redhat and Debian! Bron. Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: cpu and cyrus
Stating as if it were fact packagers are not able to keep up is somewhat of a faux pas, and seriously frowned upon by yours sincerely. It's not that you can't keep up, it's that you don't keep up. The reasons behind the lag are usually quite understandable, but regardless, those who need the most recent versions of these packages often have to look outside the distribution's packages. Faux pas or not, it's the truth. :) John -- John Madden / Sr UNIX Systems Engineer Office of Technology / Ivy Tech Community College of Indiana Free Software is a matter of liberty, not price. To understand the concept, you should think of Free as in 'free speech,' not as in 'free beer.' -- Richard Stallman Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Cyrus IMAP Packaging (was: Re: cpu and cyrus)
John Madden wrote: Stating as if it were fact packagers are not able to keep up is somewhat of a faux pas, and seriously frowned upon by yours sincerely. It's not that you can't keep up, it's that you don't keep up. The reasons behind the lag are usually quite understandable, but regardless, those who need the most recent versions of these packages often have to look outside the distribution's packages. Faux pas or not, it's the truth. :) I'm afraid you're confusing the package that is available in the 'stable' repository or repositories for a 'stable' distribution version or release with the latest version released upstream not being packaged. Distributor's policies often do not permit the type of change(s) the latest and greatest may bring along with it, to occur in their 'stable' releases. Note that I'm using the term 'stable' very lightly here, because Fedora is my turf. This would be a distributor problem more so then a packager problem. With Debian for example, in my experience, the problem has always been (different people on this mailing list can correct me if I'm wrong here) that packagers have wanted the update/upgrade/dist-upgrade database conversion and deb-conf scripting to be what could be defined as providing a seemless upgrade path. Meanwhile it seemed everyone was OK with the status quo, such resulting in little contributions to develop/test solutions to said (perceived) problems. Overall, though, I think you'll find this particular area of packaging Cyrus IMAP for OS distributions has vastly improved over the past year or so. To say packagers can't I do consider a faux pas. To say packagers don't or won't may perhaps give you reason to start doing what you need to get done in the canonical location for such efforts, from where everybody else consumes the fruits of your labour; upstream. Kind regards, Jeroen van Meeuwen -- Senior Engineer, Kolab Systems AG e: vanmeeuwen at kolabsys.com t: +44 144 340 9500 m: +44 74 2516 3817 w: http://www.kolabsys.com pgp: 9342 BF08 Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: cpu and cyrus
On Tue, 2011-09-06 at 15:16 +0200, Bron Gondwana wrote: On 09/06/2011 02:55 PM, Jeroen van Meeuwen (Kolab Systems) wrote: John Madden wrote: It's well worth your time to maintain your own compiles and even packages of Cyrus because the package maintainers can't keep up. Stating as if it were fact packagers are not able to keep up is somewhat of a faux pas, and seriously frowned upon by yours sincerely. These ideas don't grow in a vacuum - they grow partially because traditionally, packages HAVE been a long way behind with Cyrus - and even now, there's no centrally agreed and published resource with here is where you obtain up-to-date packages for your distribution/operating system. Besides, the world isn't Redhat and Debian! Debian/Ubuntu still cyrus-imapd 2.2.13 - just sayin... Craig -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean. Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: cpu and cyrus
On 9/1/11 9:54 PM, Maria McKinley wrote: On 9/1/11 5:28 PM, Dan White wrote: On 01/09/11 15:25 -0700, Maria McKinley wrote: On 9/1/11 11:49 AM, Dan White wrote: Do you use any group ACLs? It looks like your imapd process may be waiting for a group list enumeration to complete, via an nss ldap plugin. If so, and you are using the default 'auth_mech: unix' group authorization config, this is not a recommended configuration per: http://www.cyrusimap.org/docs/cyrus-imapd/2.4.10/overview.php#aclauth Doing so can cause performance issues. I don't think I am using group ACLs, as I don't remember setting it up, but haven't been able to figure this out definitively. How can I tell if I am using any group ACLs? Am I trying to index on groups? I remember setting up to index on different things, but can't find where this is set up, so maybe I am making this up. Feeling way over my head, sorry, and thanks so much for your help. IMAP group ACLs are prepended with 'group:', and would look like this in cyradm: localhost listacl user.test test lrswipkxtecdan group:wheel lrswipkxtecda Thanks. Turns out that is what I was looking at, I just don't have any groups set, so didn't see anything. I guess that isn't the problem, then. I rebuilt a few mailboxes that were showing up in the runaway processes. So far I haven't had any processes go nuts, but still too early to know for sure. ~maria Didn't work, crazy processes came back. It does seem that one particular user shows up in all of the processes that go nuts. I tried rebuilding his mailbox, but this obviously didn't have any affect. Any ideas about what else I can do? I am willing to try updating cyrus if this may help, but I was hoping to avoid this for another month or so. ~maria Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: cpu and cyrus
On Wed, 2011-08-31 at 16:34 -0400, John Madden wrote: So annoying that the stable release of debian isn't supported anymore. It seems like if you wait so long to release the stable version that it isn't supported anymore, it sort of defeats the purpose. Debian have been staying at 2.2 rather than moving up to 2.3 through two stable releases now. There is a limit to how long you can hold on to the past. Cyrus 2.3.0 was released in December 2005. There's a lesson here that I learned from the OpenLDAP folks that is re-enforced with Cyrus (and many other packages): You can't rely on your OS distributor beyond a certain scale. RHEL's OpenLDAP packages are fine a thousand objects, no good at all for a million. Some of the problems are too complex: imagine Debian delivering a functioning Murder out of the box or Red Hat combining the right version of BerkeleyDB with the right version of OpenLDAP. No thanks. It's well worth your time to maintain your own compiles and even packages of Cyrus because the package maintainers can't keep up. indeed though interestingly enough, OpenLDAP on Ubuntu is current. Once I saw that ubuntu was still featuring cyrus 2.2.13, I didn't even bother. For Ubuntu LTS, grab source for cyrus and compile - worked great. Only had to get a reasonably suitable sysv script to start/stop. Those using CentOS/Scientific Linux/RHEL should probably just rebuild Simon's excellent SRC rpms for cyrus though... patched for 'autocreate' and simple. Craig -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean. Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: cpu and cyrus
Hi, Craig White schrieb am 01.09.2011 15:44 Uhr: On Wed, 2011-08-31 at 16:34 -0400, John Madden wrote: So annoying that the stable release of debian isn't supported anymore. It seems like if you wait so long to release the stable version that it isn't supported anymore, it sort of defeats the purpose. Debian have been staying at 2.2 rather than moving up to 2.3 through two stable releases now. There is a limit to how long you can hold on to the past. Cyrus 2.3.0 was released in December 2005. There's a lesson here that I learned from the OpenLDAP folks that is re-enforced with Cyrus (and many other packages): You can't rely on your OS distributor beyond a certain scale. RHEL's OpenLDAP packages are fine a thousand objects, no good at all for a million. Some of the problems are too complex: imagine Debian delivering a functioning Murder out of the box or Red Hat combining the right version of BerkeleyDB with the right version of OpenLDAP. No thanks. It's well worth your time to maintain your own compiles and even packages of Cyrus because the package maintainers can't keep up. indeed though interestingly enough, OpenLDAP on Ubuntu is current. Once I saw that ubuntu was still featuring cyrus 2.2.13, I didn't even bother. For Ubuntu LTS, grab source for cyrus and compile - worked great. Only had to get a reasonably suitable sysv script to start/stop. Wouldn't it be great, if someone could offer a PPA for Cyrus IMAPd for Ubuntu and or Debian? I think it would really help. (I have never built a deb package myself, sorry.) Oh, at least here is a PPA with 2.4.8: https://launchpad.net/~cz.nic-labs/+archive/cyrus-imapd Marc Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: cpu and cyrus
On 8/31/11 1:09 PM, Bron Gondwana wrote: Ok - next time it goes crazy can you pick one up with gdb and get a backtrace? gdb $cyrusbinpath/imapd $pid and then when you get the prompt: bt and maybe also: p imapd_in which will give us the command that's running. There are various bugs back in the 2.2 series which could cause an infinite loop - and the most interesting bit is working out which corrupt file is causing the problem. Bron. Why does it not see the directory /usr/lib/cyrus/bin? that seems weird. Am I doing something wrong here? Tried it with a couple of different cyrus processes. ~maria ella:~# gdb /usr/lib/cyrus/bin 21626 GNU gdb (GDB) 7.0.1-debian Copyright (C) 2009 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type show copying and show warranty for details. This GDB was configured as x86_64-linux-gnu. For bug reporting instructions, please see: http://www.gnu.org/software/gdb/bugs/... /usr/lib/cyrus/bin: No such file or directory. Attaching to process 21626 Reading symbols from /usr/lib/cyrus/bin/imapd...(no debugging symbols found)...done. Reading symbols from /usr/lib/libsasl2.so.2...Reading symbols from /usr/lib/debug/usr/lib/libsasl2.so.2.0.23...done. (no debugging symbols found)...done. Loaded symbols for /usr/lib/libsasl2.so.2 Reading symbols from /usr/lib/libgssapi.so.2...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libgssapi.so.2 Reading symbols from /usr/lib/libkrb5.so.26...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libkrb5.so.26 Reading symbols from /usr/lib/libasn1.so.8...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libasn1.so.8 Reading symbols from /usr/lib/libroken.so.18...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libroken.so.18 Reading symbols from /lib/libcrypt.so.1...(no debugging symbols found)...done. Loaded symbols for /lib/libcrypt.so.1 Reading symbols from /lib/libcom_err.so.2...(no debugging symbols found)...done. Loaded symbols for /lib/libcom_err.so.2 Reading symbols from /lib/libresolv.so.2...(no debugging symbols found)...done. Loaded symbols for /lib/libresolv.so.2 Reading symbols from /usr/lib/libdb-4.7.so...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libdb-4.7.so Reading symbols from /usr/lib/libssl.so.0.9.8...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libssl.so.0.9.8 Reading symbols from /usr/lib/libcrypto.so.0.9.8...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libcrypto.so.0.9.8 Reading symbols from /lib/libwrap.so.0...(no debugging symbols found)...done. Loaded symbols for /lib/libwrap.so.0 Reading symbols from /lib/libnsl.so.1...(no debugging symbols found)...done. Loaded symbols for /lib/libnsl.so.1 Reading symbols from /lib/libc.so.6...(no debugging symbols found)...done. Loaded symbols for /lib/libc.so.6 Reading symbols from /lib/libdl.so.2...(no debugging symbols found)...done. Loaded symbols for /lib/libdl.so.2 Reading symbols from /usr/lib/libheimntlm.so.0...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libheimntlm.so.0 Reading symbols from /lib/libpthread.so.0...(no debugging symbols found)...done. [Thread debugging using libthread_db enabled] Loaded symbols for /lib/libpthread.so.0 Reading symbols from /usr/lib/libhx509.so.5...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libhx509.so.5 Reading symbols from /usr/lib/libsqlite3.so.0...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libsqlite3.so.0 Reading symbols from /usr/lib/libwind.so.0...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libwind.so.0 Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/ld-linux-x86-64.so.2 Reading symbols from /usr/lib/libz.so.1...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libz.so.1 Reading symbols from /usr/lib/sasl2/libgssapiv2.so.2...Reading symbols from /usr/lib/debug/usr/lib/sasl2/libgssapiv2.so.2.0.23...done. (no debugging symbols found)...done. Loaded symbols for /usr/lib/sasl2/libgssapiv2.so.2 Reading symbols from /usr/lib/libgssapi_krb5.so.2...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libgssapi_krb5.so.2 Reading symbols from /usr/lib/libkrb5.so.3...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libkrb5.so.3 Reading symbols from /usr/lib/libk5crypto.so.3...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libk5crypto.so.3 Reading symbols from /usr/lib/libkrb5support.so.0...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libkrb5support.so.0 Reading symbols from /lib/libkeyutils.so.1...(no debugging symbols found)...done. Loaded symbols for /lib/libkeyutils.so.1
Re: cpu and cyrus
On 9/1/11 9:38 AM, Maria McKinley wrote: On 8/31/11 1:09 PM, Bron Gondwana wrote: Ok - next time it goes crazy can you pick one up with gdb and get a backtrace? gdb $cyrusbinpath/imapd $pid and then when you get the prompt: bt and maybe also: p imapd_in which will give us the command that's running. There are various bugs back in the 2.2 series which could cause an infinite loop - and the most interesting bit is working out which corrupt file is causing the problem. Bron. Why does it not see the directory /usr/lib/cyrus/bin? that seems weird. Am I doing something wrong here? Tried it with a couple of different cyrus processes. ~maria Oops, forgot the imapd at the end of the path. Still didn't help me figure out where things are getting stuck, though. :-( ella:~# gdb /usr/lib/cyrus/bin/imapd 4416 GNU gdb (GDB) 7.0.1-debian Copyright (C) 2009 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type show copying and show warranty for details. This GDB was configured as x86_64-linux-gnu. For bug reporting instructions, please see: http://www.gnu.org/software/gdb/bugs/... Reading symbols from /usr/lib/cyrus/bin/imapd...(no debugging symbols found)...done. Attaching to program: /usr/lib/cyrus/bin/imapd, process 4416 Reading symbols from /usr/lib/libsasl2.so.2...Reading symbols from /usr/lib/debug/usr/lib/libsasl2.so.2.0.23...done. (no debugging symbols found)...done. Loaded symbols for /usr/lib/libsasl2.so.2 Reading symbols from /usr/lib/libgssapi.so.2...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libgssapi.so.2 Reading symbols from /usr/lib/libkrb5.so.26...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libkrb5.so.26 Reading symbols from /usr/lib/libasn1.so.8...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libasn1.so.8 Reading symbols from /usr/lib/libroken.so.18...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libroken.so.18 Reading symbols from /lib/libcrypt.so.1...(no debugging symbols found)...done. Loaded symbols for /lib/libcrypt.so.1 Reading symbols from /lib/libcom_err.so.2...(no debugging symbols found)...done. Loaded symbols for /lib/libcom_err.so.2 Reading symbols from /lib/libresolv.so.2...(no debugging symbols found)...done. Loaded symbols for /lib/libresolv.so.2 Reading symbols from /usr/lib/libdb-4.7.so...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libdb-4.7.so Reading symbols from /usr/lib/libssl.so.0.9.8...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libssl.so.0.9.8 Reading symbols from /usr/lib/libcrypto.so.0.9.8...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libcrypto.so.0.9.8 Reading symbols from /lib/libwrap.so.0...(no debugging symbols found)...done. Loaded symbols for /lib/libwrap.so.0 Reading symbols from /lib/libnsl.so.1...(no debugging symbols found)...done. Loaded symbols for /lib/libnsl.so.1 Reading symbols from /lib/libc.so.6...(no debugging symbols found)...done. Loaded symbols for /lib/libc.so.6 Reading symbols from /lib/libdl.so.2...(no debugging symbols found)...done. Loaded symbols for /lib/libdl.so.2 Reading symbols from /usr/lib/libheimntlm.so.0...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libheimntlm.so.0 Reading symbols from /lib/libpthread.so.0...(no debugging symbols found)...done. [Thread debugging using libthread_db enabled] Loaded symbols for /lib/libpthread.so.0 Reading symbols from /usr/lib/libhx509.so.5...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libhx509.so.5 Reading symbols from /usr/lib/libsqlite3.so.0...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libsqlite3.so.0 Reading symbols from /usr/lib/libwind.so.0...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libwind.so.0 Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/ld-linux-x86-64.so.2 Reading symbols from /usr/lib/libz.so.1...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libz.so.1 Reading symbols from /usr/lib/sasl2/libgssapiv2.so.2...Reading symbols from /usr/lib/debug/usr/lib/sasl2/libgssapiv2.so.2.0.23...done. (no debugging symbols found)...done. Loaded symbols for /usr/lib/sasl2/libgssapiv2.so.2 Reading symbols from /usr/lib/libgssapi_krb5.so.2...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libgssapi_krb5.so.2 Reading symbols from /usr/lib/libkrb5.so.3...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libkrb5.so.3 Reading symbols from /usr/lib/libk5crypto.so.3...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libk5crypto.so.3 Reading symbols from /usr/lib/libkrb5support.so.0...(no debugging symbols found)...done. Loaded
Re: cpu and cyrus
On 8/31/11 1:40 PM, Bron Gondwana wrote: On Wed, Aug 31, 2011 at 11:51:03AM -0700, Maria McKinley wrote: On 8/31/11 11:44 AM, Wesley Craig wrote: On 31 Aug 2011, at 14:36, Maria McKinley wrote: Anyway, here is an example of some processes that are getting big: PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND 24328 cyrus 20 0 147m 6236 5004 R 27.4 0.2 22:07.21 imapd 27549 cyrus 20 0 147m 6512 5116 R 27.4 0.2 155:41.26 imapd 30097 cyrus 20 0 147m 6280 5052 R 27.1 0.2 93:44.08 imapd Unfortunately I can't tell you anymore about these processes, since they are just cut and pasted from a terminal where I checked top before restarting cyrus, and cyrus has not regrown yet. It would be handy if you'd gather some metrics from a loaded but not pathological machine, and then additional metrics once it was in a bad state. Also, getting strace output from a run-away (if it is run-away) process would be handy. :wes Not sure how helpful this is, but you can see where the that I have restarted cyrus twice in the last 24 hours, and three times in the last week, and how this has affected the load, etc. of this machine. http://www.shadlen.org/munin/Servers/ella.shadlen.org/index.html I can't believe you capture the uptime graph ;) Seriously though - it's an infinite loop. There are a few different places it could be, and my money is on a bogus .seen file. There were definitely some infinite loops in there, and the bugs in skiplist db locking in 2.2 mean you could have any old rubbish show up over time. So I'm guessing it's a particular folder access that triggers the runaway process each time. Bron. I looked for the processes that were going crazy in the log files, and it seems like the problem may be a couple of .seen files and/or index files. I have some messages about SQUAT failing to open index file. What can be done to fix .seen files and index files? thanks, maria Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: cpu and cyrus
On 01/09/11 10:14 -0700, Maria McKinley wrote: On 8/31/11 1:09 PM, Bron Gondwana wrote: Ok - next time it goes crazy can you pick one up with gdb and get a backtrace? gdb $cyrusbinpath/imapd $pid and then when you get the prompt: bt and maybe also: p imapd_in which will give us the command that's running. ella:~# gdb /usr/lib/cyrus/bin/imapd 4416 (gdb) bt #0 0x7f2c049e90d8 in poll () from /lib/libc.so.6 #1 0x7f2bfe6fd31d in ?? () from /usr/lib/libldap_r-2.4.so.2 #2 0x7f2bfe6fe4fd in ldap_result () from /usr/lib/libldap_r-2.4.so.2 #3 0x7f2bfe93a505 in ?? () from /lib/libnss_ldap.so.2 #4 0x7f2bfe93a7a7 in ?? () from /lib/libnss_ldap.so.2 #5 0x7f2bfe93cff5 in _nss_ldap_endgrent () from /lib/libnss_ldap.so.2 #6 0x7f2c04a04d89 in ?? () from /lib/libc.so.6 #7 0x7f2c049c315a in endgrent () from /lib/libc.so.6 #8 0x00451d1d in ?? () #9 0x0043c008 in ?? () #10 0x00414de1 in ?? () #11 0x7f2c069cb682 in do_authorization (s_conn=0x26852f0) at server.c:1185 #12 0x7f2c069cc22d in sasl_checkpass (conn=0x26852f0, user=value optimized out, userlen=4, pass=0x2686fe0 Biko';l, passlen=7) at server.c:1746 #13 0x00414783 in ?? () #14 0x0041892a in ?? () #15 0x0041a3e4 in ?? () #16 0x00407b07 in ?? () #17 0x7f2c04943c4d in __libc_start_main () from /lib/libc.so.6 #18 0x00407229 in ?? () #19 0x7fffd439b798 in ?? () #20 0x001c in ?? () #21 0x0004 in ?? () #22 0x7fffd439be5f in ?? () #23 0x7fffd439be65 in ?? () #24 0x7fffd439be68 in ?? () #25 0x7fffd439be6b in ?? () #26 0x in ?? () (gdb) p imapd_in $1 = 40607904 (gdb) Do you use any group ACLs? It looks like your imapd process may be waiting for a group list enumeration to complete, via an nss ldap plugin. If so, and you are using the default 'auth_mech: unix' group authorization config, this is not a recommended configuration per: http://www.cyrusimap.org/docs/cyrus-imapd/2.4.10/overview.php#aclauth Doing so can cause performance issues. -- Dan White Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: cpu and cyrus
On 9/1/11 11:49 AM, Dan White wrote: On 01/09/11 10:14 -0700, Maria McKinley wrote: On 8/31/11 1:09 PM, Bron Gondwana wrote: Ok - next time it goes crazy can you pick one up with gdb and get a backtrace? gdb $cyrusbinpath/imapd $pid and then when you get the prompt: bt and maybe also: p imapd_in which will give us the command that's running. ella:~# gdb /usr/lib/cyrus/bin/imapd 4416 (gdb) bt #0 0x7f2c049e90d8 in poll () from /lib/libc.so.6 #1 0x7f2bfe6fd31d in ?? () from /usr/lib/libldap_r-2.4.so.2 #2 0x7f2bfe6fe4fd in ldap_result () from /usr/lib/libldap_r-2.4.so.2 #3 0x7f2bfe93a505 in ?? () from /lib/libnss_ldap.so.2 #4 0x7f2bfe93a7a7 in ?? () from /lib/libnss_ldap.so.2 #5 0x7f2bfe93cff5 in _nss_ldap_endgrent () from /lib/libnss_ldap.so.2 #6 0x7f2c04a04d89 in ?? () from /lib/libc.so.6 #7 0x7f2c049c315a in endgrent () from /lib/libc.so.6 #8 0x00451d1d in ?? () #9 0x0043c008 in ?? () #10 0x00414de1 in ?? () #11 0x7f2c069cb682 in do_authorization (s_conn=0x26852f0) at server.c:1185 #12 0x7f2c069cc22d in sasl_checkpass (conn=0x26852f0, user=value optimized out, userlen=4, pass=0x2686fe0 Biko';l, passlen=7) at server.c:1746 #13 0x00414783 in ?? () #14 0x0041892a in ?? () #15 0x0041a3e4 in ?? () #16 0x00407b07 in ?? () #17 0x7f2c04943c4d in __libc_start_main () from /lib/libc.so.6 #18 0x00407229 in ?? () #19 0x7fffd439b798 in ?? () #20 0x001c in ?? () #21 0x0004 in ?? () #22 0x7fffd439be5f in ?? () #23 0x7fffd439be65 in ?? () #24 0x7fffd439be68 in ?? () #25 0x7fffd439be6b in ?? () #26 0x in ?? () (gdb) p imapd_in $1 = 40607904 (gdb) Do you use any group ACLs? It looks like your imapd process may be waiting for a group list enumeration to complete, via an nss ldap plugin. If so, and you are using the default 'auth_mech: unix' group authorization config, this is not a recommended configuration per: http://www.cyrusimap.org/docs/cyrus-imapd/2.4.10/overview.php#aclauth Doing so can cause performance issues. I don't think I am using group ACLs, as I don't remember setting it up, but haven't been able to figure this out definitively. How can I tell if I am using any group ACLs? Am I trying to index on groups? I remember setting up to index on different things, but can't find where this is set up, so maybe I am making this up. Feeling way over my head, sorry, and thanks so much for your help. ~maria Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: cpu and cyrus
On 01/09/11 15:25 -0700, Maria McKinley wrote: On 9/1/11 11:49 AM, Dan White wrote: Do you use any group ACLs? It looks like your imapd process may be waiting for a group list enumeration to complete, via an nss ldap plugin. If so, and you are using the default 'auth_mech: unix' group authorization config, this is not a recommended configuration per: http://www.cyrusimap.org/docs/cyrus-imapd/2.4.10/overview.php#aclauth Doing so can cause performance issues. I don't think I am using group ACLs, as I don't remember setting it up, but haven't been able to figure this out definitively. How can I tell if I am using any group ACLs? Am I trying to index on groups? I remember setting up to index on different things, but can't find where this is set up, so maybe I am making this up. Feeling way over my head, sorry, and thanks so much for your help. IMAP group ACLs are prepended with 'group:', and would look like this in cyradm: localhost listacl user.test test lrswipkxtecdan group:wheel lrswipkxtecda -- Dan White Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: cpu and cyrus
On 9/1/11 5:28 PM, Dan White wrote: On 01/09/11 15:25 -0700, Maria McKinley wrote: On 9/1/11 11:49 AM, Dan White wrote: Do you use any group ACLs? It looks like your imapd process may be waiting for a group list enumeration to complete, via an nss ldap plugin. If so, and you are using the default 'auth_mech: unix' group authorization config, this is not a recommended configuration per: http://www.cyrusimap.org/docs/cyrus-imapd/2.4.10/overview.php#aclauth Doing so can cause performance issues. I don't think I am using group ACLs, as I don't remember setting it up, but haven't been able to figure this out definitively. How can I tell if I am using any group ACLs? Am I trying to index on groups? I remember setting up to index on different things, but can't find where this is set up, so maybe I am making this up. Feeling way over my head, sorry, and thanks so much for your help. IMAP group ACLs are prepended with 'group:', and would look like this in cyradm: localhost listacl user.test test lrswipkxtecdan group:wheel lrswipkxtecda Thanks. Turns out that is what I was looking at, I just don't have any groups set, so didn't see anything. I guess that isn't the problem, then. I rebuilt a few mailboxes that were showing up in the runaway processes. So far I haven't had any processes go nuts, but still too early to know for sure. ~maria Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
cpu and cyrus
Hi there,
I am having an issue with cyrus processes growing to use all of my cpus,
at which point I have to restart cyrus, because all other services on my
mail server grind to a halt. Below is my config file. Any ideas what may
be causing this? I noticed this behavior after the last cyrus update. I
am currently using cyrus 2.2.13-19+squeeze1, debian. I think my
processor should be plenty powerful for the mail load I handle.
thanks,
maria
# Debian defaults for Cyrus IMAP server/cluster implementation
# see cyrus.conf(5) for more information
#
# All the tcp services are tcpd-wrapped. see hosts_access(5)
# $Id: cyrus.conf 567 2006-08-14 18:19:32Z sven $
START {
# do not delete this entry!
recover cmd=/usr/sbin/ctl_cyrusdb -r
# this is only necessary if idlemethod is set to idled in
imapd.conf
#idled cmd=idled
# this is useful on backend nodes of a Murder cluster
# it causes the backend to syncronize its mailbox list with
# the mupdate master upon startup
#mupdatepush cmd=/usr/sbin/ctl_mboxlist -m
# this is recommended if using duplicate delivery suppression
delprunecmd=/usr/sbin/cyr_expire -E 3
# this is recommended if caching TLS sessions
tlsprunecmd=/usr/sbin/tls_prune
}
# UNIX sockets start with a slash and are absolute paths
# you can use a maxchild=# to limit the maximum number of forks of a service
# you can use babysit=true and maxforkrate=# to keep tight tabs on the
service
# most services also accept -U (limit number of reuses) and -T (timeout)
SERVICES {
# --- Normal cyrus spool, or Murder backends ---
# add or remove based on preferences
imapcmd=imapd -U 30 listen=imap prefork=0
maxchild=200
imaps cmd=imapd -s -U 30 listen=imaps prefork=0
maxchild=200
#pop3 cmd=pop3d -U 30 listen=pop3 prefork=0
maxchild=50
#pop3s cmd=pop3d -s -U 30 listen=pop3s prefork=0
maxchild=50
nntpcmd=nntpd -U 30 listen=nntp prefork=0
maxchild=100
#nntps cmd=nntpd -s -U 30 listen=nntps prefork=0
maxchild=100
# At least one form of LMTP is required for delivery
# (you must keep the Unix socket name in sync with imap.conf)
#lmtp cmd=lmtpd listen=localhost:lmtp prefork=0
maxchild=20
lmtpunixcmd=lmtpd listen=/var/run/cyrus/socket/lmtp
prefork=0 maxchild=20
# --
# useful if you need to give users remote access to sieve
# by default, we limit this to localhost in Debian
sieve cmd=timsieved listen=sieve prefork=0
maxchild=100
# this one is needed for the notification services
notify cmd=notifyd
listen=/var/run/cyrus/socket/notify proto=udp prefork=1
# --- Murder frontends -
# enable these and disable the matching services above,
# except for sieve (which deals automatically with Murder)
# mupdate database service - must prefork at least 1
# (mupdate slaves)
#mupdate cmd=mupdate listen=3905 prefork=1
# (mupdate master, only one in the entire cluster)
#mupdate cmd=mupdate -m listen=3905 prefork=1
# proxies that will connect to the backends
#imap cmd=proxyd listen=imap prefork=0 maxchild=100
#imaps cmd=proxyd -s listen=imaps prefork=0
maxchild=100
#pop3 cmd=pop3proxyd listen=pop3 prefork=0
maxchild=50
#pop3s cmd=pop3proxyd -s listen=pop3s prefork=0
maxchild=50
#lmtp cmd=lmtpproxyd listen=lmtp prefork=1
maxchild=20
# --
}
EVENTS {
# this is required
checkpoint cmd=/usr/sbin/ctl_cyrusdb -c period=30
# this is only necessary if using duplicate delivery suppression
delprunecmd=/usr/sbin/cyr_expire -E 3 at=0401
# this is only necessary if caching TLS sessions
tlsprunecmd=/usr/sbin/tls_prune at=0401
# prune trash
purgetrash cmd=/usr/sbin/ipurge -f -d 14 */Trash at=0301
# indexing of mailboxs for server side fulltext searches
# reindex changed mailboxes (fulltext) approximately every
other hour
#squatter_1 cmd=/usr/bin/nice -n 19 /usr/sbin/squatter -s
period=120
# reindex all mailboxes (fulltext) daily
#squatter_a cmd=/usr/sbin/squatter at=0517
}
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: cpu and cyrus
On Wed, Aug 31, 2011 at 10:33:14AM -0700, Maria McKinley wrote: Hi there, I am having an issue with cyrus processes growing to use all of my cpus, at which point I have to restart cyrus, because all other services on my mail server grind to a halt. Below is my config file. Any ideas what may be causing this? I noticed this behavior after the last cyrus update. I am currently using cyrus 2.2.13-19+squeeze1, debian. I think my processor should be plenty powerful for the mail load I handle. Which processes in particular? Cyrus 2.2 isn't supported any more, so we're unlikely to be providing any fixes if you've hit a bug that doesn't exist in more recent versions. Regards, Bron. Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: cpu and cyrus
On 8/31/11 10:56 AM, Bron Gondwana wrote: On Wed, Aug 31, 2011 at 10:33:14AM -0700, Maria McKinley wrote: Hi there, I am having an issue with cyrus processes growing to use all of my cpus, at which point I have to restart cyrus, because all other services on my mail server grind to a halt. Below is my config file. Any ideas what may be causing this? I noticed this behavior after the last cyrus update. I am currently using cyrus 2.2.13-19+squeeze1, debian. I think my processor should be plenty powerful for the mail load I handle. Which processes in particular? Cyrus 2.2 isn't supported any more, so we're unlikely to be providing any fixes if you've hit a bug that doesn't exist in more recent versions. Regards, Bron. So annoying that the stable release of debian isn't supported anymore. It seems like if you wait so long to release the stable version that it isn't supported anymore, it sort of defeats the purpose. Anyway, here is an example of some processes that are getting big: PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND 24328 cyrus 20 0 147m 6236 5004 R 27.4 0.2 22:07.21 imapd 27549 cyrus 20 0 147m 6512 5116 R 27.4 0.2 155:41.26 imapd 30097 cyrus 20 0 147m 6280 5052 R 27.1 0.2 93:44.08 imapd Unfortunately I can't tell you anymore about these processes, since they are just cut and pasted from a terminal where I checked top before restarting cyrus, and cyrus has not regrown yet. thanks, maria Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: cpu and cyrus
On 31 Aug 2011, at 14:36, Maria McKinley wrote: Anyway, here is an example of some processes that are getting big: PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND 24328 cyrus 20 0 147m 6236 5004 R 27.4 0.2 22:07.21 imapd 27549 cyrus 20 0 147m 6512 5116 R 27.4 0.2 155:41.26 imapd 30097 cyrus 20 0 147m 6280 5052 R 27.1 0.2 93:44.08 imapd Unfortunately I can't tell you anymore about these processes, since they are just cut and pasted from a terminal where I checked top before restarting cyrus, and cyrus has not regrown yet. It would be handy if you'd gather some metrics from a loaded but not pathological machine, and then additional metrics once it was in a bad state. Also, getting strace output from a run-away (if it is run-away) process would be handy. :wes Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: cpu and cyrus
On 8/31/11 11:44 AM, Wesley Craig wrote: On 31 Aug 2011, at 14:36, Maria McKinley wrote: Anyway, here is an example of some processes that are getting big: PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND 24328 cyrus 20 0 147m 6236 5004 R 27.4 0.2 22:07.21 imapd 27549 cyrus 20 0 147m 6512 5116 R 27.4 0.2 155:41.26 imapd 30097 cyrus 20 0 147m 6280 5052 R 27.1 0.2 93:44.08 imapd Unfortunately I can't tell you anymore about these processes, since they are just cut and pasted from a terminal where I checked top before restarting cyrus, and cyrus has not regrown yet. It would be handy if you'd gather some metrics from a loaded but not pathological machine, and then additional metrics once it was in a bad state. Also, getting strace output from a run-away (if it is run-away) process would be handy. :wes Not sure how helpful this is, but you can see where the that I have restarted cyrus twice in the last 24 hours, and three times in the last week, and how this has affected the load, etc. of this machine. http://www.shadlen.org/munin/Servers/ella.shadlen.org/index.html ~m Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: cpu and cyrus
On Wed, Aug 31, 2011 at 11:36:44AM -0700, Maria McKinley wrote: On 8/31/11 10:56 AM, Bron Gondwana wrote: On Wed, Aug 31, 2011 at 10:33:14AM -0700, Maria McKinley wrote: Hi there, I am having an issue with cyrus processes growing to use all of my cpus, at which point I have to restart cyrus, because all other services on my mail server grind to a halt. Below is my config file. Any ideas what may be causing this? I noticed this behavior after the last cyrus update. I am currently using cyrus 2.2.13-19+squeeze1, debian. I think my processor should be plenty powerful for the mail load I handle. Which processes in particular? Cyrus 2.2 isn't supported any more, so we're unlikely to be providing any fixes if you've hit a bug that doesn't exist in more recent versions. Regards, Bron. So annoying that the stable release of debian isn't supported anymore. It seems like if you wait so long to release the stable version that it isn't supported anymore, it sort of defeats the purpose. Debian have been staying at 2.2 rather than moving up to 2.3 through two stable releases now. There is a limit to how long you can hold on to the past. Cyrus 2.3.0 was released in December 2005. Anyway, here is an example of some processes that are getting big: PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND 24328 cyrus 20 0 147m 6236 5004 R 27.4 0.2 22:07.21 imapd 27549 cyrus 20 0 147m 6512 5116 R 27.4 0.2 155:41.26 imapd 30097 cyrus 20 0 147m 6280 5052 R 27.1 0.2 93:44.08 imapd Unfortunately I can't tell you anymore about these processes, since they are just cut and pasted from a terminal where I checked top before restarting cyrus, and cyrus has not regrown yet. Ok - next time it goes crazy can you pick one up with gdb and get a backtrace? gdb $cyrusbinpath/imapd $pid and then when you get the prompt: bt and maybe also: p imapd_in which will give us the command that's running. There are various bugs back in the 2.2 series which could cause an infinite loop - and the most interesting bit is working out which corrupt file is causing the problem. Bron. Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: cpu and cyrus
So annoying that the stable release of debian isn't supported anymore. It seems like if you wait so long to release the stable version that it isn't supported anymore, it sort of defeats the purpose. Debian have been staying at 2.2 rather than moving up to 2.3 through two stable releases now. There is a limit to how long you can hold on to the past. Cyrus 2.3.0 was released in December 2005. There's a lesson here that I learned from the OpenLDAP folks that is re-enforced with Cyrus (and many other packages): You can't rely on your OS distributor beyond a certain scale. RHEL's OpenLDAP packages are fine a thousand objects, no good at all for a million. Some of the problems are too complex: imagine Debian delivering a functioning Murder out of the box or Red Hat combining the right version of BerkeleyDB with the right version of OpenLDAP. No thanks. It's well worth your time to maintain your own compiles and even packages of Cyrus because the package maintainers can't keep up. John -- John Madden Sr UNIX Systems Engineer / Office of Technology Ivy Tech Community College of Indiana Free Software is a matter of liberty, not price. To understand the concept, you should think of Free as in 'free speech,' not as in 'free beer.' -- Richard Stallman Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: cpu and cyrus
On Wed, Aug 31, 2011 at 11:51:03AM -0700, Maria McKinley wrote: On 8/31/11 11:44 AM, Wesley Craig wrote: On 31 Aug 2011, at 14:36, Maria McKinley wrote: Anyway, here is an example of some processes that are getting big: PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND 24328 cyrus 20 0 147m 6236 5004 R 27.4 0.2 22:07.21 imapd 27549 cyrus 20 0 147m 6512 5116 R 27.4 0.2 155:41.26 imapd 30097 cyrus 20 0 147m 6280 5052 R 27.1 0.2 93:44.08 imapd Unfortunately I can't tell you anymore about these processes, since they are just cut and pasted from a terminal where I checked top before restarting cyrus, and cyrus has not regrown yet. It would be handy if you'd gather some metrics from a loaded but not pathological machine, and then additional metrics once it was in a bad state. Also, getting strace output from a run-away (if it is run-away) process would be handy. :wes Not sure how helpful this is, but you can see where the that I have restarted cyrus twice in the last 24 hours, and three times in the last week, and how this has affected the load, etc. of this machine. http://www.shadlen.org/munin/Servers/ella.shadlen.org/index.html I can't believe you capture the uptime graph ;) Seriously though - it's an infinite loop. There are a few different places it could be, and my money is on a bogus .seen file. There were definitely some infinite loops in there, and the bugs in skiplist db locking in 2.2 mean you could have any old rubbish show up over time. So I'm guessing it's a particular folder access that triggers the runaway process each time. Bron. Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: cpu and cyrus
On 31/08/11 22:40 +0200, Bron Gondwana wrote: On Wed, Aug 31, 2011 at 11:51:03AM -0700, Maria McKinley wrote: On 8/31/11 11:44 AM, Wesley Craig wrote: On 31 Aug 2011, at 14:36, Maria McKinley wrote: Anyway, here is an example of some processes that are getting big: PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND 24328 cyrus 20 0 147m 6236 5004 R 27.4 0.2 22:07.21 imapd 27549 cyrus 20 0 147m 6512 5116 R 27.4 0.2 155:41.26 imapd 30097 cyrus 20 0 147m 6280 5052 R 27.1 0.2 93:44.08 imapd Seriously though - it's an infinite loop. There are a few different places it could be, and my money is on a bogus .seen file. There were definitely some infinite loops in there, and the bugs in skiplist db locking in 2.2 mean you could have any old rubbish show up over time. So I'm guessing it's a particular folder access that triggers the runaway process each time. Maria, If the problem is caused by one or a few particular mailboxes, check the contents of configdirectory/proc/pid for each of those processes to track them down. -- Dan White Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
