Re: Miserable performance of cyrus-imapd 2.3.9 -- seems to be locking issues

2008-03-04 Thread Ian G Batten
On 28 Feb 08, at 2256, Kenneth Marshall wrote: It may be that the software RAID 5 is your problem. Without the use of NVRAM for a cache, all of the writes need all 3 disks. That will cause quite a bottle-neck. In general, RAID5 writes require two reads and two writes, independent of the

Re: Miserable performance of cyrus-imapd 2.3.9 -- seems to be locking issues

2008-02-29 Thread Nik Conwell
On Feb 28, 2008, at 4:38 PM, Jeff Fookson wrote: is about 200GB. There are typically about 200 'imapd' processes at a given time and a hugely varying number of 'lmtpds' (from about 6 to many hundreds during times of greatest pathology). System load is correspondingly in the 2-15

Re: Miserable performance of cyrus-imapd 2.3.9 -- seems to be locking issues

2008-02-29 Thread Kenneth Marshall
On Fri, Feb 29, 2008 at 07:16:24AM +0100, Pascal Gienger wrote: Jeff Fookson [EMAIL PROTECTED] wrote: Databases are all skiplist. As a rule of thumb, do not use skiplist for the duplicate delivery suppression database (deliver.db). Even if everybody hates it, use BerkeleyDB, Version

Re: Miserable performance of cyrus-imapd 2.3.9 -- seems to be locking issues

2008-02-29 Thread Gabor Gombas
On Thu, Feb 28, 2008 at 04:56:18PM -0600, Kenneth Marshall wrote: It may be that the software RAID 5 is your problem. Without the use of NVRAM for a cache, all of the writes need all 3 disks. That will cause quite a bottle-neck. It's much worse than that. Since metadata updates are almost

Re: Miserable performance of cyrus-imapd 2.3.9 -- seems to be locking issues

2008-02-29 Thread Allen Chen
I just got out of this kind of situation. If your OS is Linux, can you post /etc/syslog.conf? Allen Jeff Fookson wrote: Folks- I am hoping to get some help and guidance as to why our installation of cyrus-imapd 2.3.9 is unusably slow. Here are the specifics: The software is running on a

Re: Miserable performance of cyrus-imapd 2.3.9 -- seems to be locking issues

2008-02-29 Thread Simon Matter
Can you put a - just before /var/log/messages and /var/log/cyrus/imapd.log in your /etc/syslog.conf? (just like -/var/log/maillog) and restart syslog: service syslog restart. Another culprit can be name resolution. At least localhost and the servers own hostnames should be listed in the hosts

Re: Miserable performance of cyrus-imapd 2.3.9 -- seems to be locking issues

2008-02-29 Thread Kenneth Marshall
Delivery through the lmtpd process should not take long enough to cause this type of backlog unless there is a performance bottle-neck, such as the delivery DB format that has been suggested previously, particularly in such a small system. Cheers, Ken On Thu, Feb 28, 2008 at 04:09:58PM -0600,

Re: Miserable performance of cyrus-imapd 2.3.9 -- seems to be locking issues

2008-02-29 Thread Henrique de Moraes Holschuh
On Thu, 28 Feb 2008, Michael Bacon wrote: I've never seen drbd used for Cyrus, but it looks like other folks have done it. The combination of drbd+lvm2+ext3 might put you somewhere unpleasant, but I'll have to let the Linux-heads jump in on that one. Don't try it with 4k stacks, IMO. It

Re: Miserable performance of cyrus-imapd 2.3.9 -- seems to be locking issues

2008-02-29 Thread Simon Matter
Michael Bacon wrote: What database format are you using for the mailboxes database? What kind of storage is the metapartition (usually /var/imap) on? What kind of storage are your mail partitions on? Databases are all skiplist. Our mail partition and the metapartition are skiplist is

Miserable performance of cyrus-imapd 2.3.9 -- seems to be locking issues

2008-02-28 Thread Jeff Fookson
Folks- I am hoping to get some help and guidance as to why our installation of cyrus-imapd 2.3.9 is unusably slow. Here are the specifics: The software is running on a 1.6GHz Opteron with 2Gb memory supporting a user base of about 400 users. The average rate of arriving mail is on the order of

Re: Miserable performance of cyrus-imapd 2.3.9 -- seems to be locking issues

2008-02-28 Thread Michael Bacon
What database format are you using for the mailboxes database? What kind of storage is the metapartition (usually /var/imap) on? What kind of storage are your mail partitions on? --On Thursday, February 28, 2008 2:38 PM -0700 Jeff Fookson [EMAIL PROTECTED] wrote: Folks- I am hoping to

Re: Miserable performance of cyrus-imapd 2.3.9 -- seems to be locking issues

2008-02-28 Thread Vincent Fox
Jeff Fookson wrote: is unusably slow. Here are the specifics: You are mighty short on the SPECIFICS of your setup. Expect a slew of questions to elicit this information. Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List

Re: Miserable performance of cyrus-imapd 2.3.9 -- seems to be locking issues

2008-02-28 Thread Paul M Fleming
Limit the number of lmtpd daemons to around 10 -- that solved the issue for me.. We let sendmail handle the queuing. It is more than likely a locking issue.. Michael Bacon wrote: What database format are you using for the mailboxes database? What kind of storage is the metapartition

Re: Miserable performance of cyrus-imapd 2.3.9 -- seems to be locking issues

2008-02-28 Thread Kenneth Marshall
Jeff, Delivery database format can cause this type of problem, among other databases. For any DB that is updated with contention, use either BerkeleyDB or Skiplist format. We also had a similar issue when we did not have the expunge process running and pruning the delivery database and its size

Re: Miserable performance of cyrus-imapd 2.3.9 -- seems to be locking issues

2008-02-28 Thread Jeff Fookson
Michael Bacon wrote: What database format are you using for the mailboxes database? What kind of storage is the metapartition (usually /var/imap) on? What kind of storage are your mail partitions on? Databases are all skiplist. Our mail partition and the metapartition are both on the

Re: Miserable performance of cyrus-imapd 2.3.9 -- seems to be locking issues

2008-02-28 Thread Michael Bacon
Jeff, Just as a rule of thumb, if you've got problems with Cyrus (or any mail system), 90% of the time they're related to I/O performance. I've never seen drbd used for Cyrus, but it looks like other folks have done it. The combination of drbd+lvm2+ext3 might put you somewhere unpleasant,

Re: Miserable performance of cyrus-imapd 2.3.9 -- seems to be locking issues

2008-02-28 Thread Kenneth Marshall
It may be that the software RAID 5 is your problem. Without the use of NVRAM for a cache, all of the writes need all 3 disks. That will cause quite a bottle-neck. Ken On Thu, Feb 28, 2008 at 03:36:43PM -0700, Jeff Fookson wrote: Michael Bacon wrote: What database format are you using for

Re: Miserable performance of cyrus-imapd 2.3.9 -- seems to be locking issues

2008-02-28 Thread Scott Likens
Okay, I read over this and I felt worth commenting... There's mention of using MD, DRBD, LVM2, etc... it sounds extremely conviluted and way to complex for what you are needing. When you are doing a read or a write, each thing takes it's time before it gets commited to disk. If you are

Re: Miserable performance of cyrus-imapd 2.3.9 -- seems to be locking issues

2008-02-28 Thread Zachariah Mully
On Thu, 2008-02-28 at 16:56 -0600, Kenneth Marshall wrote: It may be that the software RAID 5 is your problem. Without the use of NVRAM for a cache, all of the writes need all 3 disks. That will cause quite a bottle-neck. Ken And if you can, try to get the mailstore over onto a RAID1. RAID5

Re: Miserable performance of cyrus-imapd 2.3.9 -- seems to be locking issues

2008-02-28 Thread Zachariah Mully
On Thu, 2008-02-28 at 18:36 -0500, Zachariah Mully wrote: On Thu, 2008-02-28 at 16:56 -0600, Kenneth Marshall wrote: It may be that the software RAID 5 is your problem. Without the use of NVRAM for a cache, all of the writes need all 3 disks. That will cause quite a bottle-neck. Ken

Re: Miserable performance of cyrus-imapd 2.3.9 -- seems to be locking issues

2008-02-28 Thread Vincent Fox
Gah my first thought was, a 3-disk RAID5? Is this 1998 or 2008? Disk is cheap. RAID-1 or RAID-10. Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Re: Miserable performance of cyrus-imapd 2.3.9 -- seems to be locking issues

2008-02-28 Thread Pascal Gienger
Jeff Fookson [EMAIL PROTECTED] wrote: Databases are all skiplist. As a rule of thumb, do not use skiplist for the duplicate delivery suppression database (deliver.db). Even if everybody hates it, use BerkeleyDB, Version 4.4.52 or higher. Give it a quite fair amount of shared memory. And run