Re: Load spikes when new email arrives
On 13-01-23 11:16 AM, francis picabia wrote: Here are more stats. Do these look average for performance? It is difficult to understand why the system was working with few load spikes before. A mailman mailing list sends 10kbyte message to 4000 users having accounts on this cyrus system. If I grep Delivered in the maillog by the minute I can see how fast the messages are stored. What backend are you using for the duplicate_db (deliver.db)? I've seen much better delivery performance using berkeley-nosync instead of the default (skiplist) for this one. e.g.: # grep Delivered /var/log/maillog | grep 'Jan 23 10:37' | wc -l 696 That is the best. This peak event pushed the load to 14 for 12 minutes, where it averages 604 messages delivered to cyrus mailboxes per minute. Is that reasonable for maximum delivery rate? I've also backed out the change (yesterday) to /sys/block/sda/queue/nr_requests I think it was pushing the load higher and there is no advantage in my hardware (SAS with Perc 5/i Raid 5 over 4 disk) to run with a low value for nr_requests. Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus -- Jean Raby jr...@inverse.ca :: +1.514.447.4918 (x120) :: www.inverse.ca Inverse inc. :: Leaders behind SOGo (www.sogo.nu) and PacketFence (www.packetfence.org) Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: Load spikes when new email arrives
On Thu, Jan 24, 2013 at 12:22 PM, Blake Hudson bl...@ispn.net wrote: There are a couple suggestions I'd like to put forth. First, improper partition alignment is generally masked by the controller cache. I strongly encourage you to check that your RAID array is making use of this cache by enabling the WriteBack caching option on this array, especially if your PERC card has a BBU (I think this was optional on perc 5). You can install the MegaCLI tool from LSI to verify this (can also be checked from OpenManage or reboot into the controller BIOS). Thanks for this tip. It put me on to what is wrong. Jan 18 07:25:39 myserv Server Administrator: Storage Service EventID: 2335 Controller event log: BBU disabled; changing WB virtual disks to WT: Controller 0 (PERC 5/i Integrated) Bingo! We had write back all along, and the performance tanked when it fell back to write through. I was wondering why my policy change attempts were flipping back when I tried testing WB this morning! This explains everything we've been seeing. Wow. Gotta call Dell. Thanks everyone for the assistance. I didn't think a battery which shows OK status in omreport could wound us! Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: Load spikes when new email arrives
francis picabia wrote the following on 1/25/2013 7:55 AM: On Thu, Jan 24, 2013 at 12:22 PM, Blake Hudson bl...@ispn.net mailto:bl...@ispn.net wrote: There are a couple suggestions I'd like to put forth. First, improper partition alignment is generally masked by the controller cache. I strongly encourage you to check that your RAID array is making use of this cache by enabling the WriteBack caching option on this array, especially if your PERC card has a BBU (I think this was optional on perc 5). You can install the MegaCLI tool from LSI to verify this (can also be checked from OpenManage or reboot into the controller BIOS). Thanks for this tip. It put me on to what is wrong. Jan 18 07:25:39 myserv Server Administrator: Storage Service EventID: 2335 Controller event log: BBU disabled; changing WB virtual disks to WT: Controller 0 (PERC 5/i Integrated) Bingo! We had write back all along, and the performance tanked when it fell back to write through. I was wondering why my policy change attempts were flipping back when I tried testing WB this morning! This explains everything we've been seeing. Wow. Gotta call Dell. Thanks everyone for the assistance. I didn't think a battery which shows OK status in omreport could wound us! The PERC cards will disable write-back caching while the BBU is charging/exercising. However, within a few hours the BBU should return to normal status. In rare instances, people on the Dell mailing list have reported that their caching status never returns to write-back - even after attempting to force write-back caching on the array. Attempts and power cycling or firmware flashing are tried, but seem to be futile in most cases. Often, replacement of the card is necessary. I'm unsure if it's the battery, the card, or some software setting, but I would definitely follow up with Dell. On the next server (or array) you configure, I would attempt to align your partitions as you've investigated. Sector 2048 seems to be a good starting position for most RAID levels. I have no conclusive evidence that a different file system or alignment improves my performance, because I've never done a fair side by side test with controlled inputs. However, we use ext4 and do align our partitions using RAID10 on 15k SAS drives for all our Cyrus installs. I have found some issues with the newer systems that I attribute to the move from ext3 to ext4 which can result in MySQL replication problems on power loss/freeze, but these issues are vary rare and usually easy to recover from in our environment. I also notice that new systems always perform better than the old systems, even with identical hardware - I've often attributed this to fragmentation. --Blake Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: Load spikes when new email arrives
On Fri, Jan 25, 2013 at 10:26 AM, Blake Hudson bl...@ispn.net wrote: francis picabia wrote the following on 1/25/2013 7:55 AM: On Thu, Jan 24, 2013 at 12:22 PM, Blake Hudson bl...@ispn.net wrote: There are a couple suggestions I'd like to put forth. First, improper partition alignment is generally masked by the controller cache. I strongly encourage you to check that your RAID array is making use of this cache by enabling the WriteBack caching option on this array, especially if your PERC card has a BBU (I think this was optional on perc 5). You can install the MegaCLI tool from LSI to verify this (can also be checked from OpenManage or reboot into the controller BIOS). Thanks for this tip. It put me on to what is wrong. Jan 18 07:25:39 myserv Server Administrator: Storage Service EventID: 2335 Controller event log: BBU disabled; changing WB virtual disks to WT: Controller 0 (PERC 5/i Integrated) Bingo! We had write back all along, and the performance tanked when it fell back to write through. I was wondering why my policy change attempts were flipping back when I tried testing WB this morning! This explains everything we've been seeing. Wow. Gotta call Dell. Thanks everyone for the assistance. I didn't think a battery which shows OK status in omreport could wound us! The PERC cards will disable write-back caching while the BBU is charging/exercising. However, within a few hours the BBU should return to normal status. In rare instances, people on the Dell mailing list have reported that their caching status never returns to write-back - even after attempting to force write-back caching on the array. Attempts and power cycling or firmware flashing are tried, but seem to be futile in most cases. Often, replacement of the card is necessary. I'm unsure if it's the battery, the card, or some software setting, but I would definitely follow up with Dell. I found the problem as I was attempting to set a policy of write back (wb) and it said success but status showed write through. Then I saw entries in the logs predating my use of the omconfig command with writepolicy, and that was the clue. This sample command is a way to dump the controller's log to disk: /opt/dell/srvadmin/sbin/omconfig storage controller controller=0 action=exportlog Then look for Absolute (at least true for Perc 5/i): grep Absolute /var/log/lsi_0125.log T27: Absolute State of Charge : 33 % T27: Absolute State of Charge : 29 % T27: Absolute State of Charge : 33 % T27: Absolute State of Charge : 29 % T27: Absolute State of Charge : 33 % T27: Absolute State of Charge : 29 % I believe at below 30% it is the threshold where write back is disabled. I'm glad we caught this as there are a number of Dell 2950 systems in a similar state or about to be. On the next server (or array) you configure, I would attempt to align your partitions as you've investigated. Sector 2048 seems to be a good starting position for most RAID levels. I have no conclusive evidence that a different file system or alignment improves my performance, because I've never done a fair side by side test with controlled inputs. However, we use ext4 and do align our partitions using RAID10 on 15k SAS drives for all our Cyrus installs. I have found some issues with the newer systems that I attribute to the move from ext3 to ext4 which can result in MySQL replication problems on power loss/freeze, but these issues are vary rare and usually easy to recover from in our environment. I also notice that new systems always perform better than the old systems, even with identical hardware - I've often attributed this to fragmentation. This exercise has been useful to compile a set of things we want to ensure on future system setups for best performance: 1. RAID 10 2. ext4 (fsync problem gone) 3. align partitions to MB, not cylinders 4. use write-back writepolicy and automate check of the BBU status (don't ever want to reach a BBU disabled state) According to Redhat's feature request bugzilla report, converting to ext4 from ext3 is unsupported mash up. They recommend only new ext4 creation. Thanks to all for their recommendations. Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: Load spikes when new email arrives
On Wed, Jan 23, 2013 at 5:25 PM, Andrew Morgan mor...@orst.edu wrote: On Wed, 23 Jan 2013, francis picabia wrote: Thanks for the response. I have been checking my iostat whenever there is a number of messages in the active queue. Here is a sample snapshot from a script I run (ignoring the first iostat output of averages): Active in queue: 193 12:47:01 up 5 days, 5:23, 6 users, load average: 14.11, 9.22, 4.67 Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util sda5 3.25 281.00 19.75 129.50 654.00 3384.0027.06 5.53 36.24 6.69 99.80 svctm is about the same as when not under load and it went above 7 only once. Then there is this comment about the validity of tracking svctm: http://www.xaprb.com/blog/**2010/09/06/beware-of-svctm-in-** linuxs-iostat/http://www.xaprb.com/blog/2010/09/06/beware-of-svctm-in-linuxs-iostat/ %util is often reaching close to %100 when there is a queue to process. sda5 is where the cyrus mail/imap lives. Our account names all begin with numbers, so almost all mail accounts are under the q folder. Okay, I didn't realize svctm could be suspect, although I guess that makes sense in a RAID array. What about your await times? Does await increase during peak loads? It seems pretty clear from iostat that you are IO bound on writes during mail delivery. As Vincent said in his reply, RAID5 performs poorly during writes. Each write actually consumes 4 disk operations (read old data, read old parity, write new data, write new parity). If you can live with the slight additional risk, turn on write caching on the Perc 5/i if you haven't already. I think they call it write-back versus write-through. If you can handle it, you would probably be a lot happier converting that RAID5 set to RAID10. You'll lose a disk worth of capacity, but get double the write performance. However, what is your real goal? Do you want to deliver mail more quickly, or do you want to reduce your load average? You can probably reduce your load average and perhaps gain a bit of speed by tweaking the lmtp maxchild limit. If you really need to deliver mail more quickly, then you need to throw more IOPS at it. Let's keep this discussion going! There are lots of ways to tune for performance. I've probably missed some. :) In another email discussion on the Redhat mailing list, I've confirmed we have an issue with partition alignment. This is getting to be quite the mess out there. I saw one posting where it is speculated there are thousands of poorly set up disk partitions for their RAID stripe size. fdisk and OS installers were late getting updated for the new TB disks and SSD disks as well. Partition alignment might account for 5 to 30% of a performance hit. I've checked and my cyrus lmtpd process count never exceeds 11 under work load. await jumps up to 150-195 at worst. If I'm already at IO saturation, I can't see how a higher lmtpd limit would help. My goal is to keep the system load reasonable so it is responsive for mailbox access by the end users. Right now we get nagios alerts about 6 times a day for excessive load. If I can move the mail queue workload into a hill instead of a sharp peak on the cacti load graph, it would be good. There are minutes around the peaks where the queue is emptied and we have only 5 messages inbound per minute. In hind sight, I agree RAID 10 should have been implemented. At the time, four years ago, getting lots of space was the priority as space needs always grow. We've never seen load issues until this month, and it seems to coincide with a general increase of all email volume and traffic. Our primary MX is also getting hit more than normal. Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: Load spikes when new email arrives
In another email discussion on the Redhat mailing list, I've confirmed we have an issue with partition alignment. This is getting to be quite the mess out there. I saw one posting where it is speculated there are thousands of poorly set up disk partitions for their RAID stripe size. fdisk and OS installers were late getting updated for the new TB disks and SSD disks as well. Partition alignment might account for 5 to 30% of a performance hit. I've checked and my cyrus lmtpd process count never exceeds 11 under work load. await jumps up to 150-195 at worst. If I'm already at IO saturation, I can't see how a higher lmtpd limit would help. My goal is to keep the system load reasonable so it is responsive for mailbox access by the end users. Right now we get nagios alerts about 6 times a day for excessive load. If I can move the mail queue workload into a hill instead of a sharp peak on the cacti load graph, it would be good. There are minutes around the peaks where the queue is emptied and we have only 5 messages inbound per minute. In hind sight, I agree RAID 10 should have been implemented. At the time, four years ago, getting lots of space was the priority as space needs always grow. We've never seen load issues until this month, and it seems to coincide with a general increase of all email volume and traffic. Our primary MX is also getting hit more than normal. There are a couple suggestions I'd like to put forth. First, improper partition alignment is generally masked by the controller cache. I strongly encourage you to check that your RAID array is making use of this cache by enabling the WriteBack caching option on this array, especially if your PERC card has a BBU (I think this was optional on perc 5). You can install the MegaCLI tool from LSI to verify this (can also be checked from OpenManage or reboot into the controller BIOS). MegaCLI Link: http://www-947.ibm.com/support/entry/portal/docdisplay?lndocid=migr-5082327 The relevant commands are as follows: MegaCli -AdpBbuCmd -aALL MegaCli -LDInfo -Lall -aALL Second, the PERC card does support RAID level migration, so if you want to add a spindle or even change RAID levels, you can. This can be done via either OpenManage (hit or miss) or the MegaCLI tool (daunting, but there are cheat sheets). You could also add a separate array to act as a dedicated mail spool. You can also replace the existing disks with faster (and/or larger) disks for additional performance without ever touching the software. To directly answer your question of If I can move the mail queue workload into a hill instead of a sharp peak on the cacti load graph, it would be good. , then lowering the LMTP limit in cyrus (or the upstream MX server) to turn the mail flow into a trickle, rather than a flood, would do this. You can adjust the concurrency rate of LMTP deliveries in postfix using lmtp_destination_concurrency_limit (default 20). The cyrus method has already been mentioned. You may also look at other ways to reduce IO wait, such as disk defragmentation or utilizing hard links in cyrus (singleinstancestore: 1). --Blake Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: Load spikes when new email arrives
In another email discussion on the Redhat mailing list, I've confirmed we have an issue with partition alignment. This is getting to be quite the mess out there. I saw one posting where it is speculated there are thousands of poorly set up disk partitions for their RAID stripe size. fdisk and OS installers were late getting updated for the new TB disks and SSD disks as well. Partition alignment might account for 5 to 30% of a performance hit. I've checked and my cyrus lmtpd process count never exceeds 11 under work load. await jumps up to 150-195 at worst. If I'm already at IO saturation, I can't see how a higher lmtpd limit would help. My goal is to keep the system load reasonable so it is responsive for mailbox access by the end users. Right now we get nagios alerts about 6 times a day for excessive load. If I can move the mail queue workload into a hill instead of a sharp peak on the cacti load graph, it would be good. There are minutes around the peaks where the queue is emptied and we have only 5 messages inbound per minute. In hind sight, I agree RAID 10 should have been implemented. At the time, four years ago, getting lots of space was the priority as space needs always grow. We've never seen load issues until this month, and it seems to coincide with a general increase of all email volume and traffic. Our primary MX is also getting hit more than normal. There are a couple suggestions I'd like to put forth. First, improper partition alignment is generally masked by the controller cache. I strongly encourage you to check that your RAID array is making use of this cache by enabling the WriteBack caching option on this array, especially if your PERC card has a BBU (I think this was optional on perc 5). You can install the MegaCLI tool from LSI to verify this (can also be checked from OpenManage or reboot into the controller BIOS). I strongly suggest to do that *ONLY* with proper BBU in place! MegaCLI Link: http://www-947.ibm.com/support/entry/portal/docdisplay?lndocid=migr-5082327 The relevant commands are as follows: MegaCli -AdpBbuCmd -aALL MegaCli -LDInfo -Lall -aALL Second, the PERC card does support RAID level migration, so if you want to add a spindle or even change RAID levels, you can. This can be done via either OpenManage (hit or miss) or the MegaCLI tool (daunting, but there are cheat sheets). You could also add a separate array to act as a dedicated mail spool. You can also replace the existing disks with faster (and/or larger) disks for additional performance without ever touching the software. To directly answer your question of If I can move the mail queue workload into a hill instead of a sharp peak on the cacti load graph, it would be good. , then lowering the LMTP limit in cyrus (or the upstream MX server) to turn the mail flow into a trickle, rather than a flood, would do this. You can adjust the concurrency rate of LMTP deliveries in postfix using lmtp_destination_concurrency_limit (default 20). The cyrus method has already been mentioned. You may also look at other ways to reduce IO wait, such as disk defragmentation or utilizing hard links in cyrus (singleinstancestore: 1). Another thing is to check partitioning here. Using separate spindles for /var/lib/imap seems a good idea, RAID1 on two small but fast disks has always worked fine for me. Simon Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: Load spikes when new email arrives
On Thu, Jan 24, 2013 at 2:06 PM, Wesley Craig wescr...@columbia.edu wrote: You might find this helpful: http://lwn.net/Articles/328363/ regarding fsync (lmtpd) latency on Linux. Lowering the lmtpd process limit ought to fix your load problem, but you should be aware how cyrus master implements that limit: your postfix processes will still make connections, but they will appear to hang when you're at the limit. This will transform your spike to a hill, but you may end up with lots of waiting postfix processes. Responding to the list where I'd like to hear more feedback on this. We are running ext3 but could easily convert it to ext4. Has anyone experienced that upgrade on their cyrus mail partitions? Does it improve latency enough to make a significant difference when IO is saturated? I have reduced the lmtpd max to 5. Now the load maxes at about 10. It does take longer to work through the queue, but it eventually settles down (e.g. 10 minutes instead of 3 or 4) Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: Load spikes when new email arrives
On Thu, 24 Jan 2013, francis picabia wrote: In another email discussion on the Redhat mailing list, I've confirmed we have an issue with partition alignment. This is getting to be quite the mess out there. I saw one posting where it is speculated there are thousands of poorly set up disk partitions for their RAID stripe size. fdisk and OS installers were late getting updated for the new TB disks and SSD disks as well. Partition alignment might account for 5 to 30% of a performance hit. Yeah, I read about partition alignment the last time I built a new Cyrus server. I don't remember how it came to my attention, but it was wrong on all of my servers too. The latest stable release of Debian Linux seems to do the right thing during installation, but previous versions did not. I followed the recommendations that I found and set the starting sector to 2048 for my partition (2048 * 512bytes = 1MB): root@cyrus-be1:~# fdisk -lu /dev/sda Disk /dev/sda: 536.9 GB, 536870912000 bytes 214 heads, 31 sectors/track, 158060 cylinders, total 1048576000 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x88aa51ee Device Boot Start End Blocks Id System /dev/sda12048 1048575999 524286976 83 Linux I don't know how much of a performance difference it would actually make, but I'm trying to squeeze all I can out of it! I've checked and my cyrus lmtpd process count never exceeds 11 under work load. await jumps up to 150-195 at worst. If I'm already at IO saturation, I can't see how a higher lmtpd limit would help. I was going to suggest setting a LOWER lmtpd limit. :) It sounds like you have already done that (reading the rest of this email thread). My goal is to keep the system load reasonable so it is responsive for mailbox access by the end users. Right now we get nagios alerts about 6 times a day for excessive load. If I can move the mail queue workload into a hill instead of a sharp peak on the cacti load graph, it would be good. There are minutes around the peaks where the queue is emptied and we have only 5 messages inbound per minute. Hmmm, what options are there that don't involve rebuilding the disk... Definitely check that you have Write-Back caching enabled on the PERC. I don't know if remounting the filesystem as ext4 would help, but that's worth a shot. Are you mounting the filesystem with the noatime option? There is no need to track atime on a Cyrus mailstore and those extra writes can add up. Here are my mount options: LABEL=be1data1 /var/spool/cyrus/mail/data1 ext4 rw,auto,data=ordered,noatime 0 2 Perhaps there are some tweaks on the Postfix side that will put less strain on Cyrus. I don't know much about Postfix though. In hind sight, I agree RAID 10 should have been implemented. At the time, four years ago, getting lots of space was the priority as space needs always grow. We've never seen load issues until this month, and it seems to coincide with a general increase of all email volume and traffic. Our primary MX is also getting hit more than normal. Well, if none of the easy stuff helps enough, then maybe you'll get to build a new Cyrus filesystem from scratch! :) Andy Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: Load spikes when new email arrives
Here are more stats. Do these look average for performance? It is difficult to understand why the system was working with few load spikes before. A mailman mailing list sends 10kbyte message to 4000 users having accounts on this cyrus system. If I grep Delivered in the maillog by the minute I can see how fast the messages are stored. e.g.: # grep Delivered /var/log/maillog | grep 'Jan 23 10:37' | wc -l 696 That is the best. This peak event pushed the load to 14 for 12 minutes, where it averages 604 messages delivered to cyrus mailboxes per minute. Is that reasonable for maximum delivery rate? I've also backed out the change (yesterday) to /sys/block/sda/queue/nr_requests I think it was pushing the load higher and there is no advantage in my hardware (SAS with Perc 5/i Raid 5 over 4 disk) to run with a low value for nr_requests. Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: Load spikes when new email arrives
On Wed, 23 Jan 2013, francis picabia wrote: Here are more stats. Do these look average for performance? It is difficult to understand why the system was working with few load spikes before. A mailman mailing list sends 10kbyte message to 4000 users having accounts on this cyrus system. If I grep Delivered in the maillog by the minute I can see how fast the messages are stored. e.g.: # grep Delivered /var/log/maillog | grep 'Jan 23 10:37' | wc -l 696 That is the best. This peak event pushed the load to 14 for 12 minutes, where it averages 604 messages delivered to cyrus mailboxes per minute. Is that reasonable for maximum delivery rate? I've also backed out the change (yesterday) to /sys/block/sda/queue/nr_requests I think it was pushing the load higher and there is no advantage in my hardware (SAS with Perc 5/i Raid 5 over 4 disk) to run with a low value for nr_requests. You can certainly achieve higher delivery rates, but that all depends on your underlying hardware and how you have partitioned your system. Why don't you start running iostat -x 5 on the system? Leave this running to give you an idea of the baseline behavior and then look at it during periods of high load. I suspect you will see that your svctm and %util will go up dramatically when a large number of messages are being delivered. But, let's not make decisions based on assumptions! :) On my Cyrus Murder frontends (3 of them), I have limited LMTP connections to 25 in cyrus.conf: lmtp cmd=/usr/local/cyrus/bin/lmtpproxyd listen=lmtp proto=tcp4 prefork=0 maxchild=25 This prevents our mail relays (Postfix) from opening too many simultaneous LMTP connections, which can cause too much I/O contention. Take a look during your periods of high load to see how many lmtpd processes are running. You may want to limit the number. Andy Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: Load spikes when new email arrives
On Wed, Jan 23, 2013 at 1:55 PM, Andrew Morgan mor...@orst.edu wrote: On Wed, 23 Jan 2013, francis picabia wrote: Here are more stats. Do these look average for performance? It is difficult to understand why the system was working with few load spikes before. A mailman mailing list sends 10kbyte message to 4000 users having accounts on this cyrus system. If I grep Delivered in the maillog by the minute I can see how fast the messages are stored. e.g.: # grep Delivered /var/log/maillog | grep 'Jan 23 10:37' | wc -l 696 That is the best. This peak event pushed the load to 14 for 12 minutes, where it averages 604 messages delivered to cyrus mailboxes per minute. Is that reasonable for maximum delivery rate? I've also backed out the change (yesterday) to /sys/block/sda/queue/nr_**requests I think it was pushing the load higher and there is no advantage in my hardware (SAS with Perc 5/i Raid 5 over 4 disk) to run with a low value for nr_requests. You can certainly achieve higher delivery rates, but that all depends on your underlying hardware and how you have partitioned your system. Why don't you start running iostat -x 5 on the system? Leave this running to give you an idea of the baseline behavior and then look at it during periods of high load. I suspect you will see that your svctm and %util will go up dramatically when a large number of messages are being delivered. But, let's not make decisions based on assumptions! :) On my Cyrus Murder frontends (3 of them), I have limited LMTP connections to 25 in cyrus.conf: lmtp cmd=/usr/local/cyrus/bin/**lmtpproxyd listen=lmtp proto=tcp4 prefork=0 maxchild=25 This prevents our mail relays (Postfix) from opening too many simultaneous LMTP connections, which can cause too much I/O contention. Take a look during your periods of high load to see how many lmtpd processes are running. You may want to limit the number. Andy Thanks for the response. I have been checking my iostat whenever there is a number of messages in the active queue. Here is a sample snapshot from a script I run (ignoring the first iostat output of averages): Active in queue: 193 12:47:01 up 5 days, 5:23, 6 users, load average: 14.11, 9.22, 4.67 Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util sda5 3.25 281.00 19.75 129.50 654.00 3384.0027.06 5.53 36.24 6.69 99.80 svctm is about the same as when not under load and it went above 7 only once. Then there is this comment about the validity of tracking svctm: http://www.xaprb.com/blog/2010/09/06/beware-of-svctm-in-linuxs-iostat/ %util is often reaching close to %100 when there is a queue to process. sda5 is where the cyrus mail/imap lives. Our account names all begin with numbers, so almost all mail accounts are under the q folder. I'll check the lmtp process numbers as well. I've put in some in_flow_delay on the postfix side so it might keep the load from peaking as sharply. Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: Load spikes when new email arrives
On 01/23/2013 08:16 AM, francis picabia wrote: I've also backed out the change (yesterday) to /sys/block/sda/queue/nr_requests I think it was pushing the load higher and there is no advantage in my hardware (SAS with Perc 5/i Raid 5 over 4 disk) to run with a low value for nr_requests. RAID 5? We use RAID 10 for our Cyrus stores. I'm in the anti-RAID 5 camp: http://www.baarf.com/ Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: Load spikes when new email arrives
On Wed, 23 Jan 2013, francis picabia wrote: Thanks for the response. I have been checking my iostat whenever there is a number of messages in the active queue. Here is a sample snapshot from a script I run (ignoring the first iostat output of averages): Active in queue: 193 12:47:01 up 5 days, 5:23, 6 users, load average: 14.11, 9.22, 4.67 Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util sda5 3.25 281.00 19.75 129.50 654.00 3384.0027.06 5.53 36.24 6.69 99.80 svctm is about the same as when not under load and it went above 7 only once. Then there is this comment about the validity of tracking svctm: http://www.xaprb.com/blog/2010/09/06/beware-of-svctm-in-linuxs-iostat/ %util is often reaching close to %100 when there is a queue to process. sda5 is where the cyrus mail/imap lives. Our account names all begin with numbers, so almost all mail accounts are under the q folder. Okay, I didn't realize svctm could be suspect, although I guess that makes sense in a RAID array. What about your await times? Does await increase during peak loads? It seems pretty clear from iostat that you are IO bound on writes during mail delivery. As Vincent said in his reply, RAID5 performs poorly during writes. Each write actually consumes 4 disk operations (read old data, read old parity, write new data, write new parity). If you can live with the slight additional risk, turn on write caching on the Perc 5/i if you haven't already. I think they call it write-back versus write-through. If you can handle it, you would probably be a lot happier converting that RAID5 set to RAID10. You'll lose a disk worth of capacity, but get double the write performance. However, what is your real goal? Do you want to deliver mail more quickly, or do you want to reduce your load average? You can probably reduce your load average and perhaps gain a bit of speed by tweaking the lmtp maxchild limit. If you really need to deliver mail more quickly, then you need to throw more IOPS at it. Let's keep this discussion going! There are lots of ways to tune for performance. I've probably missed some. :) Andy Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
