Re: [vchkpw] Silly Qmail (Queue) Syndrome and Spamcontrol Patch
Hi Tom, Thanks. At 08/09/04 20:42 (), you wrote: On Sep 7, 2004, at 11:54 PM, Devendra Singh wrote: c) what Anti-Virus and Anti-Spam tools are you using AntiVirus is clamav-0.75.1 and AntiSpam is SpamAssassin-2.63 with patched version of qmail-scanner Qmail-Scanner-1.23st (st patch) from http://xoomer.virgilio.it/j.toribio/qmail-scanner/. This patched version of qmail-scanner has been used to selectively enable only 20% of the domains to have AntiVirus/AntiSpam enabled. I am also using the --sa-reject option to have spam messages with a score higher than sa-delete (score of 16 in my case) to be rejected before the smtp session is closed. I'd probably point the finger at qmail-scanner. It's a major resource hog and starts a perl instance every time a message comes in. I do agree, in fact I knew. Weighing Options. I use clamav and SpamAssassin as well, but use qscanq (google for it) and qmail-spamc (included with SpamAssassin) to block viruses and score spam on messages at the qmail-queue stage. Unfortunately, without patching, you won't be able to selectively enable it per domain or have an sa-reject option. I would look at qscanq as well as QMVC suggested by Dr Erwin. I would also look for any other options if possible. But, I need to enable / disable AntiVirus AntiSpam for Selective email-addresses (and domains). Also, Clients requirements have forced me to quarantine Spam above certain level and should be intimated to the sender (just in case its's a real sender). If we bounce the spam it will result into a double bounce mostly. Hence rejection is required. You could look at some of the patches Ken Jones of Inter7 has put together to add SpamAssassin integration to vdelivermail. This would offload the spam processing from qmail-smptd, and can be enabled on a per-domain basis. You could then replace qmail-scanner with qscanq to block viruses (for all domains) at the smtpd level. Where can I find those patches by Ken? Any URL please (I tried searching the archives). Some hints: - It might me worthwilhe to reduce the incoming-concurrency. Drop it to 30. Any figures less than 80 would cause lot many Servers not to get smtp connect to our Server during peak time of 0100 to 0500 hrs EDT. Maybe not. You need to determine whether a lower concurrency will reduce the amount of time spent on each message and ultimately allow more connections per hour. Once you start hitting virtual memory, all of the current connections will get bogged down. Take a look at how many messages are processed per hour at 100, and then at 80. If the queue is growing and messages aren't getting delivered, there's not much benefit to queueing the message instead of just not accepting the connection. I am experimenting on it. However, I have got sigh of relief (probably for the time being), by adding sbl-xbl.spamhaus.org to rblsmtpd (I was already using bl.spamcop.net). This has reduced the SMTPD threads a bit. Dr Erwin, May I request Dr Erwin to get reply on my reply to his reply in this thread. I have already tried the newanalyse package on a development (Fedora) Server. It works, great. The qmFind did not compile. One more question Dr Erwin, The SMTP log is more informative after your SpamControl Patch but it lacks the IP addresses in front of the entries. Have I missed something or its like that only. Here is a sample. @400041416592147f8924 Accept::SNDR::Relay_Client: MailFrom: [EMAIL PROTECTED] RcptTo: [EMAIL PROTECTED] @4000414165921589e0bc tcpserver: deny 3214 x.x.xxx:111.222.333.444:25 :218.20.230.246::63771 MAXCONNIP:5 @40004141659217e0a9cc Accept::ORIG::Local_Sender: MailFrom: [EMAIL PROTECTED] RcptTo: [EMAIL PROTECTED] @4000414165921e41b07c Accept::ORIG::Local_Sender: MailFrom: [EMAIL PROTECTED] RcptTo: [EMAIL PROTECTED] @4000414165cb2f16d51c tcpserver: status: 69/80 @4000414165cc0cfb055c Accept::RCPT::Rcpthosts_Rcptto: MailFrom: [EMAIL PROTECTED] RcptTo: [EMAIL PROTECTED] @4000414165cc181127dc tcpserver: status: 70/80 @4000414165f0227f90dc tcpserver: ok 5737 x.x.xxx:111.222.333.444:25 :61.11.87.37::2812 @4000414165f025019c24 tcpserver: ok 5736 x.x.xxx:111.222.333.444:25 :203.145.134.238:dvromafh:2447 @4000414165f026fdc5fc Reject::SNDR::Invalid_Relay: MailFrom: [EMAIL PROTECTED] RcptTo: [EMAIL PROTECTED] @4000414165f02792fd34 tcpserver: ok 5174 x.x.xxx:111.222.333.444:25 :61.1.220.108::1854 @4000414166381540b25c Reject::ORIG::No_DNSMX: MailFrom: [EMAIL PROTECTED] RcptTo: [EMAIL PROTECTED] Thanks. Devendra Singh
Re: [vchkpw] Silly Qmail (Queue) Syndrome and Spamcontrol Patch
Hi Dr Erwin, At 07/09/04 13:23 (), you wrote: Hi, At 11:15 07.09.04 +0530, you wrote: At 06/09/04 22:15 (), Erwin Hoffmann wrote: Hi, At 20:11 06.09.04 +0530, you wrote: Dear Erwin, Sorry for question not really related to Vpopmail. It seems that I am hit by Silly Qmail (Queue) Syndrome. I am using the Spamcontrol Patch v2.2.12 along with vpopmail-5.4.6, but have not used the experimental bigtodo. Wished to apply the bigtodo. I would like to get clarified that whether you bigtodo is based on ext_todo patch or big-todo patch or both. I had not initially compiled the bigtodo thinking that it is experimental. What do you suggest. Well. At first you have to tell why you think you are hit by the Silly Qmail Syndrom. Any hints ? Second. Apart from the big-todo enhencement, my implementation of Andre Oppermann's performance enhancements dont work well. After investigation a look of time and testing I didn't find any significant performance improvement. Note: The code in SPAMCONTROL is not the ext-big-todo; however it is based of Andre's first suggestion to influence qmail's scheduler for mail processing; which was buggy by itself. Third. The best thing is to avoid bounces to non-existing accounts. Use my RECIPIENTS extension as part of Qmail or perhaps the real-rcptto patch. The forthcoming SPAMCONTROL version will include verion 0.42 of the RECIPIENTS extension; check my Qmail page (http://www.fehcom.de/qmail.html). regards. --eh. Hi Erwin, Thanks for nice reply. I am attaching Queue Size graph (5 Minute Average) updated Tuesday, 7 September 2004 at 0:50 (EDT). You can notice between 0400 - 1000 hrs (EDT) a quite high Mail Queue. During that time period the smtpd is running to the tune of 100/100. But the send is running to the tune of local 3/15 remote 5/40. The messages in queue but not yet preprocessed goes on increasing in wild. When the smtpd runs to the tune of 85/100 its all okay. This has started happening on almost every start of the week, when huge volume of genuine + virus infected customers mails start pouring in. Ok. Until now, you did not tell us what hardware and network connection you have. Anyway. My experience using a 2*1G PIII and fast SCSI Disks on FreeBSD show some similar behavior. Its Linux slsp-da4p21 2.4.18-18.7.x #1 [Red Hat Linux release 7.3 (Valhalla)] Intel(R) Pentium(R) CPU 2.40GHz cache size : 512 KB RAM:1GB SWAP: 2GB HDD: Barracuda 7200.7 (It's an IDE Drive) Model Number:ST380011A Capacity:80 GB Speed:7200 rpm Seek time:8.5 ms avg Interface:Ultra ATA/100 df -m Filesystem 1M-blocks Used Available Use% Mounted on /dev/hda373990 16422 53810 24% / /dev/hda1 114 9 999% /boot none 441 0 4400% /dev/shm fdisk -l Disk /dev/hda: 255 heads, 63 sectors, 9729 cylinders Units = cylinders of 16065 * 512 bytes Device BootStart EndBlocks Id System /dev/hda1 * 115120456 83 Linux /dev/hda216 146 1052257+ 82 Linux swap /dev/hda3 147 9729 76975447+ 83 Linux The /home/vpopmail/domains and /var/qmail/queue both are on /dev/hda3 Network Card: Realtek|RTL-8139/8139C The Server is connected to a 100 MBPS Network Port limited to 10 MBPS (10 M/s is equal to over 3 terabytes of traffic per month). mii-tool -v eth0: negotiated 10baseT-FD, link ok product info: vendor 00:00:00, model 0 rev 0 basic mode: autonegotiation enabled basic status: autonegotiation complete, link ok capabilities: 100baseTx-FD 100baseTx-HD 10baseT-FD 10baseT-HD advertising: 100baseTx-FD 100baseTx-HD 10baseT-FD 10baseT-HD link partner: 10baseT-FD 10baseT-HD I have not yet noticed any signs of Network Bottleneck. I am not using RECIPIENTS extension, but using badrcptto for whitelisting mechanism, which works very well (might be a bit slow due to the reason that lookup is being done into txt database). Ok. Good choice. I am also using http://linux.voyager.hr/ucspi-tcp/tcpserver-limits-2004-07-25.diff patch to limit concurrent connection from single IP. This helps identifying Virus trodden computers and denying them connection (it's a boon). Good. I also have Caching-DNS on this Server (djbdns). Excellent. About the todo patches the comments of Dave Sill (of Qmail Handbook fame) are interesting to note in the thread: Outbound email rate slows when inbound rate is high http://groups.google.com/groups?hl=enlr=ie=UTF-8c2coff=1threadm=e6c47de 7.0310091325.147cade4%40posting.google.comrnum=2prev=/groups%3Fq%3Dext-tod o%26hl%3Den%26lr%3D%26ie%3DUTF-8%26c2coff%3D1%26selm%3De6c47de7.0310091325.1 47cade4%2540posting.google.com%26rnum%3D2 Dave is right. No doubt. Also one can have a look at the thread ext-todo and big-todo patches http://groups.google.com/groups?hl=enlr=ie=UTF-8c2coff=1threadm=wx0lm56 pfo0.fsf%40sws5.ctd.ornl.govrnum=1prev=/groups%3Fhl%3Den%26lr%3D%26ie%3DUT
Re: [vchkpw] Silly Qmail (Queue) Syndrome and Spamcontrol Patch
On Sep 7, 2004, at 11:54 PM, Devendra Singh wrote: c) what Anti-Virus and Anti-Spam tools are you using AntiVirus is clamav-0.75.1 and AntiSpam is SpamAssassin-2.63 with patched version of qmail-scanner Qmail-Scanner-1.23st (st patch) from http://xoomer.virgilio.it/j.toribio/qmail-scanner/. This patched version of qmail-scanner has been used to selectively enable only 20% of the domains to have AntiVirus/AntiSpam enabled. I am also using the --sa-reject option to have spam messages with a score higher than sa-delete (score of 16 in my case) to be rejected before the smtp session is closed. I'd probably point the finger at qmail-scanner. It's a major resource hog and starts a perl instance every time a message comes in. I use clamav and SpamAssassin as well, but use qscanq (google for it) and qmail-spamc (included with SpamAssassin) to block viruses and score spam on messages at the qmail-queue stage. Unfortunately, without patching, you won't be able to selectively enable it per domain or have an sa-reject option. You could look at some of the patches Ken Jones of Inter7 has put together to add SpamAssassin integration to vdelivermail. This would offload the spam processing from qmail-smptd, and can be enabled on a per-domain basis. You could then replace qmail-scanner with qscanq to block viruses (for all domains) at the smtpd level. Some hints: - It might me worthwilhe to reduce the incoming-concurrency. Drop it to 30. Any figures less than 80 would cause lot many Servers not to get smtp connect to our Server during peak time of 0100 to 0500 hrs EDT. Maybe not. You need to determine whether a lower concurrency will reduce the amount of time spent on each message and ultimately allow more connections per hour. Once you start hitting virtual memory, all of the current connections will get bogged down. Take a look at how many messages are processed per hour at 100, and then at 80. If the queue is growing and messages aren't getting delivered, there's not much benefit to queueing the message instead of just not accepting the connection. -- Tom Collins - [EMAIL PROTECTED] QmailAdmin: http://qmailadmin.sf.net/ Vpopmail: http://vpopmail.sf.net/ Info on the Sniffter hand-held Network Tester: http://sniffter.com/
Re: [vchkpw] Silly Qmail (Queue) Syndrome and Spamcontrol Patch
Hi, At 11:15 07.09.04 +0530, you wrote: At 06/09/04 22:15 (), Erwin Hoffmann wrote: Hi, At 20:11 06.09.04 +0530, you wrote: Dear Erwin, Sorry for question not really related to Vpopmail. It seems that I am hit by Silly Qmail (Queue) Syndrome. I am using the Spamcontrol Patch v2.2.12 along with vpopmail-5.4.6, but have not used the experimental bigtodo. Wished to apply the bigtodo. I would like to get clarified that whether you bigtodo is based on ext_todo patch or big-todo patch or both. I had not initially compiled the bigtodo thinking that it is experimental. What do you suggest. Well. At first you have to tell why you think you are hit by the Silly Qmail Syndrom. Any hints ? Second. Apart from the big-todo enhencement, my implementation of Andre Oppermann's performance enhancements dont work well. After investigation a look of time and testing I didn't find any significant performance improvement. Note: The code in SPAMCONTROL is not the ext-big-todo; however it is based of Andre's first suggestion to influence qmail's scheduler for mail processing; which was buggy by itself. Third. The best thing is to avoid bounces to non-existing accounts. Use my RECIPIENTS extension as part of Qmail or perhaps the real-rcptto patch. The forthcoming SPAMCONTROL version will include verion 0.42 of the RECIPIENTS extension; check my Qmail page (http://www.fehcom.de/qmail.html). regards. --eh. Hi Erwin, Thanks for nice reply. I am attaching Queue Size graph (5 Minute Average) updated Tuesday, 7 September 2004 at 0:50 (EDT). You can notice between 0400 - 1000 hrs (EDT) a quite high Mail Queue. During that time period the smtpd is running to the tune of 100/100. But the send is running to the tune of local 3/15 remote 5/40. The messages in queue but not yet preprocessed goes on increasing in wild. When the smtpd runs to the tune of 85/100 its all okay. This has started happening on almost every start of the week, when huge volume of genuine + virus infected customers mails start pouring in. Ok. Until now, you did not tell us what hardware and network connection you have. Anyway. My experience using a 2*1G PIII and fast SCSI Disks on FreeBSD show some similar behavior. I am not using RECIPIENTS extension, but using badrcptto for whitelisting mechanism, which works very well (might be a bit slow due to the reason that lookup is being done into txt database). Ok. Good choice. I am also using http://linux.voyager.hr/ucspi-tcp/tcpserver-limits-2004-07-25.diff patch to limit concurrent connection from single IP. This helps identifying Virus trodden computers and denying them connection (it's a boon). Good. I also have Caching-DNS on this Server (djbdns). Excellent. About the todo patches the comments of Dave Sill (of Qmail Handbook fame) are interesting to note in the thread: Outbound email rate slows when inbound rate is high http://groups.google.com/groups?hl=enlr=ie=UTF-8c2coff=1threadm=e6c47de 7.0310091325.147cade4%40posting.google.comrnum=2prev=/groups%3Fq%3Dext-tod o%26hl%3Den%26lr%3D%26ie%3DUTF-8%26c2coff%3D1%26selm%3De6c47de7.0310091325.1 47cade4%2540posting.google.com%26rnum%3D2 Dave is right. No doubt. Also one can have a look at the thread ext-todo and big-todo patches http://groups.google.com/groups?hl=enlr=ie=UTF-8c2coff=1threadm=wx0lm56 pfo0.fsf%40sws5.ctd.ornl.govrnum=1prev=/groups%3Fhl%3Den%26lr%3D%26ie%3DUT F-8%26c2coff%3D1%26q%3Ddave%2Bsill%2Bext-todo%2Band%2Bbig-todo Ok. I tried to apply the patch http://www.nrg4u.com/qmail/ext_todo-20030105.patch over and above spamcontrol but it failed at: patch p1 ext_todo-20030105.patch patching file p1 patching file EXTTODO-INFO patching file FILES patching file Makefile Hunk #2 succeeded at 713 (offset 8 lines). Hunk #4 succeeded at 818 (offset 8 lines). Hunk #5 succeeded at 1598 (offset 87 lines). Hunk #6 succeeded at 1585 (offset 9 lines). Hunk #7 succeeded at 1694 (offset 87 lines). patching file TARGETS Hunk #1 succeeded at 405 (offset 20 lines). patching file hier.c Hunk #1 succeeded at 115 with fuzz 1 (offset 7 lines). patching file install-big.c patching file qmail-send.c Hunk #1 FAILED at 1215. Hunk #2 succeeded at 1527 (offset 88 lines). Hunk #3 succeeded at 1662 (offset 20 lines). Hunk #4 succeeded at 1797 (offset 88 lines). 1 out of 4 hunks FAILED -- saving rejects to file qmail-send.c.rej patching file qmail-start.c patching file qmail-todo.c Yes. I did not dare to include the ext-big-todo into SPAMCONTROL (yet). It breaks the philosphy of Qmail. It also might be possible that my disk speed be contributing a bit to the bottleneck. Current iostat figures are (not taken at the silly qmail syndrome time, I would take that figure to when I am hit again). iostat -d -x /dev/hda2 2 Linux 2.4.18-18.7.x (slsp-da4p21) 09/07/2004 Device:rrqm/s wrqm/s r/s w/s rsec/s wsec/srkB/swkB/s avgrq-sz avgqu-sz await svctm %util /dev/hda20.03 2.27 1.85 0.37 14.98 21.73
[vchkpw] Silly Qmail (Queue) Syndrome and Spamcontrol Patch
Dear Erwin, Sorry for question not really related to Vpopmail. It seems that I am hit by Silly Qmail (Queue) Syndrome. I am using the Spamcontrol Patch v2.2.12 along with vpopmail-5.4.6, but have not used the experimental bigtodo. Wished to apply the bigtodo. I would like to get clarified that whether you bigtodo is based on ext_todo patch or big-todo patch or both. I had not initially compiled the bigtodo thinking that it is experimental. What do you suggest. Thanks. Devendra Singh
Re: [vchkpw] Silly Qmail (Queue) Syndrome and Spamcontrol Patch
Hi, At 20:11 06.09.04 +0530, you wrote: Dear Erwin, Sorry for question not really related to Vpopmail. It seems that I am hit by Silly Qmail (Queue) Syndrome. I am using the Spamcontrol Patch v2.2.12 along with vpopmail-5.4.6, but have not used the experimental bigtodo. Wished to apply the bigtodo. I would like to get clarified that whether you bigtodo is based on ext_todo patch or big-todo patch or both. I had not initially compiled the bigtodo thinking that it is experimental. What do you suggest. Well. At first you have to tell why you think you are hit by the Silly Qmail Syndrom. Any hints ? Second. Apart from the big-todo enhencement, my implementation of Andre Oppermann's performance enhancements dont work well. After investigation a look of time and testing I didn't find any significant performance improvement. Note: The code in SPAMCONTROL is not the ext-big-todo; however it is based of Andre's first suggestion to influence qmail's scheduler for mail processing; which was buggy by itself. Third. The best thing is to avoid bounces to non-existing accounts. Use my RECIPIENTS extension as part of Qmail or perhaps the real-rcptto patch. The forthcoming SPAMCONTROL version will include verion 0.42 of the RECIPIENTS extension; check my Qmail page (http://www.fehcom.de/qmail.html). regards. --eh. Dr. Erwin Hoffmann | FEHCom | http://www.fehcom.de/ Wiener Weg 8, 50858 Cologne | T: +49 221 484 4923 | F: ...24
Re: [vchkpw] Silly Qmail (Queue) Syndrome and Spamcontrol Patch
At 06/09/04 22:15 (), Erwin Hoffmann wrote: Hi, At 20:11 06.09.04 +0530, you wrote: Dear Erwin, Sorry for question not really related to Vpopmail. It seems that I am hit by Silly Qmail (Queue) Syndrome. I am using the Spamcontrol Patch v2.2.12 along with vpopmail-5.4.6, but have not used the experimental bigtodo. Wished to apply the bigtodo. I would like to get clarified that whether you bigtodo is based on ext_todo patch or big-todo patch or both. I had not initially compiled the bigtodo thinking that it is experimental. What do you suggest. Well. At first you have to tell why you think you are hit by the Silly Qmail Syndrom. Any hints ? Second. Apart from the big-todo enhencement, my implementation of Andre Oppermann's performance enhancements dont work well. After investigation a look of time and testing I didn't find any significant performance improvement. Note: The code in SPAMCONTROL is not the ext-big-todo; however it is based of Andre's first suggestion to influence qmail's scheduler for mail processing; which was buggy by itself. Third. The best thing is to avoid bounces to non-existing accounts. Use my RECIPIENTS extension as part of Qmail or perhaps the real-rcptto patch. The forthcoming SPAMCONTROL version will include verion 0.42 of the RECIPIENTS extension; check my Qmail page (http://www.fehcom.de/qmail.html). regards. --eh. Hi Erwin, Thanks for nice reply. I am attaching Queue Size graph (5 Minute Average) updated Tuesday, 7 September 2004 at 0:50 (EDT). You can notice between 0400 - 1000 hrs (EDT) a quite high Mail Queue. During that time period the smtpd is running to the tune of 100/100. But the send is running to the tune of local 3/15 remote 5/40. The messages in queue but not yet preprocessed goes on increasing in wild. When the smtpd runs to the tune of 85/100 its all okay. This has started happening on almost every start of the week, when huge volume of genuine + virus infected customers mails start pouring in. I am not using RECIPIENTS extension, but using badrcptto for whitelisting mechanism, which works very well (might be a bit slow due to the reason that lookup is being done into txt database). I am also using http://linux.voyager.hr/ucspi-tcp/tcpserver-limits-2004-07-25.diff patch to limit concurrent connection from single IP. This helps identifying Virus trodden computers and denying them connection (it's a boon). I also have Caching-DNS on this Server (djbdns). About the todo patches the comments of Dave Sill (of Qmail Handbook fame) are interesting to note in the thread: Outbound email rate slows when inbound rate is high http://groups.google.com/groups?hl=enlr=ie=UTF-8c2coff=1threadm=e6c47de7.0310091325.147cade4%40posting.google.comrnum=2prev=/groups%3Fq%3Dext-todo%26hl%3Den%26lr%3D%26ie%3DUTF-8%26c2coff%3D1%26selm%3De6c47de7.0310091325.147cade4%2540posting.google.com%26rnum%3D2 Also one can have a look at the thread ext-todo and big-todo patches http://groups.google.com/groups?hl=enlr=ie=UTF-8c2coff=1threadm=wx0lm56pfo0.fsf%40sws5.ctd.ornl.govrnum=1prev=/groups%3Fhl%3Den%26lr%3D%26ie%3DUTF-8%26c2coff%3D1%26q%3Ddave%2Bsill%2Bext-todo%2Band%2Bbig-todo I tried to apply the patch http://www.nrg4u.com/qmail/ext_todo-20030105.patch over and above spamcontrol but it failed at: patch p1 ext_todo-20030105.patch patching file p1 patching file EXTTODO-INFO patching file FILES patching file Makefile Hunk #2 succeeded at 713 (offset 8 lines). Hunk #4 succeeded at 818 (offset 8 lines). Hunk #5 succeeded at 1598 (offset 87 lines). Hunk #6 succeeded at 1585 (offset 9 lines). Hunk #7 succeeded at 1694 (offset 87 lines). patching file TARGETS Hunk #1 succeeded at 405 (offset 20 lines). patching file hier.c Hunk #1 succeeded at 115 with fuzz 1 (offset 7 lines). patching file install-big.c patching file qmail-send.c Hunk #1 FAILED at 1215. Hunk #2 succeeded at 1527 (offset 88 lines). Hunk #3 succeeded at 1662 (offset 20 lines). Hunk #4 succeeded at 1797 (offset 88 lines). 1 out of 4 hunks FAILED -- saving rejects to file qmail-send.c.rej patching file qmail-start.c patching file qmail-todo.c It also might be possible that my disk speed be contributing a bit to the bottleneck. Current iostat figures are (not taken at the silly qmail syndrome time, I would take that figure to when I am hit again). iostat -d -x /dev/hda2 2 Linux 2.4.18-18.7.x (slsp-da4p21) 09/07/2004 Device:rrqm/s wrqm/s r/s w/s rsec/s wsec/srkB/swkB/s avgrq-sz avgqu-sz await svctm %util /dev/hda20.03 2.27 1.85 0.37 14.98 21.73 7.4910.86 16.57 0.13 208.66 171.08 3.79 Device:rrqm/s wrqm/s r/s w/s rsec/s wsec/srkB/swkB/s avgrq-sz avgqu-sz await svctm %util /dev/hda20.00 0.00 2.00 0.00 16.000.00 8.00 0.00 8.00 0.15 75.00 50.00 1.00 Device:rrqm/s wrqm/s r/s w/s rsec/s wsec/srkB/swkB/s avgrq-sz avgqu-sz await svctm %util /dev/hda20.00 0.00 0.00 0.000.00
