At 11:15 07.09.04 +0530, you wrote:
>At 06/09/04 22:15 (), Erwin Hoffmann wrote:
>>At 20:11 06.09.04 +0530, you wrote:
>> >Dear Erwin,
>> >
>> >Sorry for question not really related to Vpopmail.
>> >
>> >It seems that I am hit by "Silly Qmail (Queue) Syndrome".
>> >
>> >I am using the Spamcontrol Patch v2.2.12 along with vpopmail-5.4.6, but
>> >have not used the experimental "bigtodo".
>> >
>> >Wished to apply the bigtodo. I would like to get clarified that whether
>> >bigtodo is based on ext_todo patch or big-todo patch or both. I had not
>> >initially compiled the bigtodo thinking that it is experimental.
>> >
>> >What do you suggest.
>>Well. At first you have to tell why you think you are hit by the "Silly
>>Qmail Syndrom". Any hints ?
>>Second. Apart from the big-todo enhencement, my implementation of Andre
>>Oppermann's performance enhancements dont work well. After investigation a
>>look of time and testing I didn't find any significant performance
>>Note: The code in SPAMCONTROL is not the ext-big-todo; however it is based
>>of Andre's first suggestion to influence qmail's scheduler for mail
>>processing; which was buggy by itself.
>>Third. The best thing is to avoid bounces to non-existing accounts.
>>Use my RECIPIENTS extension as part of Qmail or perhaps the "real-rcptto"
>>The forthcoming SPAMCONTROL version will include verion 0.42 of the
>>RECIPIENTS extension; check my Qmail page (http://www.fehcom.de/qmail.html).
>Hi Erwin,
>Thanks for nice reply.
>I am attaching "Queue Size" graph (5 Minute Average) updated Tuesday, 7 
>September 2004 at 0:50 (EDT).
>You can notice between 0400 - 1000 hrs (EDT) a quite high Mail Queue. 
>During that time period the smtpd is running to the tune of 100/100. But 
>the send is running to the tune of local 3/15 remote 5/40. The "messages in 
>queue but not yet preprocessed" goes on increasing in wild. When the smtpd 
>runs to the tune of 85/100 its all okay. This has started happening on 
>almost every start of the week, when huge volume of genuine + virus 
>infected customers mails start pouring in.

Ok. Until now, you did not tell us what hardware and network connection you
have. Anyway. My experience using a 2*1G PIII and fast SCSI Disks on
FreeBSD show some similar behavior. 

>I am not using "RECIPIENTS extension", but using "badrcptto" for 
>whitelisting mechanism, which works very well (might be a bit slow due to 
>the reason that lookup is being done into txt database).

Ok. Good choice.

>I am also using 
>http://linux.voyager.hr/ucspi-tcp/tcpserver-limits-2004-07-25.diff patch to 
>limit concurrent connection from single IP. This helps identifying Virus 
>trodden computers and denying them connection (it's a boon).


>I also have Caching-DNS on this Server (djbdns).


>About the todo patches the comments of Dave Sill (of Qmail Handbook fame) 
>are interesting to note in the thread:
>"Outbound email rate slows when inbound rate is high"

Dave is right. No doubt.

>Also one can have a look at the thread
>"ext-todo and big-todo patches"


>I tried to apply the patch 
>http://www.nrg4u.com/qmail/ext_todo-20030105.patch over and above 
>spamcontrol but it failed at:
>patch p1 < ext_todo-20030105.patch
>patching file p1
>patching file EXTTODO-INFO
>patching file FILES
>patching file Makefile
>Hunk #2 succeeded at 713 (offset 8 lines).
>Hunk #4 succeeded at 818 (offset 8 lines).
>Hunk #5 succeeded at 1598 (offset 87 lines).
>Hunk #6 succeeded at 1585 (offset 9 lines).
>Hunk #7 succeeded at 1694 (offset 87 lines).
>patching file TARGETS
>Hunk #1 succeeded at 405 (offset 20 lines).
>patching file hier.c
>Hunk #1 succeeded at 115 with fuzz 1 (offset 7 lines).
>patching file install-big.c
>patching file qmail-send.c
>Hunk #1 FAILED at 1215.
>Hunk #2 succeeded at 1527 (offset 88 lines).
>Hunk #3 succeeded at 1662 (offset 20 lines).
>Hunk #4 succeeded at 1797 (offset 88 lines).
>1 out of 4 hunks FAILED -- saving rejects to file qmail-send.c.rej
>patching file qmail-start.c
>patching file qmail-todo.c

Yes. I did not dare to include the ext-big-todo into SPAMCONTROL (yet).
It breaks the philosphy of Qmail.

>It also might be possible that my disk speed be contributing a bit to the 
>bottleneck. Current iostat figures are (not taken at the silly qmail 
>syndrome time, I would take that figure to when I am hit again).

>iostat -d -x /dev/hda2 2
>Linux 2.4.18-18.7.x (slsp-da4p21)       09/07/2004
>Device:    rrqm/s wrqm/s   r/s   w/s  rsec/s  wsec/s    rkB/s    wkB/s 
>avgrq-sz avgqu-sz   await  svctm  %util
>/dev/hda2    0.03   2.27  1.85  0.37   14.98   21.73     7.49    10.86 
>16.57     0.13  208.66 171.08   3.79
>Device:    rrqm/s wrqm/s   r/s   w/s  rsec/s  wsec/s    rkB/s    wkB/s 
>avgrq-sz avgqu-sz   await  svctm  %util
>/dev/hda2    0.00   0.00  2.00  0.00   16.00    0.00     8.00     0.00 
>8.00     0.15   75.00  50.00   1.00
>Device:    rrqm/s wrqm/s   r/s   w/s  rsec/s  wsec/s    rkB/s    wkB/s 
>avgrq-sz avgqu-sz   await  svctm  %util
>/dev/hda2    0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00 
>0.00     0.00    0.00   0.00   0.00
>Device:    rrqm/s wrqm/s   r/s   w/s  rsec/s  wsec/s    rkB/s    wkB/s 
>avgrq-sz avgqu-sz   await  svctm  %util
>/dev/hda2    0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00 
>0.00     0.00    0.00   0.00   0.00
>I am getting more and more convinced that ext-todo might be a possible 
>solution in this situation.

>What do you feel !

Well. Its hard to tell and certainly a "feeling" is not enough to solve
your situation. You did already a lot of diagnosis and most of your
attempts are ok (as far as I can tell).

As mentioned before, you have to explain a little

a) what hardware you are using
b) what is your average (and not mean) email volume
c) what Anti-Virus and Anti-Spam tools are you using
d) what is your general Qmail setup
e) what is the rate of inbound and outbound traffic

>From what I can see from your graph, you have typically around 400 email in
queue. Use my "newanalyse" package + qmailanaloge and you get a nice
summary statistics.

Some hints:

- It might me worthwilhe to reduce the incoming-concurrency. Drop it to 30.
Actually, the result of this is opposite from what it seems: 
This helps to better "arbiter" qmail-smtpd and thus improves overall

- Most of load originates presumably from I/0. You should try everything
to reduce it. In particular, use SPAMCONTROLs badmimetype filter in the
first place. However, if you use the qmail-queue extension and lets say
qmail-scanner, this wont help much (for reasons I will explain in
Use my QMVC instead.


PS: Very interesting discussion.

Dr. Erwin Hoffmann | FEHCom | http://www.fehcom.de/
Wiener Weg 8, 50858 Cologne | T: +49 221 484 4923 | F: ...24

Reply via email to