Re: Qmail (as seen on a not so good day).
On Thu, 18 Feb 1999, Jere Cassidy wrote: -| If i do a "netstat -n|grep ":25 " I get almost 700 connections although -| most of these are in the "CLOSE WAIT" stage or something similar. What kernel revision are you running? I know some of the 2.0.3X kernels pre .36 supposedly have a problem with never dropping connections. So, even though the client side has dropped, the server keeps it alive, and this is counted in tcp servers max. Here's an example: tcp32696645 ns1.netmdc.com:17117pm3-1.netmdc.com:telnet FIN_WAIT1 tcp32696555 ns1.netmdc.com:17162pm3-1.netmdc.com:telnet FIN_WAIT1 tcp32696675 ns1.netmdc.com:25151pm3-1.netmdc.com:telnet FIN_WAIT1 tcp32216638 ns1.netmdc.com:25351pm3-1.netmdc.com:telnet FIN_WAIT1 tcp32696670 ns1.netmdc.com:26194pm3-1.netmdc.com:telnet FIN_WAIT1 tcp32696625 ns1.netmdc.com:14393pm3-1.netmdc.com:telnet FIN_WAIT1 tcp32696625 ns1.netmdc.com:16940pm3-1.netmdc.com:telnet FIN_WAIT1 tcp32696630 ns1.netmdc.com:16948pm3-1.netmdc.com:telnet FIN_WAIT1 tcp32696615 ns1.netmdc.com:17031pm3-1.netmdc.com:telnet FIN_WAIT1 tcp32696689 ns1.netmdc.com:17204pm3-1.netmdc.com:telnet FIN_WAIT1 tcp32696615 ns1.netmdc.com:17531pm3-1.netmdc.com:telnet FIN_WAIT1 tcp32696576 ns1.netmdc.com:18093pm3-1.netmdc.com:telnet FIN_WAIT1 tcp0 14706 ns1.netmdc.com:telnet byu006601wks.rn.b:61165 ESTABLISHED tcp 19 0 ns1.netmdc.com:5299 pm2-4.netmdc.com:telnet CLOSE_WAIT This happened when i had to reboot the portmaster, the connections never cleaned up properly. This was WELL over a week ago ;) ___ _ __ _ __ /___ ___ /__ John Gonzalez/Net.Tech __ __ \ __ \ __/_ __ `__ \/ __ /_ ___/ MDC Computers/netMDC! _ / / / `__/ /_ / / / / / / /_/ / / /__ (505)437-7600/fax-437-3052 /_/ /_/\___/\__/ /_/ /_/ /_/\__,_/ \___/ http://www.netmdc.com [-[system info]---] 12:30pm up 13 days, 19:10, 3 users, load average: 0.25, 0.48, 0.27
Re: Qmail (as seen on a not so good day).
Jere Cassidy writes: > Mark, > We removed the -c options because we are currently using > qmail/control/concurrencyremote and qmail/control/concurrencylocal. should we also > add -c options to tcpserver? Bad boy! concurrencyremote/local are for outgoing E-mail. -c option to tcpserver sets the limit for incoming E-mail. You were defaulting to a maximum of 40 inbound connections at the same time. I'd say you should bump it up to about 300. -- Sam
Re: Qmail (as seen on a not so good day).
At 09:01 AM 2/18/99 -0500, Jere Cassidy wrote: >Mark, >We removed the -c options because we are currently using >qmail/control/concurrencyremote and qmail/control/concurrencylocal. should we also >add -c options to tcpserver? Correct. They are two different controls. The -c option defines how many inbound connections you will accept at one time. concurrencyremote defines how many *outbound* connetions you will initiate at one time. >Now that i think about it... it seems if the connections eclispse approximately 200, >we start to have incredible delay on the smtp connects. (this leads me to believe >that 128 limit mentioned in Redhat Linux 5.2's `man listen` might be our problem). I >am unsure what to do next is there a way to up this limit (besides hacking the >kernel somewhere)? Do other unices perform better in this area? Doesn't anyone else >have this problem ? As I've said previously. I'd be surprised if backlog is a problem. Now that I think about it, limits may be a problem if tcpserver has inherited low limits. But that's additional to changing to -c. One thing at a time. >It would be difficult to take it out at this point. It has helped us immensely in Right. Well let's just get the local settings right first and see what happens. >system availability, but I truly believe that our main server (an alpha633) would be >able to handle _all_ the load if it didnt run into these tcpserver/max connection >issues. I agree. You have a lot more horsepower than I've used for much larger user bases. So, change the tcpserver to use -c and restart tcpserver. There is no need to restart qmail. Regards.
Re: Qmail (as seen on a not so good day).
Doh! I did not realize that this was the issue. my esteemed co-worker decided to start using the concurrency files in qmail/control and i assumed these were the same. ( a really stupid assumption) I will set these up and perform test on the servers. Thanks Mark John and Tim. I guess I really should have rechecked the tcpserver man pages as you guys suggested! RTFM! Thanks again. If this doesn't work I am going to join my Luddite pals in Canada! "Timothy L. Mayo" wrote: > YES, USE THE -c OPTION TO TCPSERVER. The -b sets the backlog. -c sets > the number of simultaneous TCP sessions that tcpserver will process. You > have increased the backlog but have not increased the number of sessions > each server will accept. The default is 40! > > On Thu, 18 Feb 1999, Jere Cassidy wrote: > > > > > What is wrong with the following setup: > > > > less than 30K customers: > > > > qmail 1.03 running on 3 high speed alpha's (each with 128MB ram) > > Running 4 TCPSERVER daemon processes. > > > > > > 1 SMTP (port 25) > > 1 POP3 (port 110) > > 1 SMTP (port 2001) > > 1 POP3 (port 2002) > > > > These 3 servers running these 4 daemons share a Netapp filer for backend > > storage. > > > > We have done major tuning to these servers time after time. Here is the > > current situation: > > 3 of the daemons run fine. the SMTP (on regular port 25) does not > > respond. > > > > I have set the -b option for TCPserver(this helped us immensely before) > > to 5000 (supposedly allowing tcpserver to respond to 5000 connections). > > > > Is there some default limit somewhere that would only allow tcpserver to > > pass so many connections to qmail-smtpd? The downtime on the servers is > > getting rediculous because of this problem. > > > > If I run /var/qmail/bin/qmail-smtpd it comes right up. > > If I telnet localhost 2002 (simply another instance of tcpserver) it > > comes right up. > > Both POP3 connections come right up > > > > If i do a "netstat -n|grep ":25 " I get almost 700 connections although > > most of these are in the "CLOSE WAIT" stage or something similar. > > > > On one of the servers, when this happens and qmail is totally > > unresponsive on port 25, the load drops to 0.00 and the server just sits > > there. > > > > restarting qmail seems to help for about 5 minutes... then the imaginary > > limit is hit and everything goes to hell. > > > > Anyone have any suggestions for the current situation? > > > > > > > > > > > > > > > > > > -- > > > > > > // Jere Cassidy - System Administration - D&E SuperNet > > email: [EMAIL PROTECTED]phone: (717)738-7054 > > web: http://www.desupernet.net/jere > > pager/pcs: [EMAIL PROTECTED] - (717)203-0042 > > ~~~ "While sowing the seeds of Utopia, > > you invoked a convenient amnesia" -BR ~~~ > > > > > > > > > > > > - > Timothy L. Mayo mailto:[EMAIL PROTECTED] > Senior Systems Manager > localconnect(sm) > http://www.localconnect.net/ > > The National Business Network Inc. http://www.nb.net/ > One Monroeville Center, Suite 850 > Monroeville, PA 15146 > (412) 810- Phone > (412) 810-8886 Fax -- // Jere Cassidy - System Administration - D&E SuperNet email: [EMAIL PROTECTED]phone: (717)738-7054 web: http://www.desupernet.net/jere pager/pcs: [EMAIL PROTECTED] - (717)203-0042 ~~~ "While sowing the seeds of Utopia, you invoked a convenient amnesia" -BR ~~~
Re: Qmail (as seen on a not so good day).
Yes, you NEED the -c option. The qmail concurency limits are for OUTGOING mail (either local or remote). The tcpserver -c option set concurrency for INCOMING TCP connections which is where you are having the problem. The default is 40, I would try 80 or 100 and see what happens. On Thu, 18 Feb 1999, Jere Cassidy wrote: > Mark, > We removed the -c options because we are currently using > qmail/control/concurrencyremote and qmail/control/concurrencylocal. should we also > add -c options to tcpserver? > > I know that the backlog _is_ an issue. Our performance increased so much when we did > -b30 instead of letting the default take over (5). Every time we have upped this > number, we get performance gains. The problem is that if only one server is left in > the Alteon's rotation, it seems to get more than 128 connections in a matter of > minutes. These then seem to have a logarythmic effect. The more connections queue > up, the more connections are delayed, the more connections are queued up... etc... > This leads to the 700 number (which i do admit is extermely high-- netstat showed > some 590 smtp connects). During the slow time this morning after i got all 3 servers > handling mail, we had about 24 connections total (not just smtp). > > Now that i think about it... it seems if the connections eclispse approximately 200, > we start to have incredible delay on the smtp connects. (this leads me to believe > that 128 limit mentioned in Redhat Linux 5.2's `man listen` might be our problem). I > am unsure what to do next is there a way to up this limit (besides hacking the > kernel somewhere)? Do other unices perform better in this area? Doesn't anyone else > have this problem ? > > The alteon takes out servers that are not responsive to its SMTP and/or POP3 > connects. > It would be difficult to take it out at this point. It has helped us immensely in > system availability, but I truly believe that our main server (an alpha633) would be > able to handle _all_ the load if it didnt run into these tcpserver/max connection > issues. > Thanks again for the help Mark and anyone else who wishes to contribute! > > -Jere > > > Mark Delany wrote: > > > >Yes, 700 connections seems high, but after some period of down time, it seems to > > > > Just to emphasize what I was saying. I reckon for 30K users (was that the > > number-ish?), 700 is far higher than normal. I know there is no such thing > > as normal, but... > > > > >tcpserver -l$hostvalue -q -b100 -H -R -D 0 pop-3 > > >tcpserver -l$hostvalue -q -b50 -H -R -D 0 2001 > > >tcpserver -l$hostvalue -t8 -q -b5000 -D -u502 -g2108 > > >tcpserver -l$hostvalue -t8 -q -b50 -D -u502 -g2108 > > > > In all cases, change the -b to a -c > > > > >After reading "man listen" I am reminded of the help this list gave us when we > > >had this problem before. Our connections were being artificially limited by > > >Linux to 5 at a time! This was solved with adding the -b20 (then later upping > > > > It *may* be worth upping it beyond the default, but the listen queue really > > only comes into effect if tcpserver isn't keeping up with the inbound > > connection rate. > > > > As long as tcpserver is doing the accept and passoff to qmail-smtpd in > > enough time, the backlog doesn't apply. the concurrency with -c does of course. > > > > >If one server reaches this limit, it is overloaded and if lucky it is dropped > > >out of the rotation by the alteon. This causes the other servers to overload > > >and reach the same state. > > > > Is the alteon configured to load balance of switch on no response? > > > > Also, is it possible to bypass the Alteon for a while? There may be some > > unknown interaction there. > > > > Regards. > > -- > > // Jere Cassidy - System Administration - D&E SuperNet > email: [EMAIL PROTECTED]phone: (717)738-7054 > web: http://www.desupernet.net/jere > pager/pcs: [EMAIL PROTECTED] - (717)203-0042 > ~~~ "While sowing the seeds of Utopia, > you invoked a convenient amnesia" -BR ~~~ > > > > - Timothy L. Mayo mailto:[EMAIL PROTECTED] Senior Systems Manager localconnect(sm) http://www.localconnect.net/ The National Business Network Inc. http://www.nb.net/ One Monroeville Center, Suite 850 Monroeville, PA 15146 (412) 810- Phone (412) 810-8886 Fax
Re: Qmail (as seen on a not so good day).
Mark, We removed the -c options because we are currently using qmail/control/concurrencyremote and qmail/control/concurrencylocal. should we also add -c options to tcpserver? I know that the backlog _is_ an issue. Our performance increased so much when we did -b30 instead of letting the default take over (5). Every time we have upped this number, we get performance gains. The problem is that if only one server is left in the Alteon's rotation, it seems to get more than 128 connections in a matter of minutes. These then seem to have a logarythmic effect. The more connections queue up, the more connections are delayed, the more connections are queued up... etc... This leads to the 700 number (which i do admit is extermely high-- netstat showed some 590 smtp connects). During the slow time this morning after i got all 3 servers handling mail, we had about 24 connections total (not just smtp). Now that i think about it... it seems if the connections eclispse approximately 200, we start to have incredible delay on the smtp connects. (this leads me to believe that 128 limit mentioned in Redhat Linux 5.2's `man listen` might be our problem). I am unsure what to do next is there a way to up this limit (besides hacking the kernel somewhere)? Do other unices perform better in this area? Doesn't anyone else have this problem ? The alteon takes out servers that are not responsive to its SMTP and/or POP3 connects. It would be difficult to take it out at this point. It has helped us immensely in system availability, but I truly believe that our main server (an alpha633) would be able to handle _all_ the load if it didnt run into these tcpserver/max connection issues. Thanks again for the help Mark and anyone else who wishes to contribute! -Jere Mark Delany wrote: > >Yes, 700 connections seems high, but after some period of down time, it seems to > > Just to emphasize what I was saying. I reckon for 30K users (was that the > number-ish?), 700 is far higher than normal. I know there is no such thing > as normal, but... > > >tcpserver -l$hostvalue -q -b100 -H -R -D 0 pop-3 > >tcpserver -l$hostvalue -q -b50 -H -R -D 0 2001 > >tcpserver -l$hostvalue -t8 -q -b5000 -D -u502 -g2108 > >tcpserver -l$hostvalue -t8 -q -b50 -D -u502 -g2108 > > In all cases, change the -b to a -c > > >After reading "man listen" I am reminded of the help this list gave us when we > >had this problem before. Our connections were being artificially limited by > >Linux to 5 at a time! This was solved with adding the -b20 (then later upping > > It *may* be worth upping it beyond the default, but the listen queue really > only comes into effect if tcpserver isn't keeping up with the inbound > connection rate. > > As long as tcpserver is doing the accept and passoff to qmail-smtpd in > enough time, the backlog doesn't apply. the concurrency with -c does of course. > > >If one server reaches this limit, it is overloaded and if lucky it is dropped > >out of the rotation by the alteon. This causes the other servers to overload > >and reach the same state. > > Is the alteon configured to load balance of switch on no response? > > Also, is it possible to bypass the Alteon for a while? There may be some > unknown interaction there. > > Regards. -- // Jere Cassidy - System Administration - D&E SuperNet email: [EMAIL PROTECTED]phone: (717)738-7054 web: http://www.desupernet.net/jere pager/pcs: [EMAIL PROTECTED] - (717)203-0042 ~~~ "While sowing the seeds of Utopia, you invoked a convenient amnesia" -BR ~~~
Re: Qmail (as seen on a not so good day).
YES, USE THE -c OPTION TO TCPSERVER. The -b sets the backlog. -c sets the number of simultaneous TCP sessions that tcpserver will process. You have increased the backlog but have not increased the number of sessions each server will accept. The default is 40! On Thu, 18 Feb 1999, Jere Cassidy wrote: > > What is wrong with the following setup: > > less than 30K customers: > > qmail 1.03 running on 3 high speed alpha's (each with 128MB ram) > Running 4 TCPSERVER daemon processes. > > > 1 SMTP (port 25) > 1 POP3 (port 110) > 1 SMTP (port 2001) > 1 POP3 (port 2002) > > These 3 servers running these 4 daemons share a Netapp filer for backend > storage. > > We have done major tuning to these servers time after time. Here is the > current situation: > 3 of the daemons run fine. the SMTP (on regular port 25) does not > respond. > > I have set the -b option for TCPserver(this helped us immensely before) > to 5000 (supposedly allowing tcpserver to respond to 5000 connections). > > Is there some default limit somewhere that would only allow tcpserver to > pass so many connections to qmail-smtpd? The downtime on the servers is > getting rediculous because of this problem. > > If I run /var/qmail/bin/qmail-smtpd it comes right up. > If I telnet localhost 2002 (simply another instance of tcpserver) it > comes right up. > Both POP3 connections come right up > > If i do a "netstat -n|grep ":25 " I get almost 700 connections although > most of these are in the "CLOSE WAIT" stage or something similar. > > On one of the servers, when this happens and qmail is totally > unresponsive on port 25, the load drops to 0.00 and the server just sits > there. > > restarting qmail seems to help for about 5 minutes... then the imaginary > limit is hit and everything goes to hell. > > Anyone have any suggestions for the current situation? > > > > > > > > > -- > > > // Jere Cassidy - System Administration - D&E SuperNet > email: [EMAIL PROTECTED]phone: (717)738-7054 > web: http://www.desupernet.net/jere > pager/pcs: [EMAIL PROTECTED] - (717)203-0042 > ~~~ "While sowing the seeds of Utopia, > you invoked a convenient amnesia" -BR ~~~ > > > > > - Timothy L. Mayo mailto:[EMAIL PROTECTED] Senior Systems Manager localconnect(sm) http://www.localconnect.net/ The National Business Network Inc. http://www.nb.net/ One Monroeville Center, Suite 850 Monroeville, PA 15146 (412) 810- Phone (412) 810-8886 Fax
Re: Qmail (as seen on a not so good day).
>Yes, 700 connections seems high, but after some period of down time, it seems to Just to emphasize what I was saying. I reckon for 30K users (was that the number-ish?), 700 is far higher than normal. I know there is no such thing as normal, but... >tcpserver -l$hostvalue -q -b100 -H -R -D 0 pop-3 >tcpserver -l$hostvalue -q -b50 -H -R -D 0 2001 >tcpserver -l$hostvalue -t8 -q -b5000 -D -u502 -g2108 >tcpserver -l$hostvalue -t8 -q -b50 -D -u502 -g2108 In all cases, change the -b to a -c >After reading "man listen" I am reminded of the help this list gave us when we >had this problem before. Our connections were being artificially limited by >Linux to 5 at a time! This was solved with adding the -b20 (then later upping It *may* be worth upping it beyond the default, but the listen queue really only comes into effect if tcpserver isn't keeping up with the inbound connection rate. As long as tcpserver is doing the accept and passoff to qmail-smtpd in enough time, the backlog doesn't apply. the concurrency with -c does of course. >If one server reaches this limit, it is overloaded and if lucky it is dropped >out of the rotation by the alteon. This causes the other servers to overload >and reach the same state. Is the alteon configured to load balance of switch on no response? Also, is it possible to bypass the Alteon for a while? There may be some unknown interaction there. Regards.
Re: Qmail (as seen on a not so good day).
Thanks for the help guys. I am going to post this info to the list for general review: Mark Delany wrote: > Sounds particularly nasty. It may well be a bug associated with establishing > tcp sessions. > > 700 outstanding sessions sounds awfully high. > > It may also be a router issue. Is it possible that the tcp sessions are > starting and not completing because certain packet types are being rejected? > > Btw. How are you getting 700 sessions? Do you have the concurrency on > tcpserver set *that* high? > > Regards. > John White wrote: > Uhhh... You definitely want to check out that tcpserver manpage again. > Especially the difference between -c and -b. > > Also, the listen manpage. > > It would greatly help if you would give me your current tcpserver > invocation for qmail-smtpd > > -- > John White > [EMAIL PROTECTED] > Yes, 700 connections seems high, but after some period of down time, it seems to me to be within the realm of possibilities. Our Alteon ACEdirector (load balancing switch) seemed to indicate approximately 600 connections and I had originally thgought that this number was artificially inflated due to some time-out value. The number that the Alteon showed was up as high as 1200 connections on a single server (when the other 2 were down). Here is the invocation of tcpserver: tcpserver -l$hostvalue -q -b100 -H -R -D 0 pop-3 /var/qmail/bin/qmail-popup $hostvalue /var/qmail/bin/CheckPasswd /var/qmail/bin/qmail-pop3d Maildir & tcpserver -l$hostvalue -q -b50 -H -R -D 0 2001 /var/qmail/bin/qmail-popup $hostvalue /var/qmail/bin/CheckPasswdVirtual cyb /var/qmail/bin/qmail-pop3d Maildir & tcpserver -l$hostvalue -t8 -q -b5000 -D -u502 -g2108 -x/var/qmail/control/tcprules.dat 0 smtp /var/qmail/bin/qmail-smtpd & tcpserver -l$hostvalue -t8 -q -b50 -D -u502 -g2108 -x/var/qmail/control/tcprules.dat 0 2002 /var/qmail/bin/qmail-smtpd & In case you are wondering, CheckPasswdVirtual (on port 2001) and SMTP (on port 2002) are actually accepting connections to a different switched address on the Alteon. CheckPasswdVirtual attaches a "cyb-" to their username so that they did not have to make any changes to accomodate a VirtualDomain type login. ohh.. and $hostvalue = mailX.desupernet.net where X = [124] After reading "man listen" I am reminded of the help this list gave us when we had this problem before. Our connections were being artificially limited by Linux to 5 at a time! This was solved with adding the -b20 (then later upping it to 30, then 50, then 100) . Last night, I grew frustrated with this and set it to 5000 on the port that was unresponsive (25). This had no effect. 'man listen' seems to indicate that 128 may be a max value this can be set to? Is that my actual limit? If one server reaches this limit, it is overloaded and if lucky it is dropped out of the rotation by the alteon. This causes the other servers to overload and reach the same state. SMTP connections will eventually get through, it just may take 5 or 10 or 15+ minutes for the 220 mail1.desupernet.net ESMTP line to come up. Thanks again for the help (I have been sleeping under my desk awaiting helpful replies :) ) If you need any other info let me know. > At 01:16 18/02/99 -0500, Jere Cassidy wrote: > > > >What is wrong with the following setup: > > > >less than 30K customers: > > > >qmail 1.03 running on 3 high speed alpha's (each with 128MB ram) > >Running 4 TCPSERVER daemon processes. > > > > > >1 SMTP (port 25) > >1 POP3 (port 110) > >1 SMTP (port 2001) > >1 POP3 (port 2002) > > > >These 3 servers running these 4 daemons share a Netapp filer for backend > >storage. > > > >We have done major tuning to these servers time after time. Here is the > >current situation: > >3 of the daemons run fine. the SMTP (on regular port 25) does not > >respond. > > > >I have set the -b option for TCPserver(this helped us immensely before) > >to 5000 (supposedly allowing tcpserver to respond to 5000 connections). > > > >Is there some default limit somewhere that would only allow tcpserver to > >pass so many connections to qmail-smtpd? The downtime on the servers is > >getting rediculous because of this problem. > > > >If I run /var/qmail/bin/qmail-smtpd it comes right up. > >If I telnet localhost 2002 (simply another instance of tcpserver) it > >comes right up. > >Both POP3 connections come right up > > > >If i do a "netstat -n|grep ":25 " I get almost 700 connections although > >most of these are in the "CLOSE WAIT" stage or something similar. > > > >On one of the servers, when this happens and qmail is totally > >unresponsive on port 25, the load drops to 0.00 and the server just sits > >there. > > > >restarting qmail seems to help for about 5 minutes... then the imaginary > >limit is hit and everything goes to hell. > > > >Anyone have any suggestions for the current situation? > > > > > > > > > > > > > > > > > >-- > >---
