Re: [vchkpw] Qmail with Simscan, SA and ClamAv
"but I now have a Core2Duo server dedicated to SpamAssassin" Same here, I have a dual xeon w/simscan/clam/SA. 5700 email users. About 20% busy most of the time. It scans then passes mail onto a old p3-4cpu main mailbox. Never miss an email again! Yahoo! Toolbar alerts you the instant new Mail arrives. http://tools.search.yahoo.com/toolbar/features/mail/
Re: [vchkpw] Qmail with Simscan, SA and ClamAv
On Wed, 2007-01-31 at 13:11 -0500, Rick Macdougall wrote: > Max Esquivel wrote: > > Thanks all for the suggestions. Been looking at things in more detail: > > > > 1) Im not sure how many sessions we are handling. I do now we were > > maxing out at 120 connections per sec at peak times. > > > > 2) we do have spamc and spamd running. > > spamd --max-children 25 -x -v -d --pidfile=/var/run/spamd.pid > > 25 childs enough? > > > > max children 25 and 1 gig of memory is not going to work. Once you > start swapping every thing is going to slow to a halt. Just to elaborate on that - I figure 50MB per child (mine currently vary from 12 - 40), so 25 children would require at least a gig. Honestly, mine is set to 32, and I only have 1 gb - but it's a dedicated server, and it was a pain to tweak it to where it is now. I definitely need more Ram. > I'd increase the amount of ram (we run 4 gig on all our SA servers) and > see if it would be possible to get a separate machine to just run spamd on. > > Regards, > > Rick >
Re: [vchkpw] Qmail with Simscan, SA and ClamAv
Max Esquivel wrote: Thanks all for the suggestions. Been looking at things in more detail: 1) Im not sure how many sessions we are handling. I do now we were maxing out at 120 connections per sec at peak times. 2) we do have spamc and spamd running. spamd --max-children 25 -x -v -d --pidfile=/var/run/spamd.pid 25 childs enough? max children 25 and 1 gig of memory is not going to work. Once you start swapping every thing is going to slow to a halt. I'd increase the amount of ram (we run 4 gig on all our SA servers) and see if it would be possible to get a separate machine to just run spamd on. Regards, Rick
Re: [vchkpw] Qmail with Simscan, SA and ClamAv
Thanks all for the suggestions. Been looking at things in more detail: 1) Im not sure how many sessions we are handling. I do now we were maxing out at 120 connections per sec at peak times. 2) we do have spamc and spamd running. spamd --max-children 25 -x -v -d --pidfile=/var/run/spamd.pid 25 childs enough? 3) Running vpopmail and not using mysql. 4)We do have todo patch installed. 5) LOG Files: a) Mail.log >>at a glance these are all legitimate users with hosted domains on the server. looks pretty normal. b) Mail.err >> theres a lot of this entry: pop3d: Maximum connection limit reached for :::201.194.10.118 Looking at these IP's they correspond to the IP numbers of my country's ISP's through which most of my users connect to the internet, so that would seem to makes sense. Leaving number of max connections per IP as is for now. I found a lot of this as well: imapd: /usr/lib/courier-imap/etc/ shared/index: No such file or directory. (PS. all catch-alls are set to bounce, but I dont know if this is related in any way). c) simlog: there are quite a few, actually a LOT, of connect error 2 messages. Traced it back to p0f fingerprinting. Have turned it off and have also disabled checking mail from local users to the outside. Have simscan/SA/Clam running smoothly for about an hour now. Will wait for a peak in email traffic see how it handles it. Again. Thanks to all for observations and suggestions so far. I will continue to look at this and post back anything that may be useful. Max
Re: [vchkpw] Qmail with Simscan, SA and ClamAv
On Wed, 2007-01-31 at 11:22 -0500, Darrel O'Pry wrote: > On Tue, 2007-01-30 at 19:14 -0600, Max Esquivel wrote: > > I have also posted this to vchkpw list: > > > > I have a server with qmail running some 600 email accounts over some > > 30 domains. I recently installed simscan, Spamassassin and ClamAv. > > It all works really well, but during peak hours (say 300 to 500k per > > sec inbound traffic) Thee server starts to bog down and progressively > > gets slower and slower until 120 connections are maxed out and the > > server starts rejecting smtp connections first and then pop > > connections. This is a new AMD 64 bit with 1Gig Ram running on > > Debian and running also Apache with php, mysql, and Horde webmail > > (with very very few hits per day). If I turn off simscan, situation > > returns to normal after a while. I have tried finding some > > documentation about how many users and traffic qmail with simscan, SA > > and Clam may handle, but it seems there is nothing out there other > > than very general stuff like "many users", "thousands of users" > > > > Perhaps the problem is in my setup and some configuration for > > simscan, SA or clam that I have set/not set incorrectly, ot I have > > not realized this number of users and trafffic is just too much for > > one server. Any suggestions or links to appropriate docs will be most > > appreciated. > > > > Thanks! > > > > Max Esquivel > > I was having a similar issue. For my environment it was being caused by > large attachments. I ended up only using simscan to call clamav and set > some basic attachment blocking policy. I was also having that issue, about 2000 regular users. I have multiple servers, but the 'last' thing I did might help you the most. I used to share SpamAssassin with my secondary MX, and data store, but I now have a Core2Duo server dedicated to SpamAssassin. It was just eating up way too many resources - even on that low usage box. My primary MX averages about 300k per sec, 1000 msgs/hour and is only a Duron 1400. Looks like it peaked at 2800 msgs yesterday at 3pm CST on the dot ;) No more slow scanning issues. There are also some SA optimizations to speed up scans. Put: dns_available yes rbl_timeout 10 razor_timeout 5 pyzor_timeout 5 check_mx_attempts 1 in your SpamAssassin local.cf and run dnscache locally. And make sure the SpamAssassin logs don't have 'Unable to read bayes_seen' errors - that's a killer too. Rick
Re: [vchkpw] Qmail with Simscan, SA and ClamAv
On Tue, 2007-01-30 at 19:14 -0600, Max Esquivel wrote: > I have also posted this to vchkpw list: > > I have a server with qmail running some 600 email accounts over some > 30 domains. I recently installed simscan, Spamassassin and ClamAv. > It all works really well, but during peak hours (say 300 to 500k per > sec inbound traffic) Thee server starts to bog down and progressively > gets slower and slower until 120 connections are maxed out and the > server starts rejecting smtp connections first and then pop > connections. This is a new AMD 64 bit with 1Gig Ram running on > Debian and running also Apache with php, mysql, and Horde webmail > (with very very few hits per day). If I turn off simscan, situation > returns to normal after a while. I have tried finding some > documentation about how many users and traffic qmail with simscan, SA > and Clam may handle, but it seems there is nothing out there other > than very general stuff like "many users", "thousands of users" > > Perhaps the problem is in my setup and some configuration for > simscan, SA or clam that I have set/not set incorrectly, ot I have > not realized this number of users and trafffic is just too much for > one server. Any suggestions or links to appropriate docs will be most > appreciated. > > Thanks! > > Max Esquivel I was having a similar issue. For my environment it was being caused by large attachments. I ended up only using simscan to call clamav and set some basic attachment blocking policy. I call SpamAssassin through procmail, but only on messages smaller than 250k to avoid scanning large media attachments. I also get user_prefs working this way. I still haven't figured out hot to get user prefs to work on aliases though. :( .darrel.
Re: [vchkpw] Qmail with Simscan, SA and ClamAv
> I have also posted this to vchkpw list: > > I have a server with qmail running some 600 email accounts over some > 30 domains. I recently installed simscan, Spamassassin and ClamAv. > It all works really well, but during peak hours (say 300 to 500k per > sec inbound traffic) Thee server starts to bog down and progressively > gets slower and slower until 120 connections are maxed out and the > server starts rejecting smtp connections first and then pop > connections. This is a new AMD 64 bit with 1Gig Ram running on > Debian and running also Apache with php, mysql, and Horde webmail > (with very very few hits per day). If I turn off simscan, situation > returns to normal after a while. I have tried finding some > documentation about how many users and traffic qmail with simscan, SA > and Clam may handle, but it seems there is nothing out there other > than very general stuff like "many users", "thousands of users" > > Perhaps the problem is in my setup and some configuration for > simscan, SA or clam that I have set/not set incorrectly, ot I have > not realized this number of users and trafffic is just too much for > one server. Any suggestions or links to appropriate docs will be most > appreciated. > One thing that I found helpful was to put the simscan temporary directory onto a ramdisk. I have /var/qmail/simscan mounted as tmpfs, forcing the mode to 750, uid to simscan and gid to vchkpw (I use Vpopmail, YMMV), and specifying the size to 1G (my box has 1G ram and 2G swap, so the default tmpfs size is only 512M). Since anything put there is transient by definition, if I have a power failure and the contents die I lose nothing. And since simscan cleans up after itself, as long as you're not getting large numbers of very large emails all at once, it rarely forces the tmpfs to hit swap. I actually have a server with the same memory config that runs 4 separate instances of qmail on separate IPs (consolidation of multiple servers into one box), and "idling" it only uses roughly half a gig of RAM. That's for 4 instances each of qmail and clamd, plus an assortment of other daemons used only for one instance or another and a MySql DB to hold it all together. It bogs down a little on occasion when a large list goes through, but that's all CPU from virus scanning hundreds of emails at once. One other suggestion - if you don't already use it, patch qmail with the external-todo patch - it speeds up send of mail while processing the queue... Josh -- Joshua Megerman SJGames MIB #5273 - OGRE AI Testing Division You can't win; You can't break even; You can't even quit the game. - Layman's translation of the Laws of Thermodynamics [EMAIL PROTECTED]
RE: [vchkpw] Qmail with Simscan, SA and ClamAv
whats your incomming and out going sessions are u using spamc and a spamd ? if so r u running sufficient spamd clients for the qty of incoming... most mail we receive is proced calmav/spamassasin in under 5s with a box <50% your power... but id increase your ram by 2x... are u running rbls b4 the spam checks? kenneth gf brown ceo shadowplay.net > -Original Message- > From: Max Esquivel [mailto:[EMAIL PROTECTED] > Sent: January 30, 2007 20:15 > To: vchkpw@inter7.com > Subject: [vchkpw] Qmail with Simscan, SA and ClamAv > > > I have also posted this to vchkpw list: > > I have a server with qmail running some 600 email accounts over some > 30 domains. I recently installed simscan, Spamassassin and ClamAv. > It all works really well, but during peak hours (say 300 to 500k per > sec inbound traffic) Thee server starts to bog down and > progressively > gets slower and slower until 120 connections are maxed out and the > server starts rejecting smtp connections first and then pop > connections. This is a new AMD 64 bit with 1Gig Ram running on > Debian and running also Apache with php, mysql, and Horde webmail > (with very very few hits per day). If I turn off simscan, situation > returns to normal after a while. I have tried finding some > documentation about how many users and traffic qmail with > simscan, SA > and Clam may handle, but it seems there is nothing out there other > than very general stuff like "many users", "thousands of users" > > Perhaps the problem is in my setup and some configuration for > simscan, SA or clam that I have set/not set incorrectly, ot I have > not realized this number of users and trafffic is just too much for > one server. Any suggestions or links to appropriate docs will > be most > appreciated. > > Thanks! > > Max Esquivel > >
