Thanks all for the suggestions. Been looking at things in more detail:
1) Im not sure how many sessions we are handling. I do now we were
maxing out at 120 connections per sec at peak times.
2) we do have spamc and spamd running.
spamd --max-children 25 -x -v -d --pidfile=/var/run/spamd.pid
25 childs enough?
3) Running vpopmail and not using mysql.
4)We do have todo patch installed.
5) LOG Files:
a) Mail.log >>at a glance these are all legitimate users with hosted
domains on the server. looks pretty normal.
b) Mail.err >> theres a lot of this entry: pop3d: Maximum connection
limit reached for ::ffff:188.8.131.52
Looking at these IP's they correspond to the IP numbers of my
country's ISP's through which most of my users connect to the
internet, so that would seem to makes sense. Leaving number of max
connections per IP as is for now.
I found a lot of this as well: imapd: /usr/lib/courier-imap/etc/
shared/index: No such file or directory. (PS. all catch-alls are
set to bounce, but I dont know if this is related in any way).
c) simlog: there are quite a few, actually a LOT, of connect error 2
Traced it back to p0f fingerprinting. Have turned it off and have
also disabled checking mail from local users to the outside.
Have simscan/SA/Clam running smoothly for about an hour now. Will
wait for a peak in email traffic see how it handles it.
Again. Thanks to all for observations and suggestions so far. I
will continue to look at this and post back anything that may be useful.