Re: polw falling behind?

2006-12-04 Thread Robert Felber
On Sat, Dec 02, 2006 at 07:17:03AM -0600, Len Conrad wrote:
 What are the average delays from a client connecting to the 
 smtpd
 and an answer from policyd-weight?
 
 Where do I find that info?

This can only be guessed by lines like:

Dec  4 00:07:40 fpsvr1z150 postfix/smtpd[31885]: connect from 
unknown[85.137.11.234]

Dec  4 00:07:44 fpsvr1z150 postfix/policyd-weight[65787]: weighted check:  
IN_SBL_XBL_SPAMHAUS=4.35 NOT_IN_BL_NJABL=-1.5 NOT_IN_SPAMCOP=-1.5 
CL_IP_NE_HELO=5.85 RESOLVED_IP_IS_NOT_HELO=1.5 (check from: 
.01-flash-web-templates. - helo: .fran-r0pedpv5aw. - helo-domain: 
.fran-r0pedpv5aw.)  FROM_NOT_FAILED_HELO(DOMAIN)=7.35 client=85.137.11.234 
helo=fran-r0pedpv5aw [EMAIL PROTECTED] [EMAIL PROTECTED], rate: 16.05


In above case it took 4 seconds to answer.

 
 What's your kernel setting for so_maxconn (linux: sysctl 
 net.core.somaxconn)
 
 kern.ipc.somaxconn: 128

You should increase this value to 512 or 1024.

From your Recv-Q entries it seems to run ok. It seems that your machine runs
out of tcp slots. Thus increase kern.ipc.somaxconn.

As of the moment policyd-weight cannot do asynchronous DNS lookups. Thus
one policyd-weight instance cannot handle 2 smtpd clients at once but must
process them serial. Doing that async with one process is a very hard task
and I fear that it will impossible with the current design.
However, if a policyd-weight instance is busy then a new smtpd client will
cause policyd-weight to create a new instance, including required sockets.


-- 
Robert Felber (PGP: 896CF30B)
Munich, Germany


Policyd-weight Mailinglist - http://www.policyd-weight.org/


Re: polw falling behind?

2006-12-02 Thread Robert Felber
On Fri, Dec 01, 2006 at 12:23:08PM -0600, Len Conrad wrote:
 For today:
 mx1# egrep -ic ^... .. 10.*problem talking to server 127.0.0.1:12525 
 /var/log/maillog
 3849
 
 ... stop/start policyd-weight, then
 
 mx1# egrep -ic ^... .. 11.*problem talking to server 127.0.0.1:12525 
 /var/log/maillog
 0
 
 What's going on?

No one can tell by that line of log.
 
How many smtpd processes exist at such times,
how many policyd-weight process exist at such times?

What's you operating system?

What are the average delays from a client connecting to the smtpd
and an answer from policyd-weight?


Where there warning:s or err:s before?

What's your kernel setting for so_maxconn (linux: sysctl net.core.somaxconn)

whats the output of 
bsd:netstat -na | grep '.12525'
linux:  netstat -pna | grep '.12525'
at such times.


If you see in the netstat output above a process with a huge Recv-Q
like:

PROTO   Recv-Q Send-Q  Local Address  Foreign AddressState  
 PID/Program name
tcp1234567  0  127.0.0.1.12525127.0.0.1.55710
ESTABLISHED 24216/policyd-weight

then I would appreciate an strace -fp $PID output 
where $PID is the PID of the process with the huge Recv-Q


Please cosnider that there is alaways a different approach when using different 
OSes, so please
tell allways your OS and kernel version (uname -a). The Perl version is handy, 
too!


P.s.: I am busy and not available until monday.



-- 
Robert Felber (PGP: 896CF30B)
Munich, Germany


Policyd-weight Mailinglist - http://www.policyd-weight.org/


Re: polw falling behind?

2006-12-02 Thread Len Conrad



How many smtpd processes exist at such times,
how many policyd-weight process exist at such times?


the problem is not occurring today in the 1000s, but we know what you 
want to know for the next time


the problem is pretty much continuous but so bad as reported 
yesterday. For today:


mx1# egrep -i problem talking to server 127.0.0.1:12525 /var/log/maillog
Dec  2 00:16:05 mx1 postfix/smtpd[58432]: warning: problem talking to 
server 127.0.0.1:12525: Operation timed out
Dec  2 00:16:12 mx1 postfix/smtpd[58383]: warning: problem talking to 
server 127.0.0.1:12525: Operation timed out
Dec  2 00:17:46 mx1 postfix/smtpd[58432]: warning: problem talking to 
server 127.0.0.1:12525: Operation timed out
Dec  2 00:17:46 mx1 postfix/smtpd[58414]: warning: problem talking to 
server 127.0.0.1:12525: Operation timed out
Dec  2 01:06:48 mx1 postfix/smtpd[59002]: warning: problem talking to 
server 127.0.0.1:12525: Operation timed out
Dec  2 01:06:50 mx1 postfix/smtpd[58966]: warning: problem talking to 
server 127.0.0.1:12525: Operation timed out
Dec  2 01:08:29 mx1 postfix/smtpd[59002]: warning: problem talking to 
server 127.0.0.1:12525: Operation timed out
Dec  2 03:50:30 mx1 postfix/smtpd[60786]: warning: problem talking to 
server 127.0.0.1:12525: Operation timed out
Dec  2 03:50:45 mx1 postfix/smtpd[60774]: warning: problem talking to 
server 127.0.0.1:12525: Operation timed out


mx1# ps aux | egrep -ic policy
32

mx1# ps aux | egrep -ic smtpd
66

the traffic per hour:

Per-Hour Traffic Summary
timerecevd delivd deferd bouncd rj_4xx rj_5xx errlim rj_anv
-0100 1658   1885 94 38   2960   5063420   2213
0100-0200 1477   1620105 42   2682   4703454   5021
0200-0300 1290   1325 98  0   2722   5517573   4315
0300-0400 1523   1524142  2   2967   6312492  10121
0400-0500 1479   1480123  3   2668   7222492   3004
0500-0600 1398   1400120  0   2507   7691448   5568
0600-0700 1632   1669 40  3   2754   6300435   8563
0700-0800 1655   1672112  2   2436   6368508   9950
0800-09000  0  0  0  0  0  0  0

cpu is 1.7 GHz, 1 GB RAM, disk is ATA100, OS is fbsd 6.1R

policyd-weight is:

# 
#  policyd-weight configuration (defaults) Version 0.1.14 beta
# 

#my $DEBUG= 0;   # 1 or 0 - don't comment
   $DEBUG   = 0;

   $REJECTMSG= 550 Mail appeared to be SPAM or forged. Ask your 
Mail/DNS-Administrator to correct HELO and DNS MX settings or to get 
removed from DNSBLs;


   $REJECTLEVEL  = 15;
#my $REJECTLEVEL  = 1;   # Mails with scores which exceed this
# REJECTLEVEL will be rejected


   $DEFER_ACTION = '450';   # Possible values: DEFER_IF_PERMIT,
# DEFER_IFREJECT,
# 4xx response codes. See also access(5)
# DEFAULT: 450
   $DEFER_LEVEL  = 20;
#my $DEFER_LEVEL  = 5;   # DEFER mail only up to this level
# scores greater than DEFER_LEVEL will be
# rejected
# DEFAULT: 5

   $DNSERRMSG = '450 No DNS entries for your MTA, HELO and 
Domain. Contact YOUR administrator';

   $dnsbl_checks_only = 0;  # 1: ON, 0: OFF (default)
   $LOG_BAD_RBL_ONLY  = 1;  # 1: ON (default), 0: OFF

## DNSBL settings
   @dnsbl_score = (
#HOST,BAD SCORE,  GOOD SCORE,   LOG NAME
'dynablock.njabl.org',2.00,  0,'DYN_NJABL',
#'sbl-xbl.spamhaus.org',   4.35,   0,'SBL_XBL_SPAMHAUS',
'zen.spamhaus.org',   13.00,   0,   'ZEN_SPAMHAUS',
'bl.spamcop.net', 13.00,   0,'SPAMCOP',
'dnsbl.njabl.org',4.00,   0,'BL_NJABL',
'dnsbl.sorbs.net',0.50,   0,'BL_SORBS',
'list.dsbl.org',  4.35,  0,'DSBL_ORG',
#'ix.dnsbl.manitu.net',4.35,  0,'IX_MANITU',
'pub.mxrate.net', 2.25,   0,'MXRATE',
'relays.ordb.org',3.25,  0,'ORDB_ORG'
);

   $MAXDNSBLHITS  = 2;  # If Client IP is listed in MORE
# DNSBLS than this var, it gets
# REJECTed immediately

  $MAXDNSBLSCORE = 15;
#  $MAXDNSBLSCORE = 8;  # alternatively, if the score of
# DNSBLs is ABOVE this
# level, reject immediately


   $MAXDNSBLMSG   = '550 Your MTA is listed in too many DNSBLs';

## RHSBL settings
   @rhsbl_score = (
#'multi.surbl.org', 4,0,'SURBL',
'multi.surbl.org',  0,  0,  'SURBL',