I am a bit concerned about the performance of amavisd.
The server will be lightly loaded and then I'll get a number of these
deferrals. When there is a lot of traffic this can grow to several hundred.
Also the amavis scanning is taking its sweet time.
I have 2 front-end MX servers and 2-Outbound smtp servers. These servers are
the content scanning servers. I originally had 2 but at peak hours they
would run far behind. I have since then added 2 more servers for a total of
4. Mail traffic is about 150,452 per day averaging 6,200 per hour. Peaks are
around 10,000 an hour. Average message size is 32K. During peak hours the
average message size will be between 45K and 50K. Below are relevant postfix
settings. I currently have bayes and auto white listing turned off. No extra
rules other than the ones that come with the install. One of the servers I'm
running 20 max servers, One of the other I'm running 7 max servers. Not much
difference in problems with either server. Since I have lightened the load
on the servers they do not get behind very often. But I had hoped they could
process more than a few thousand per hour. I wrote my own awk script to
create a report on the server performance. I will included the awk code so
you can see what I am counting to get my numbers.
Any help would be appreciated. Sorry if this email is lengthy, but you need
to see some data to make any observations.
Ed
Content scanning Configuration
------------------------------
Dell 1850
2G memory
1-Xeon 3.4 ghz processor
Mirrored Ultra Scsi 10,000 rpm disk drives.
OS is FreeBSD 6 running a SMP enabled kernel.
Postfix 2.3-20060126
Amavisd-new amavisd-new-2.3.3 (20050822)
Clamav ClamAV 0.88/1289/Tue Feb 14 06:36:44 2006
Awk Report
----------
Feb 14 Content Scanning for cs1
Time to deliver to amavis via smtp
Time
Hour Messages Average Max 30-59 60-119 120+
--------------------------------------------------------------
00 1808 1.6 8.0 0 0 0
01 1359 1.8 6.6 0 0 0
02 1169 1.8 7.5 0 0 0
03 1344 1.8 8.5 0 0 0
04 1467 1.8 15.0 0 0 0
05 1648 1.9 8.9 0 0 0
06 2247 1.7 6.2 0 0 0
07 2290 1.7 8.0 0 0 0
08 2373 2.0 20.0 0 0 0
09 2348 1.8 7.5 0 0 0
10 2524 4.3 1203.0 0 0 6
11 2293 1.9 23.0 0 0 0
12 2276 6.0 1206.0 0 0 8
13 2165 1.7 8.0 0 0 0
14 2193 1.7 7.7 0 0 0
15 1955 1.7 7.7 0 0 0
16 1671 2.1 603.0 0 0 1
17 1585 1.7 7.8 0 0 0
18 2102 1.4 7.7 0 0 0
19 2016 1.4 7.8 0 0 0
20 1239 3.6 1197.0 0 0 2
21 1233 1.8 7.0 0 0 0
22 1274 1.7 5.6 0 0 0
23 1632 1.5 6.7 0 0 0
----------------------------------------------------------------------------
----------
Total 44211 2.2 1206.0
Time to deliver messages to mail servers
Time
Hour Messages Average Max 30-59 60-119 120+
--------------------------------------------------------------
00 1929 2.0 1193.0 0 0 2
01 1399 0.7 12.0 0 0 0
02 1237 0.7 12.0 0 0 0
03 1389 0.7 10.0 0 0 0
04 1536 0.7 7.6 0 0 0
05 1739 0.8 9.8 0 0 0
06 2220 25.4 643.0 149 145 175
07 2346 19.8 1280.0 129 84 126
08 2336 118.0 1504.0 50 85 707
09 2510 25.2 1005.0 1 4 107
10 2853 38.6 1745.0 0 0 105
11 2681 0.6 16.0 0 0 0
12 2609 0.6 9.6 0 0 0
13 2388 0.6 11.0 0 0 0
14 2436 0.8 33.0 1 0 0
15 2106 0.6 15.0 0 0 0
16 1747 0.4 4.5 0 0 0
17 1637 0.4 5.1 0 0 0
18 2110 7.6 1285.0 124 93 1
19 2118 5.4 1071.0 50 17 4
20 1298 0.4 6.8 0 0 0
21 1257 0.4 7.0 0 0 0
22 1300 0.4 2.7 0 0 0
23 1686 3.6 120.0 54 13 0
----------------------------------------------------------------------------
----------
Total 46867 12.9 1745.0
Content scanning performance
Scan Time Passed
Blocked
Hour Messages Average Max Clean Spam Bad_Header Infected
Banned Spam
----------------------------------------------------------------------------
----------
00 1818 1.6 7.7 1384 413 16 3
2 0
01 1360 1.8 6.6 874 464 20 0
2 0
02 1222 1.7 7.5 825 385 11 1
0 0
03 1352 1.8 8.5 820 515 13 3
1 0
04 1493 1.8 15.0 972 500 17 2
2 0
05 1687 1.9 8.9 1164 497 16 8
2 0
06 2411 1.6 6.2 1883 505 10 10
3 0
07 2597 1.6 8.0 2087 486 15 5
4 0
08 2818 2.4 20.8 2273 515 19 7
2 0
09 2676 1.7 7.5 1987 664 21 3
1 0
10 2536 1.8 9.5 1942 577 15 1
0 0
11 2516 1.9 25.1 2048 457 7 4
0 0
12 2522 1.8 13.3 1957 536 19 8
2 0
13 2205 1.7 8.0 1761 428 10 2
4 0
14 2284 1.7 8.6 1719 537 24 2
2 0
15 1957 1.7 7.7 1418 479 52 8
0 0
16 1660 1.7 14.5 1133 434 89 3
1 0
17 1562 1.7 7.8 1020 466 71 3
2 0
18 2264 1.3 7.7 1849 366 43 3
3 0
19 2148 1.3 9.3 1742 388 16 2
0 0
20 1251 1.8 12.8 838 394 10 7
2 0
21 1192 1.7 6.9 775 397 15 3
2 0
22 1224 1.7 5.6 802 412 8 1
1 0
23 1736 1.4 6.7 1369 359 2 4
2 0
----------------------------------------------------------------------------
----------
Total 46491 1.7 25.1 34642 11174 539 93
40 0
Awk Code
--------
{
if (NR==1){printf("%s %s Content Scanning for %s\n\n",$1,$2,SYS)}
HR=substr($3,1,2)
if (substr($5,9,5)=="smtp[" && substr($8,1,26) ==
"relay=127.0.0.1[127.0.0.1]" && substr($9,1,6) == "delay=") {
ia=length($9)-7
iMsg=substr($9,7,ia)
iTot[HR]+=iMsg
iCnt[HR]++
iAT+=iMsg
iCT++
if ( iMsg > 30 && iMsg < 60 ) iGT1[HR]++
if ( iMsg > 60 && iMsg < 120 ) iGT2[HR]++
if ( iMsg > 120 ) iGT3[HR]++
if ( iMsg > iMax[HR] ) { iMax[HR]=iMsg }
if ( iMsg > iTM ) { iTM=iMsg }
}
if (substr($5,9,5)=="smtp[" && substr($8,1,26) !=
"relay=127.0.0.1[127.0.0.1]" && substr($9,1,6) == "delay=") {
oa=length($9)-7
oMsg=substr($9,7,oa)
oHR=substr($3,1,2)
oTot[HR]+=oMsg
oCnt[HR]++
oAT+=oMsg
oCT++
if ( oMsg > 30 && oMsg < 60 ) oGT1[HR]++
if ( oMsg > 60 && oMsg < 120 ) oGT2[HR]++
if ( oMsg > 120 ) oGT3[HR]++
if ( oMsg > oMax[HR] ) { oMax[HR]=oMsg }
if ( oMsg > oTM ) { oTM=oMsg }
}
if ($7=="Passed" || $7=="Blocked") {
Msg=$(NF-1)
HR=substr($3,1,2)
Tot[HR]+=Msg
Cnt[HR]++
AT+=Msg
CT++
a=index($8,",")
if (a==0) {w=$8}
else {w=substr($8,1,length($8)-1)}
TYPE=HR "-" $7 "-" w
T[TYPE]++
TTA=$7 "-" w
TT[TTA]++
if ( Msg > Max[HR] ) { Max[HR]=Msg }
if ( Msg > TM ) { TM=Msg }
}
}
END {
print "Time to deliver to amavis via smtp"
print " "
print " Time"
print " Hour Messages Average Max 30-59 60-119 120+"
print "--------------------------------------------------------------"
for (y=0;y<=23;y++) {
i=sprintf("%02d",y)
if(iCnt[i]==0){continue}
printf(" %s %7d %5.1f %6.1f %6d %6d
%6d\n",i,iCnt[i],iTot[i]/iCnt[i],iMax[i],iGT1[i],iGT2[i],iGT3[i])
}
print
"---------------------------------------------------------------------------
-----------"
printf("Total %7d %5.1f %6.1f\n\n",iCT,iAT/iCT,iTM)
print "Time to deliver messages"
print " "
print " Time"
print " Hour Messages Average Max 30-59 60-119 120+"
print "--------------------------------------------------------------"
for (y=0;y<=23;y++) {
i=sprintf("%02d",y)
if(oCnt[i]==0){continue}
printf(" %s %7d %5.1f %6.1f %6d %6d
%6d\n",i,oCnt[i],oTot[i]/oCnt[i],oMax[i],oGT1[i],oGT2[i],oGT3[i])
}
print
"---------------------------------------------------------------------------
-----------"
printf("Total %7d %5.1f %6.1f\n\n",oCT,oAT/oCT,oTM)
print "Content scanning performance"
print " "
print " Scan Time Passed
Blocked"
print " Hour Messages Average Max Clean Spam Bad_Header
Infected Banned Spam"
print
"---------------------------------------------------------------------------
-----------"
for (y=0;y<=23;y++) {
i=sprintf("%02d",y)
if(Cnt[i]==0){continue}
printf(" %s %7d %4.1f
%4.1f",i,Cnt[i],(Tot[i]/Cnt[i])/1000,Max[i]/1000)
printf(" %6d %6d
%6d",T[i"-Passed-CLEAN"],T[i"-Passed-SPAM"],T[i"-Passed-BAD-HEADER"])
printf(" %6d %6d
%6d\n",T[i"-Blocked-INFECTED"],T[i"-Blocked-BANNED"],T[i"-Blocked-SPAM"])
}
print
"---------------------------------------------------------------------------
-----------"
printf("Total %7d %4.1f %4.1f",CT,(AT/CT)/1000,TM/1000)
printf(" %6d %6d
%6d",TT["Passed-CLEAN"],TT["Passed-SPAM"],TT["Passed-BAD-HEADER"])
printf(" %6d %6d
%6d\n",TT["Blocked-INFECTED"],TT["Blocked-BANNED"],TT["Blocked-SPAM"])
}
Amavisd.conf
------------
use strict;
#
# $Header: /usr/local/ic/config/files/RCS/amavisd.conf,v 1.3 2005/12/20
20:10:37 root Exp root $
#
# a minimalistic configuration file for amavisd-new with all necessary
settings
#
# see amavisd.conf-default for a list of all variables with their
defaults;
# see amavisd.conf-sample for a traditional-style commented file;
# for more details see documentation in INSTALL, README_FILES/*
# and at http://www.ijs.si/software/amavisd/amavisd-new-docs.html
# COMMONLY ADJUSTED SETTINGS:
# @bypass_virus_checks_maps = (1); # uncomment to DISABLE anti-virus code
# @bypass_spam_checks_maps = (1); # uncomment to DISABLE anti-spam code
$max_servers = 20; # number of pre-forked children (2..15 is
common)
$daemon_user = 'vscan'; # (no default; customary: vscan or amavis)
$daemon_group = 'vscan'; # (no default; customary: vscan or amavis)
$mydomain = 'sandiegort.com'; # a convenient default for other settings
$MYHOME = '/var/amavis'; # a convenient default for other settings
$TEMPBASE = "$MYHOME/tmp"; # working directory, needs to be created
manually
#$TEMPBASE = "/var/mwk"; # working directory, needs to be created manually
$ENV{TMPDIR} = $TEMPBASE; # environment variable TMPDIR
$QUARANTINEDIR = '/var/amavis/virusmails';
# $quarantine_subdir_levels = 1; # add level of subdirs to disperse
quarantine
# $daemon_chroot_dir = $MYHOME; # chroot directory or undef
# $db_home = "$MYHOME/db";
# $helpers_home = "$MYHOME/var"; # prefer $MYHOME clean and owned by root?
# $pid_file = "$MYHOME/var/amavisd.pid";
# $lock_file = "$MYHOME/var/amavisd.lock";
#NOTE: create directories $MYHOME/tmp, $MYHOME/var, $MYHOME/db manually
[EMAIL PROTECTED] = ( [".$mydomain", '.barnett.net'] );
@local_domains_maps = ( read_hash("$MYHOME/domains.map") ); # using hash
# @mynetworks = qw( 127.0.0.0/8 [::1] [FE80::]/10 [FEC0::]/10
# 10.0.0.0/8 172.16.0.0/12 192.168.0.0/16 );
$log_level = 2; # verbosity 0..5
$log_recip_templ = undef; # disable by-recipient level-0 log entries
$DO_SYSLOG = 1; # log via syslogd (preferred)
$enable_db = 1; # enable use of BerkeleyDB/libdb (SNMP and
nanny)
$enable_global_cache = 1; # enable use of libdb-based cache if
$enable_db=1
$inet_socket_port = 10024; # listen on this local TCP port(s) (see
$protocol)
# $unix_socketname = "$MYHOME/amavisd.sock"; # when using sendmail milter
$sa_tag_level_deflt = 2.0; # add spam info headers if at, or above that
level
$sa_tag2_level_deflt = 6.31; # add 'spam detected' headers at that level
$sa_kill_level_deflt = 6.31; # triggers spam evasive actions
$sa_dsn_cutoff_level = 9; # spam level beyond which a DSN is not sent
# $sa_quarantine_cutoff_level = 20; # spam level beyond which quarantine is
off
$sa_mail_body_size_limit = 200*1024; # don't waste time on SA if mail is
larger
$sa_local_tests_only = 0; # only tests which do not require internet
access?
$sa_auto_whitelist = 1; # turn on AWL in SA 2.63 or older (irrelevant
# for SA 3.0, cf option is
'use_auto_whitelist')
# @lookup_sql_dsn =
# ( ['DBI:mysql:database=mail;host=127.0.0.1;port=3306', 'user1',
'passwd1'],
# ['DBI:mysql:database=mail;host=host2', 'username2', 'password2'],
# ["DBI:SQLite:dbname=$MYHOME/sql/mail_prefs.sqlite", '', ''] );
# @storage_sql_dsn = @lookup_sql_dsn; # none, same, or separate database
$virus_admin = "[EMAIL PROTECTED]"; # notifications
recip.
$mailfrom_notify_admin = "[EMAIL PROTECTED]"; # notifications
sender
$mailfrom_notify_recip = "[EMAIL PROTECTED]"; # notifications
sender
$mailfrom_notify_spamadmin = "[EMAIL PROTECTED]"; # notifications
sender
$mailfrom_to_quarantine = ''; # null return path; uses original sender if
undef
@addr_extension_virus_maps = ('virus');
@addr_extension_spam_maps = ('spam');
@addr_extension_banned_maps = ('banned');
@addr_extension_bad_header_maps = ('badh');
# $recipient_delimiter = '+'; # undef disables address extensions
altogether
# when enabling addr extensions do also Postfix/main.cf:
recipient_delimiter=+
$path = '/usr/local/sbin:/usr/local/bin:/usr/sbin:/sbin:/usr/bin:/bin';
# $dspam = 'dspam';
$MAXLEVELS = 14;
$MAXFILES = 1500;
$MIN_EXPANSION_QUOTA = 100*1024; # bytes (default undef, not
enforced)
$MAX_EXPANSION_QUOTA = 300*1024*1024; # bytes (default undef, not
enforced)
$sa_spam_subject_tag = ' ';
#$sa_spam_subject_tag = '***SPAM*** ';
$defang_virus = 1; # MIME-wrap passed infected mail
$defang_banned = 1; # MIME-wrap passed mail containing banned name
# OTHER MORE COMMON SETTINGS (defaults may suffice):
# $myhostname = 'host.example.com'; # must be a fully-qualified domain
name!
# $notify_method = 'smtp:[127.0.0.1]:10025';
# $forward_method = 'smtp:[127.0.0.1]:10025'; # set to undef with milter!
# $final_virus_destiny = D_DISCARD;
# $final_banned_destiny = D_BOUNCE;
$final_spam_destiny = D_PASS;
# $final_bad_header_destiny = D_PASS;
Postfix Configuration
---------------------
main.cf
# Amavis Configuration
content_filter=smtp-amavis:[127.0.0.1]:10024
master.cf
#
# Amavis configuration
#
smtp-amavis unix - - n - 20 smtp
-o smtp_data_done_timeout=1200
-o smtp_send_xforward_command=yes
-o disable_dns_lookups=yes
127.0.0.1:10025 inet n - n - - smtpd
-o content_filter=
-o myhostname=localhost.sandiegort.com
-o local_recipient_maps=
-o relay_recipient_maps=
-o smtpd_restriction_classes=
-o smtpd_client_restrictions=
-o smtpd_helo_restrictions=
-o smtpd_sender_restrictions=
-o smtpd_recipient_restrictions=permit_mynetworks,reject
-o mynetworks=127.0.0.0/8
-o strict_rfc821_envelopes=yes
-o smtpd_error_sleep_time=0
-o smtpd_soft_error_limit=1001
-o smtpd_hard_error_limit=1000
-o smtpd_client_connection_count_limit=0
-o smtpd_client_connection_rate_limit=0
-o receive_override_options=no_unknown_recipient_checks
What causes these refusal messages?
-----------------------------------
-Queue ID- --Size-- ----Arrival Time---- -Sender/Recipient-------
723E64AC8D* 4706 Wed Feb 15 14:06:50
[EMAIL PROTECTED]
[EMAIL PROTECTED]
4B3DB4ACBF* 11029 Wed Feb 15 14:06:51
[EMAIL PROTECTED]
[EMAIL PROTECTED]
0F8974ACC1 22366 Wed Feb 15 14:00:02 [EMAIL PROTECTED]
(lost connection with 127.0.0.1[127.0.0.1] while sending end of data --
message may be sent more than once)
[EMAIL PROTECTED]
1E8F44ACC8 2489 Wed Feb 15 14:00:06 [EMAIL PROTECTED]
(connect to 127.0.0.1[127.0.0.1]: Connection
refused)
[EMAIL PROTECTED]
CC0544ACD9 38880 Wed Feb 15 14:00:01 [EMAIL PROTECTED]
(host 127.0.0.1[127.0.0.1] said: 421 4.3.2 Service shutting down, closing
channel (in reply to end of DATA command))
[EMAIL PROTECTED]
8A7644ACFA 129256 Wed Feb 15 14:00:02 [EMAIL PROTECTED]
(lost connection with 127.0.0.1[127.0.0.1] while receiving the initial
server greeting)
[EMAIL PROTECTED]
A5A694AD0E 38864 Wed Feb 15 14:00:02 [EMAIL PROTECTED]
(connect to 127.0.0.1[127.0.0.1]: Connection
refused)
[EMAIL PROTECTED]
-- 244 Kbytes in 7 Requests.
-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
for problems? Stop! Download the new AJAX search engine that makes
searching your log files as easy as surfing the web. DOWNLOAD SPLUNK!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=103432&bid=230486&dat=121642
_______________________________________________
AMaViS-user mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/amavis-user
AMaViS-FAQ:http://www.amavis.org/amavis-faq.php3
AMaViS-HowTos:http://www.amavis.org/howto/
