i will just report spamassassin trunc breaks amavisd-new
going back to 3.4.6 as now until 4.0.1 is released, in amavisd logs hits is always - with imho means spamtest is skipped, can you verify sa trunk does work still with amavisd ?
Re: amavisd 100% cpu load - 470 queued messages...
In this moment I have more than 400 delivery of a 178kb text/html message... no attachment... For specific senders I may: - apply a very restrictive throttling - skip spamassassin check On Fri, Jun 28, 2019 at 3:18 PM Matus UHLAR - fantomas wrote: > >> On 28.06.19 12:03, hg user wrote: > >> >I did investigate a bit and was able to link spikes in emails received > on > >> >the external MTAs to spikes in messages queued in amavisd. > >> > > >> >As soon as I receive a mailing list/newsletter kind of email addressed > to > >> >several hundreds recipients and the message size is more than 70kb, the > >> >messages starts to queue up. > >> > > >> >A message I was able to isolate is 94kb, has text and html part and > >> >processing via command line takes about 6 seconds, I received 500+ of > them. > >> >Due to unsubscribe urls containing the mail address, all messages have > >> >different hashes > > >On Fri, Jun 28, 2019 at 01:23:38PM +0200, Matus UHLAR - fantomas wrote: > >> 6 seconds message scanning is quite fast. I have set up > >> razor,pyzor,dcc,dnsbl timeouts to 20 seconds to improve > >> acurracy (maybe marginally, but I don't care). > >> > >> If you receive that much mail often, you need faster computer. > >> Maybe sa-compile would help you a bit, but only with CPU usage, network > >> checks need time (and they are effective so don't mess them up). > > On 28.06.19 15:09, Henrik K wrote: > >If the bottleneck is network checks, one should just increase parallel > >processes. But OP probably has some other problems if regex are taking so > >long (suggested by the 100% CPU usage). > > processing 370 of 70k emails within a few seconds of course results in 100% > cpu usage for those few seconds :-) > > >Biggest improvement comes from razor/pyzor/dcc being asynchronous in > trunk, > >I suggest trying it if possible. :-) > > great news. > -- > Matus UHLAR - fantomas, uh...@fantomas.sk ; http://www.fantomas.sk/ > Warning: I wish NOT to receive e-mail advertising to this address. > Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu. > Eagles may soar, but weasels don't get sucked into jet engines. >
Re: amavisd 100% cpu load - 470 queued messages...
On 28.06.19 12:03, hg user wrote: >I did investigate a bit and was able to link spikes in emails received on >the external MTAs to spikes in messages queued in amavisd. > >As soon as I receive a mailing list/newsletter kind of email addressed to >several hundreds recipients and the message size is more than 70kb, the >messages starts to queue up. > >A message I was able to isolate is 94kb, has text and html part and >processing via command line takes about 6 seconds, I received 500+ of them. >Due to unsubscribe urls containing the mail address, all messages have >different hashes On Fri, Jun 28, 2019 at 01:23:38PM +0200, Matus UHLAR - fantomas wrote: 6 seconds message scanning is quite fast. I have set up razor,pyzor,dcc,dnsbl timeouts to 20 seconds to improve acurracy (maybe marginally, but I don't care). If you receive that much mail often, you need faster computer. Maybe sa-compile would help you a bit, but only with CPU usage, network checks need time (and they are effective so don't mess them up). On 28.06.19 15:09, Henrik K wrote: If the bottleneck is network checks, one should just increase parallel processes. But OP probably has some other problems if regex are taking so long (suggested by the 100% CPU usage). processing 370 of 70k emails within a few seconds of course results in 100% cpu usage for those few seconds :-) Biggest improvement comes from razor/pyzor/dcc being asynchronous in trunk, I suggest trying it if possible. :-) great news. -- Matus UHLAR - fantomas, uh...@fantomas.sk ; http://www.fantomas.sk/ Warning: I wish NOT to receive e-mail advertising to this address. Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu. Eagles may soar, but weasels don't get sucked into jet engines.
Re: amavisd 100% cpu load - 470 queued messages...
On Fri, Jun 28, 2019 at 12:03:23PM +0200, hg user wrote: > > A message I was able to isolate is 94kb, has text and html part and processing > via command line takes about 6 seconds, I received 500+ of them. Due to > unsubscribe urls containing the mail address, all messages have different > hashes If it happens to be large pathological body for some regexes, upcoming 3.4.3 might help here with it's body_part_scan_size option (truncates body length to 50k by default, fine to drop to 10-20k on slow server too). 3.4.3-rc3 is available here if you or someone wants to try: http://talon2.pccc.com/~kmcgrail/devel/ To identify slow rules, download http://svn.apache.org/repos/asf/spamassassin/trunk/masses/plugins/HitFreqsRuleTiming.pm And run spamassassin --cf 'loadplugin HitFreqsRuleTiming /path/to/HitFreqsRuleTiming.pm' -L messagefile See resulting timing.log file
Re: amavisd 100% cpu load - 470 queued messages...
On Fri, Jun 28, 2019 at 01:23:38PM +0200, Matus UHLAR - fantomas wrote:
> On 28.06.19 12:03, hg user wrote:
> >I did investigate a bit and was able to link spikes in emails received on
> >the external MTAs to spikes in messages queued in amavisd.
> >
> >As soon as I receive a mailing list/newsletter kind of email addressed to
> >several hundreds recipients and the message size is more than 70kb, the
> >messages starts to queue up.
> >
> >A message I was able to isolate is 94kb, has text and html part and
> >processing via command line takes about 6 seconds, I received 500+ of them.
> >Due to unsubscribe urls containing the mail address, all messages have
> >different hashes
>
> 6 seconds message scanning is quite fast. I have set up
> razor,pyzor,dcc,dnsbl timeouts to 20 seconds to improve
> acurracy (maybe marginally, but I don't care).
>
> If you receive that much mail often, you need faster computer.
> Maybe sa-compile would help you a bit, but only with CPU usage, network
> checks need time (and they are effective so don't mess them up).
If the bottleneck is network checks, one should just increase parallel
processes. But OP probably has some other problems if regex are taking so
long (suggested by the 100% CPU usage).
Here's some figures for trunk/4.0.0 with razor,pyzor,dcc,ixhash and bunch of
other stuff.
# grep TIMING-SA mail.log mail.log.1 | pcregrep -o '\d+ ms' | awk '{print $1}'
| histogram
Count: 4375
Range: 7.000 - 8551.000; Mean: 1720.447; Median: 1252.000; Stddev: 1632.929
Percentiles: 90th: 4422.000; 95th: 4589.000; 99th: 8101.000
7.000 - 15.069:36 ##
15.069 - 31.276: 184 #
31.276 - 63.831: 426
63.831 - 129.221: 335
129.221 - 260.565: 113 #
260.565 - 524.384:14 #
524.384 - 1054.296: 545 #
1054.296 - 2118.689: 1138 #
2118.689 - 4256.650: 945
4256.650 - 8551.000: 639 ##
<500ms are shortcircuits. But mostly 1-4 sec for normal stuff.
Biggest improvement comes from razor/pyzor/dcc being asynchronous in trunk,
I suggest trying it if possible. :-)
Re: amavisd 100% cpu load - 470 queued messages...
On 28.06.19 12:03, hg user wrote: I did investigate a bit and was able to link spikes in emails received on the external MTAs to spikes in messages queued in amavisd. As soon as I receive a mailing list/newsletter kind of email addressed to several hundreds recipients and the message size is more than 70kb, the messages starts to queue up. A message I was able to isolate is 94kb, has text and html part and processing via command line takes about 6 seconds, I received 500+ of them. Due to unsubscribe urls containing the mail address, all messages have different hashes 6 seconds message scanning is quite fast. I have set up razor,pyzor,dcc,dnsbl timeouts to 20 seconds to improve acurracy (maybe marginally, but I don't care). If you receive that much mail often, you need faster computer. Maybe sa-compile would help you a bit, but only with CPU usage, network checks need time (and they are effective so don't mess them up). -- Matus UHLAR - fantomas, uh...@fantomas.sk ; http://www.fantomas.sk/ Warning: I wish NOT to receive e-mail advertising to this address. Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu. Atheism is a non-prophet organization.
Re: amavisd 100% cpu load - 470 queued messages...
I did investigate a bit and was able to link spikes in emails received on the external MTAs to spikes in messages queued in amavisd. As soon as I receive a mailing list/newsletter kind of email addressed to several hundreds recipients and the message size is more than 70kb, the messages starts to queue up. A message I was able to isolate is 94kb, has text and html part and processing via command line takes about 6 seconds, I received 500+ of them. Due to unsubscribe urls containing the mail address, all messages have different hashes On Fri, Jun 28, 2019 at 11:38 AM Matus UHLAR - fantomas wrote: > >> On Fri, Jun 28, 2019 at 10:49 AM hg user > wrote: > >>> I'm not able to lower cpu usage of amavisd. > >>> 4 cpus are used 100% and messages queue up to 15 minutes before being > >>> processed. > >>> > >>> mailq reports up to 470 queued messages... and this is bad, really bad. > >>> > >>> The most part of SA work is spent here: > >>> tests_pri_0: 7371 (93.7%) > >>> and I know that priority 0 includes almost everything > >>> > >>> Other than disabling some rules (which one?!!?!) and adding some cpus, > >>> what can I really do? > > >> Messages reported by mailq decreased to about 370 and then, in a few > >> seconds, to 0... from 370 to 0 in a few seconds... > > On 28.06.19 10:03, Dominic Raferd wrote: > >As a workaround you could set (in amavis conf file) something like: > > > >$child_timeout = 20; > > > >which restricts any child process to (roughly) 20 seconds after which > >amavis will abort it > > I don't think that's a good idea. That would highly decreate accuracy. > > -- > Matus UHLAR - fantomas, uh...@fantomas.sk ; http://www.fantomas.sk/ > Warning: I wish NOT to receive e-mail advertising to this address. > Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu. > Nothing is fool-proof to a talented fool. >
Re: amavisd 100% cpu load - 470 queued messages...
On Fri, Jun 28, 2019 at 10:49 AM hg user wrote: I'm not able to lower cpu usage of amavisd. 4 cpus are used 100% and messages queue up to 15 minutes before being processed. mailq reports up to 470 queued messages... and this is bad, really bad. The most part of SA work is spent here: tests_pri_0: 7371 (93.7%) and I know that priority 0 includes almost everything Other than disabling some rules (which one?!!?!) and adding some cpus, what can I really do? Messages reported by mailq decreased to about 370 and then, in a few seconds, to 0... from 370 to 0 in a few seconds... On 28.06.19 10:03, Dominic Raferd wrote: As a workaround you could set (in amavis conf file) something like: $child_timeout = 20; which restricts any child process to (roughly) 20 seconds after which amavis will abort it I don't think that's a good idea. That would highly decreate accuracy. -- Matus UHLAR - fantomas, uh...@fantomas.sk ; http://www.fantomas.sk/ Warning: I wish NOT to receive e-mail advertising to this address. Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu. Nothing is fool-proof to a talented fool.
Re: amavisd 100% cpu load - 470 queued messages...
On Fri, 28 Jun 2019 at 09:56, hg user wrote: > Messages reported by mailq decreased to about 370 and then, in a few > seconds, to 0... from 370 to 0 in a few seconds... > > > > On Fri, Jun 28, 2019 at 10:49 AM hg user wrote: > >> I'm not able to lower cpu usage of amavisd. >> 4 cpus are used 100% and messages queue up to 15 minutes before being >> processed. >> >> mailq reports up to 470 queued messages... and this is bad, really bad. >> >> The most part of SA work is spent here: >> tests_pri_0: 7371 (93.7%) >> and I know that priority 0 includes almost everything >> >> Other than disabling some rules (which one?!!?!) and adding some cpus, >> what can I really do? >> > As a workaround you could set (in amavis conf file) something like: $child_timeout = 20; which restricts any child process to (roughly) 20 seconds after which amavis will abort it
Re: amavisd 100% cpu load - 470 queued messages...
Messages reported by mailq decreased to about 370 and then, in a few seconds, to 0... from 370 to 0 in a few seconds... On Fri, Jun 28, 2019 at 10:49 AM hg user wrote: > I'm not able to lower cpu usage of amavisd. > 4 cpus are used 100% and messages queue up to 15 minutes before being > processed. > > mailq reports up to 470 queued messages... and this is bad, really bad. > > The most part of SA work is spent here: > tests_pri_0: 7371 (93.7%) > and I know that priority 0 includes almost everything > > Other than disabling some rules (which one?!!?!) and adding some cpus, > what can I really do? >
amavisd 100% cpu load - 470 queued messages...
I'm not able to lower cpu usage of amavisd. 4 cpus are used 100% and messages queue up to 15 minutes before being processed. mailq reports up to 470 queued messages... and this is bad, really bad. The most part of SA work is spent here: tests_pri_0: 7371 (93.7%) and I know that priority 0 includes almost everything Other than disabling some rules (which one?!!?!) and adding some cpus, what can I really do?
Re: Debian 8: Postfix -> amavisd-new -> spamassassin -> Bayes : not scanning?
On 10 Jul 2017 at 16:56, RW wrote: > > On Mon, 10 Jul 2017 14:48:29 +0200 > Frantisek Rysanek wrote: > > > Dear fellow Debian users, > > > > it seems that I've found the correct answer. > > > > In /etc/spamassassin/local.cf, > > in addition to the aforementioned: > > use_bayes 1 > > bayes_auto_learn 1 > > I have added: > > > > use_bayes_rules 1 > > > > Found when trawling the /usr/share/perl5/Mail directory, > > namely discovered in SpamAssassin/Conf.pm. > > That's the basis of the Mail::SpamAssassin::Conf man page. > > use_bayes_rules is 1 by default, it only exists so you can turn-off > the rules without disabling learning; if it made any difference then > amavis has changed something. > Yes. Thanks for pointing that out. I did find it in the documentation sections of the Conf.pm Perl module that get exported as a man page during a manual module installation. This is not the first time that I've taken a rather lengthy cruise to give myself the right advice: RTFM :-) Not sure exactly why, but in the Amavis + Spamassassin combo, somehow it was a little unclear to me, "where to start" :-) Then again, in the end it seems it was actually the -T switch to Perl that made a key difference. Debian spoils me. Makes me lazy :-) Frank
Re: Debian 8: Postfix -> amavisd-new -> spamassassin -> Bayes : not scanning?
...oops, forgot to forward to the SA mailing list, apologies...
-
There's more, it seems like that wasn't the end-game yet :-)
Just after I sent the previous optimistic message,
I got a cold shower: the BAYES scores were gone again.
So I went back to some serious level of debug,
tried removing some config related to auto-expiry
that I was playing with at the same time, but even
as I got the config back to where it used to work,
the Bayes was gone again. Same symptoms.
While I was fumbling sadly through the debug log, I noticed another
promising warning:
_WARN: plugin: eval failed: Insecure dependency in sprintf while
running with -T switch at
/usr/share/perl5/Mail/SpamAssassin/Logger.pm line 241.
Now what the hell is the -T switch...
man perl
cannot find it right there (wish I knew the right chapter).
The source code wasn't much help either.
But after a bit of Googling, after I narrowed down the query,
I got this:
http://search.cpan.org/~bdfoy/PerlPowerTools-1.012/bin/printf
And several other pointers to an
"Insecure dependency in eval while running setuid"
Same thing? Probably.
The -T switch is for "taint mode".
https://perldoc.perl.org/perlsec.html#Taint-mode
And it's a security measure, so that your casual "evals with a printf
inside" are not easily hijacked for "code injection".
Now where the hell does that -T switch get into play.
SpamAssassin is running as a module of Amavis.
I already knew that Amavis was really a Perl script.
The Perl interpreter probably gets called using the #! shell
specification on the first line in /usr/sbin/amavisd-new .
You betcha.
>From there, the workaround is simple.
But ... OOPS!
I probably shouldn't tell anyone :->
Still... I don't understand why it suddenly worked for a while,
and then suddenly no longer, not anymore. Where's the hidden state?
I did restart Amavis after each change in the config files,
meaning I restarted the Perl interpreter all over each time...
"This is some spooky $#|t we got here, sarge..."
(to paraphrase Henry Rollins in the Lost Highway)
Frank
On 10 Jul 2017 at 14:44, debian-u...@lists.debian.org wrote:
> Dear fellow Debian users,
>
> it seems that I've found the correct answer.
>
> In /etc/spamassassin/local.cf,
> in addition to the aforementioned:
> use_bayes 1
> bayes_auto_learn 1
> I have added:
>
> use_bayes_rules 1
>
> Found when trawling the /usr/share/perl5/Mail directory,
> namely discovered in SpamAssassin/Conf.pm.
> Looked promising, so I tried it. How silly.
>
> That one line has caused some difference on the inside,
> as a result of which, I now have a BAYES score in the
> X-Spam-Status header in every message.
> A remaining trouble is that all the scores so far come out as
> BAYES_00 :-) so I may have to work on that some more.
> No SPAM has arrived yet, to provide a proper test.
> (I get 2-3 a day in my inbox - the rest is taken care of
> by greylisting and the general SpamAssassin scoring rules.)
>
> Other possibly interesting options:
>bayes_use_hapaxes
>bayes_auto_expire
>bayes_token_ttl
>bayes_seen_ttl
>
> Actually I've managed to get a backtrace from one function that I
> could identify as getting called: in
> /usr/share/perl5/Mail/SpamAssassin/BayesStore/DBM.pm : sub
> tie_db_readonly { ...
> my $iii = 1;
> print dbg("Stack Trace:");
> while ( (my @call_details = (caller($iii++))) ){
> dbg( $call_details[1].":".$call_details[2]." in function" . \
> $call_details[3] );
> }
>
> ...which did produce a neat stack trace. I'm attaching it, if
> anyone's interested.
> The code was taken almost verbatim from
> https://stackoverflow.com/questions/229009/how-can-i-get-a-call-stack-
> listing-in-perl
>
> In the stack trace I could see that something inside Amavis goes "have
> this message scanned", but some lower layers (across several
> indirections) got asked "is_scan_available" and
> "learner_is_scan_available". Funny, that...
>
> I've also noticed that
> /usr/share/perl5/Mail/SpamAssassin/Bayes.pm contains a note, saying
>
> # This is the general class used to train a learning classifier with
> # new samples of spam and ham mail, and classify based on prior
> # training.
> #
> # Prior to version 3.3.0, the default Bayes implementation was here;
> # if you're looking for information on that, it has moved to
> #Mail::SpamAssassin::Plugin::Bayes .
>
> And yes indeed, there's another file:
> /usr/share/perl5/Mail/SpamAssassin/Plugin/Bayes.pm
> containing the function check_bayes() where I'd previously
> put my dbg() trap...
>
> ...so I thought: "maybe SpamAssassin.pm was 'requiring' the wrong
> module?" But that doe
Re: Debian 8: Postfix -> amavisd-new -> spamassassin -> Bayes : not scanning?
On Mon, 10 Jul 2017 14:48:29 +0200 Frantisek Rysanek wrote: > Dear fellow Debian users, > > it seems that I've found the correct answer. > > In /etc/spamassassin/local.cf, > in addition to the aforementioned: > use_bayes 1 > bayes_auto_learn 1 > I have added: > > use_bayes_rules 1 > > Found when trawling the /usr/share/perl5/Mail directory, > namely discovered in SpamAssassin/Conf.pm. That's the basis of the Mail::SpamAssassin::Conf man page. use_bayes_rules is 1 by default, it only exists so you can turn-off the rules without disabling learning; if it made any difference then amavis has changed something.
Re: Debian 8: Postfix -> amavisd-new -> spamassassin -> Bayes : not scanning?
Dear fellow Debian users,
it seems that I've found the correct answer.
In /etc/spamassassin/local.cf,
in addition to the aforementioned:
use_bayes 1
bayes_auto_learn 1
I have added:
use_bayes_rules 1
Found when trawling the /usr/share/perl5/Mail directory,
namely discovered in SpamAssassin/Conf.pm.
Looked promising, so I tried it. How silly.
That one line has caused some difference on the inside,
as a result of which, I now have a BAYES score in the
X-Spam-Status header in every message.
A remaining trouble is that all the scores so far come out as
BAYES_00 :-) so I may have to work on that some more.
No SPAM has arrived yet, to provide a proper test.
(I get 2-3 a day in my inbox - the rest is taken care of
by greylisting and the general SpamAssassin scoring rules.)
Other possibly interesting options:
bayes_use_hapaxes
bayes_auto_expire
bayes_token_ttl
bayes_seen_ttl
Actually I've managed to get a backtrace from one function that I
could identify as getting called:
in /usr/share/perl5/Mail/SpamAssassin/BayesStore/DBM.pm :
sub tie_db_readonly {
...
my $iii = 1;
print dbg("Stack Trace:");
while ( (my @call_details = (caller($iii++))) ){
dbg( $call_details[1].":".$call_details[2]." in function" . \
$call_details[3] );
}
...which did produce a neat stack trace. I'm attaching it, if
anyone's interested.
The code was taken almost verbatim from
https://stackoverflow.com/questions/229009/how-can-i-get-a-call-stack-
listing-in-perl
In the stack trace I could see that something inside Amavis goes
"have this message scanned", but some lower layers (across several
indirections) got asked "is_scan_available" and
"learner_is_scan_available". Funny, that...
I've also noticed that
/usr/share/perl5/Mail/SpamAssassin/Bayes.pm contains a note, saying
# This is the general class used to train a learning classifier with
# new samples of spam and ham mail, and classify based on prior
# training.
#
# Prior to version 3.3.0, the default Bayes implementation was here;
# if you're looking for information on that, it has moved to
#Mail::SpamAssassin::Plugin::Bayes .
And yes indeed, there's another file:
/usr/share/perl5/Mail/SpamAssassin/Plugin/Bayes.pm
containing the function check_bayes() where I'd previously
put my dbg() trap...
...so I thought: "maybe SpamAssassin.pm was 'requiring' the wrong
module?"
But that doesn't seem to be the case... (I've tried :-)
Instead, after I added
use_bayes_rules 1
I started to get BAYES scores in the mail headers.
And my trap in check_bayes() gets logged.
That's probably a good start :-)
Frank
On 9 Jul 2017 at 23:40, users@spamassassin.apache.org wrote:
>
> Dear polite people in the SA users' mailing list,
>
> I would appreciate any help with the following setup.
> For the record, I'm sending this same text to the
> debian-users mailing list - I'm not technically
> cross-posting, as that would probably earn me a bad
> reputation (or a kick).
>
> I've just built a new mailserver based on Debian 8.8,
> with Postfix + Cyrus. I have a long history of using
> Amavis with SpamAssassin for SPAM filtering.
> On the newly installed machine, there is
> SpamAssassin 3.4.0-6 = the current version for Jessie.
>
> And within SpamAssassin, my previous server (based on
> Debian Squeeze) was using the Bayesian filter.
> Using
> sa-learn --backup
> sa-learn --restore=...
> I have migrated the Bayes database to the new machine,
> and after a few path tweaks and privilege adjustments,
> I got sa-learn-cyrus to do its job.
>
> Curiously to me, I don't see any BAYES scores
> in the X-Spam-Status header. I suspect that the Bayes
> plugin does not actually get called to evaluate
> the messages passing through my server.
>
> In /etc/spamassassin/local.cf, I have the following:
> use_bayes 1
> bayes_auto_learn 1
> bayes_path /var/lib/spamassassin/.spamassassin/bayes
> ...a couple of whitelist_from rules, and
> add_header all Report _REPORT_
>
>
> In /etc/amavis/conf.d/15-content_filter_mode, I have UNcommented this:
>
> @bypass_spam_checks_maps = (
>\%bypass_spam_checks, \@bypass_spam_checks_acl,
> \$bypass_spam_checks_re);
>
>
> In /etc/amavis/conf.d/50-user , I have the following:
>
> $DO_SYSLOG = 0;
> $LOGFILE = "/var/log/amavis.log";
> $sa_tag_level_deflt = -; # always add spam info headers
>
> $log_level = 1;
> $sa_debug = 1;
>
> I've also tried log_level = 2, which showed me a privilege problem,
> where the SA's Bayes plugin couldn't create a lock file... so that's
> handled too. I'm getting *some* notes about the Bayes plugin in the
> amavis log:
>
> Jul 9 21:25:54 mail.x.y.z /usr/sbin/amavisd-new[8868]: (08868-01) SA
> dbg: bayes: tie-in
Debian 8: Postfix -> amavisd-new -> spamassassin -> Bayes : not scanning?
Dear polite people in the SA users' mailing list, I would appreciate any help with the following setup. For the record, I'm sending this same text to the debian-users mailing list - I'm not technically cross-posting, as that would probably earn me a bad reputation (or a kick). I've just built a new mailserver based on Debian 8.8, with Postfix + Cyrus. I have a long history of using Amavis with SpamAssassin for SPAM filtering. On the newly installed machine, there is SpamAssassin 3.4.0-6 = the current version for Jessie. And within SpamAssassin, my previous server (based on Debian Squeeze) was using the Bayesian filter. Using sa-learn --backup sa-learn --restore=... I have migrated the Bayes database to the new machine, and after a few path tweaks and privilege adjustments, I got sa-learn-cyrus to do its job. Curiously to me, I don't see any BAYES scores in the X-Spam-Status header. I suspect that the Bayes plugin does not actually get called to evaluate the messages passing through my server. In /etc/spamassassin/local.cf, I have the following: use_bayes 1 bayes_auto_learn 1 bayes_path /var/lib/spamassassin/.spamassassin/bayes ...a couple of whitelist_from rules, and add_header all Report _REPORT_ In /etc/amavis/conf.d/15-content_filter_mode, I have UNcommented this: @bypass_spam_checks_maps = ( \%bypass_spam_checks, \@bypass_spam_checks_acl, \$bypass_spam_checks_re); In /etc/amavis/conf.d/50-user , I have the following: $DO_SYSLOG = 0; $LOGFILE = "/var/log/amavis.log"; $sa_tag_level_deflt = -; # always add spam info headers $log_level = 1; $sa_debug = 1; I've also tried log_level = 2, which showed me a privilege problem, where the SA's Bayes plugin couldn't create a lock file... so that's handled too. I'm getting *some* notes about the Bayes plugin in the amavis log: Jul 9 21:25:54 mail.x.y.z /usr/sbin/amavisd-new[8868]: (08868-01) SA dbg: bayes: tie-ing to DB file R/O /var/lib/spamassassin/.spamassassin/bayes_toks Jul 9 21:25:54 mail.x.y.z /usr/sbin/amavisd-new[8868]: (08868-01) SA dbg: bayes: tie-ing to DB file R/O /var/lib/spamassassin/.spamassassin/bayes_seen Jul 9 21:25:54 mail.x.y.z /usr/sbin/amavisd-new[8868]: (08868-01) SA dbg: bayes: found bayes db version 3 Jul 9 21:25:55 mail /usr/sbin/amavisd-new[8868]: (08868-01) SA dbg: plugin: Mail::SpamAssassin::Plugin::Bayes=HASH(0x6bc65b0) implements 'learn_message', priority 0 Jul 9 21:25:55 mail.x.y.z /usr/sbin/amavisd-new[8868]: (08868-01) SA dbg: locker: safe_lock: created /var/lib/spamassassin/.spamassassin/bayes.lock.mail.fccps.cz.8868 Jul 9 21:25:55 mail.x.y.z /usr/sbin/amavisd-new[8868]: (08868-01) SA dbg: locker: safe_lock: trying to get lock on /var/lib/spamassassin/.spamassassin/bayes with 0 retries Jul 9 21:25:55 mail.x.y.z /usr/sbin/amavisd-new[8868]: (08868-01) SA dbg: locker: safe_lock: link to /var/lib/spamassassin/.spamassassin/bayes.lock: link ok Jul 9 21:25:55 mail.x.y.z /usr/sbin/amavisd-new[8868]: (08868-01) SA dbg: bayes: tie-ing to DB file R/W /var/lib/spamassassin/.spamassassin/bayes_toks Jul 9 21:25:55 mail.x.y.z /usr/sbin/amavisd-new[8868]: (08868-01) SA dbg: bayes: tie-ing to DB file R/W /var/lib/spamassassin/.spamassassin/bayes_seen Jul 9 21:25:55 mail.x.y.z /usr/sbin/amavisd-new[8868]: (08868-01) SA dbg: bayes: found bayes db version 3 Jul 9 21:25:55 mail.x.y.z /usr/sbin/amavisd-new[8868]: (08868-01) SA dbg: bayes: learned 'd963c4a7f11e91c3bd3317ea92408c2013c99dad@sa_generated', atime: 1499628354 Jul 9 21:25:55 mail.x.y.z /usr/sbin/amavisd-new[8868]: (08868-01) SA dbg: bayes: untie-ing Jul 9 21:25:55 mail.x.y.z /usr/sbin/amavisd-new[8868]: (08868-01) SA dbg: bayes: files locked, now unlocking lock Jul 9 21:25:55 mail.x.y.z /usr/sbin/amavisd-new[8868]: (08868-01) SA dbg: locker: safe_unlock: unlink /var/lib/spamassassin/.spamassassin/bayes.lock Makes me wonder if the "implements" messages can mean something (no "scan" operation?): Jul 9 21:25:21 mail.x.y.z /usr/sbin/amavisd-new[8850]: SA dbg: plugin: Mail::SpamAssassin::Plugin::Bayes=HASH(0x6bc65b0) implements 'learner_new', priority 0 Jul 9 21:25:21 mail.x.y.z /usr/sbin/amavisd-new[8850]: SA dbg: plugin: Mail::SpamAssassin::Plugin::Bayes=HASH(0x6bc65b0) implements 'learner_is_scan_available', priority 0 Jul 9 21:25:22 mail.x.y.z /usr/sbin/amavisd-new[8850]: SA dbg: plugin: Mail::SpamAssassin::Plugin::Bayes=HASH(0x6bc65b0) implements 'learner_close', priority 0 Jul 9 21:25:22 mail.x.y.z /usr/sbin/amavisd-new[8850]: SA dbg: plugin: Mail::SpamAssassin::Plugin::Bayes=HASH(0x6bc65b0) implements 'prefork_init', priority 0 Jul 9 21:25:22 mail.x.y.z /usr/sbin/amavisd-new[8868]: SA dbg: plugin: Mail::SpamAssassin::Plugin::Bayes=HASH(0x6bc65b0) implements 'spamd_child_init', priority 0 Jul 9 21:25:22 mail.x.y.z /usr/sbin/amavisd-new[8869]: SA dbg: plugin: Mail::SpamAssassin::Plugin::Bayes=HASH(0x6bc65b0) implements 'spamd_child_init', priority 0 Jul 9 21:25:5
Re: Spamassassin and amavisd-new wont' check (faked) bounce with zip-archive/exe (maleware)
On Mon, 2015-10-26 at 13:09 +0100, Django[BOfH] wrote: > Hello list, dear Marc! > > I had have a "little problem" with a mailsystem. > > A few days agoe a colleague received over 200 bounce-messages and > this > over 10 minutes. O.K., that was all backscatter from a software > -company > in Redmond :( All those messages had have an attachment (zip archive) > with maleware. > Obvious first question: Are you using SPF? - If so, what did SA have to say about these messages? - If not, what are you using to detect messages with forged sender details and what did it say? IMO the prime purpose of SPF is to detect bounce spam with forged sender details that correspond to your site. Are you using DKIM? - the same SPF follow-up questions also apply to DKIM. It would be helpful if we could see the message headers plus the plaintest and HTML MIME parts, but don't post it here because that may cause your message to be taken as spam: post it on DropBox or a similar website and post the URL here. Martin
Re: Spamassassin and amavisd-new wont' check (faked) bounce with zip-archive/exe (maleware)
On 26.10.15 13:09, Django [BOfH] wrote: Hello list, dear Marc! correction: Helo spamassassn-users list - it has nothing to do with attachment or virus scanning. you should have contacted amavisd-new list http://lists.amavis.org/cgi-bin/mailman/listinfo/amavis-users So I tried to understand, why our AMaVis's allowed those faked bounce-messages with mailware. -- Matus UHLAR - fantomas, uh...@fantomas.sk ; http://www.fantomas.sk/ Warning: I wish NOT to receive e-mail advertising to this address. Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu. Spam = (S)tupid (P)eople's (A)dvertising (M)ethod
Re: Spamassassin and amavisd-new wont' check (faked) bounce with zip-archive/exe (maleware)
On Mon, 26 Oct 2015, Django [BOfH] wrote: A few days agoe a colleague received over 200 bounce-messages and this over 10 minutes. O.K., that was all backscatter from a software-company in Redmond :( All those messages had have an attachment (zip archive) with maleware. http://impsec.org/email-tools/procmail-security.html This does coexist with SA. I've been gonna create an archive scanning plugin for a while now, no round tuits tho. -- John Hardin KA7OHZhttp://www.impsec.org/~jhardin/ jhar...@impsec.orgFALaholic #11174 pgpk -a jhar...@impsec.org key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C AF76 D822 E6E6 B873 2E79 --- ...the Fates notice those who buy chainsaws... -- www.darwinawards.com --- 5 days until Halloween
Spamassassin and amavisd-new wont' check (faked) bounce with zip-archive/exe (maleware)
Hello list, dear Marc! I had have a "little problem" with a mailsystem. A few days agoe a colleague received over 200 bounce-messages and this over 10 minutes. O.K., that was all backscatter from a software-company in Redmond :( All those messages had have an attachment (zip archive) with maleware. A few minutes I was shocked, 'cause how could all AMaVis-hosts at customer site, transport maleware in a zip-archive?! So, I tried to send a new mail, with this zip-archive to all of our 5 MX and nowhere it was possible to trespass our borderfilters. :) So I tried to understand, why our AMaVis's allowed those faked bounce-messages with mailware. The only thing I found was those maillog-entries: Sep 8 13:17:10 amavis-cluster-by amavis[23088]: (23088-10) bounce rescued by domain (DSN), <> -> <redac...@example.com>, date: Tue, 8 Sep 2015 12:41:24 +0200, from: Rosenbaum Group <redac...@example.com>, message-id: <hdmuibrrpv7zej2q0r2t...@example.com>, return-path: redac...@example.com "bounce rescued by domain (DSN)"? What's that? So I tried to ask google, wether or not there are existing news known by others. The only things I found was: https://www.mail-archive.com/amavis-user@lists.sourceforge.net/msg11245. html http://sourceforge.net/p/amavis/mailman/amavis-user/thread/201010051713. 38050.ste...@localside.net/ and http://www.ijs.si/software/amavisd/ " ... bounce killer feature (requires pen pals SQL logging) checks a header section attached to received non-delivery status notifications, and discards bounces to fake mail which do not refer to our genuine outgoing mail;" I'm not so fimilar with this, how p@trick told it "spam and maleware over backscatter as esoteric problem ;)", and your "bounce killer feature". May you tell me a few more points, what this feature can do and if it the right point, to ban those attacks? Or there exists an unknown feature for banning attachments (i.e. zip-archives with maleware)? Every hint is useful! On AMaViS 2.10 have you marked "do_ascii": @decoders = ( ['mail', \_mime_decode], # [[qw(asc uue hqx ync)], \_ascii], # not safe In RELEASE_NOTES you wrote: - amavisd.conf: commented-out calls to do_ascii to match defaults in the amavisd program; the uulib code (as invoked by Convert::UUlib) has a history of stability problems, seems it is causing more grief compared to the benefits it brings; Safe or stability? What happens if I activate this encoder for recognize those faked bounces? Is the prize high? Thanx4help! Have a nice day! Django -- "Bonnie & Clyde der Postmaster-Szene!" approved by Postfix-God http://wetterstation-pliening.info http://dokuwiki.nausch.org http://wiki.piratenpartei.de/Benutzer:Django
Re: First start with Spamassassin Debian 7 wheezy postfix and dovecot amavisd
On 10.01.15 00:34, Steffen Mutter wrote: I just started fiddling around with spamassassin, very good software but a high level of complexity. Already run into several problems because mail headers were not rewritten and whitelisting of mails did fail - had to trigger amavis to do the job. sa-learn is nice, but it does not work with dovecots mdbox, does dovecot's antispam plugin (http://wiki2.dovecot.org/Plugins/Antispam) work with amavisd? I would start searching here... -- Matus UHLAR - fantomas, uh...@fantomas.sk ; http://www.fantomas.sk/ Warning: I wish NOT to receive e-mail advertising to this address. Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu. Spam is for losers who can't get business any other way.
Re: First start with Spamassassin Debian 7 wheezy postfix and dovecot amavisd
On Sat, 10 Jan 2015 00:34:49 +0100 Steffen Mutter wrote: Hi, I just started fiddling around with spamassassin, very good software but a high level of complexity. Already run into several problems because mail headers were not rewritten and whitelisting of mails did fail - had to trigger amavis to do the job. ... spamassassin configuration files reside in /et/spamassassin/ so I think I have to do the configuration here, why do I have to repeat configuration in amavis?!? http://www.ijs.si/software/amavisd/#faq-spam Okay, here's now a problem I couldn't solve: debian-spamd -c sa-update --nogpg --channel spamassassin.heinlein-support.de replies: http: GET 3.3 request failed: 400 URL must be absolute: 400 URL must be absolute error: no mirror data available for channel spamassassin.heinlein-support.de channel: MIRRORED.BY contents were missing, channel failed Huh? There is a file in /var/lib/spamassassin/3.003002/updates_spamassassin_spamassassin.org/ named MIRRORED.BY That's for the default channel. If that updates successfully then presumably it's a problem with spamassassin.heinlein-support.de. Do I have to insert something here, to get the channel working? What do I have to do to add some channels? It's a long time since I've used a third-party channel, but I don't recall having to do anything.
First start with Spamassassin Debian 7 wheezy postfix and dovecot amavisd
Hi, I just started fiddling around with spamassassin, very good software but a high level of complexity. Already run into several problems because mail headers were not rewritten and whitelisting of mails did fail - had to trigger amavis to do the job. sa-learn is nice, but it does not work with dovecots mdbox, so I wrote a wrapper to store mails in mbox, which works like this: all mails in INBOX are treated as ham all mails in INBOX.Junk are treated as spam, mails savedbefore 3w are marked as deleted, mails savedbefore 1d marked as read works with the mailowners very well so far. spamassassin configuration files reside in /et/spamassassin/ so I think I have to do the configuration here, why do I have to repeat configuration in amavis?!? Okay, here's now a problem I couldn't solve: debian-spamd -c sa-update --nogpg --channel spamassassin.heinlein-support.de replies: http: GET 3.3 request failed: 400 URL must be absolute: 400 URL must be absolute error: no mirror data available for channel spamassassin.heinlein-support.de channel: MIRRORED.BY contents were missing, channel failed Huh? There is a file in /var/lib/spamassassin/3.003002/updates_spamassassin_spamassassin.org/ named MIRRORED.BY Do I have to insert something here, to get the channel working? What do I have to do to add some channels? Where do I find debian specific howtos? Please forgive me, I am sure you already answered questions like mine a thousand times - I am just a newbie... Kind regards form Germany, Steffem
Re: Fake amavisd-new header lines in recent spam
/do we have your permission to add this rule to SA's masscheck / autopromoting ?/ Yes, by all means, go ahead. -- *Rich Wales* ri...@richw.org
Re: Fake amavisd-new header lines in recent spam
On 11/10/2014 09:01 AM, Rich Wales wrote: /do we have your permission to add this rule to SA's masscheck / autopromoting ?/ Yes, by all means, go ahead. Thanks, Commited to http://svn.apache.org/repos/asf/spamassassin/trunk/rulesrc/sandbox/emailed/sa_users_contrib.cf masscheck results will show up in http://ruleqa.spamassassin.org/ hopefully, starting off tomorrow, after today's masschecks
Fake amavisd-new header lines in recent spam
Hi. Recently, I've noticed that some spam arriving on my mail server contains a Received: header line citing amavisd-new -- possibly an attempt to trick spam filters into concluding the message has already been scanned and is presumably free of problems. Here is an example of one of these -- the physically last (i.e., chronologically first) Received: in the message. Received: by 03112d50.rn56dss9.lunafutral.com (amavisd-new, port 9150) with ESMTP id 03MBRTVHDVT112DXUHRJKRWD50; for rande...@richw.org; Sat, 8 Nov 2014 17:41:05 -0700 The above line contains several clues that can distinguish it from a real Received: line generated by amavisd-new, so I imagine a rule could be created to detect this and increase a message's spam score accordingly. Should I go ahead and discuss this in greater depth here on this list? Or would it be better to go off-list with a smaller group of developers, so as not to give too many ideas to the black hats? :-) Rich Wales ri...@richw.org
Re: Fake amavisd-new header lines in recent spam
On 11/09/2014 06:45 PM, Rich Wales wrote: Hi. Recently, I've noticed that some spam arriving on my mail server contains a Received: header line citing amavisd-new -- possibly an attempt to trick spam filters into concluding the message has already been scanned and is presumably free of problems. Here is an example of one of these -- the physically last (i.e., chronologically first) Received: in the message. Received: by 03112d50.rn56dss9.lunafutral.com (amavisd-new, port 9150) with ESMTP id 03MBRTVHDVT112DXUHRJKRWD50; for rande...@richw.org; Sat, 8 Nov 2014 17:41:05 -0700 The above line contains several clues that can distinguish it from a real Received: line generated by amavisd-new, so I imagine a rule could be created to detect this and increase a message's spam score accordingly. Should I go ahead and discuss this in greater depth here on this list? Or would it be better to go off-list with a smaller group of developers, so as not to give too many ideas to the black hats? :-) rule is sandbox waiting to be promoted http://svn.apache.org/repos/asf/spamassassin/trunk/rulesrc/sandbox/axb/20_axb_misc.cf AXB_XRCVD_8B8
Re: Fake amavisd-new header lines in recent spam
On 11/09/2014 06:59 PM, Axb wrote: On 11/09/2014 06:45 PM, Rich Wales wrote: Hi. Recently, I've noticed that some spam arriving on my mail server contains a Received: header line citing amavisd-new -- possibly an attempt to trick spam filters into concluding the message has already been scanned and is presumably free of problems. Here is an example of one of these -- the physically last (i.e., chronologically first) Received: in the message. Received: by 03112d50.rn56dss9.lunafutral.com (amavisd-new, port 9150) with ESMTP id 03MBRTVHDVT112DXUHRJKRWD50; for rande...@richw.org; Sat, 8 Nov 2014 17:41:05 -0700 The above line contains several clues that can distinguish it from a real Received: line generated by amavisd-new, so I imagine a rule could be created to detect this and increase a message's spam score accordingly. Should I go ahead and discuss this in greater depth here on this list? Or would it be better to go off-list with a smaller group of developers, so as not to give too many ideas to the black hats? :-) rule is sandbox waiting to be promoted http://svn.apache.org/repos/asf/spamassassin/trunk/rulesrc/sandbox/axb/20_axb_misc.cf AXB_XRCVD_8B8 hitting like crazy and safe http://ruleqa.spamassassin.org/20141108-r1637525-n/AXB_XRCVD_8B8/detail pick it up from the sandbox and score it as high as you want.
RE: Fake amavisd-new header lines in recent spam
hitting like crazy and safe Confirmed, thank you. /MJ
Re: Fake amavisd-new header lines in recent spam
Yeah they tried a similar trick with MailScanner years ago, basically dont trust someone elses mail to tell the truth as per usual On Sunday, 9 November 2014, Marieke Janssen mjans...@myguard.nl wrote: hitting like crazy and safe Confirmed, thank you. /MJ -- -- Martin Hepworth, CISSP Oxford, UK
RE: Fake amavisd-new header lines in recent spam
Yeah they tried a similar trick with MailScanner years ago, basically dont trust someone elses mail to tell the truth as per usual You are right about trust, but in this case we can detect fake amavis-headers and score bigtime in a safe way. And from what I can tell from my logs it hits really hard. /MJ
Re: Fake amavisd-new header lines in recent spam
This *AXB_XRCVD_8B8* rule seems excessively broad to me. It seems it could wrongly catch e-mail that was legitimately Amavis-scanned on its way out by a server whose name just happened to be eight characters long. I think a better rule would take advantage of other anomalies with these fake header lines, such as the following: * There is an *extraneous semicolon* before the for clause. There should be only one semicolon in a Received: line -- namely, the one just before the date/time stamp. * There is *no from clause*. A valid Received: line from an amavisd-new scan will always have a from clause -- and further, I believe a valid from clause from amavisd-new will always reference localhost. * The Received: line from a real amavisd-new scan *shouldn't be the chronologically first* (physically last) Received: line. The first Received: line (time-wise) happens when a message is initially delivered to the local mail software; a genuine outbound amavisd-new scan will generate the chronologically *second* (physically second-to-last) Received: line. * The *port number is strange*. While it is not absolutely mandatory for an amavisd-new installation to use port 10024, I believe it is pretty much unheard of for amavisd-new to be set up to listen on ports like 7693, 7686, 7684, or 17196. Here is a sample rule which will detect the extraneous semicolon. header BOGUS_RCVD_AMAVIS Received =~ /\(amavisd-new,\s+port\s+\d+\).+;\s*for\b/ -- *Rich Wales* Palo Alto, CA ri...@richw.org
Re: Fake amavisd-new header lines in recent spam
On 11/10/2014 02:32 AM, Rich Wales wrote: This *AXB_XRCVD_8B8* rule seems excessively broad to me. It seems it could wrongly catch e-mail that was legitimately Amavis-scanned on its way out by a server whose name just happened to be eight characters long. I think a better rule would take advantage of other anomalies with these fake header lines, such as the following: * There is an *extraneous semicolon* before the for clause. There should be only one semicolon in a Received: line -- namely, the one just before the date/time stamp. * There is *no from clause*. A valid Received: line from an amavisd-new scan will always have a from clause -- and further, I believe a valid from clause from amavisd-new will always reference localhost. * The Received: line from a real amavisd-new scan *shouldn't be the chronologically first* (physically last) Received: line. The first Received: line (time-wise) happens when a message is initially delivered to the local mail software; a genuine outbound amavisd-new scan will generate the chronologically *second* (physically second-to-last) Received: line. * The *port number is strange*. While it is not absolutely mandatory for an amavisd-new installation to use port 10024, I believe it is pretty much unheard of for amavisd-new to be set up to listen on ports like 7693, 7686, 7684, or 17196. Here is a sample rule which will detect the extraneous semicolon. header BOGUS_RCVD_AMAVIS Received =~ /\(amavisd-new,\s+port\s+\d+\).+;\s*for\b/ do we have your permission to add this rule to SA's masscheck / autopromoting ?
AWL in SQL with amavisd-new
Hi, I am using the auto-whitelist feature of SpamAssassin stored into a PostgreSQL database. It works fine but I have got one issue: as I am calling SA from amavisd-new, the username stored in the AWL SQL table is always amavis. Now this renders my AWL useless as the username should actually be the From: header field of an email. So my question here really is: how do I get to log the From: as uername and not the amavis username in my AWL table? Regards ML
Re: AWL in SQL with amavisd-new
On 6/26/2014 10:31 AM, ML mail wrote: I am using the auto-whitelist feature of SpamAssassin stored into a PostgreSQL database. It works fine but I have got one issue: as I am calling SA from amavisd-new, the username stored in the AWL SQL table is always amavis. Now this renders my AWL useless as the username should actually be the From: header field of an email. So my question here really is: how do I get to log the From: as uername and not the amavis username in my AWL table? This is known as site-wide AWL because by default, you are using the username that invokes the call to SA. What I do is I use MIMEDefang and call spamc from MIMEDefang passing the username based on code that I run to determine. Not exactly easy... But the reason I'm posting is that many servers run sitewide AWL without issue. Why do you feel it is useless? Also, we're testing TxRep as a replacement to AWL in trunk if you want to look at that. Regards, KAM
Re: AWL in SQL with amavisd-new
Ok so if I understand you correctly you are saying that it is possible to use AWL as site-wide having just one part of the e-mail exchange (the To: field) and this works fine/reliabily? On Thursday, June 26, 2014 4:34 PM, Kevin A. McGrail kmcgr...@pccc.com wrote: On 6/26/2014 10:31 AM, ML mail wrote: I am using the auto-whitelist feature of SpamAssassin stored into a PostgreSQL database. It works fine but I have got one issue: as I am calling SA from amavisd-new, the username stored in the AWL SQL table is always amavis. Now this renders my AWL useless as the username should actually be the From: header field of an email. So my question here really is: how do I get to log the From: as uername and not the amavis username in my AWL table? This is known as site-wide AWL because by default, you are using the username that invokes the call to SA. What I do is I use MIMEDefang and call spamc from MIMEDefang passing the username based on code that I run to determine. Not exactly easy... But the reason I'm posting is that many servers run sitewide AWL without issue. Why do you feel it is useless? Also, we're testing TxRep as a replacement to AWL in trunk if you want to look at that. Regards, KAM
Re: AWL in SQL with amavisd-new
On Thu, 26 Jun 2014 07:42:50 -0700 ML mail wrote: Ok so if I understand you correctly you are saying that it is possible to use AWL as site-wide having just one part of the e-mail exchange (the To: field) and this works fine/reliabily? To: isn't relevant, you either have site-wide or per user AWL. There's not much point to the latter unless you also have per user BAYES as well. SA gets the sender from the headers.
Re: AWL in SQL with amavisd-new
Kevin A. McGrail skrev den 2014-06-26 16:34: But the reason I'm posting is that many servers run sitewide AWL without issue. Why do you feel it is useless? multi recipient is handled better in amavisd-new, but its not very well dokumented, if you always just get single recipient spams its not an issue with speed Mark did a well job of scanning once, and score pr recipent, with settings in sa_user sql prefs, nearly as good spamd/spamc does, or might even be better, dont know yet :( but i will love to get back to amavisd-new with amavisd-milter now currently using spampd here without any issue but to the awl being global or pr recipient is another story that imho not make sense, but bayes username does if not whising to use global bayes more qeustions later for txrep
Re: AWL in SQL with amavisd-new
ML mail skrev den 2014-06-26 16:42: Ok so if I understand you correctly you are saying that it is possible to use AWL as site-wide having just one part of the e-mail exchange (the To: field) and this works fine/reliabily? incorrect question, incorrect answer :=) the username in awl is the unix running user, not email addresses, therefor amavisd is single user pr default, if you like to chain it to be fully virtual you must ask more qestions here on how spamd/spamc works first the awl user must be mapped to email before it can be used as virtual user i am lost aswell
Re: AWL in SQL with amavisd-new
I got it all wrong: I was assuming that AWL works by using a tuple consisting of to/from (in the database: username/mail). Now thanks to your explanation I got it that the username is in fact only used for user-bound AWL. This means that I can simply use site-wide AWL. TxRep sounds quite promising btw! On Thursday, June 26, 2014 6:26 PM, Benny Pedersen m...@junc.eu wrote: ML mail skrev den 2014-06-26 16:42: Ok so if I understand you correctly you are saying that it is possible to use AWL as site-wide having just one part of the e-mail exchange (the To: field) and this works fine/reliabily? incorrect question, incorrect answer :=) the username in awl is the unix running user, not email addresses, therefor amavisd is single user pr default, if you like to chain it to be fully virtual you must ask more qestions here on how spamd/spamc works first the awl user must be mapped to email before it can be used as virtual user i am lost aswell
Re: SpamAssassin with Amavisd-new
Timothy Murphy skrev den 2013-07-12 15:09: I have Postfix/Dovecot running on my CentOS-6.4 server, and I'm trying to add SpamAssassin through Amavisd. I have followed meticulously the instructions in http://wiki.centos.org/HowTos/Amavisd. see 2. Installation As far as I can tell, amavisd and clamav are running, as also is spamd. spamd is not used in amavisd, waste of resources I have set ok_languages en it fr de ga in /etc/mail/spamassassin/local.cf spamassassin 21 -D --lint | less is textcat plugin loaded ? is there missing perlmodules ? in dependice ?, that part can be seen in --lint but am still getting oodles of spam (I assume) in Chinese. What am I doing wrong? reading incorrect docs, using incorrect distros, i stop here :=)
SpamAssassin with Amavisd-new
I have Postfix/Dovecot running on my CentOS-6.4 server, and I'm trying to add SpamAssassin through Amavisd. I have followed meticulously the instructions in http://wiki.centos.org/HowTos/Amavisd. As far as I can tell, amavisd and clamav are running, as also is spamd. I have set ok_languages en it fr de ga in /etc/mail/spamassassin/local.cf but am still getting oodles of spam (I assume) in Chinese. What am I doing wrong? -- Timothy Murphy e-mail: gayleard /at/ eircom.net tel: +353-86-2336090, +353-1-2842366 School of Mathematics, Trinity College, Dublin 2, Ireland
Re: SpamAssassin with Amavisd-new
On Fri, 12 Jul 2013 15:09:10 +0200 Timothy Murphy wrote: I have Postfix/Dovecot running on my CentOS-6.4 server, and I'm trying to add SpamAssassin through Amavisd. I have followed meticulously the instructions in http://wiki.centos.org/HowTos/Amavisd. As far as I can tell, amavisd and clamav are running, as also is spamd. I have set ok_languages en it fr de ga in /etc/mail/spamassassin/local.cf but am still getting oodles of spam (I assume) in Chinese. What am I doing wrong? The textcat plugin doesn't work with unicode.
Re: SpamAssassin with Amavisd-new
On Fri, 2013-07-12 at 15:09 +0200, Timothy Murphy wrote: I have Postfix/Dovecot running on my CentOS-6.4 server, and I'm trying to add SpamAssassin through Amavisd. I have followed meticulously the instructions in http://wiki.centos.org/HowTos/Amavisd. As far as I can tell, amavisd and clamav are running, as also is spamd. I have set ok_languages en it fr de ga in /etc/mail/spamassassin/local.cf but am still getting oodles of spam (I assume) in Chinese. What am I doing wrong? Be aware that all SA does is to add a header to every message giving its spammminess score. Something else has to use that score to recognise and deal with spam. So, what program(s) look at the message after SA has scored it and what do they with spam that differs from what they do with ham? Martin
Re: SpamAssassin with Amavisd-new
please send logs to see what is happening BR DEJan On Fri, Jul 12, 2013 at 3:09 PM, Timothy Murphy gayle...@alice.it wrote: I have Postfix/Dovecot running on my CentOS-6.4 server, and I'm trying to add SpamAssassin through Amavisd. I have followed meticulously the instructions in http://wiki.centos.org/HowTos/Amavisd. As far as I can tell, amavisd and clamav are running, as also is spamd. I have set ok_languages en it fr de ga in /etc/mail/spamassassin/local.cf but am still getting oodles of spam (I assume) in Chinese. What am I doing wrong? -- Timothy Murphy e-mail: gayleard /at/ eircom.net tel: +353-86-2336090, +353-1-2842366 School of Mathematics, Trinity College, Dublin 2, Ireland
Re: SpamAssassin with Amavisd-new
On 7/12/2013 9:09 AM, Timothy Murphy wrote: I have Postfix/Dovecot running on my CentOS-6.4 server, and I'm trying to add SpamAssassin through Amavisd. I have followed meticulously the instructions in http://wiki.centos.org/HowTos/Amavisd. As far as I can tell, amavisd and clamav are running, as also is spamd. I have set ok_languages en it fr de ga in /etc/mail/spamassassin/local.cf but am still getting oodles of spam (I assume) in Chinese. What am I doing wrong? Well, this shouldn't affect the results, but if you followed those instructions, you should NOT have spamd running. SA is being run from amavisd, running spamd as well is a waste of resources. To determine why you are getting the spam, take a look at the headers. Are there amavis/spamd headers there? Do the headers say it is spam? -- Bowie
Re: Greylisting (was Re: Fairly-Secure Anti-SPAM Gateway Using OpenBSD, Postfix, Amavisd-new, SpamAssassin, Razor and DCC ? Can I get your opinion?)
On 11/29/12 14:46:25, David F. Skoll wrote: We greylist after the end of DATA. This wastes bandwidth, but lets us use the Subject: line as an additional mix in the greylisting tuple. This catches ratware that retries in the face of greylisting, but mutates the subject line with each retry. We use grey listing on our low volume server, and as others have noted, it works well because a high percentage of spam bots do not bother to retry. But as others have mentioned, it can be painful waiting for the delayed confirmation on a registration to a web site to come in an hour/two later, or email from a new client who is waiting on a response. Since this is a Spam Assassin list: Is there a way of disabling grey listing, but still receiving some benefit from the principle that mail received from a first time or infrequent sender should be looked upon with some suspicion? Assume that either some to-be-implemented SA filter, or some mail gateway front-end (like MIMEDefang), adds a new tag/two, for example: SENDER_FIRST_RCPT, SENDER_LOW_FREQ, SENDER_HI_FREQ, or SENDER_HI_AVE_SA_SCORE? All these tags might be based upon some look back period (say: 90 days). Theoretically, these new tags could be calculated after the fact when passing through a spam corpus. And since many/most grey listing systems differentiate by some form of (sender, recipient) pairing this analysis can be reliably/repeatably performed by an SA plug-in at the point of delivery to the user, if needed. It would need to be shown that these new tags improve the ability to discriminate spam from ham. If the scheme worked well, there might be no need for grey listing at all.
Re: Fairly-Secure Anti-SPAM Gateway Using OpenBSD, Postfix, Amavisd-new, SpamAssassin, Razor and DCC ? Can I get your opinion?
On 11/29/12 10:44:54, John Hardin wrote: You will probably want to put a little effort into maintaining lists of regular correspondents who can bypass greylisting. There may be tools to automate that, e.g. to whitelist someone a local user has sent mail to. Has anyone looked into the use of a DNS-based white listing service? For example: http://www.dnswl.org/ It might be interesting to make a pass over a grey list database and see if the sites white listed there appear in the registry. And that sites that were black listed or simply did not retry are _not_ listed in the white list.
Re: Greylisting (was Re: Fairly-Secure Anti-SPAM Gateway Using OpenBSD, Postfix, Amavisd-new, SpamAssassin, Razor and DCC ? Can I get your opinion?)
We greylist after the end of DATA. This wastes bandwidth, but lets us use the Subject: line as an additional mix in the greylisting tuple. This catches ratware that retries in the face of greylisting, but mutates the subject line with each retry. We use grey listing on our low volume server, and as others have noted, it works well because a high percentage of spam bots do not bother to retry. But as others have mentioned, it can be painful waiting for the delayed confirmation on a registration to a web site to come in an hour/two later, or email from a new client who is waiting on a response. Using dnswl.org to whitelist against greylisting might help some. Since this is a Spam Assassin list: Is there a way of disabling grey listing, but still receiving some benefit from the principle that mail received from a first time or infrequent sender should be looked upon with some suspicion? Assume that either some to-be-implemented SA filter, or some mail gateway front-end (like MIMEDefang), adds a new tag/two, for example: SENDER_FIRST_RCPT, SENDER_LOW_FREQ, SENDER_HI_FREQ, or SENDER_HI_AVE_SA_SCORE? All these tags might be based upon some look back period (say: 90 days). Theoretically, these new tags could be calculated after the fact when passing through a spam corpus. And since many/most grey listing systems differentiate by some form of (sender, recipient) pairing this analysis can be reliably/repeatably performed by an SA plug-in at the point of delivery to the user, if needed. It would need to be shown that these new tags improve the ability to discriminate spam from ham. If the scheme worked well, there might be no need for grey listing at all.
Re: Fairly-Secure Anti-SPAM Gateway Using OpenBSD, Postfix, Amavisd-new, SpamAssassin, Razor and DCC ? Can I get your opinion?
You will probably want to put a little effort into maintaining lists of regular correspondents who can bypass greylisting. There may be tools to automate that, e.g. to whitelist someone a local user has sent mail to. Has anyone looked into the use of a DNS-based white listing service? For example: http://www.dnswl.org/ It might be interesting to make a pass over a grey list database and see if the sites white listed there appear in the registry. And that sites that were black listed or simply did not retry are _not_ listed in the white list. Been using it at least couple years to bypass greylisting. Seems to give no negative impact. Be sure to add the IP of your servers there.
Re: Greylisting (was Re: Fairly-Secure Anti-SPAM Gateway Using OpenBSD, Postfix, Amavisd-new, SpamAssassin, Razor and DCC ? Can I get your opinion?)
On Mon, 2012-12-03 at 07:23 -0800, Gary Funck wrote: Since this is a Spam Assassin list: Is there a way of disabling grey listing, but still receiving some benefit from the principle that mail received from a first time or infrequent sender should be looked upon with some suspicion? Yes. If you keep a list of the recipients of outgoing mail its easy to whitelist any mail you receive from them. This approach does what you want: a sender is treated as suspicious until you've sent mail to them and recipient list maintenance is easy to automate. I use a mail archive system as my recipients list because it has a record of everybody I've sent mail to. I use an SA plugin to access the archive. The combination of it and an associated rule will whitelist anybody who is recorded in the archive as having received mail from me. However, the database archives messages at 4-6 /sec, so this and/or the storage requirements (4.3 GB to store 143,000 messages) may mean that, if you're a high volume site and/or don't need an archive, you'd be better off just keeping a list of the recipient(s) of outgoing messages. I wrote my archive for personal use because I can find an old e-mail with the archive search tool faster than I can by ferreting though a set of mail folders: it was never designed as a high volume solution, but should manage small business volumes quite easily with both it and SA running on a typical desktop PC. Up to early this year I was using an 866 MHz P3 with 512MB RAM that easily kept up while PostgreSQL,the archive, Postfix and SA. That is all now running on a 3GHz dual Athlon with 4 GB RAM but not going any faster - an upgrade to Fedora 16 forced the change because its installer wouldn't run in less than 1GB RAM. If you think my SA plugin or the mail archive would be of use to you, contact me off-list. Martin
Re: Fairly-Secure Anti-SPAM Gateway Using OpenBSD, Postfix, Amavisd-new, SpamAssassin, Razor and DCC ? Can I get your opinion?
On Mon, 2012-12-03 at 07:27 -0800, Gary Funck wrote: On 11/29/12 10:44:54, John Hardin wrote: You will probably want to put a little effort into maintaining lists of regular correspondents who can bypass greylisting. There may be tools to automate that, e.g. to whitelist someone a local user has sent mail to. Has anyone looked into the use of a DNS-based white listing service? Everybody's mail stream is different (I don't see any of the spam types discussed over the last week or two) so my guess is that any public whitelister would not be specific enough for any particular site. Its quite likely that stuff you and your users don't want would be whitelisted by it and OTOH you probably have a few mail sources that you want to see but aren't being whitelisted. For instance, I doubt that a US-based whitelister would whitelist customer information sent out by, say, Australian energy companies or British telcos. Martin
Re: Greylisting (was Re: Fairly-Secure Anti-SPAM Gateway Using OpenBSD, Postfix, Amavisd-new, SpamAssassin, Razor and DCC ? Can I get your opinion?)
On Mon, 3 Dec 2012 07:23:59 -0800 Gary Funck wrote: Since this is a Spam Assassin list: Is there a way of disabling grey listing, but still receiving some benefit from the principle that mail received from a first time or infrequent sender should be looked upon with some suspicion? Personally I wouldn't want to do it that way round - with a positive score for unknown rather than a negative score for known. YMMV but almost all of the FPs I've had in the last ten years have been that sort of mail because it's less likely to be recognised by Bayes.
Re: Fairly-Secure Anti-SPAM Gateway Using OpenBSD, Postfix, Amavisd-new, SpamAssassin, Razor and DCC ? Can I get your opinion?
Ed, I'm looking to set up a spam filtering server to replace our ISP's spam filtering service. I've seen this tutorial ( ftp://orn.mpg.de/pub/unix/mail/Fairly-Secure_Anti-SPAM_Gateway_Using_SpamAssassin.html#antivirus ) and I'd be very interested in YOUR opinion; do you think, fundamentally, a server with these software packages could be an effective combination at fighting spam? We're a (I guess) medium size organization with appx. 1000 end users. What about weaving clam-av into the mix? Although this tutorial uses OpenBSD, I'll probably be using FreeBSD. Thank you for your input! I use the same setting on FreeBSD with good enought results. Most of the products are from the ports. I have added to the scheme: - postgrey: grey listing is a very effective way to drop spam, at the cost of a 15 to 60 minutes delay in incoming email; - ClamAV and Kaspersky for viruses (even though there are not that many lately); they fit well in amavis as amavis was preliminarily designed to catch viruses... - procmail to handle the mail delivery and quarantine and daily summary of spam. I have 250 users. Good luk, Olivier
Re: Fairly-Secure Anti-SPAM Gateway Using OpenBSD, Postfix, Amavisd-new, SpamAssassin, Razor and DCC ? Can I get your opinion?
Gentlemen, Thank you for your feedback! I'll be sure to check into Postgrey. Are there any special considerations to installing/configuring it or is it simply a matter of installing, reading the docs and configuring? Ed
Re: Fairly-Secure Anti-SPAM Gateway Using OpenBSD, Postfix, Amavisd-new, SpamAssassin, Razor and DCC ? Can I get your opinion?
Am 29.11.2012 17:04, schrieb Ed Flecko: Gentlemen, Thank you for your feedback! I'll be sure to check into Postgrey. Are there any special considerations to installing/configuring it or is it simply a matter of installing, reading the docs and configuring? Ed yes dont do greylist all, use selective also for other checks like rbl, spf etc i.e http://www.arschkrebs.de/postfix/postfix_greylisting.shtml i dont use amavis on gateways i use spamass-milter with sanesecurity antispam sigs and clamav-milter but thats mostly a matter of taste amavis has tons of more features but therefor its more complex anyway in milter mode you are able to reject on smtp income stage Best Regards MfG Robert Schetterer -- [*] sys4 AG http://sys4.de, +49 (89) 30 90 46 64 Franziskanerstraße 15, 81669 München Sitz der Gesellschaft: München, Amtsgericht München: HRB 199263 Vorstand: Patrick Ben Koetter, Axel von der Ohe, Marc Schiffbauer Aufsichtsratsvorsitzender: Joerg Heidrich
Re: Fairly-Secure Anti-SPAM Gateway Using OpenBSD, Postfix, Amavisd-new, SpamAssassin, Razor and DCC ? Can I get your opinion?
On Thu, 29 Nov 2012, Ed Flecko wrote: I'll be sure to check into Postgrey. Are there any special considerations to installing/configuring it or is it simply a matter of installing, reading the docs and configuring? The biggest consideration is not technical, it's managing the expectations of your users. You will need to educate your users that email is *not* instant messaging. You will probably want to put a little effort into maintaining lists of regular correspondents who can bypass greylisting. There may be tools to automate that, e.g. to whitelist someone a local user has sent mail to. Some users are extremely allergic to any delays in their email; you may have to maintain a list of exception destination addresses to keep them happy, or for addresses where no delay is acceptable, e.g. support@... or sales@... -- John Hardin KA7OHZhttp://www.impsec.org/~jhardin/ jhar...@impsec.orgFALaholic #11174 pgpk -a jhar...@impsec.org key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C AF76 D822 E6E6 B873 2E79 --- Bother, said Pooh as he struggled with /etc/sendmail.cf, it never does quite what I want. I wish Christopher Robin was here. -- Peter da Silva in a.s.r --- 26 days until Christmas
Re: Fairly-Secure Anti-SPAM Gateway Using OpenBSD, Postfix, Amavisd-new, SpamAssassin, Razor and DCC ? Can I get your opinion?
Good thoughts...thank you John. Ed
Re: Fairly-Secure Anti-SPAM Gateway Using OpenBSD, Postfix, Amavisd-new, SpamAssassin, Razor and DCC ? Can I get your opinion?
From: John Hardin jhar...@impsec.org Some users are extremely allergic to any delays in their email; you may have to maintain a list of exception destination addresses to keep them happy, or for addresses where no delay is acceptable, e.g. support@... or sales@... I fully agree. When I purchase an air-line ticket, I want the mail immediately in my inbox. If the greylisting software replies a 4xx Please come back in 299 seconds, the truth is that you will have to wait an undetermined amount of time, depending on the sending server setup, and not at all under your control. Very frustrating. Use good blacklists such as zen.spamhaus.org (free for small installations). Frédéric De Mees Brussels
Re: Fairly-Secure Anti-SPAM Gateway Using OpenBSD, Postfix, Amavisd-new, SpamAssassin, Razor and DCC ? Can I get your opinion?
From: John Hardin jhar...@impsec.org I fully agree. When I purchase an air-line ticket, I want the mail immediately in my inbox. If the greylisting software replies a 4xx Please come back in 299 seconds, the truth is that you will have to wait an undetermined amount of time, depending on the sending server setup, and not at all under your control. Very frustrating. I use a blend of greylisting and spamassassin, so that only mails which are close to the margin by SA score get greylisted; lower-scoring mails are accepted immediately, and high-scoring mails are rejected outright. It works pretty well. I've never had any complaints about delivery speed, but some senders have broken mail servers that don't retry on receiving a temporary failure. Greylisting.org maintains an incomplete list of such servers: http://www.greylisting.org/whitelisting.shtml --Ian
Greylisting (was Re: Fairly-Secure Anti-SPAM Gateway Using OpenBSD, Postfix, Amavisd-new, SpamAssassin, Razor and DCC ? Can I get your opinion?)
On Thu, 29 Nov 2012 14:36:45 -0500 vec...@vectro.org wrote: I've never had any complaints about delivery speed, but some senders have broken mail servers that don't retry on receiving a temporary failure. Many such servers use broken SMTP implementations that can't handle a 4xx code in response to RCPT properly. We greylist after the end of DATA. This wastes bandwidth, but lets us use the Subject: line as an additional mix in the greylisting tuple. This catches ratware that retries in the face of greylisting, but mutates the subject line with each retry. Also, once a given IP passes greylisting, we remember that and we don't greylist that server for 40 days. If you have a large-enough user population, this can greatly mitigate the problems caused by initial greylisting delays. Regards, David.
Re: Fairly-Secure Anti-SPAM Gateway Using OpenBSD, Postfix, Amavisd-new, SpamAssassin, Razor and DCC ? Can I get your opinion?
I'll expand a little on John's comments below On 29/11/12 18:44, John Hardin wrote: On Thu, 29 Nov 2012, Ed Flecko wrote: I'll be sure to check into Postgrey. Are there any special considerations to installing/configuring it or is it simply a matter of installing, reading the docs and configuring? The biggest consideration is not technical, it's managing the expectations of your users. You will need to educate your users that email is *not* instant messaging. Indeed. But do also play around with the delays in postgrey (--delay). A minimal delay of 60 seconds is enough to force a retry and is adequate - legit hosts will retry, non-legit hosts won't so a longer delay is generally unnecessary. You will probably want to put a little effort into maintaining lists of regular correspondents who can bypass greylisting. There may be tools to automate that, e.g. to whitelist someone a local user has sent mail to. Postgrey has an auto-whitelisting mechanism that can be fine tuned by reducing the number of times a client must successfully retry (--auto-whitelist-clients) before auto-whitelisting and adjusting the age of the cache (--max-age) so whitelisted clients are cached for longer. Generally after a couple weeks of normal mail flow, all regular hosts should be cached so only new contacts will get greylisted. Also don't be afraid to whitelist big clients that you receive correspondence from - you know they are legit and will resend so it's pointless greylisting them. Postgrey is very configurable and all the options above are documented in the manpage. Some users are extremely allergic to any delays in their email; you may have to maintain a list of exception destination addresses to keep them happy, or for addresses where no delay is acceptable, e.g. support@... or sales@...
Re: Fairly-Secure Anti-SPAM Gateway Using OpenBSD, Postfix, Amavisd-new, SpamAssassin, Razor and DCC ? Can I get your opinion?
On 11/29/2012 12:01, Ned Slider wrote: Indeed. But do also play around with the delays in postgrey (--delay). A minimal delay of 60 seconds is enough to force a retry and is adequate - legit hosts will retry, non-legit hosts won't so a longer delay is generally unnecessary. This is only one of the benefits of greylisting; it's one that spammers can trivially bypass by implementing a retry mechanism of their own. The other benefit of greylisting is that you can defer (or re-check) DNSBLs before making the final decision to accept or decline, so a fresh zombie or new spam sender doesn't get a free bite at the inbox. Instead, fact-acting DNSBLs have a chance to get the new sender listed before a greylist retry period expires. Here we do a combination of the two approaches, immediately whitelisting any address to which the user has sent mail in the past, as well as a fairly large list of known senders. After that, we only look at greylisting if the session or message is otherwise a bit suspicious, be it missing or mismatching rDNS, SPF softfail or worse, DK/DKIM failures, BAYES 70+ or SpamAssassin 4+, etc. If it trips one of these normally-too-sensitive-to-use-for-blocking rules, it gets passed over to the greylisting subsystem and then can try again after a few minutes before getting through. This has proved to work very well since it allows a majority of legitimate mail through without greylisting even on the first attempt, but still nets us most of the benefits of greylisting in the end. -- Dave Warren http://www.hireahit.com/ http://ca.linkedin.com/in/davejwarren
Re: Greylisting (was Re: Fairly-Secure Anti-SPAM Gateway Using OpenBSD, Postfix, Amavisd-new, SpamAssassin, Razor and DCC ? Can I get your opinion?)
On 11/29/2012 08:46 PM, David F. Skoll wrote: [...] Also, once a given IP passes greylisting, we remember that and we don't greylist that server for 40 days. If you have a large-enough user population, this can greatly mitigate the problems caused by initial greylisting delays. Do you treat yahoo like spam sources in the same way?
Re: Greylisting (was Re: Fairly-Secure Anti-SPAM Gateway Using OpenBSD, Postfix, Amavisd-new, SpamAssassin, Razor and DCC ? Can I get your opinion?)
On 11/29/2012 12:27, Andrzej A. Filip wrote: On 11/29/2012 08:46 PM, David F. Skoll wrote: [...] Also, once a given IP passes greylisting, we remember that and we don't greylist that server for 40 days. If you have a large-enough user population, this can greatly mitigate the problems caused by initial greylisting delays. Do you treat yahoo like spam sources in the same way? There's almost no point in greylisting an IP that you know will retry properly anyway, so why wouldn't you allow that IP to bypass greylisting in the future? -- Dave Warren http://www.hireahit.com/ http://ca.linkedin.com/in/davejwarren
Re: Greylisting (was Re: Fairly-Secure Anti-SPAM Gateway Using OpenBSD, Postfix, Amavisd-new, SpamAssassin, Razor and DCC ? Can I get your opinion?)
Am 29.11.2012 20:46, schrieb David F. Skoll: On Thu, 29 Nov 2012 14:36:45 -0500 vec...@vectro.org wrote: I've never had any complaints about delivery speed, but some senders have broken mail servers that don't retry on receiving a temporary failure. Many such servers use broken SMTP implementations that can't handle a 4xx code in response to RCPT properly. We greylist after the end of DATA. This wastes bandwidth, but lets us use the Subject: line as an additional mix in the greylisting tuple. This catches ratware that retries in the face of greylisting, but mutates the subject line with each retry. Also, once a given IP passes greylisting, we remember that and we don't greylist that server for 40 days. If you have a large-enough user population, this can greatly mitigate the problems caused by initial greylisting delays. Regards, David. greylisting isnt state of art, however it might helpfull in some domains ( everyone has its own spam), using postscreen with postfix before selective greylisting is a good choice Best Regards MfG Robert Schetterer -- [*] sys4 AG http://sys4.de, +49 (89) 30 90 46 64 Franziskanerstraße 15, 81669 München Sitz der Gesellschaft: München, Amtsgericht München: HRB 199263 Vorstand: Patrick Ben Koetter, Axel von der Ohe, Marc Schiffbauer Aufsichtsratsvorsitzender: Joerg Heidrich
Re: Greylisting (was Re: Fairly-Secure Anti-SPAM Gateway Using OpenBSD, Postfix, Amavisd-new, SpamAssassin, Razor and DCC ? Can I get your opinion?)
On 11/29/2012 09:31 PM, Dave Warren wrote: On 11/29/2012 12:27, Andrzej A. Filip wrote: On 11/29/2012 08:46 PM, David F. Skoll wrote: [...] Also, once a given IP passes greylisting, we remember that and we don't greylist that server for 40 days. If you have a large-enough user population, this can greatly mitigate the problems caused by initial greylisting delays. Do you treat yahoo like spam sources in the same way? There's almost no point in greylisting an IP that you know will retry properly anyway, so why wouldn't you allow that IP to bypass greylisting in the future? I assume that greylisting of yahoo like spam sources increases chances of bulk detectors detecting spam. Is not it trues? [based on real data]
Re: Greylisting (was Re: Fairly-Secure Anti-SPAM Gateway Using OpenBSD, Postfix, Amavisd-new, SpamAssassin, Razor and DCC ? Can I get your opinion?)
On Thu, 29 Nov 2012 21:27:19 +0100 Andrzej A. Filip andrzej.fi...@gmail.com wrote: Do you treat yahoo like spam sources in the same way? With respect to greylisting, of course. If a machine passes greylisting once, it's extremely likely to pass it in future and it's an utter waste of time to greylist it. Regards, David.
Re: Greylisting (was Re: Fairly-Secure Anti-SPAM Gateway Using OpenBSD, Postfix, Amavisd-new, SpamAssassin, Razor and DCC ? Can I get your opinion?)
On 11/29/2012 09:53 PM, David F. Skoll wrote: On Thu, 29 Nov 2012 21:27:19 +0100 Andrzej A. Filip andrzej.fi...@gmail.com wrote: Do you treat yahoo like spam sources in the same way? With respect to greylisting, of course. If a machine passes greylisting once, it's extremely likely to pass it in future and it's an utter waste of time to greylist it. Does greylisting increase chances of bulk detectors (razor/pyzor/dcc) in case of yahoo like spam sources? [ based on your experience ]
Re: Greylisting (was Re: Fairly-Secure Anti-SPAM Gateway Using OpenBSD, Postfix, Amavisd-new, SpamAssassin, Razor and DCC ? Can I get your opinion?)
On Thu, 29 Nov 2012 21:59:45 +0100 Andrzej A. Filip andrzej.fi...@gmail.com wrote: Does greylisting increase chances of bulk detectors (razor/pyzor/dcc) in case of yahoo like spam sources? [ based on your experience ] I suppose it might, but I don't use razor, pyzor, dcc or anything similar so I have no personal experience. Regards, David.
Re: Greylisting (was Re: Fairly-Secure Anti-SPAM Gateway Using OpenBSD, Postfix, Amavisd-new, SpamAssassin, Razor and DCC ? Can I get your opinion?)
I've never had any complaints about delivery speed, but some senders have broken mail servers that don't retry on receiving a temporary failure. Many such servers use broken SMTP implementations that can't handle a 4xx code in response to RCPT properly. We greylist after the end of DATA. This wastes bandwidth, but lets us use the Subject: line as an additional mix in the greylisting tuple. This catches ratware that retries in the face of greylisting, but mutates the subject line with each retry. Also, once a given IP passes greylisting, we remember that and we don't greylist that server for 40 days. If you have a large-enough user population, this can greatly mitigate the problems caused by initial greylisting delays. Every 60 seconds we look at all messages that arrived in last 60 seconds. If there Spamassassin score is less the 1 we add that server to a whitelist for 6 months. If its already on whitelist we update the last message time. If a message scores over 5 we remove it from whitelist if its on it. We do not greylist servers on the whitelist. Works very well. Even though we use greylisting our users very rarely notice if at all due to this.
Re: Greylisting (was Re: Fairly-Secure Anti-SPAM Gateway Using OpenBSD, Postfix, Amavisd-new, SpamAssassin, Razor and DCC ? Can I get your opinion?)
Just wondering how many boxes: rcpt domains: rcpt users: you guys are sending through greylisting. Axb
Re: Greylisting (was Re: Fairly-Secure Anti-SPAM Gateway Using OpenBSD, Postfix, Amavisd-new, SpamAssassin, Razor and DCC ? Can I get your opinion?)
On Thu, 29 Nov 2012, David F. Skoll wrote: On Thu, 29 Nov 2012 21:27:19 +0100 Andrzej A. Filip andrzej.fi...@gmail.com wrote: Do you treat yahoo like spam sources in the same way? With respect to greylisting, of course. If a machine passes greylisting once, it's extremely likely to pass it in future and it's an utter waste of time to greylist it. Modulo spamvertised URIs and spam checksums sent via such hosts, particularly if they are freemail. Filtering out the spambots who don't retry (and as trivial as that is to defeat, a large amount still gets blocked by this in my experience) is not the _only_ reason to greylist. Giving the URIBLs a chance to list a new URI and the checksum services a chance to recognize a new body are also benefits of greylisting. (But, as you said, you don't take advantage of those tools.) Also, greylisting generally keys on host+sender, not just host. -- John Hardin KA7OHZhttp://www.impsec.org/~jhardin/ jhar...@impsec.orgFALaholic #11174 pgpk -a jhar...@impsec.org key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C AF76 D822 E6E6 B873 2E79 --- Bother, said Pooh as he struggled with /etc/sendmail.cf, it never does quite what I want. I wish Christopher Robin was here. -- Peter da Silva in a.s.r --- 26 days until Christmas
Re: Greylisting (was Re: Fairly-Secure Anti-SPAM Gateway Using OpenBSD, Postfix, Amavisd-new, SpamAssassin, Razor and DCC ? Can I get your opinion?)
On Thu, 29 Nov 2012 22:47:45 +0100 Axb axb.li...@gmail.com wrote: boxes: About 50 000 rcpt domains: About 2000 rcpt users: Lots. I don't have an exact figure. you guys are sending through greylisting. This is on our machines. Our larger customers have significantly higher numbers. Regards, David.
Re: Greylisting (was Re: Fairly-Secure Anti-SPAM Gateway Using OpenBSD, Postfix, Amavisd-new, SpamAssassin, Razor and DCC ? Can I get your opinion?)
Does greylisting increase chances of bulk detectors (razor/pyzor/dcc) in case of yahoo like spam sources? No. A remarkable fraction of ratware still doesn't bother to retry, so the most simple minded greylister will deter them. That's why it's useful. I've never seen any support for the theory that greylisting delays make it more likely that the host will be blacklisted when it retries. I haven't seen many legit senders that don't retry as David says he has, but I don't have his volume of mail, either.
Re: Greylisting (was Re: Fairly-Secure Anti-SPAM Gateway Using OpenBSD, Postfix, Amavisd-new, SpamAssassin, Razor and DCC ? Can I get your opinion?)
On Thu, 30 Nov 2012, John Levine wrote: Does greylisting increase chances of bulk detectors (razor/pyzor/dcc) in case of yahoo like spam sources? No. A remarkable fraction of ratware still doesn't bother to retry, so the most simple minded greylister will deter them. That's why it's useful. I've never seen any support for the theory that greylisting delays make it more likely that the host will be blacklisted when it retries. It's not so much the host being blacklisted, as a checksum of the spam being published by pyzor et. al., or for spamvertised websites in the spam being published by URIBLs, so that when the sender tries again the score for that message will be higher than it would the first time around, hopefully high enough to classify it as spam rather than a FN. -- John Hardin KA7OHZhttp://www.impsec.org/~jhardin/ jhar...@impsec.orgFALaholic #11174 pgpk -a jhar...@impsec.org key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C AF76 D822 E6E6 B873 2E79 --- Bother, said Pooh as he struggled with /etc/sendmail.cf, it never does quite what I want. I wish Christopher Robin was here. -- Peter da Silva in a.s.r --- 26 days until Christmas
Re: Greylisting (was Re: Fairly-Secure Anti-SPAM Gateway Using OpenBSD, Postfix, Amavisd-new, SpamAssassin, Razor and DCC ? Can I get your opinion?)
On Thu, 29 Nov 2012 18:01:38 -0800 (PST) John Hardin jhar...@impsec.org wrote: It's not so much the host being blacklisted, as a checksum of the spam being published by pyzor et. al., or for spamvertised websites in the spam being published by URIBLs, so that when the sender tries again the score for that message will be higher than it would the first time around, hopefully high enough to classify it as spam rather than a FN. I would love to gather some hard data on this. Maybe a research project for the future... since we do our greylisting post-DATA, we could in principle run all the content-filtering and URIBL lookups and check if the score changes between the first attempt and the final attempt after greylisting. Or those who use SA without greylisting could reprocess messages after an hour or two and see if the score goes up. [My gut instinct says that a reasonable greylisting interval is too short for most DNSBLs to react. Pyzor/Razor/DCC may be somewhat more adept at reacting quickly.] Regards, David.
Re: Greylisting (was Re: Fairly-Secure Anti-SPAM Gateway Using OpenBSD, Postfix, Amavisd-new, SpamAssassin, Razor and DCC ? Can I get your opinion?)
On 11/29/2012 17:37, John Levine wrote: Does greylisting increase chances of bulk detectors (razor/pyzor/dcc) in case of yahoo like spam sources? No. A remarkable fraction of ratware still doesn't bother to retry, so the most simple minded greylister will deter them. That's why it's useful. I've never seen any support for the theory that greylisting delays make it more likely that the host will be blacklisted when it retries. If I run my accepted-and-quarantined spam corpus through a filter to test against DNSBL effectiveness, I always see higher effectiveness ratings than what was shown during the SMTP phase. I haven't done so in recent enough memory to have any actual numbers, but when I last did a comparison, slow moving DNSBLs showed little/no change at all, fast-acting trap-driven ones show more of a difference. Now I've not studied the exactly amount of time it takes for hosts to start getting listed, but since I only greylist questionable stuff already and since I whitelist aggressively, I've been able to set my greylisting in the 30-60 minute range without too many seizures from users and with higher rejection counts -- Since greylisting doesn't cause higher reject counts, I assume (yes, just assume) that it's due to higher hit rates. I admit that it would make sense to do further testing, but for fast-acting DNSBLs, and body-hash based systems, it makes sense that the longer one defers a message, the greater the odds of a hit against a new zombie or a new spam-run. -- Dave Warren http://www.hireahit.com/ http://ca.linkedin.com/in/davejwarren
Re: Greylisting (was Re: Fairly-Secure Anti-SPAM Gateway Using OpenBSD, Postfix, Amavisd-new, SpamAssassin, Razor and DCC ? Can I get your opinion?)
On 11/29/2012 18:54, David F. Skoll wrote: [My gut instinct says that a reasonable greylisting interval is too short for most DNSBLs to react. Pyzor/Razor/DCC may be somewhat more adept at reacting quickly.] Something trap-driven like NIX is a candidate. No, it's not safe enough to reject based on it's output, but it was worth use in a scoring system. Invalument too responds reasonably quickly, enough that it sometimes tripped during the greylist period. The other trick is how you define reasonable. A reasonable greylist period for greylisting all mail is about 3 seconds, otherwise you'll have users screaming. However, if you only greylist questionable stuff to start with (rDNS failures, mismatches, etc, SPF fails, borderline-spammy stuff, DUL hits), you can get away with much longer times since most of it is crap anyway but a greylist period can help let the odd gem through. -- Dave Warren http://www.hireahit.com/ http://ca.linkedin.com/in/davejwarren
Fairly-Secure Anti-SPAM Gateway Using OpenBSD, Postfix, Amavisd-new, SpamAssassin, Razor and DCC ? Can I get your opinion?
I'm looking to set up a spam filtering server to replace our ISP's spam filtering service. I've seen this tutorial ( ftp://orn.mpg.de/pub/unix/mail/Fairly-Secure_Anti-SPAM_Gateway_Using_SpamAssassin.html#antivirus ) and I'd be very interested in YOUR opinion; do you think, fundamentally, a server with these software packages could be an effective combination at fighting spam? We're a (I guess) medium size organization with appx. 1000 end users. What about weaving clam-av into the mix? Although this tutorial uses OpenBSD, I'll probably be using FreeBSD. Thank you for your input! :-) Ed
Re: Fairly-Secure Anti-SPAM Gateway Using OpenBSD, Postfix, Amavisd-new, SpamAssassin, Razor and DCC ? Can I get your opinion?
On 28/11/12 23:32, Ed Flecko wrote: I'm looking to set up a spam filtering server to replace our ISP's spam filtering service. I've seen this tutorial ( ftp://orn.mpg.de/pub/unix/mail/Fairly-Secure_Anti-SPAM_Gateway_Using_SpamAssassin.html#antivirus ) and I'd be very interested in YOUR opinion; do you think, fundamentally, a server with these software packages could be an effective combination at fighting spam? We're a (I guess) medium size organization with appx. 1000 end users. What about weaving clam-av into the mix? Although this tutorial uses OpenBSD, I'll probably be using FreeBSD. Thank you for your input! :-) Ed I use Postfix with Amavisd-new which allows SpamAssassin and Clam-AV to be easily integrated. I also use Postgrey for greylisting. I find this setup very flexible and efficient. Clam-AV doesn't catch a huge amount on my mail flow - email borne trojans/viruses don't seem to be overly popular these days. You can get 3rd party signatures for things like phishing although I've never tried these as I've trained SA to do a good job on catching phishing emails. I'm running on Linux (RHEL5) but I guess the base OS is largely irrelevant so I'd use what you are comfortable with. I guess there are many ways to skin this particular cat but the above setup works very well for me. In other words, I suspect you will get a number of different answers all providing effective solutions based around the use of SpamAssassin and/or Clam-AV. The difference mostly seems to be how you choose to integrate them into your mail server.
Re: setup spamassassin without amavisd
On Sun, Jul 08, 2012 at 04:40:31PM -0500, Dave Funk wrote: On 07/08/2012 12:49 PM, David Kentwood wrote: Hi, I want to setup spamassassin + clamav + postfix. Many internet guides use Amavisd to integrate them together. However, my vps has only 516mb ram so I don't want to install Amavisd unless it's really recommended. So would the setup work well without using Amavisd? Would you recommend using Amavisd? One thing to keep in mind are the various factors that influence memory usage in spamassassin clamav (and by how much). For example (on a SLES-11 x86_64 box) clamd with just the stock ClamAV rules has a RSS of 155MB, with a number of 3'rd party add in rulesets (EG Sanesecurity, SecureiteInfo, etc) its RSS is over 500MB. However the Clam + added rulesets has a hit rate that is 50x~100x higher than just stock ClamAv rules I just ditch main.cld which seems pointless, I think it saved something like 40-50MB. If there are actually ever any new viruses, daily.cld should catch them. With this and most 3rd party sigs, clamd is only 80MB RSS. spamd's memory size is influenced by added rules and by scanned message size. As spamd keeps in memory multiple copies of a message (the raw form, the parsed 'full' form, the cleaned normalized form, etc) its memory usage grows nonlinearly with message size. EG if you restrict spamd to only scanning small ( 64KB) messages it might be no more than 100MB RSS but when you feed it larger messages (say 350KB) it can easily hit 150MB RSS per instance. I've never seen my amavisd RSS over 100MB (512k msg size). On a 64-bit box it can be something like 1.5x more since Perl likes to spend a whole lot more memory there. But not something to worry usually on a VPS. So if you limit scanned message size you use less memory but then bloated spams will slip thru. They won't if you use amavisd. It just truncates messages to the limit and scans that. :-) Depending upon your mail flow rate you may want to keep multiple spamd children around. Each child uses up memory but multiple children help thruput during bursts of incoming messages. You can easily run many children since amavisd or spamd forks are copy-on-writed pretty well. So only extra memory used is the per scan state and file data etc.
Perl, fork, and copy-on-write (was Re: setup spamassassin without amavisd)
On Mon, 9 Jul 2012 09:06:39 +0300 Henrik K h...@hege.li wrote: You can easily run many children since amavisd or spamd forks are copy-on-writed pretty well. So only extra memory used is the per scan state and file data etc. Have you actually measured this? My experience is that forked Perl processes end up sharing surprisingly little memory. The reason is that Perl uses reference-counting to track liveness (so it can free memory when the reference count reaches zero.) This turns a lot of what should be read-only accesses into writes and severely hurts memory page sharing. Be careful. Regards, David.
Re: Perl, fork, and copy-on-write (was Re: setup spamassassin without amavisd)
On Mon, Jul 09, 2012 at 04:06:48AM -0400, David F. Skoll wrote: On Mon, 9 Jul 2012 09:06:39 +0300 Henrik K h...@hege.li wrote: You can easily run many children since amavisd or spamd forks are copy-on-writed pretty well. So only extra memory used is the per scan state and file data etc. Have you actually measured this? My experience is that forked Perl processes end up sharing surprisingly little memory. The reason is that Perl uses reference-counting to track liveness (so it can free memory when the reference count reaches zero.) This turns a lot of what should be read-only accesses into writes and severely hurts memory page sharing. Yes I tested it quite a bit with amavis, sadly I don't have my notes anymore. I think each active child added around 5-10MB to used system memory seen with top/free commands. Since childs can be configured to be shut down after x requests, it also reduces any risks. Of course YMMV and there could be flaws in my testing. But if 20 childs happily start and run on my small VPS, it should already tell something. ;-)
Re: setup spamassassin without amavisd
Thank you all for your replies. I have carefully considered all of your suggestions and decided to try the Amavisd setup. According to some suggestions here and various online sources, Amavisd is fast, scalable, easy to use and highly configurable. It loads up without spamd, and can handles spam and virus filtering simultaneously. I did the setup on a centos 6.2 vps running nginx/php/mysql/bind together with postfix/dovecot/clamav/spamassassin/amavisd. I didn't use any custom rule set for clamav/spamassassin. Most of the settings were based on default install using yum. However, after starting all the services, the entire memory usage shot up to around 900mb (the openvz vps only has 512mb ram + 512mb burst memory) while the server was not even live! free -k total used free shared buffers cached Mem: 1048576 894248 154328 0 0 0 -/+ buffers/cache:894248 154328 Swap: 0 0 0 Moreover, the 'top' command showed that Amavisd and clamd each reserved about 300mb of virtual memory. I am not sure if this is typical for such a setup. Perhaps high memory allocation is caused by some intrinsic issue with the openvz kernel?! I'll be doing some customization per suggestions from some of the replies here. And if there aren't ways to significally reduce the memory usage, I might just upgrade the memory to 1gb. Regardless, thank you very much for all your help!
Re: setup spamassassin without amavisd
David Kentwood wrote: Hi, I want to setup spamassassin + clamav + postfix. Many internet guides use Amavisd to integrate them together. However, my vps has only 516mb ram so I don't want to install Amavisd unless it's really recommended. So would the setup work well without using Amavisd? Yes, it works very well without amavisd: http://jessen.ch/articles/spamassassin-and-postfix/ (a bit old, but still valid). -- Per Jessen, Zürich (20.5°C)
Re: setup spamassassin without amavisd
Den 2012-07-09 08:06, Henrik K skrev: I just ditch main.cld which seems pointless, I think it saved something like 40-50MB. If there are actually ever any new viruses, daily.cld should catch them. With this and most 3rd party sigs, clamd is only 80MB RSS. so you think that virus that are very old are moved from main into daily if seen daily ? if thats the case i can save some mem aswell :=) freshclam will just try to keep atleast main and daily :(
Re: setup spamassassin without amavisd
On 7/9/2012 5:42 AM, Benny Pedersen wrote: Den 2012-07-09 08:06, Henrik K skrev: I just ditch main.cld which seems pointless, I think it saved something like 40-50MB. If there are actually ever any new viruses, daily.cld should catch them. With this and most 3rd party sigs, clamd is only 80MB RSS. so you think that virus that are very old are moved from main into daily if seen daily ? if thats the case i can save some mem aswell :=) freshclam will just try to keep atleast main and daily :( I'm not sure this is safe. The viruses in main.cld may be older than the ones in daily.cld, but that doesn't mean they aren't still out there. In answer to your question, I would very much doubt that virus definitions are every moved from main back to daily. Since you are expected to have the main.cld database, which already contains the definition, it would be a waste of bandwidth to force everyone to download it again in daily.cld. You should ask on the ClamAV list for a definitive answer. -- Bowie
Re: setup spamassassin without amavisd
On Mon, Jul 09, 2012 at 11:29:08AM -0400, Bowie Bailey wrote: On 7/9/2012 5:42 AM, Benny Pedersen wrote: Den 2012-07-09 08:06, Henrik K skrev: I just ditch main.cld which seems pointless, I think it saved something like 40-50MB. If there are actually ever any new viruses, daily.cld should catch them. With this and most 3rd party sigs, clamd is only 80MB RSS. so you think that virus that are very old are moved from main into daily if seen daily ? if thats the case i can save some mem aswell :=) freshclam will just try to keep atleast main and daily :( I'm not sure this is safe. The viruses in main.cld may be older than the ones in daily.cld, but that doesn't mean they aren't still out there. In answer to your question, I would very much doubt that virus definitions are every moved from main back to daily. Since you are expected to have the main.cld database, which already contains the definition, it would be a waste of bandwidth to force everyone to download it again in daily.cld. You should ask on the ClamAV list for a definitive answer. It's been months since last main generation. 30750647 Mar 23 13:15 main.cvd Since I couldn't care less about any viruses which are pretty much non-existent today, I have no problem of doing this. I mainly use ClamAV for all the 3rd party sigs. MTA rules and SpamAssassin catches anything resembling a virus here anyway, but keeping daily sigs is fine as backup. And yes you need a custom download directory for freshclam to keep all the sigs and move daily.cld manually to the real directory.
setup spamassassin without amavisd
Hi, I want to setup spamassassin + clamav + postfix. Many internet guides use Amavisd to integrate them together. However, my vps has only 516mb ram so I don't want to install Amavisd unless it's really recommended. So would the setup work well without using Amavisd? Would you recommend using Amavisd? Thank you, Dave
Re: setup spamassassin without amavisd
On 07/08/2012 12:49 PM, David Kentwood wrote: Hi, I want to setup spamassassin + clamav + postfix. Many internet guides use Amavisd to integrate them together. However, my vps has only 516mb ram so I don't want to install Amavisd unless it's really recommended. So would the setup work well without using Amavisd? Would you recommend using Amavisd? First, you'd need to check if you need any features which only Amavisd can offer. If the answer is no, I'd go for Fuglu. It uses spamd, interfaces with Clamav, etc. Written in Python, real easy to setup manage. Well documented and under active development. Been using in production for quite a while, on a dozen of high traffic boxes and happy :) Docs: http://sourceforge.net/apps/trac/fuglu/ Download: http://sourceforge.net/projects/fuglu/ Axb
Re: setup spamassassin without amavisd
On Sun, July 8, 2012 13:49, David Kentwood wrote: Hi, I want to setup spamassassin + clamav + postfix. Many internet guides use Amavisd to integrate them together. However, my vps has only 516mb ram so I don't want to install Amavisd unless it's really recommended. So would the setup work well without using Amavisd? Would you recommend using Amavisd? Thank you, Dave I use it without amavisd, and works fine. I have different installations, - integrated via using postfix/spamass-milter - integrated via postfix/maildrop Those have differences, and I have different requiremens for each installation and those do it fine. However, amavisd. It links to SpamAssasin by using it as an add-on library. It does not need spamd up, amavisd is the daemon itself. I think SpamAssassin + amavisd memory requirement does not differ much from using plain SpamAssassin. I have not verified this, but this is my understanding.
Re: setup spamassassin without amavisd
From: Jari Fredriksson ja...@iki.fi On Sun, July 8, 2012 13:49, David Kentwood wrote: Hi, I want to setup spamassassin + clamav + postfix. Many internet guides use Amavisd to integrate them together. However, my vps has only 516mb ram so I don't want to install Amavisd unless it's really recommended. So would the setup work well without using Amavisd? Would you recommend using Amavisd? Thank you, Dave I use it without amavisd, and works fine. I have different installations, - integrated via using postfix/spamass-milter - integrated via postfix/maildrop Those have differences, and I have different requiremens for each installation and those do it fine. However, amavisd. It links to SpamAssasin by using it as an add-on library. It does not need spamd up, amavisd is the daemon itself. I think SpamAssassin + amavisd memory requirement does not differ much from using plain SpamAssassin. I have not verified this, but this is my understanding. I use a combination with Postfix and spampd as a policy daemon. In this setup the policy daemon is called in real time during the SMTP transaction, and the incoming mail message which exhibits a SA score above a certain value is rejected at the end of DATA. My installation runs without any incident since 2 or 3 years on boxes with 512M RAM. But the message volume is not high, no more than 10'000 ~ 20'000 messages a day, including rejected spam. See this information regarding spampd: http://wiki.apache.org/spamassassin/IntegratePostfixViaSpampd I have posted a description of my setup, in French: http://blog.demees.net/?p=7 Frédéric De Mees Brussels
Re: setup spamassassin without amavisd
On Sun, 2012-07-08 at 06:49 -0400, David Kentwood wrote: Hi, I want to setup spamassassin + clamav + postfix. Many internet guides use Amavisd to integrate them together. However, my vps has only 516mb ram You don't say what your mail volume is, but until recently I ran SA successfully on a 512MB, 866 MHz P3 house server box which also runs Postgres and Apache. Mail volume was 150 msgs/day +/-50. I'm still using the same mail handling chain but for reasons not related to mail, the box recently got bigger and faster by a factor of 8 for RAM and 12 for CPU cycles. The set up is that SA only handles incoming mail: getmail fetches the incoming mail from my ISP and the getmail mda script defines a pipeline consisting of: spamc | spamkiller | sendmail where (obviously) spamc passes messages to spamd for processing. Spamkiller is a locally developed program that looks at the SA headers, quarantines spam and hands ham to the Postfix sendmail utility for delivery the local copy of Postfix. If this type of set-up floats your boat, you can download spamkiller and friends from here: http://www.libelle-systems.com/free/ HTH Martin
Re: setup spamassassin without amavisd
Den 2012-07-08 12:49, David Kentwood skrev: I want to setup spamassassin + clamav + postfix. good choice Many internet guides use Amavisd to integrate them together. this is the most common setup, it does not mean there is not any other options However, my vps has only 516mb i have being there with freebsd 4.9 with just 256mb ram so I dont want to install Amavisd unless its really recommended. amavisd itself is not ram hungry, most ram usa is from clamd and spamassasin perl usages So would the setup work well without using Amavisd? i would start seeing what is left after clamd is started Would you recommend using Amavisd? i would upgrade to 1024mb and try using spampd and clamav-milter, spampd can be used as proxy in postfix so you dont need an quarantine if harddisk space is small but there is one caviat, proxy scanning limits speeds in postfix since it queue up before in queue, later versions of postfix have better support for proxy scanning
Re: setup spamassassin without amavisd
On 07/08/2012 12:49 PM, David Kentwood wrote:
Hi,
I want to setup spamassassin + clamav + postfix. Many internet guides use
Amavisd to integrate them together. However, my vps has only 516mb ram so I
don't want to install Amavisd unless it's really recommended. So would the
setup work well without using Amavisd? Would you recommend using Amavisd?
One thing to keep in mind are the various factors that influence memory
usage in spamassassin clamav (and by how much).
For example (on a SLES-11 x86_64 box) clamd with just the stock ClamAV
rules has a RSS of 155MB, with a number of 3'rd party add in rulesets
(EG Sanesecurity, SecureiteInfo, etc) its RSS is over 500MB.
However the Clam + added rulesets has a hit rate that is 50x~100x higher
than just stock ClamAv rules
spamd's memory size is influenced by added rules and by scanned message
size. As spamd keeps in memory multiple copies of a message (the raw form,
the parsed 'full' form, the cleaned normalized form, etc) its memory
usage grows nonlinearly with message size. EG if you restrict spamd to
only scanning small ( 64KB) messages it might be no more than 100MB RSS
but when you feed it larger messages (say 350KB) it can easily hit
150MB RSS per instance.
So if you limit scanned message size you use less memory but then
bloated spams will slip thru. Also 3'rd party rulesets can be
quite helpful hitting fast mutating spams.
Depending upon your mail flow rate you may want to keep multiple spamd
children around. Each child uses up memory but multiple children help
thruput during bursts of incoming messages.
--
Dave Funk University of Iowa
dbfunk (at) engineering.uiowa.eduCollege of Engineering
319/335-5751 FAX: 319/384-0549 1256 Seamans Center
Sys_admin/Postmaster/cell_adminIowa City, IA 52242-1527
#include std_disclaimer.h
Better is not better, 'standard' is better. B{
anti virus EICAR file is not detected by the couple clamd amavisd
hi folks in my station anti virus EICAR file is not detected by the couple clamd amavisd all testimonials are welcome -- http://pgp.mit.edu:11371/pks/lookup?op=getsearch=0x092164A7 gpg --keyserver pgp.mit.edu --recv-key 092164A7 pgpiBHw6zHTrd.pgp Description: PGP signature
Re: anti virus EICAR file is not detected by the couple clamd amavisd
On 7/5/11 3:15 PM, m...@smtp.fakessh.eu wrote: hi folks in my station anti virus EICAR file is not detected by the couple clamd amavisd all testimonials are welcome works fine here. you must be doing something wrong. find out what you are doing wrong and it will work. -- Michael Scheidell, CTO o: 561-999-5000 d: 561-948-2259 *| *SECNAP Network Security Corporation * Best Mobile Solutions Product of 2011 * Best Intrusion Prevention Product * Hot Company Finalist 2011 * Best Email Security Product * Certified SNORT Integrator __ This email has been scanned and certified safe by SpammerTrap(r). For Information please see http://www.secnap.com/products/spammertrap/ __
Re: anti virus EICAR file is not detected by the couple clamd amavisd
On Tue, 2011-07-05 at 21:15 +0200, m...@smtp.fakessh.eu wrote:
in my station
anti virus EICAR file is not detected by the couple clamd amavisd
^^
all testimonials are welcome
Oh, sure... Naughty boy. Bad cross-posting.
And the SA users list shouldn't have been included in the first place,
don't you think?
--
char *t=\10pse\0r\0dtu\0.@ghno\x4e\xc8\x79\xf4\xab\x51\x8a\x10\xf4\xf4\xc4;
main(){ char h,m=h=*t++,*x=t+2*h,c,i,l=*x,s=0; for (i=0;il;i++){ i%8? c=1:
(c=*++x); c128 (s+=h); if (!(h=1)||!t[s+h]){ putchar(t[s]);h=m;s=0; }}}
Re: anti virus EICAR file is not detected by the couple clamd amavisd
On Wed, 06 Jul 2011 00:38:46 +0200, Karsten Bräckelmann wrote: And the SA users list shouldn't have been included in the first place, don't you think? first off all, welcome back Karsten :) yes i think that some users have turned spamassassin into a virus scanner with clamav plugin, and visa versa amavisd-new into a spam scanner without the help of spamassassin, its mostly how windows 7 uses gigabyte ram and not just megabyte fun aside from me now, be free dont use windows
