Re: Porn spam killer PCRE

[Benny Pedersen]

On 2016-08-27 16:27, Nikolaos Milas wrote:


What may be a problem here?


you miss update of SPF plugin to a never version

you may need to if version >  around rules there

so spf none only is used on versions that support it

upstream bug here

Re: Porn spam killer PCRE

[Sean Greenslade]

On Sat, Aug 27, 2016 at 05:27:18PM +0300, Nikolaos Milas wrote:
> Thank you Sean for all your help.
> 
> I am focusing on amavisd-new / spamassassin for my efforts.
> 
> I have started a thread in the amavisd-new users mailing list, since
> bayesian filtering is off-topic here.
> 
> However, before ending this thread, I would like to ask: all these rules for
> which you have defined scores:
> 
>score LOTS_OF_MONEY 0.9
>score FREEMAIL_FORGED_FROMDOMAIN 0.5
>score WEIRD_PORT 1.5
>score TO_IN_SUBJ 1.0
> 
>score RCVD_IN_MSPIKE_BL 1.0
>score RCVD_IN_MSPIKE_L5 2.5
>score RCVD_IN_MSPIKE_L4 2.0
>...
> 
> ...are custom rules you have set up? If so, what do they look like?(You may
> want to reply off-list, if you deem it is more approrpriate.)

These are not custom rules. You may need to look into the amavisd
documentation to see if they do anything unusual with SA, but those
rules are part of the normal SA install.

> The same question goes for the SPF rules.
> 
> I am asking because I see:
> 
># spamassassin -D --lint
>...
>Aug 27 17:13:56.860 [10384] dbg: config: warning: score set for
>non-existent rule RCVD_IN_MSPIKE_H4
>Aug 27 17:13:56.860 [10384] dbg: config: warning: score set for
>non-existent rule RCVD_IN_MSPIKE_WL
> 
> Also note that re. SPF tests, I see:
> 
># spamassassin -D --lint
>...
>Aug 27 17:13:58.228 [10384] warn: rules: failed to run SPF_NONE
>test, skipping:
>Aug 27 17:13:58.229 [10384] warn:  (Can't locate object method
>"check_for_spf_none" via package "Mail:
> 
> 
> What may be a problem here?

Since you're not using SA in the same way I am, I can't really say. Note
that SA has a rules update mechanism that may need to be triggered to
get the full rule set.

--Sean

Re: Porn spam killer PCRE

[Nikolaos Milas]

On 25/8/2016 7:46 μμ, Sean Greenslade wrote:


Make sure the SPF rules have weights set, then check the spam report for
a message. If SPF is working, you should see at least one of the SPF
rules trigger for every message.


Thank you Sean for all your help.

I am focusing on amavisd-new / spamassassin for my efforts.

I have started a thread in the amavisd-new users mailing list, since 
bayesian filtering is off-topic here.


However, before ending this thread, I would like to ask: all these rules 
for which you have defined scores:


   score LOTS_OF_MONEY 0.9
   score FREEMAIL_FORGED_FROMDOMAIN 0.5
   score WEIRD_PORT 1.5
   score TO_IN_SUBJ 1.0

   score RCVD_IN_MSPIKE_BL 1.0
   score RCVD_IN_MSPIKE_L5 2.5
   score RCVD_IN_MSPIKE_L4 2.0
   ...

...are custom rules you have set up? If so, what do they look like?(You 
may want to reply off-list, if you deem it is more approrpriate.)


The same question goes for the SPF rules.

I am asking because I see:

   # spamassassin -D --lint
   ...
   Aug 27 17:13:56.860 [10384] dbg: config: warning: score set for
   non-existent rule RCVD_IN_MSPIKE_H4
   Aug 27 17:13:56.860 [10384] dbg: config: warning: score set for
   non-existent rule RCVD_IN_MSPIKE_WL
   Aug 27 17:13:56.861 [10384] dbg: config: warning: score set for
   non-existent rule FREEMAIL_FORGED_FROMDOMAIN
   Aug 27 17:13:56.861 [10384] dbg: config: warning: score set for
   non-existent rule RCVD_IN_MSPIKE_L5
   Aug 27 17:13:56.861 [10384] dbg: config: warning: score set for
   non-existent rule RCVD_IN_MSPIKE_H3
   Aug 27 17:13:56.861 [10384] dbg: config: warning: score set for
   non-existent rule PP_TOO_MUCH_UNICODE02
   Aug 27 17:13:56.861 [10384] dbg: config: warning: score set for
   non-existent rule HEADER_HOST_IN_WHITELIST
   Aug 27 17:13:56.861 [10384] dbg: config: warning: score set for
   non-existent rule URI_HOST_IN_BLACKLIST
   Aug 27 17:13:56.861 [10384] dbg: config: warning: score set for
   non-existent rule RCVD_IN_MSPIKE_H2
   Aug 27 17:13:56.861 [10384] dbg: config: warning: score set for
   non-existent rule HEADER_FROM_DIFFERENT_DOMAINS
   Aug 27 17:13:56.861 [10384] dbg: config: warning: score set for
   non-existent rule URI_HOST_IN_WHITELIST
   Aug 27 17:13:56.862 [10384] dbg: config: warning: score set for
   non-existent rule HELO_LH_HOME
   Aug 27 17:13:56.862 [10384] dbg: config: warning: score set for
   non-existent rule HEADER_HOST_IN_BLACKLIST
   Aug 27 17:13:56.862 [10384] dbg: config: warning: score set for
   non-existent rule RCVD_IN_MSPIKE_ZBI
   Aug 27 17:13:56.862 [10384] dbg: config: warning: score set for
   non-existent rule RCVD_IN_MSPIKE_L2
   Aug 27 17:13:56.862 [10384] dbg: config: warning: score set for
   non-existent rule HK_NAME_MR_MRS
   Aug 27 17:13:56.862 [10384] dbg: config: warning: score set for
   non-existent rule RCVD_IN_MSPIKE_BL
   Aug 27 17:13:56.862 [10384] dbg: config: warning: score set for
   non-existent rule PP_TOO_MUCH_UNICODE05
   Aug 27 17:13:56.862 [10384] dbg: config: warning: score set for
   non-existent rule RCVD_IN_MSPIKE_L4
   Aug 27 17:13:56.862 [10384] dbg: config: warning: score set for
   non-existent rule RCVD_IN_MSPIKE_H5
   Aug 27 17:13:56.863 [10384] dbg: config: warning: score set for
   non-existent rule URIBL_SBL_A
   Aug 27 17:13:56.863 [10384] dbg: config: warning: score set for
   non-existent rule PP_MIME_FAKE_ASCII_TEXT
   Aug 27 17:13:56.863 [10384] dbg: config: warning: score set for
   non-existent rule RCVD_IN_MSPIKE_L3
   ...

Also note that re. SPF tests, I see:

   # spamassassin -D --lint
   ...
   Aug 27 17:13:58.228 [10384] warn: rules: failed to run SPF_NONE
   test, skipping:
   Aug 27 17:13:58.229 [10384] warn:  (Can't locate object method
   "check_for_spf_none" via package "Mail:
   [...]:SpamAssassin::PerMsgStatus" at (eval 1241) line 343.
   Aug 27 17:13:58.229 [10384] warn: )
   ...
   Aug 27 17:13:58.231 [10384] warn: rules: failed to run SPF_HELO_NONE
   test, skipping:
   Aug 27 17:13:58.231 [10384] warn:  (Can't locate object method
   "check_for_spf_helo_none" via package "Mail:
   [...]:SpamAssassin::PerMsgStatus" at (eval 1241) line 959.
   Aug 27 17:13:58.231 [10384] warn: )
   ...

What may be a problem here?

Nick

Re: Porn spam killer PCRE

[Sean Greenslade]

On Thu, Aug 25, 2016 at 05:28:35PM +0300, Nikolaos Milas wrote:
> On 23/8/2016 11:58 μμ, Sean Greenslade wrote:
> 
> > Hope this is helpful,
> 
> Thanks Sean for your time and eagerness to help. I appreciate it.
> 
> I am planning to try your suggestions.
> 
> I am using CentOS 6. Can you please let me know about the perl package
> SPF-related bug you mentioned?

The bug is a CentOS 7-specific bug as far as I know. Here is the bug
page if you're interested (see comment 29):
https://bugzilla.redhat.com/show_bug.cgi?id=1200167

I would also suggest considering upgrading to CentOS 7, as 6 is going to
end full updates next year.

> I have installed:
> 
>yum --enablerepo=epel install perl-Mail-SPF perl-Sys-Hostname-Long
> 
> Would this be OK, or I should add/change something?

Make sure the SPF rules have weights set, then check the spam report for
a message. If SPF is working, you should see at least one of the SPF
rules trigger for every message.

> Finally, I note that in my setup I am loading/running spamassassin from
> within amavis and not directly in postfix.

I'm not familiar with amavis, but I doubt that would be a problem.
SpamAssassin runs as the spamd daemon, which loads its configuration
directly from the /etc files.

--Sean

Re: Porn spam killer PCRE

[Nikolaos Milas]

On 23/8/2016 11:58 μμ, Sean Greenslade wrote:


Hope this is helpful,


Thanks Sean for your time and eagerness to help. I appreciate it.

I am planning to try your suggestions.

I am using CentOS 6. Can you please let me know about the perl package 
SPF-related bug you mentioned?


I have installed:

   yum --enablerepo=epel install perl-Mail-SPF perl-Sys-Hostname-Long

Would this be OK, or I should add/change something?

Finally, I note that in my setup I am loading/running spamassassin from 
within amavis and not directly in postfix.


Thanks again,
Nick

Re: Porn spam killer PCRE

[Sean Greenslade]

On Tue, Aug 23, 2016 at 11:27:39PM +0300, Nikolaos Milas wrote:
> On 20/8/2016 7:56 μμ, Sean Greenslade wrote:
> 
> > ...
> > - Enable & configure per-user bayesan filtering
> > - Increase allowed storage space for bayesan databases
> > - Update a particular perl package to make SPF work (CentOS / RHEL
> >specific bug)
> > ...
> > - Re-weight a bunch of internal rules, in particular the bayes, SPF,
> >and mailspike rules
> > 
> > Another thing I did was enable the spam report to be added to all
> > messages, that way I could more easily debug why spam that was getting
> > past the filter didn't trigger it.
> > 
> > ...I slowly ramped up rule weights and waited to see what would happen. ...
> 
> Thank you Sean for your advice. I would appreciate if you provide more
> technical details on how to try the above and/or indicate documentation
> links with associated guide(s).
> 
> Your experience might help avoid re-inventing the wheel while trying to make
> users life easier in our really congested daily routine.

As I said, it's going to need to be tailored to your particular use
case, so my config files probably won't be useful as-is to you. Fighting
spam is a constantly changing battle, and it really depends on what
types of spam you're currently suffering from.  I'll provide some
snippets so that you can see what I'm doing. Any filenames will be the
CentOS variants or my own personal systems.

The main spamassassin conf (/etc/mail/spamassassin/local.cf) has all of
my spamassassin modifications.

I've added these settings to help analyze SA's decisions:
> report_safe 0 # Don't encapsulate spam, just tag it.
> add_header all Report _REPORT_ # Provide spam report in all mails.

My bayesian filter settings:
> # Bayesian filtering yeah!
> bayes_path /var/mail/bayes_db/bayes
> bayes_file_mode 0775
> bayes_expiry_max_db_size 30
> # 15 = ~4 MiB

The main tweak is the size increase. See this page for bayes help:
https://wiki.apache.org/spamassassin/BayesFaq

One of my custom rules, this one for .docm files:
> loadplugin Mail::SpamAssassin::Plugin::MIMEHeader
> mimeheader S_DOCM_ATTACHED Content-Type =~ 
> /docm|ms-word\.document\.macroEnabled/i
> describe   S_DOCM_ATTACHED email contains a docm file attachment
> score  S_DOCM_ATTACHED 4.5

Something to note is that none of my rules ever have enough weight on
their own to trigger the filter. But don't worry, most spam will trigger
many, many rules. I routinely get messages that have a spam score of
60+ points, with a trigger threshold of 5.2 points.

My custom rule for the sales@* spam:
> header   S_SALESFORCETo =~ /sales\@/i
> describe S_SALESFORCEThe sales team has been complaining about 
> all the spam.
> scoreS_SALESFORCE2.0

Note that this rule has a lower weight, because I'm less certain that
this alone is indicative of spam. 

And finally, a few of my custom rule weights. These came from analysing
incoming spam that was missed and figuring out which rules triggered on
it most reliably:
> score LOTS_OF_MONEY 0.9
> score FREEMAIL_FORGED_FROMDOMAIN 0.5
> score WEIRD_PORT 1.5
> score TO_IN_SUBJ 1.0

And these I added to make sure they work, since SA has some weird rule
weights that disable certain checks if bayesian filtering is on:
> score RCVD_IN_MSPIKE_BL 1.0
> score RCVD_IN_MSPIKE_L5 2.5
> score RCVD_IN_MSPIKE_L4 2.0
> score RCVD_IN_MSPIKE_L3 1.5
> score RCVD_IN_MSPIKE_L2 1.0
> score RCVD_IN_MSPIKE_H5 -1.0
> score RCVD_IN_MSPIKE_H4 -0.8
> score RCVD_IN_MSPIKE_H3 -0.6
> score RCVD_IN_MSPIKE_H2 -0.4
> score RCVD_IN_MSPIKE_WL -0.5
> score SPF_NONE 0.001
> score SPF_HELO_NONE 0.001
> score SPF_PASS -0.1
> score SPF_HELO_PASS -0.1
> score SPF_FAIL 3.0
> score SPF_HELO_FAIL 3.0
> score SPF_NEUTRAL 0.001
> score SPF_HELO_NEUTRAL 0.001
> score SPF_SOFTFAIL 1.0
> score SPF_HELO_SOFTFAIL 1.0


To tie spamassassin into postfix, I use these settings in
/etc/postfix/master.cf:
> smtp  inet  n   -   n   -   -   smtpd
>   -o content_filter=spamfilter
>   -o smtpd_tls_security_level=may
>   -o smtpd_sasl_auth_enable=no
> # ...
> spamfilter
>   unix  -   n   n   -   -   pipe
>flags=Rq user=mailfilter argv=/opt/postfix_spamcheck.sh -oi -f ${sender} 
> ${recipient}

And /opt/postfix_spamcheck.sh:
> #!/bin/bash
> # Simple filter to plug SpamAssassin into the Postfix MTA
> # File locations:
> # (CHANGE AS REQUIRED TO SUIT YOUR SERVER)
> SENDMAIL=/usr/sbin/sendmail
> SPAMASSASSIN=/usr/bin/spamc
>
> #logger <<<"Spam filter piping to SpamAssassin, then to: $SENDMAIL $@"
> ${SPAMASSASSIN} | ${SENDMAIL} "$@"
>
> exit $?


Then I have some scripts that read mails out of a specific maildir
(isspam) and runs sa-learn on them to allow users to train the bayesian
filter from thir mail clients. I won't post those since they are heavily
tied into my system for sorting and processing emails, which is quite
unusual and full of python.

Hope this is helpful,

--Sean

Re: Porn spam killer PCRE

[Nikolaos Milas]

On 20/8/2016 7:56 μμ, Sean Greenslade wrote:


...
- Enable & configure per-user bayesan filtering
- Increase allowed storage space for bayesan databases
- Update a particular perl package to make SPF work (CentOS / RHEL
   specific bug)
...
- Re-weight a bunch of internal rules, in particular the bayes, SPF,
   and mailspike rules

Another thing I did was enable the spam report to be added to all
messages, that way I could more easily debug why spam that was getting
past the filter didn't trigger it.

...I slowly ramped up rule weights and waited to see what would happen. ...


Thank you Sean for your advice. I would appreciate if you provide more 
technical details on how to try the above and/or indicate documentation 
links with associated guide(s).


Your experience might help avoid re-inventing the wheel while trying to 
make users life easier in our really congested daily routine.


Thanks in advance,
Nick

Re: Porn spam killer PCRE

[Sean Greenslade]

On Fri, Aug 19, 2016 at 04:30:38PM +0300, Nikolaos Milas wrote:
> 
> We are already using postscreen, many RBLs, the fqrdns.pcre, amavis,
> spamassassin with scamp and we are filtering about 60-70% of total incoming
> mail as spam, but there is still much more that should be filtered out.
> 
> Any additional suggestions?
> 
> Thanks,
> Nick

I have personally found SpamAssassin to work very well, filtering out
over 95% of my spam. That said, it does not work perfectly out of the
box. The main alterations I have made are:
- Enable & configure per-user bayesan filtering
- Increase allowed storage space for bayesan databases
- Update a particular perl package to make SPF work (CentOS / RHEL
  specific bug)
- Add some custom rules based on specific addresses being targeted (e.g.
  we don't have a sales dept., but we get spam sent to sa...@domain.tld)
- Add custom rule to detect suspicious attachments (e.g. .exe, .docm)
- Re-weight a bunch of internal rules, in particular the bayes, SPF,
  and mailspike rules

Another thing I did was enable the spam report to be added to all
messages, that way I could more easily debug why spam that was getting
past the filter didn't trigger it.

Note that by default, SpamAssassin has a pretty conservative ruleset
that is much more happy to allow false negatives than false positives.
In my opinion this is a good thing, as users will be more unhappy to see
legitimate messages in spam than the other way around. My approach to
tuning SpamAssassin was also rather conservative: I slowly ramped up
rule weights and waited to see what would happen. I would also test
new rules by giving them a very small weight at first, just to make sure
they trigger correctly, then giving them an appropriate real weight.

Hope this is helpful,

--Sean

Re: Porn spam killer PCRE

[Wietse Venema]

Nikolaos Milas:
> Hello,
> 
> Has anyone developed and/or maintains PCRE filtering for porn spam senders?
> 
> (Something like 
> https://github.com/stevejenkins/hardwarefreak.com-fqrdns.pcre)
> 
> I guess many sender / client domains could be filtered-away if they 
> include keywords like "kiss", "girl", "date", "adult", "cute", etc.
> 
> Any advice on that? (Note: Some users suffer more than others by porn spam.)
> 
> We are already using postscreen, many RBLs, the fqrdns.pcre, amavis, 
> spamassassin with scamp and we are filtering about 60-70% of total 
> incoming mail as spam, but there is still much more that should be 
> filtered out.
> 
> Any additional suggestions?

Consider: there are only about 10 characters in the domain name,
and 100-1000 characters in message content. Based on these numbers,
filtering based on content should be an order of magnitude more
precise than filtering based on the domain name.

Wietse

Re: Porn spam killer PCRE

[Christian Recktenwald]

On Fri, Aug 19, 2016 at 04:30:38PM +0300, Nikolaos Milas wrote:
> Hello,
> 
> Has anyone developed and/or maintains PCRE filtering for porn spam senders?
> 
> (Something like
> https://github.com/stevejenkins/hardwarefreak.com-fqrdns.pcre)
> 
> I guess many sender / client domains could be filtered-away if they
> include keywords like "kiss", "girl", "date", "adult", "cute", etc.
> 
> Any advice on that? (Note: Some users suffer more than others by porn spam.)

please take into consideration that this simple approach will produce
false positive hits on domains like:
www.hotchkiss.co.uk
www.girlscouts.org
www.nerdgirl.com
updates.polar.com
cutebit.de
...

you get the idea

> We are already using postscreen, many RBLs, the fqrdns.pcre, amavis,
> spamassassin with scamp and we are filtering about 60-70% of total
> incoming mail as spam, but there is still much more that should be
> filtered out.
> 
> Any additional suggestions?

Yeah well. That's a complex topic which goes way beyond the scope of this 
mailing list.

If you're interested, you're welcome to send me a personal email.

Re: Porn spam killer PCRE

[Steve Jenkins]

On Fri, Aug 19, 2016 at 7:30 AM, Nikolaos Milas  wrote:

> Hello,
>
> Has anyone developed and/or maintains PCRE filtering for porn spam senders?
>
> (Something like https://github.com/stevejenkins/hardwarefreak.com-fqrdns.
> pcre)
>
> I guess many sender / client domains could be filtered-away if they
> include keywords like "kiss", "girl", "date", "adult", "cute", etc.
>
> Any advice on that? (Note: Some users suffer more than others by porn
> spam.)
>
> We are already using postscreen, many RBLs, the fqrdns.pcre, amavis,
> spamassassin with scamp and we are filtering about 60-70% of total incoming
> mail as spam, but there is still much more that should be filtered out.
>
> Any additional suggestions?


Hi, Nikolaos. It's certainly possible to do that with PCRE filtering, but I
think it would block too many false positives from legitimate domains such
as kissimmee.org, blackgirlsvote.com, savethedate.com, and cuteoverload.com
(based on your examples).

Maybe some other form(s) of content filtering?

Steve Jenkins

Re: Porn spam killer PCRE

[Benny Pedersen]

On 2016-08-19 19:56, Nikolaos Milas wrote:

[no porn seen]


I understand your suggestion to query the fail2ban db directly from
postfix but I need to research more on how to implement that.


yes if that could be done, it save alot, but if not fail2ban could 
possible just call a wrapper that updates postfix sqlite maps based on 
what is in the logs seen, so far so good aswell here, i self have to 
rethink on how to make it best



Yet, I guess we could get better results by fail2ban itself? Could you
suggest how to expand/improve filtering rules and policy so as to get
better results from fail2ban itself?


not much, i have not used fail2ban very much to be of much help with it, 
its not long time since i posted a fail2ban config for blocking based on 
spamhaus rbls with diffrent ban time pr result, this is the best so far 
i have made yet



Any directions on how to better leverage fail2ban with any or both
ways will be appreciated!


do more static firewalling, eg one week if dul ips or pbl listed, such 
ips does not change daily, but being abused hourly :(



All the best,
Nick


yes back to my gentoo ebuild work :-)

Re: Porn spam killer PCRE

[Nikolaos Milas]

On 19/8/2016 5:29 μμ, Benny Pedersen wrote:


fail2ban ?


Thank you,

I am already using fail2ban directly with the following rules:

  /etc/fail2ban/filter.d/:

   failregex = reject: RCPT from (.*)\[\]: 554
reject: RCPT from (.*)\[\]: 450
reject: RCPT from (.*)\[\]:([0-9]{4,5}:)? 550
too many errors after AUTH from (.*)\[\]

   ignoreregex =

  /etc/fail2ban/jail.conf:

   [postfix]

   enabled  = true
   filter   = postfix
   action   = iptables-multiport[name=Postfix, port="smtp,submission",
   protocol=tcp]
   logpath  = /var/log/maillog
   maxretry = 6
   findtime = 1200
   bantime  = 7200

I understand your suggestion to query the fail2ban db directly from 
postfix but I need to research more on how to implement that.


Yet, I guess we could get better results by fail2ban itself? Could you 
suggest how to expand/improve filtering rules and policy so as to get 
better results from fail2ban itself?


Any directions on how to better leverage fail2ban with any or both ways 
will be appreciated!


All the best,
Nick

Re: Porn spam killer PCRE

[Benny Pedersen]

On 2016-08-19 15:30, Nikolaos Milas wrote:


Any additional suggestions?


fail2ban ?

(permenent blocking is easy, hint, fail2ban uses sqlite db, its pretty 
simple to query it from postfix sqlite map)

Porn spam killer PCRE

[Nikolaos Milas]

Hello,

Has anyone developed and/or maintains PCRE filtering for porn spam senders?

(Something like 
https://github.com/stevejenkins/hardwarefreak.com-fqrdns.pcre)


I guess many sender / client domains could be filtered-away if they 
include keywords like "kiss", "girl", "date", "adult", "cute", etc.


Any advice on that? (Note: Some users suffer more than others by porn spam.)

We are already using postscreen, many RBLs, the fqrdns.pcre, amavis, 
spamassassin with scamp and we are filtering about 60-70% of total 
incoming mail as spam, but there is still much more that should be 
filtered out.


Any additional suggestions?

Thanks,
Nick