Re: How to reject mails with special message-id (Debian, Amavis, Spamassassin)

[Joe Quinn]

On 9/20/2016 9:46 AM, Thomas Barth wrote:



Am 20.09.2016 um 15:27 schrieb Bowie Bailey:


X-Spam-Status: Yes, score=14.009 tag=2 tag2=6.31 kill=6.31
tests=[HTML_MESSAGE=0.001, MESSAGEID_LOCAL=8,
MIME_HTML_ONLY=1.105,
PYZOR_CHECK=1.985, RCVD_IN_BRBL_LASTEXT=1.644, RDNS_NONE=1.274]
autolearn=no autolearn_force=no


The base SA ruleset is optimized to detect spam with a score of 5.0.  If
you raise that score, you will allow more spam to come through. If you
lower that score, you will see more legitimate messages blocked as
spam.  Make sure you know what you are doing before you change this 
score.




I read that 5.0 is aggressive and suitable for single user setup, 
conservative values are 8.0 or 11.0.


required_score n.nn (default: 5)
https://spamassassin.apache.org/full/3.2.x/doc/Mail_SpamAssassin_Conf.html 




I ve checked most of the mails recognized as spam. The lowest score 
was 8.6x so far.


Here is another mail from ...local. It definitely was spam with zip 
attachment. Common is a sender address with digits.
 -> , quarantine: 
l/spam-lEHVGcheLkyq.gz, Message-ID: 
<20160920202635.6b90ec7...@allfromboats.com.local>, mail_id: 
lEHVGcheLkyq, Hits: 19.118


May be I also should block sender adresses with more than 2 digits in 
the name?
My experience has been that spam scoring gets error-dominated pretty 
rapidly outside the range near 5.0. That is to say, the difference in 
actual spamminess between messages scored 4 and 6 is far more 
predictable and significant than between -1 and 1, or 10 and 12. Even a 
score of 8.0 I would expect to take months of tuning to get right, 
between rescoring rules and RBLs appropriately and then giving the bayes 
thresholds accurate scores on top of that. The furthest I would probably 
go is 4.5 to 6.0. Outside that range, it's easy to run into 
unpredictable "why was this spam blocked and that spam wasn't" scenarios.


Many of the stock published rules are scored by AI, which runs an 
optimization problem to get the most spam on the right side of 5.0 and 
the most ham on the left side. For the purposes of solving that problem, 
the difference between a message scoring 4.8 and 4.9 is the same as the 
difference between 4.0 and 4.9, or -50 and 4.9. Developers smooth out 
the scoring curve by determining what rules the AI gets to score and for 
how much, but that effect is strongest where we can quantify its 
usefulness (near the default threshold).


Bayes is scored with a similar consideration, built around probability.

Re: How to reject mails with special message-id (Debian, Amavis, Spamassassin)

[RW]

On Tue, 20 Sep 2016 15:46:21 +0200
Thomas Barth wrote:

> Am 20.09.2016 um 15:27 schrieb Bowie Bailey:
> 
> >> X-Spam-Status: Yes, score=14.009 tag=2 tag2=6.31 kill=6.31
> >> tests=[HTML_MESSAGE=0.001, MESSAGEID_LOCAL=8,
> >> MIME_HTML_ONLY=1.105,
> >> PYZOR_CHECK=1.985, RCVD_IN_BRBL_LASTEXT=1.644,
> >> RDNS_NONE=1.274] autolearn=no autolearn_force=no  
> >
> > The base SA ruleset is optimized to detect spam with a score of
> > 5.0.  If you raise that score, you will allow more spam to come
> > through.  If you lower that score, you will see more legitimate
> > messages blocked as spam.  Make sure you know what you are doing
> > before you change this score. 
> 
> I read that 5.0 is aggressive and suitable for single user setup, 
> conservative values are 8.0 or 11.0.

It depends what you are doing; 5.0 is low for outright rejection, but
what's often done is to have two thresholds. You reject at the higher
level and anything over the lower level is filed into a spam folder or
marked as spam. 5.0 is a sensible value for this.

Re: How to reject mails with special message-id (Debian, Amavis, Spamassassin)

[Bowie Bailey]

On 9/20/2016 9:46 AM, Thomas Barth wrote:



Am 20.09.2016 um 15:27 schrieb Bowie Bailey:


X-Spam-Status: Yes, score=14.009 tag=2 tag2=6.31 kill=6.31
tests=[HTML_MESSAGE=0.001, MESSAGEID_LOCAL=8,
MIME_HTML_ONLY=1.105,
PYZOR_CHECK=1.985, RCVD_IN_BRBL_LASTEXT=1.644, RDNS_NONE=1.274]
autolearn=no autolearn_force=no


The base SA ruleset is optimized to detect spam with a score of 5.0.  If
you raise that score, you will allow more spam to come through. If you
lower that score, you will see more legitimate messages blocked as
spam.  Make sure you know what you are doing before you change this 
score.




I read that 5.0 is aggressive and suitable for single user setup, 
conservative values are 8.0 or 11.0.


required_score n.nn (default: 5)
https://spamassassin.apache.org/full/3.2.x/doc/Mail_SpamAssassin_Conf.html 



Depends on your situation.  I've been using 5.0 for years in a 
tag-and-deliver setup.  I delete spam messages at a score of 10 - 15 for 
a few users who receive large amounts of spam.


Also, as I said before, remember that SA's required_score setting is 
ignored in an Amavis setup.  You should use Amavis's tag_level, 
tag2_level, and kill_level settings instead.  According to the header 
shown above, you currently appear to be blocking spam at a score of 6.31.


I ve checked most of the mails recognized as spam. The lowest score 
was 8.6x so far.


It is impossible to block all spam.  There will always be some that 
slips through.  The objective of a spam blocker such as SA is to block 
the most spam possible while keeping false positives near zero.  Users 
will complain about a few spam that get through -- they will scream 
about a single important message that gets blocked.


Here is another mail from ...local. It definitely was spam with zip 
attachment. Common is a sender address with digits.
 -> , quarantine: 
l/spam-lEHVGcheLkyq.gz, Message-ID: 
<20160920202635.6b90ec7...@allfromboats.com.local>, mail_id: 
lEHVGcheLkyq, Hits: 19.118


May be I also should block sender adresses with more than 2 digits in 
the name?


VERY bad idea.  Especially if you deal with the general public. There 
are tons and tons of people who have emails like jim...@gmail.com.  You 
might get away with a low scoring rule for messages with 4 or more 
digits, but I would give it a very low score to start with and watch it 
for a week or so to see how many hams vs spams it hits.


--
Bowie

Re: How to reject mails with special message-id (Debian, Amavis, Spamassassin)

[li...@rhsoft.net]

Am 20.09.2016 um 15:46 schrieb Thomas Barth:

I read that 5.0 is aggressive and suitable for single user setup,
conservative values are 8.0 or 11.0


depends on your glue, setup and bayes-training

many setups tag spam with 5.0 or 5.5 while the glue like a milter 
rejects spam above 8.0 points



I ve checked most of the mails recognized as spam. The lowest score was
8.6x so far.


that don't say anything as i recall from other posts your bayes is 
currently not working - the point is not what was detected but what 
slipped through and why or became a false-postive and why



Here is another mail from ...local. It definitely was spam with zip
attachment. Common is a sender address with digits.
 -> , quarantine:
l/spam-lEHVGcheLkyq.gz, Message-ID:
<20160920202635.6b90ec7...@allfromboats.com.local>, mail_id:
lEHVGcheLkyq, Hits: 19.118

May be I also should block sender adresses with more than 2 digits in
the name?


you should not block anything by single rules, that thread sounds like 
you are a absolute beginner and in that case you should refrain from 
blindly setup rules because you think you have found a spam sign somewehere


anyways, i can assure you that .local in a message-id is *nothing 
unusual* and frankly i had even to step back from reject from-headers 
with .local because a large part of mailadmins configure their systems 
as 'mail.company.local' and in case of bounces (mailbox full as example) 
the envelope is a null-sender and the from-header postmaster@fool.local


well, and all that systems have a message-id ending with .local and if 
you want numbers - we would have rejected or tagged 981 *100% ham* 
messages with a message-id ending with .local and my users would have 
crucified me for such a setup

Re: How to reject mails with special message-id (Debian, Amavis, Spamassassin)

[Thomas Barth]

Am 20.09.2016 um 15:27 schrieb Bowie Bailey:


X-Spam-Status: Yes, score=14.009 tag=2 tag2=6.31 kill=6.31
tests=[HTML_MESSAGE=0.001, MESSAGEID_LOCAL=8,
MIME_HTML_ONLY=1.105,
PYZOR_CHECK=1.985, RCVD_IN_BRBL_LASTEXT=1.644, RDNS_NONE=1.274]
autolearn=no autolearn_force=no


The base SA ruleset is optimized to detect spam with a score of 5.0.  If
you raise that score, you will allow more spam to come through.  If you
lower that score, you will see more legitimate messages blocked as
spam.  Make sure you know what you are doing before you change this score.



I read that 5.0 is aggressive and suitable for single user setup, 
conservative values are 8.0 or 11.0.


required_score n.nn (default: 5)
https://spamassassin.apache.org/full/3.2.x/doc/Mail_SpamAssassin_Conf.html


I ve checked most of the mails recognized as spam. The lowest score was 
8.6x so far.


Here is another mail from ...local. It definitely was spam with zip 
attachment. Common is a sender address with digits.
 -> , quarantine: 
l/spam-lEHVGcheLkyq.gz, Message-ID: 
<20160920202635.6b90ec7...@allfromboats.com.local>, mail_id: 
lEHVGcheLkyq, Hits: 19.118


May be I also should block sender adresses with more than 2 digits in 
the name?

Re: How to reject mails with special message-id (Debian, Amavis, Spamassassin)

[Bowie Bailey]

On 9/20/2016 7:31 AM, Thomas Barth wrote:



Am 20.09.2016 um 12:23 schrieb Matus UHLAR - fantomas:


Message-Id: <20160920154140.f5a976c...@static.vnpt.vn.local>


you can put this in /etc/spamassassin/local.cf

header   MESSAGEID_LOCAL  Message-Id =~ /\.local>$/
scoreMESSAGEID_LOCAL  1
describe MESSAGEID_LOCAL  Message-Id contains ".local" TLD

and reload whatever you plug into your MTA.


It works. I already got a mail by ...vn.local

X-Spam-Status: Yes, score=14.009 tag=2 tag2=6.31 kill=6.31
tests=[HTML_MESSAGE=0.001, MESSAGEID_LOCAL=8, 
MIME_HTML_ONLY=1.105,

PYZOR_CHECK=1.985, RCVD_IN_BRBL_LASTEXT=1.644, RDNS_NONE=1.274]
autolearn=no autolearn_force=no


The base SA ruleset is optimized to detect spam with a score of 5.0.  If 
you raise that score, you will allow more spam to come through.  If you 
lower that score, you will see more legitimate messages blocked as 
spam.  Make sure you know what you are doing before you change this score.


I would score your MESSAGEID_LOCAL rule closer to 3 or 4 at the most.  
This way, it still makes a big contribution to the score, but messages 
will not be blocked based on it unless there are other spam signs.  The 
best idea with new rules is to add them with a low score at first and 
keep an eye on them for at least a couple of days to make sure that they 
are performing as expected before raising the score.


Remember that Amavis has its own settings for required_score and message 
markup.  Any of those settings in the SA config files will be ignored.


--
Bowie

Re: How to reject mails with special message-id (Debian, Amavis, Spamassassin)

[Paul Stead]

On 20/09/16 12:31, Thomas Barth wrote:
Am 20.09.2016 um 12:23 schrieb Matus UHLAR - fantomas:

Message-Id: 
<20160920154140.f5a976c...@static.vnpt.vn.local>

you can put this in /etc/spamassassin/local.cf

header   MESSAGEID_LOCAL  Message-Id =~ /\.local>$/
scoreMESSAGEID_LOCAL  1
describe MESSAGEID_LOCAL  Message-Id contains ".local" TLD

and reload whatever you plug into your MTA.

It works. I already got a mail by ...vn.local

X-Spam-Status: Yes, score=14.009 tag=2 tag2=6.31 kill=6.31
   tests=[HTML_MESSAGE=0.001, MESSAGEID_LOCAL=8, MIME_HTML_ONLY=1.105,
   PYZOR_CHECK=1.985, RCVD_IN_BRBL_LASTEXT=1.644, RDNS_NONE=1.274]
   autolearn=no autolearn_force=no

Be aware that there are an awful lot of Microsoft Exchange servers that are set 
up with .local, .lan and .corp TLDs (for a number of reasons).

You may find you're blocking legitimate email from an Exchange server (poorly?) 
configured in this way.

Paul
--
Paul Stead
Systems Engineer
Zen Internet

Re: How to reject mails with special message-id (Debian, Amavis, Spamassassin)

[Matus UHLAR - fantomas]

Am 20.09.2016 um 12:23 schrieb Matus UHLAR - fantomas:


Message-Id: <20160920154140.f5a976c...@static.vnpt.vn.local>


you can put this in /etc/spamassassin/local.cf

header   MESSAGEID_LOCAL  Message-Id =~ /\.local>$/
scoreMESSAGEID_LOCAL  1
describe MESSAGEID_LOCAL  Message-Id contains ".local" TLD

and reload whatever you plug into your MTA.


On 20.09.16 13:31, Thomas Barth wrote:

It works. I already got a mail by ...vn.local

X-Spam-Status: Yes, score=14.009 tag=2 tag2=6.31 kill=6.31
   tests=[HTML_MESSAGE=0.001, MESSAGEID_LOCAL=8, MIME_HTML_ONLY=1.105,
   PYZOR_CHECK=1.985, RCVD_IN_BRBL_LASTEXT=1.644, RDNS_NONE=1.274]
   autolearn=no autolearn_force=no


I see you don't have bayes DB set up. With amavis you can have site-wide
bayes database, stored under amavis user.
proper training should get this spam caught.

--
Matus UHLAR - fantomas, uh...@fantomas.sk ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
You have the right to remain silent. Anything you say will be misquoted,
then used against you. 

Re: How to reject mails with special message-id (Debian, Amavis, Spamassassin)

[Thomas Barth]

Am 20.09.2016 um 12:23 schrieb Matus UHLAR - fantomas:


Message-Id: <20160920154140.f5a976c...@static.vnpt.vn.local>


you can put this in /etc/spamassassin/local.cf

header   MESSAGEID_LOCAL  Message-Id =~ /\.local>$/
scoreMESSAGEID_LOCAL  1
describe MESSAGEID_LOCAL  Message-Id contains ".local" TLD

and reload whatever you plug into your MTA.


It works. I already got a mail by ...vn.local

X-Spam-Status: Yes, score=14.009 tag=2 tag2=6.31 kill=6.31
tests=[HTML_MESSAGE=0.001, MESSAGEID_LOCAL=8, MIME_HTML_ONLY=1.105,
PYZOR_CHECK=1.985, RCVD_IN_BRBL_LASTEXT=1.644, RDNS_NONE=1.274]
autolearn=no autolearn_force=no

Re: How to reject mails with special message-id (Debian, Amavis, Spamassassin)

[RW]

On Tue, 20 Sep 2016 12:50:26 +0200
Thomas Barth wrote:

> Thanks for your help Matus
> 

> bayes_auto_learn 1
> 
> For autolearning do I have to put these mails to a special folder?

No, but autolearning is not very good. If you are scanning your own
mail you are better-off not using it. IIWY I'd turn it off, clear the
database and train it manually using sa-learn. 

Just be sure that you are training as the correct unix user.  

> >> Message-Id: <20160920154140.f5a976c...@static.vnpt.vn.local>  
> >
> > you can put this in /etc/spamassassin/local.cf
> >
> > header   MESSAGEID_LOCAL  Message-Id =~ /\.local>$/
> > scoreMESSAGEID_LOCAL  1
> > describe MESSAGEID_LOCAL  Message-Id contains ".local" TLD  
> 
> 1 is the value added to the hits? I ve set required_score to 8.0. To 
> make sure it ll be rejected as spam do I have to set to
> 
>score MESSAGEID_LOCAL 8.0

It's pretty common for legitimate email to match this in my experience.

Re: How to reject mails with special message-id (Debian, Amavis, Spamassassin)

[Thomas Barth]

Thanks for your help Matus


Am 20.09.2016 um 12:23 schrieb Matus UHLAR - fantomas:

there are many ways to make SA better - configure BAYES database, enable
network tests (razor, pyzor, DCC), and not use DNS server for resolution
that is shared with other companies...


I have already enabled razor, pyzor, but it seems not to be enough. DCC 
cant be found in the repository.


#pyzor
use_pyzor 1
pyzor_path /usr/bin/pyzor

#razor
use_razor2 1
razor_config /etc/razor/razor-agent.conf

#   Use Bayesian classifier (default: 1)
#
use_bayes 1
use_bayes_rules 1

#   Bayesian classifier auto-learning (default: 1)
#
bayes_auto_learn 1

For autolearning do I have to put these mails to a special folder?





Message-Id: <20160920154140.f5a976c...@static.vnpt.vn.local>


you can put this in /etc/spamassassin/local.cf

header   MESSAGEID_LOCAL  Message-Id =~ /\.local>$/
scoreMESSAGEID_LOCAL  1
describe MESSAGEID_LOCAL  Message-Id contains ".local" TLD


1 is the value added to the hits? I ve set required_score to 8.0. To 
make sure it ll be rejected as spam do I have to set to


  score MESSAGEID_LOCAL 8.0

as well?


and reload whatever you plug into your MTA.


My system is debian, amavis, spamassassin, clamav. How can I extend
Spamassassin? Or can I just call /usr/bin/sa-update --no-gpg regularly
and wait to get better ruleset?


you should do that. Debian SA does that if you set CRON=1 in
/etc/default/spamassassin


Ah, ok, I have already added it to the crontab

# m h  dom mon dow   command
0 6 */3 * * /usr/bin/sa-update --no-gpg &> /dev/null

but I will use the debian way then.

Re: How to reject mails with special message-id (Debian, Amavis, Spamassassin)

[Matus UHLAR - fantomas]

On 20.09.16 12:15, Thomas Barth wrote:
I get mails not recognized as spam and I would like to extend 
spamassassin to reject these mails.


there are many ways to make SA better - configure BAYES database, enable
network tests (razor, pyzor, DCC), and not use DNS server for resolution
that is shared with other companies...

The mails look very normal, but 
the message-id is conspicuous. I want to reject the mail if it 
contains .local at the end of message-id.


you can create local SA rule and score it.
You just have to be careful so that rule won't score on ham mail.


Message-Id: <20160920154140.f5a976c...@static.vnpt.vn.local>


you can put this in /etc/spamassassin/local.cf

header   MESSAGEID_LOCAL  Message-Id =~ /\.local>$/
scoreMESSAGEID_LOCAL  1
describe MESSAGEID_LOCAL  Message-Id contains ".local" TLD

and reload whatever you plug into your MTA.

My system is debian, amavis, spamassassin, clamav. How can I extend 
Spamassassin? Or can I just call /usr/bin/sa-update --no-gpg 
regularly and wait to get better ruleset?


you should do that. Debian SA does that if you set CRON=1 in
/etc/default/spamassassin

--
Matus UHLAR - fantomas, uh...@fantomas.sk ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
I feel like I'm diagonally parked in a parallel universe. 

How to reject mails with special message-id (Debian, Amavis, Spamassassin)

[Thomas Barth]

Hello,

I get mails not recognized as spam and I would like to extend 
spamassassin to reject these mails. The mails look very normal, but the 
message-id is conspicuous. I want to reject the mail if it contains 
.local at the end of message-id.


Subject: Tracking data
From: "Paula Booker" 
MIME-Version: 1.0
Content-Type: multipart/related;
boundary="part_a9cf4dac64cf39e24ba76020748b62f0"
Message-Id: <20160920154140.f5a976c...@static.vnpt.vn.local>
Date: Tue, 20 Sep 2016 15:41:40 +0700


My system is debian, amavis, spamassassin, clamav. How can I extend 
Spamassassin? Or can I just call /usr/bin/sa-update --no-gpg regularly 
and wait to get better ruleset?