Re: Problem with txrep database

[Mark Martinec]

On 2016-07-04 11:16, Ralf Hildebrandt wrote:

Using SpamAssassin version 3.4.1 on Ubuntu precise (14.04)

My /etc/mail/spamassassin/local.cf has:

use_txrep1
normalize_charset1
txrep_factoryMail::SpamAssassin::SQLBasedAddrList
user_awl_dsn DBI:mysql:spamassassin:127.0.0.1
user_awl_sql_usernameroot
user_awl_sql_passwordsecret
user_awl_sql_table   txrep

My /etc/mail/spamassassin/v341.pre contains:

loadplugin Mail::SpamAssassin::Plugin::TxRep

Yet I'm seeing no entries in the sql database:

# mysql -uroot -psecret
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 80
Server version: 5.5.49-MariaDB-1ubuntu0.14.04.1 (Ubuntu)

Copyright (c) 2000, 2016, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MariaDB [(none)]> use spamassassin;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed
MariaDB [spamassassin]> select * FROM txrep;
Empty set (0.04 sec)

SpamAssassin is used from within amavisd-new; it seems that the config is being 
read/used:

# su - amavis
$ spamassassin --lint -D 2>&1 | fgrep -i txrep
Jul  4 11:15:18.924 [17820] dbg: plugin: loading 
Mail::SpamAssassin::Plugin::TxRep from @INC
Jul  4 11:15:18.934 [17820] dbg: TxRep: new object created
Jul  4 11:15:19.247 [17820] dbg: config: fixed relative path: 
/var/lib/spamassassin/3.004001/updates_spamassassin_org/60_txrep.cf
Jul  4 11:15:19.247 [17820] dbg: config: using 
"/var/lib/spamassassin/3.004001/updates_spamassassin_org/60_txrep.cf" for 
included file
Jul  4 11:15:19.247 [17820] dbg: config: read file 
/var/lib/spamassassin/3.004001/updates_spamassassin_org/60_txrep.cf
Jul  4 11:15:20.212 [17820] dbg: plugin: 
Mail::SpamAssassin::Plugin::TxRep=HASH(0x4ea0c08) implements 'learner_new', 
priority 0
Jul  4 11:15:20.553 [17820] dbg: TxRep: no scan in lint mode, quitting



Try enabling debug logging for plugins TxRep and auto-whitelist.

Amavis will log SA debug messages at log level 3.

amavisd.conf:
  $log_level = 3;  # or higher
  $sa_debug = 'TxRep,auto-whitelist';

make sure syslogd is not filtering out debug level messages,
then reload amavisd and grep through the log:

  tail -f /var/log/amavisd-debug.log | egrep '(TxRep|auto-whitelist): '


Mark

Re: Numerous problems with SA under Raspbian jessie & Ubuntu 15.10

[Mark Martinec]

On 2015-12-20 16:15, Jari Fredriksson wrote:

Dec 20 17:04:08.149 [32761] dbg: timing: total 128808 ms -
load_scoreonly_sql: 0.27 (0.0%), signal_user_changed: 127278 (98.8%),
b_tie_ro: 127275 (98.8%), parse: 6 (0.0%), extract_message_metadata:
187 (0.1%), get_uri_detail_list: 8 (0.0%), tests_pri_-1000: 234
(0.2%), tests_pri_-950: 2.4 (0.0%), tests_pri_-900: 2.9 (0.0%),
tests_pri_-400: 130 (0.1%), check_bayes: 116 (0.1%), b_tokenize: 18
(0.0%), b_tok_get_all: 44 (0.0%), b_comp_prob: 4.4 (0.0%),
b_tok_touch_all: 45 (0.0%), b_finish: 2.1 (0.0%), tests_pri_0: 830
(0.6%), check_spf: 66 (0.1%), poll_dns_idle: 49 (0.0%), check_pyzor:
0.58 (0.0%), tests_pri_500: 27 (0.0%), tests_pri_1000: 2.7 (0.0%),
rewrite_mail: 0.85 (0.0%), copy_config: 66 (0.1%)

signal_user_changed and b_tie_ro? What are those?


SpamAssassin/BayesStore/SQL.pm

It is trying to connect to your SQL database, which takes two minutes.

  Mark

Re: Numerous problems with SA under Raspbian jessie & Ubuntu 15.10

[Mark Martinec]

Btw, the fix for:

  "each on reference is experimental at ..."
  "keys on reference is experimental at
 /usr/share/perl5/Mail/SpamAssassin/Plugin/URILocalBL.pm"

is at Bug 7208:
  https://bz.apache.org/SpamAssassin/show_bug.cgi?id=7208

Mark

Re: Google redirects

[Mark Martinec]

On 2015-12-18 16:29, Axb wrote:

On 12/18/2015 04:17 PM, Mark Martinec wrote:

On 2015-12-17 22:41, Axb wrote:

could you make a version using redirector_pattern so the redirected
target can be looked up via URIBL plugin?


Isn't this already the case? Redirect targets are added
to a list of URIs and are subject to same rules as
directly collected URIs.



I suggested converting the rawbody rule John was working on into a
redirector_pattern


Note that the following rule as posted by John:

  uri __GOOG_MALWARE_DNLD 
m;^https?://[^/]*\.google\.com/[^?]*url\?.*[\?&]download=1;i


would not currently work as a redirector_pattern due to the problem
I posted in my today's reply (Re: redirector_pattern question);
i.e. where the redirector target contains "http:", followed
by other URI arguments (like "=1" here).

  Mark

Re: Google redirects

[Mark Martinec]

On 2015-12-17 22:41, Axb wrote:

could you make a version using redirector_pattern so the redirected
target can be looked up via URIBL plugin?


Isn't this already the case? Redirect targets are added
to a list of URIs and are subject to same rules as
directly collected URIs.

  Mark

Re: redirector_pattern question

[Mark Martinec]

On 2015-12-18 11:19, Paul Stead wrote:

After the messages last night I've been looking into the
redirector_pattern config option - I'm seeing weird results...

Given the redirector_pattern of:

redirector_pattern m'https?://www.google.com/url?q=([^&]+).*'i

I've noticed that spamassassin can sometimes miss - I don't think it's
to do with the regex above as I've tried multiple variations - I've 
made

it reproducible but I'm not sure why this is happening:

p1: http://pastebin.com/w93ZZp9h
p2: http://pastebin.com/p2pciGC3

[...]

# spaspamassassin -D -t < p2 2>&1 | grep baddomain

p2 doesn't pick up on baddomain.com

Any thoughts or have I stumbled upon a problem?



Two problems there, one is in your regexp, the other is in
the SpamAssassin logic of dealing with redirects.

The parameter of redirector_pattern is a regular expression,
dots and a question mark have a special meaning in a regexp.
If this special meaning is not wanted, they need to be quoted.
Also, SpamAssassin does not add anchoring to a redirector_pattern
regexp, so you need to add it yourself when needed.
And the .* at the end is redundant for this reason.

So it should be something like:

  redirector_pattern m{^https?://www\.google\.com/url\?q=([^&]+)}i


The other problem is in 
Mail::SpamAssassin::Util::uri_list_canonicalize()

near line 1346:

  # deal with http redirectors.  strip off one level of redirector
  # and add back to the array.  the foreach loop will go over those
  # and deal appropriately.
  # bug 3308: redirectors like yahoo only need one '/' ... 
  if ($rest =~ m{(https?:/{0,2}.+)$}i) {
push(@uris, $1);
  }

  # resort to redirector pattern matching if the generic https? 
check

  # doesn't result in a match -- bug 4176
  else {
foreach (@{$redirector_patterns}) {

Note that it tries the hard-coded check first, and skips
evaluating redirector patterns when the hard-coded match
was successful.

So your redirector pattern was not tried at all, and at the same time
the hard-coded check obtained an invalid URL: from an URL
  "/url?q=http://baddomain.com=1;
it collected as "http://baddomain.com=1; instead of 
"http://baddomain.com;

The URI syntax after a '?' should treat "=1" as a second argument
and not glue it with the first argument "http://baddomain.com;.

So the resulting invalid URL "http://baddomain.com=1;
does not match any URI rules for baddomain.com .


Please try the attached patch (to 3.4.1 or trunk), it reverses
the order of checks: tries redirector_patterns first, and only
falls back to a sloppy hard-coded check as a last resort.
(and that hard-coded check would better be fixed too)

... and please open a bug report in bugzilla.


MarkIndex: lib/Mail/SpamAssassin/Util.pm
===
--- lib/Mail/SpamAssassin/Util.pm	(revision 1720791)
+++ lib/Mail/SpamAssassin/Util.pm	(working copy)
@@ -1342,24 +1342,28 @@
   # deal with http redirectors.  strip off one level of redirector
   # and add back to the array.  the foreach loop will go over those
   # and deal appropriately.
-  # bug 3308: redirectors like yahoo only need one '/' ... 
-  if ($rest =~ m{(https?:/{0,2}.+)$}i) {
-push(@uris, $1);
-  }
 
-  # resort to redirector pattern matching if the generic https? check
-  # doesn't result in a match -- bug 4176
-  else {
-	foreach (@{$redirector_patterns}) {
-	  if ("$proto$host$rest" =~ $_) {
-	next unless defined $1;
-	dbg("uri: parsed uri pattern: $_");
-	dbg("uri: parsed uri found: $1 in redirector: $proto$host$rest");
-	push (@uris, $1);
-	last;
-	  }
-	}
+  # try redirector pattern matching first
+  # (but see also bug 4176)
+  my $found_redirector_match;
+  foreach my $re (@{$redirector_patterns}) {
+if ("$proto$host$rest" =~ $re) {
+  next unless defined $1;
+  dbg("uri: parsed uri pattern: $re");
+  dbg("uri: parsed uri found: $1 in redirector: $proto$host$rest");
+  push (@uris, $1);
+  $found_redirector_match = 1;
+  last;
+}
   }
+  if (!$found_redirector_match) {
+# try generic https? check if redirector pattern matching failed
+# bug 3308: redirectors like yahoo only need one '/' ... 
+if ($rest =~ m{(https?:/{0,2}.+)$}i) {
+  push(@uris, $1);
+  dbg("uri: parsed uri found: $1 in hard-coded redirector");
+}
+  }
 
   
   ## TVD: known issue, if host has multiple combinations of the following,

Re: DNS lookups fail with SpamAssassin since Net::DNS 1.03

[Mark Martinec]

Not sure about SPF. It's supposed to be fixed in the current 3.4 branch
and in trunk, which is why I'm not seeing a problem with Net::DNS 1.03
or Net::DNS 1.04. Will check how the released version of 3.4.1 fares
with Net::DNS 1.04 regarding SPF. The emergency patches were applied
to FreeBSD ports, don't know about other distributions.

Here are the relevant bug entries:
  https://bz.apache.org/SpamAssassin/show_bug.cgi?id=7231
  https://bz.apache.org/SpamAssassin/show_bug.cgi?id=7265


I forgot about the Bug 7223, which also affects Net::DNS 1.01
or later:

https://bz.apache.org/SpamAssassin/show_bug.cgi?id=7223


Tried it now with 3.4.1 and Net::DNS 1.04.

You still need to apply the patch from Bug 7223 (in addition
to a patch from Bug 7231), then it passes all tests with
Net::DNS 1.04 (even without patches from Bug 7265).

Seems easiest to install SpamAssassin from a svn 3.4 branch
( svn checkout http://svn.apache.org/repos/asf/spamassassin/branches/3.4 
spamassassin-3.4 )

or downgrade Net::DNS to a pre-1.* version (i.e. 0.83).

  Mark

Re: DNS lookups fail with SpamAssassin since Net::DNS 1.03

[Mark Martinec]

Ian Eiloart wrote:
I had this problem after upgrading from a rather old version of SA. 
After
upgrading to Net::DNS 1.04, the errors aren’t logged, but SpamAssassin 
isn’t

finding SPF records. I wonder whether anyone can offer any suggestions.

[...]

Yesterday, I upgraded Net::DNS 1.03 to Net::DNS 1.04, and the
"Can't locate object method "handles"" errors are not longer happening,
but I still didn’t see any SPF_ results in my logs. I should see about
30,000 a day, judging by the volume before I upgraded.


Not sure about SPF. It's supposed to be fixed in the current 3.4 branch
and in trunk, which is why I'm not seeing a problem with Net::DNS 1.03
or Net::DNS 1.04. Will check how the released version of 3.4.1 fares
with Net::DNS 1.04 regarding SPF. The emergency patches were applied
to FreeBSD ports, don't know about other distributions.

Here are the relevant bug entries:
  https://bz.apache.org/SpamAssassin/show_bug.cgi?id=7231
  https://bz.apache.org/SpamAssassin/show_bug.cgi?id=7265

There were three changes in New::DNS that affected SpamAssassin.
The change in 1.01 affected SpamAssassin due to our sloppy
parsing of Net::DNS results. The changes brought by 1.03
affected SpamAssassin on two fronts, both are due to an
incompoatible API change in Net::DNS: different object class
expected by bgread (which affected a handful of other
Perl modules too), and a change in semantics of "retry" and
"retrans" options, which affected DKIM plugin.

  Mark

Re: Re-4: A rule to check X-ASN header

[Mark Martinec]

My eventual goal is to test for "Has google in the sender name OR
domain" and "is NOT from a ASN owned by Google".

https://www.ultratools.com/tools/asnInfoResult?domainName=Google

Am I'm not explaining myself correctly?


... nevertheless ... a valid DKIM signature by google is as good
if not a better proof that a mail is coming from Google:

  full _DKIM_VALID_GOOGLE eval:check_dkim_valid(gmail.com, 
googlemail.com, googlegroups.com)

  header _GOOGLE_ANYWHERE_IN FROM  From =~ /google|gmail/i
  meta GOTTA _GOOGLE_ANYWHERE_IN FROM && !_DKIM_VALID_GOOGLE

Anyway, an ASN test would fail on mailing list mail by google senders.
A DKIM test would also likely but not necessarily fail in such mail,
depending how a mailing list is configured. For example this
SpamAssassin mailing list preserves DKIM signature validity just fine.

  Mark

Re: DNS lookups fail with SpamAssassin since Net::DNS 1.03

[Mark Martinec]

Net::DNS 1.03 breaks compatibility with SpamAssassin:
DNS lookups no longer work, and warnings like the following pop up:


On 2015-11-13 19:22, Quanah Gibson-Mount wrote:

Well, IO::Socket::IP support is new in Net::DNS 1.03, but it is only
used if IO::Socket::INET6 is not present.  I would assume you can use
it as long as you have IO::Socket::INET6 installed, but I haven't
tested that assumption.


They changed the object class that their bgsend() returns,
and bgread() expects such object in its argument.

SpamAssassin provides it's own bgsend (for multiple historical reasons,
mainly to do with deficiencies or bugs in older versions of Net::DNS)
and gives an object of class IO::Socket to bgread(), which is fine
according to what the 1.02 documentation specifies:

$ man Net::DNS::Resolver :
  bgread
Reads the answer from a background query (see "bgsend").
The argument is an "IO::Socket" object returned by "bgsend".

This is no longer so in 1.03, as the Net::DNS bgsend() now provides
an IO::Select object and bgread expects a IO::Select object,
no longer happy with an IO::Socket object that SpamAssassin provides.

The new 1.03 docs say:
   bgread
 Reads the answer from a background query (see "bgsend").
 The argument is an "IO::Socket" object returned by "bgsend".

To me, this is an incompatible documented change - not something
one would expect in an 1.02 -> 1.03 update.

  Mark

DNS lookups fail with SpamAssassin since Net::DNS 1.03

[Mark Martinec]

Net::DNS 1.03 breaks compatibility with SpamAssassin:
DNS lookups no longer work, and warnings like the following pop up:

  lookup failed: Can't locate object method "handles" via package 
"IO::Socket::IP"

at /usr/local/lib/perl5/site_perl/Net/DNS/Resolver/Base.pm line 735.

There is a CPAN ticket open for this:
  https://rt.cpan.org/Public/Bug/Display.html?id=108745

Please stick to Net::DNS 1.02 until this is resolved.

  Mark

Re: Relay Country Plugin GEOIP issue

[Mark Martinec]

2015-10-15 00:09, a...@satester.com wrote:

Does the Geo::IP package need
to be recompiled for the change to go into effect?


No.



Use of uninitialized value $hasStructureInfo in numeric eq (==)
at (eval 31) line 5520


According to comments in Geo/IP.pm your GeoIP database files
must be very old if the program followed that codepath:

# no structure info, must be pre Sep 2002 database, go back to


Check your database:

  $ spamassassin --lint -D metadata' 2>&1 | fgrep RelayCountry

should yield something like:

Oct 15 01:26:45.584 [78315] dbg: metadata: RelayCountry:
  Using database: Geo::IP GEO-106FREE 20151006 Build 1
  Copyright (c) 2015 MaxMind Inc All Rights Reserved


Fresh files can be downloaded from:

  http://dev.maxmind.com/geoip/legacy/geolite/

unzip them and place them to their expected location,
typically /usr/local/share/GeoIP/ .
You need files GeoIP.dat and GeoIPv6.dat there.

  Mark

Re: charset=utf-16 tricks out SA

[Mark Martinec]

2015-10-10 03:03, RW wrote:


I'm not seeing any body tokens, even after training.

I was expecting that the text would be tokenized as individual UTF-8
sequences. ASCII characters encoded as UTF-16 and decoded with the
wrong endianness are still valid UTF-16. Normalizing them into
UTF-8 should produce completely multi-byte UTF-8 without whitespace or
punctuation (not counting U+2000 inside UTF-8).

If I add John Hardin's diagnostic rule

body __ALL_BODY /.*/
tflags   __ALL_BODY multiple

I get:

ran body rule __ALL_BODY ==> got hit: " _ _D_e_a_r_
_p_o_t_e_n_c_i_a_l_ _p_a_r_t_n_e_r_,_ _ _W_e_ _a_r_e_
_p_r_o_f_e_s_s_i_o_n_a_l_ _i_n_ _e_n_g_i_n_e_e_r_i_n_g_,_
_...

It looks like it's still UTF-16, and Bayes is seeing individual
letters (which are too short to be tokens) separated by nulls.


The way it works now is if the decoding as declared fails,
and some guessing fails too, it falls back to Windows-1252,
which are single byte characters (superset of ISO-8859-1),
which can't fail, and gives you the result you are seeing
(spaced out by null characters).


If I change the mime to utf-16le it works correctly, except that the
subject isn't converted - including the copy in the body.  If I set the
mime to utf-16le I get what appears to be the multi-byte UTF-8 I was
expecting.


The endoded-word in the Subject header field needs to be
declared as utf-16le too, then it works (tried on trunk).


So SA isn't falling back to big-endian, it wont normalize without an
explicit endianess.


It tries as BE, and when Encode::decode reports a failure, it
decodes as Windows-1252.


BTW with normalize_charset 0 it looks like a spammer can effectively
turn-off body tokenization by using UTF-16 (with correct endianness).


Yes. There are also other tricks that a spammer can't play.
It's not possible to emulate all different behaviours of
various mail reading programs. Still, in the case we have
it would make sense to try also the utf-16le, since this is
a default endianness in Windows.

  Mark

Re: charset=utf-16 tricks out SA

[Mark Martinec]

Reindl Harald wrote:


no custom body rules hit like they do for ISO/UTF8 :-(

What is your normalize_charsets setting?


enabled, that's what i meant with "like they do for ISO/UTF8" and
adding "dear potencial partner" to CUST_BODY_17 did not change the
score

see attached sample and rule below

body  CUST_BODY_17/.*(1st page ranking of google|dear
potencial partner).*/i
score CUST_BODY_171.0
describe  CUST_BODY_17Contains Low


The problem with this message is that it declares encoding
as UTF-16, i.e. not explicitly stating endianness like
UTF-16BE or UTF-16LE, and there is no BOM mark at the
beginning of each textual part, so endianness cannot be
determined. The RFC 2781 says that big-endian encoding
should be assumed in absence of BOM.
See https://en.wikipedia.org/wiki/UTF-16

In the provided message the actual endianness is LE, and
BOM is missing, so decoding as UTF-16BE fails and the
rule does not hit. Garbage-in, garbage-out.

If you manually edit the sample and replace UTF-16
with UTF-16LE (and normalize is enabled), your rule should
hit - at least it does so in the current trunk code.

If this seems to be common in the wild, please open a
bug ticket, as Kevin suggested, and attach the sample there.

  Mark

Re: command arguments for spamd on FreeBSD

[Mark Martinec]

I recently updated SpamAssassin to version 3.4.1 on FreeBSD
8.4-release p36, running Postfix/Dovecot2.
My question is simple, how do I pass command arguments to spamd? (and
did this change since 3.3.x?)
I am quite sure that before upgrading the following lines in rc.conf
worked correctly to pass flags to spamd:
spamd_enable="YES"
spamd_flags="-s local5"
spamd_command_args="-d -A [2a03:6000:::xxx] -u spamd -x -q -r 
${pidfile}"

But after the update the line "spamd_command_args" seems to be ignored.


Put all command line options for spamd in spamd_flags.

  Mark

Re: Heads up: Net::DNS update may have quietly broken your SpamAssassin.

[Mark Martinec]

Olivier Nicole wrote:

"Bill Cole"  writes:

I noticed today that the hit rate on URIBL* rules had dropped to to 
zero

since my last round of updates, and after many hours of trying to
determine why which included reviewing BIND configs and packet 
captures

and dissection, I nailed it down to SA making DNS queries without the
"recursion desired" flag. Since my local nameservers isn't 
authoritative

for much, this meant a whole lot of "no answer, no error" DNS replies.

It turns out that this is due to an internal change introduced in 
recent

versions of Net::DNS, which SA relied upon to set the RD flag
automatically. See
https://bz.apache.org/SpamAssassin/show_bug.cgi?id=7223 for details 
and

a patch.


Thank you. I just check and for what it is worth, recent installations 
of

SA on FreeBSD do include the patch.

Best regards,
Olivier


Not to forget the fix at:
  https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=202283
which is also needed with Net::DNS 1.01 or later.

Already cherrypicked in the FreeBSD port of SpamAssassin:
  https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=202283

Mark

Re: Heads up: Net::DNS update may have quietly broken your SpamAssassin.

[Mark Martinec]

Not to forget the fix at:
  https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=202283
which is also needed with Net::DNS 1.01 or later.


Sorry, wrong link, should be:
  https://bz.apache.org/SpamAssassin/show_bug.cgi?id=7231


Already cherrypicked in the FreeBSD port of SpamAssassin:
  https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=202283



Mark

Re: New warnings after Perl upgrade to 5.20?

[Mark Martinec]

Larry Rosenman wrote:


Getting the following on sa-learn:



each on reference is experimental at
  /usr/local/lib/perl5/site_perl/Mail/SpamAssassin/Plugin/URILocalBL.pm
  line 353.
keys on reference is experimental at
  /usr/local/lib/perl5/site_perl/Mail/SpamAssassin/Plugin/URILocalBL.pm
  line 377.
keys on reference is experimental at
  /usr/local/lib/perl5/site_perl/Mail/SpamAssassin/Plugin/URILocalBL.pm
  line 406.

this is after I upgraded my PERL to 5.20.
(SA 3.4.1 on FreeBSD 10.2-STABLE from Ports)


Just warnings fortunately.

  https://bz.apache.org/SpamAssassin/show_bug.cgi?id=7208

fix in revision 1684653:

  
https://svn.apache.org/viewvc/spamassassin/trunk/lib/Mail/SpamAssassin/Plugin/URILocalBL.pm?r1=1684653=1684652=1684653



Mark

Re: ALL_TRUSTED triggering _intermittently_ on external mails?

[Mark Martinec]

PGNd wrote:


I'm running
postfix 3.0.1
amavisd-new-2.10.1 (20141025)
SpamAssassin version 3.4.1
on linux/64.

amavisd/spamassasin is invoked as a postfix prequeue proxy filter.

Spam is getting scanned and scored.  Usually correctly.

Intermittenly, I get an email that gets SHORTCIRCUITED on ALL_TRUSTED
(example below).

I've read through https://wiki.apache.org/spamassassin/TrustPath, and
am missing something; I haven't yet managed to pick out the problem.

My SA config includes the following
internal_networks  127.0.0.0/8 10.2.2.0/24 10.1.1.0/24 
X.X.X.X/29
trusted_networks   10.2.2.0/24 10.1.1.0/24 
X.X.X.X/29


Here are headers for the received message.  Note the shortcircuited 
score:


X-Spam-Status: No, score=-101 tagged_above=- required=5
tests=[ALL_TRUSTED=-1, SC_HAM=-100, SHORTCIRCUIT=-0.0001]
autolearn=disabled



the msg's received-from headers are _not_ all on my internal networks,


Are the 196.28.80.29, 196.28.80.61, and 196.28.66.13 in your
trusted X.X.X.X/29 ? If they are, then hitting ALL_TRUSTED is expected.


grep Received: spam1.txt
INT Received: from mail-backend..com (LHLO 
mail-backend..com)
INT		Received: from relay-vpn.mail..com (internal.mail..com 
[10.1.1.10])

INT Received: from localhost (localhost [127.0.0.1])
INT Received: from amavis-feed.mail..com ([10.1.1.10])
INT Received: from localhost (localhost [127.0.0.1])
INT Received: from mail..com ([127.0.0.1])
EXT Received: from relay09.smp.mweb.co.za (relay09.smp.mweb.co.za
[196.28.80.29])
EXT Received: from relay-mc01.smp.mweb.co.za ([196.28.80.61])
EXT Received: from [196.28.66.13] (helo=MWSJHBMC03)

If I 'spamassassin -D  spam1.txt' the message, it scores completely 
differently



Jun 18 22:38:19.967 [19747] dbg: check:
  tests=BAYES_50,DCC_CHECK,HTML_MESSAGE,UNPARSEABLE_RELAY


See the UNPARSEABLE_RELAY here?  It prevents ALL_TRUSTED from firing.



Received: from mail-backend..com (LHLO mail-backend..com)
 (10.2.2.20) by mail-backend..com with LMTP; Thu, 18 Jun 2015
 16:50:56 -0700 (PDT)


It is this Received header field above that is supposedly unparsable,
and it was added _after_ a message went though a content filter.

Remove it and re-try the command-line spamassassin test in the message.

  Mark

Re: ALL_TRUSTED triggering _intermittently_ on external mails?

[Mark Martinec]

PGNd wrote:


The LHLO/LMTP header still is added at the backend, and
UNPARSEABLE_RELAY still hits.  I've not yet determined what the actual
problem with the parsing is.


It's a shortcoming/bug in the SpamAssassin ad-hoc parser.

Please open a Bugzilla ticket and provide a sample of
your Received header field (which is perfectly valid
according to RFC 5321).

  Mark

Re: Confused about Bayes expiry

[Mark Martinec]

 Ian Zimmerman wrote:


I am very confused by the various features involving expiry from Bayes.

perldoc Mail::SpamAssassin::Conf :

   bayes_expiry_max_db_size  (default: 15)

   What should be the maximum size of the Bayes tokens 
database?

   When expiry occurs, the Bayes system will keep either 75% of
   the maximum value, or 100,000 tokens, whichever has a larger
   value.  150,000 tokens is roughly equivalent to a 8Mb
   database file.

   bayes_auto_expire (default: 1)

   If enabled, the Bayes system will try to automatically 
expire

   old tokens from the database.  Auto-expiry occurs when the
   number of tokens in the database surpasses the
   bayes_expiry_max_db_size value. If a bayes datastore backend
   does not implement individual key/value expirations, the
   setting is silently ignored.

   bayes_token_ttl   (default: 3w, i.e. 3 weeks)

   Time-to-live / expiration time in seconds for tokens kept in
   a Bayes database.  A numeric value is optionally suffixed by
   a time unit (s, m, h, d, w, indicating seconds (default),
   minutes, hours, days, weeks).

   If bayes_auto_expire is true and a Bayes datastore backend
   supports it (currently only Redis), this setting controls
   deletion of expired tokens from a bayes database. The value
   is observed on a best-effort basis, exact timing promises 
are

   not necessarily kept. If a bayes datastore backend does not
   implement individual key/value expirations, the setting is
   silently ignored.

This really sounds as if expiry is a no-op for backends other than
Redis.  And yet Debian bug #334829 [1] exists, and has spawned a whole
subculture of solutions and work-arounds.  (Sorry for the slight
exaggeration.)  Clearly the users reporting these problems do not use
Redis, in fact by all signs they use the default DB backend, as I do.
So should I be worried about the expiry overhead and set up a separate
--force-expire job?  I am confused.
  [1] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=334829


The redis backend takes advantage of an auto-expiry mechanism of
key/value pairs as provided by a redis server internally (transparently
and automatically), so with this backend the bayes_token_ttl is the
only setting that matters, and SpamAssassin (auto)expiration runs
are not needed, if fact they are a no-op and should not be used.

With other bayes back-ends the traditional expiration mechanisms
need to be used, either auto-expiration runs triggered from time
to time by SpamAssassin, or explicit expiration runs, e.g. from
a cron job. With these traditional back-ends the bayes_token_ttl
setting has no effect.


and has spawned a whole subculture of solutions and work-arounds


Indeed. These mostly pre-date the availability of a Redis back-end.

  Mark

RE: PerMsgStatus Util warnings

[Mark Martinec]

Jim Barber wrote:


From: Mark Martinec [mailto:mark.martinec...@ijs.si]
Are you using some third-party SpamAssasin plugin that relies on the
deprecated subroutine Mail::SpamAssassin::Util::uri_to_domain ?


I'm getting the same error:

May 15 12:34:41 smtp-syd mimedefang-multiplexor[30108]:
t4F2YYjZ003229: Slave 6 stderr: plugin: eval failed: Undefined
subroutine Mail::SpamAssassin::Util::RegistrarBoundaries::trim_domain
called at /usr/share/perl5/Mail/SpamAssassin/Util.pm line 1236.

I'm using The SpamAssassin Debian package (which is version 3.4.1-1).

I do have a third party plugin that is calling uri_to_domain.
It is located at the following URL:

https://github.com/smfreegard/DecodeShortURLs

And is linked to from the SpamAssassin Custom Plugins wiki page:

https://wiki.apache.org/spamassassin/CustomPlugins

Here is the offending line:

	jimb@smtp-syd:~$ grep uri_to_domain 
/etc/spamassassin/DecodeShortURLs.pm

my ($dom, $host) = Mail::SpamAssassin::Util::uri_to_domain($_);

I'm mentioning this because the bug that was raised for this has the
following comment:

KAM: Please let us know if you are using a 3rd party plugin because
  we'd like to make sure the authors noticed the change in 3.4.1.
https://bz.apache.org/SpamAssassin/show_bug.cgi?id=7195



Thanks, useful. If you have a chance please try the small patch there
(just adds the:  use Mail::SpamAssassin::Util::RegistrarBoundaries;
line in Util.pm) and see if it helps.

  Mark

Re: PerMsgStatus Util warnings

[Mark Martinec]

Alex Regan wrote:


I have v3.4.1 with amavisd v2.9.1 on fedora20 and receiving the
following warnings:

May 13 23:32:31 mail01 amavis[17306]: (17306-10) _WARN: plugin: eval
failed: Undefined subroutine
Mail::SpamAssassin::Util::RegistrarBoundaries::trim_domain called at
/usr/share/perl5/vendor_perl/Mail/SpamAssassin/Util.pm line 1236.

May 13 23:35:43 mail01 amavis[17386]: (17386-12) _WARN: Use of
uninitialized value in concatenation (.) or string at
/usr/share/perl5/vendor_perl/Mail/SpamAssassin/PerMsgStatus.pm line
3082.

I thought I remembered this issue being discussed some time ago, but
couldn't find any recent references to it. It looks like it happens
just in the course of processing messages with no other information
around these lines. I also don't know when it first started. Thought
someone might have some ideas?


Are you using some third-party SpamAssasin plugin that relies on the
deprecated subroutine Mail::SpamAssassin::Util::uri_to_domain ?

Please try the patch below:

===
--- Mail/SpamAssassin/Util.pm~  2015-04-28 21:56:49.0 +0200
+++ Mail/SpamAssassin/Util.pm   2015-05-14 16:26:23.198104251 +0200
@@ -49,4 +49,5 @@

 use Mail::SpamAssassin::Logger;
+use Mail::SpamAssassin::Util::RegistrarBoundaries;  # deprecated

 BEGIN {
===


Mark

Re: PerMsgStatus Util warnings

[Mark Martinec]

Alex Regan wrote:


Please open a bug report (one should do, even though these are two

independent issues).


Bug filed, thanks so much.

https://bz.apache.org/SpamAssassin/show_bug.cgi?id=7195



Great, thanks. Please also try the posted patch and see if that
resolves the first type of a warning (the 'Undefined subroutine').

  Mark

Re: PerMsgStatus Util warnings

[Mark Martinec]

I have v3.4.1 with amavisd v2.9.1 on fedora20 and receiving the
following warnings:

May 13 23:32:31 mail01 amavis[17306]: (17306-10) _WARN: plugin: eval
failed: Undefined subroutine
Mail::SpamAssassin::Util::RegistrarBoundaries::trim_domain called at
/usr/share/perl5/vendor_perl/Mail/SpamAssassin/Util.pm line 1236.

May 13 23:35:43 mail01 amavis[17386]: (17386-12) _WARN: Use of
uninitialized value in concatenation (.) or string at
/usr/share/perl5/vendor_perl/Mail/SpamAssassin/PerMsgStatus.pm line
3082.


The second warning is independent from the first, I'm seeing these too.

It's because the RegistryBoundaries::uri_to_domain returns undef
when it sees %hh encoding in an URL, and the caller then tries to
concatenate it with 'dummy@', which produces this warning.
Seems harmless but annoying, should be fixed.

Please open a bug report (one should do, even though these are two
independent issues).

  Mark

Re: interesting spammer trick (bayes)

[Mark Martinec]

On 2015-05-03 10:55, Reindl Harald wrote:

recently i observed by playing around with bayes-training that some junk
(maybe unintentional) is using the mimetype 'application/octet-stream'
instead 'text/html' containing the payload of a form with javascript
prevets the attachment from tokenizing


the new feature in 3.4.1 will take care of that while i am not sure how
much impact in classifying a trained attachment at the end has

SHA1 digests of all MIME parts (including non-textual) can now be
contributed to Bayes tokens, which allows the bayes classifier to assess
also the non-textual content. The set of sources of bayes tokens is
configurable with a new configuration option 'bayes_token_sources'
as documented in the Mail::SpamAssassin::Conf man page. (Bug 7115)
It is disabled by default for backward compatibility.


i am not sure here in context of backward compatibility


Just a cautionary speech. There were some concerns whether
it is beneficial or not to contribute digests of non-textual
parts or not, and not much experience has been gained yet,
so to avoid any potential surprise the default is the same
as with 3.4.0, i.e. digests are not included.

In my experience it can be valuable to include these, and
I haven't seen any ill effect while observing top-10
bayes tokens containing digests, as logged by a debug log,
for several weeks.


correct me but IMHO bayes_token_sources all should not have a side
effect when you train a bayes on SA 3.4.1 and share it with a setup
using 3.4.0 - the 3.4.0 setup just should not benefit from the new
mimeparts-tokens in the database but still from all others?


That is correct, learned digest tokens as inserted by 3.4.1 are
ignored by 3.4.0 code.

Btw, note that spamd does not process messages larger than some
pre-set size limit. Even if truncated messages are passed to
spamd, it would not see MIME parts beyond the truncation limit.
This is unlike what the current (to-be-released) version of
amavisd does: regardless of mail size amavisd would compute
digests of *all* pristine mail parts, and pass them to SpamAssassin
out-of-band, already ready-to-use, even if a message is truncated.
This also avoids some pre-processing 'corruption' of MIME digests
when computed by SpamAssassin, as a 'pristine' mail as understood
by SpamAssassin is sometimes a little less 'pristine' than ideal,
e.g. due to squashing long runs of empty lines in a message,
and splitting long paragraphs into chunks.

With MIME digests it's the same approach as with DKIM signatures,
which are also pre-computed by amavisd on the complete (non-truncated)
pristine message, and passed to SpamAssassin for use in the DKIM
plugin.

  Mark

Re: need_tags

[Mark Martinec]

On 5/3/2015 2:40 AM, Robert Schetterer wrote:

Hi , i try to understand
this

http://search.cpan.org/dist/Mail-SpamAssassin/lib/Mail/SpamAssassin.pm

need_tags
...
Currently the only tag that needs to be explicitly requested is 'TIMING'
...

the goal would be to have time stats of every rule matching in logs,
without
advanced debug levels on production servers for better analysis


On 2015-05-04 15:42, Kevin A. McGrail wrote:

I believe this setting is solely for profiling of spamassassin.  See
https://bz.apache.org/SpamAssassin/show_bug.cgi?id=5356 but I doubt it's
of much use to the general public.


The requirement for timing can be requested by a application using
the SpamAssassin library. Currently amavisd does turn it on and
the SpamAssassin timing report is included in the amavisd log,
but the spamd does not include the timing report in its log.

  Mark

Re: need_tags

[Mark Martinec]

The requirement for timing can be requested by a application using
the SpamAssassin library. Currently amavisd does turn it on and
the SpamAssassin timing report is included in the amavisd log,
but the spamd does not include the timing report in its log.


On 2015-05-05 1:20, Kevin A. McGrail wrote:

Neat.  Can you cut and paste an example log and perhaps we can mimic it
in spamd easily?


Here's and example
(I hope my mailer won't wrap it unnecessarily):

May  5 01:34:51 dorothy amavis[8286]: (08286-06) TIMING-SA [total 2178 
ms, cpu 2
89 ms] - parse: 0.87 (0.0%), extract_message_metadata: 13 (0.6%), 
get_uri_detail
_list: 0.65 (0.0%), tests_pri_-1000: 29 (1.3%), tests_pri_-950: 1.56 
(0.1%), tes
ts_pri_-900: 1.40 (0.1%), tests_pri_0: 1251 (57.5%), check_spf: 0.13 
(0.0%), che
ck_dkim_adsp: 5 (0.2%), check_bayes: 9 (0.4%), b_tokenize: 3.4 (0.2%), 
b_tok_get
_all: 1.93 (0.1%), b_comp_prob: 2.0 (0.1%), b_tok_touch_all: 0.62 
(0.0%), b_fini
sh: 0.13 (0.0%), check_razor2: 202 (9.3%), check_dcc: 838 (38.5%), 
check_pyzor:

162 (7.4%), tests_pri_500: 5 (0.3%), get_report: 0.58 (0.0%)

Most of it (elapsed times) is produced by a call to:

  $pms-get_tag('TIMING');

The CPU usage is obtained by calling Unix::Getrusage .

  Mark

Re: TxRep $msgscore warning

[Mark Martinec]

  my $sa_args = {
 local_tests_only   = $SALocalTestsOnly,
 dont_copy_prefs= 1,
 userprefs_filename = $config,
 user_dir   = $Features{'Path:QUARANTINEDIR'},
 debug  = 'TxRep'
 };


 amavisd.conf:
 $sa_debug = 1;
 Is there a way to tighten that to just show TxRep debug messages?

The:

$sa_debug = 'TxRep';

is the equivalent of the above  debug='TxRep'.
A comma-separated list of SpamAssassin debug facilities can
be provided and is passed to the 'debug' argument of SA.

  Mark

Re: SA 3.4.1 - error messages in log?

[Mark Martinec]

 On May 2, 2015 7:08:10 PM Mark Martinec wrote:
  May  2 06:45:29 sunshine spamd[22293]: Use of uninitialized value
  $hasStructureInfo in numeric eq (==) at (eval 46) line 5520.

 This one seems to come from a module Geo::IP, called form a
 SpamAssassin plugin URILocalBL.
 [...] Try disabling loading of a plugin URILocalBL as a start
 (in config file v341.pre).


On 2015-05-03 2:26, Art Greenberg wrote:

The line for URILocalBL is commented out - I had not enabled it.


There is also the RelayCountry plugin that is using the Geo::IP module.

  Mark

Re: dkim invalid and 3.4.1

[Mark Martinec]

1.0 T_DKIM_INVALID  DKIM-Signature header exists but is not valid



The score for this rule should be a zero or a near-zero.
There must be some problem with assigning a score to
such test rule (the 1.0 is a default value if a score line
is missing).


T_DKIM_INVALID is a test rule, as such its score should be 0.01
by default. Make sure the sa-update has provided an up-to-date
version of rules.

  Mark

Re: dkim invalid and 3.4.1

[Mark Martinec]

On 2015-05-03 5:34, Nick Edwards wrote:

Is there any reason
reason=invalid (public key: not available)  is declared as error
to fail t_dkim_invalid

1.0 T_DKIM_INVALID DKIM-Signature header exists but is not valid

This is published a neutral so should not be considered invalid

This only occurs since upgrade 3.4.0 - 3.4.1, no changed made by the
sender - its a govt dept, who doesnt change things at all let alone in
middle of weekend.

Be a shame to have to score that 0 simply because the code over reacts.


The score for this rule should be a zero or a near-zero.
There must be some problem with assigning a score to
such test rule (the 1.0 is a default value if a score line
is missing).

An invalid or unverifiable DKIM signature is supposed to be
treated equivalent to a missing signature.

  Mark

Re: SA 3.4.1 - error messages in log?

[Mark Martinec]

On 2015-05-02 14:39, Art Greenberg wrote:

I just moved from 3.4.0 to 3.4.1 and am seeing messages from SA in
maillog that I've not seen before. Do they indicate an issue with my
installation?

Here is a segment from the log. This pattern repeats each time SA
processes a message. The use of uninitialized value and copy_config
timeout are new, as is the change of child states.

May  2 06:45:29 sunshine spamd[22293]: Use of uninitialized value
$hasStructureInfo in numeric eq (==) at (eval 46) line 5520.


This one seems to come from a module Geo::IP, called form a
SpamAssassin plugin URILocalBL.


May  2 06:46:29 sunshine spamd[22290]: spamd: copy_config timeout,
respawning child process after 1 messages at /usr/local/bin/spamd line
1447.

[...]

May  2 06:46:30 sunshine spamd[21741]: spamd: handled cleanup of child
pid [22290] due to SIGCHLD: exit 0
May  2 06:46:40 sunshine spamd[22293]: spamd: copy_config timeout,
respawning child process after 1 messages at /usr/local/bin/spamd line
1447.

[...]

SA does seem to be processing messages correctly. But starting SA does
seem to take forever ... I've had to increase the timeout on line 2977
in spamd from 180 to 600 seconds. I'm running SA just for personal email
on a machine pretty much dedicated to handling email so performance is
not a concern. Centos 6.6 64 bit, Intel E7300 2.66GHz with 4GB RAM and
500GB disk.


No idea. Try disabling loading of a plugin URILocalBL as a start
(in config file v341.pre).

  Mark

Re: ANNOUNCE: Apache SpamAssassin 3.4.1 available (bug)

[Mark Martinec]

On 2015-05-01 20:34, Forrest wrote:

Upgrading from a simple 3.4.0 installation, 3.4.1 refuses to start, with
this error:

Starting spamd: child process [3723] exited or timed out without signaling
  production of a PID file:
  exit 255 at /usr/local/perl/bin/spamd line 2986.

I've seen this before, but I did check for any leftover PID files (none
exist).  I also rebooted our system, to no avail.Going to attempt
downgrading to see if that fixes the bug.


spamd --debug


  Mark

Re: spam score question

[Mark Martinec]

 On April 22, 2015 8:44:59 PM EDT, Thom Miller t...@cagroups.com
 wrote:
 On Sat, 18 Apr 2015 08:16:40 -0700
 Michael Williamson michael.h.william...@gmail.com wrote:
 It appears to me that spamassassin can produce different spam
 scores for the same email.
 In particular, I have noticed that points are omitted for
 RCVD_IN_SBL_CSS (Spamhaus blacklist) sometimes. Why?
 In the past I noticed that network tests were sometimes completely
 omitted. I believe sa checks for network connectivity before
 perfoming these tests, and incorrectly determines that there is no
 network available.

 In my case, adding:
   dns_available yes
 to my local.cf solved this issue.



2015-04-24 01:38, Thom Miller wrote:

Kevin A. McGrail kmcgr...@pccc.com wrote:


On 4/22/2015 11:19 PM, Thom Miller wrote:
 According to
 https://spamassassin.apache.org/full/3.2.x/doc/Mail_SpamAssassin_Conf.html :

 By default, SpamAssassin will query some default hosts on the
 internet to attempt to check if DNS is working or not. The problem
 is that it can introduce some delay if your network connection is
 down, and in some cases it can wrongly guess that DNS is
 unavailable because the test connections failed.

 I decided that since the network should always be available,
 there's no reason for spamassassin to test it.

Interesting.  What version of SA are you using?


I'm running 3.4.0 now, but I made this addition to local.cf when I was
running 3.2. I don't know if the change is still necessary in my case,
but I haven't bothered to remove it.

-Thom



The 'dns_available yes' is a default since 3.4.0.


3.4.0 release notes:

 * A default setting for option 'dns_available' was changed from 'test' 
to

 'yes' (bug 6770, bug 6769), so SpamAssassin now assumes by default that
 it is running on a host with an internet connection and a working DNS
 resolver. If this is not the case, please configure this option 
explicitly.


 The change avoids surprises on an otherwise well connected host which 
may
 experience a temporary DNS unavailability at the system startup time or 
a
 temporary network outage when spamd was starting, and the initial 
failed

 test would disable DNS queries permanently. The option is documented in
 the Mail::SpamAssassin::Conf POD or man page.



Mark

Re: FPs on RCVD_ILLEGAL_IP

[Mark Martinec]

Dianne Skoll wrote:

On Mon, 20 Apr 2015 17:02:09 -0700 (PDT)
John Hardin jhar...@impsec.org wrote:


I suggest that this rule should treat 0/8 as equivalent to 127/8.
That's essentially what it's reserved for, just local to the LAN
vs. local to the host.


Does 0/8 really mean that?  On at least one OS (Linux), the TCP stack
treats it specially:

$ telnet 0.1.2.3
Trying 0.1.2.3...
telnet: Unable to connect to remote host: Invalid argument

The EINVAL return is certainly not the same as trying a nonexistent
host:

$ telnet 10.11.12.13
Trying 10.11.12.13...
[hangs]

I don't think 0/8 was intended for real traffic.  I understood it to be
intended only for hosts trying to discover their real IP addresses.


The 0.0.0.0/8 is an 'unspecified' address. In principle a host is
free to use it for whatever purposes internally (or for network 
discovery),

but is not routable outside the host. In that sense it behaves the
same as 127.0.0.0/8 and I think for the purposes of seeing it in
a Received header field outside of the 'trusted' zone it should be
treated as any foreign private IP address space, i.e. it may be
valid for the host which inserted it, but has no value for us
the receiving party.

Btw, the same should apply to addresses ::/128 and ::1/128 .

  Mark

Re: FPs on RCVD_ILLEGAL_IP

[Mark Martinec]

In any case, I think that RCVD_ILLEGAL_IP should not be adding
score points if it sees an 0.0.0.0/8 address in a Received header field.

  Mark

Re: FPs on RCVD_ILLEGAL_IP

[Mark Martinec]

shanew wrote:

I presume detecting forged Received headers was the point of this rule
all along, so if we all toss this rule out the window (or adjust to
exclude this edge case), aren't we potentially encouraging spammers to
hide their true networks in the same way?


There is no benefit to spammers (and a likely disservice to them)
for forging a non-trustworthy external Received header field
and providing some unusual IP address there, and they cannot forge
the boundary Received header field inserted by recipient's own mailer.
I can only conclude that a rule like RCVD_ILLEGAL_IP will hit
mostly on misconfigured or misguided sending mailers, not primarily
on spam.

Reindl Harald wrote:

my problem with that rule is that it hits practically no spam
but only ham and so it goes in the wrong direction entirely


Most likely.


  Mark

Re: FPs on RCVD_ILLEGAL_IP

[Mark Martinec]

Dianne Skoll wrote:

Mark Martinec mark.martinec...@ijs.si wrote:

I can only conclude that a rule like RCVD_ILLEGAL_IP will hit
mostly on misconfigured or misguided sending mailers, not primarily
on spam.


I disagree.  Now that the Microsoft issue has been fixed, well over 95%
of the mail on our system that hits RCVD_ILLEGAL_IP is spam.


You are right, I checked our logs and the RCVD_ILLEGAL_IP does
indeed mostly hit on spam.


... although there's a funny twist there. Some of these illegal
IP addresses are not really a claimed-to-be IP address of a mailer,
but come from an embedded e-mail address in a comment:

Received: from unknown (HELO localhost)
  (jennifer_pr...@sbcglobal.net@236.192.200.84)
  by mm-36-150-122-178.brest.dynamic.pppoe.byfly.by with ESMTPA;
  Tue, 21 Apr 2015 23:55:53 +0300

Received: from unknown (HELO localhost)
  (bsobolew...@stockton-house.com@236.139.213.194)
  by 76.172.150.91 with ESMTPA; Tue, 21 Apr 2015 11:41:10 -0800

so by a lucky coincidence a misparsed Received ends up
yielding a useful-but-wrong result.

  Mark

Re: FPs on RCVD_ILLEGAL_IP

[Mark Martinec]

John Hardin wrote:

I suggest that this rule should treat 0/8 as equivalent to 127/8.
That's essentially what it's reserved for, just local to the LAN vs.
local to the host.


I fully agree.

  Mark

Re: Fwd: Fwd: Re: effectiveness of DCC checks?

[Mark Martinec]

Kevin A. McGrail wrote:

Vernon, do you have a  recommended score for the implementation of
DCC with SA? There are concerns that bulk mail from good senders has
been hit by DCC which is completely by design.


Vernon replied off-list so I wanted to bring the relevant portion back
to the list:

My general suggestion is that SA+DCC clients that don't use whitelists
for bulk mail (I still say whitelists are required) should use a DCC
score that is not large enough to tell SA to mark a message spam
purely from the DCC result, but large enough to push other signs of 
spam

over the threshold. vjs


With which the current 1.1 score points complies just fine.

The whitelisting for bulk mailers in the DCC config file is 
advised/required

by the DCC docs anyway.


So I think DCC users should consider moving DCC to an __ test and
identify things like DNSBLs with whitelist entries and other tests that
can use DCC to meta more effective classification rules.
Anyone has any thoughts on rules to submit, let me know!


I don't agree with moving a DCC rule into a __* rule or setting
its score to a near zero. I find DCC hits useful as they are now:
contributing to the overall score, bit not so large as to make
a major effect by themselves.

  Mark

Re: effectiveness of DCC checks?

[Mark Martinec]

John Hardin wrote:

[...] The 1.1 points is hardcoded:
  50_scores.cf:score DCC_CHECK0  1.1   0  1.1
It's reasonable to argue that this score should be informational only,
and that it should only be scored meaningfully in metas.


Steve Freegard wrote:

However - I'll readily agree with you that DCC_CHECK adding score
to all bulk mail isn't that useful


I find it quite reasonable (and useful) to add 1.1 score points to
all bulk mail. There's still plenty of headroom left towards 5 points
for well-intended contents.

  Mark

Re: Spamassassin not catching spam (Follow-up)

[Mark Martinec]

well, a better setup would run spamassassin via milter *before-queue*
and proper reject junk at SMTP level - so you have a tag level let say
between 5.5 and 7.9 points and reject above 8.0

the flagged ones can go in a seperate folder via sieve and the 
absolute
high score junk is proper rejected and with some luck the spam 
attempts

go down at all
http://www.postfix.org/MILTER_README.html


If you're already using amavisd with postfix and postscreen, would
there be any benefit to considering a milter in this way?

Spamassassin already verifies authenticity, although not pre-queue,
but I'm not sure that's enough for me to introduce another set of
applications to manage...


The usual amavis setup with postfix nowadays is a before-queue setup:

smtpd  pass  -  -  n  -  150  smtpd
-o smtpd_proxy_filter=inet:[::1]:10024
-o smtpd_proxy_options=speed_adjust

A milter setup has no advantage over a before-queue proxy setup,
and has some disadvantages (e.g. unable to individually adjust
mail header section in multi-recipient mail messages).

Don't use a content filter in an after queue setup, unless you
give up mail rejections. Non-delivery notifications due to spam/malware
*must* be avoided, discarding mail without letting sender or
recipient be aware about it is not acceptable (and may not be legal),
and delivering tagged junk (or delivering to a dedicated recipient's
folder) may not be appreciated by recipients.

  Mark

Re: Recommendations for ASF SA Implementation

[Mark Martinec]

2015-03-17 22:16, Kevin A. McGrail wrote:


I am working on recommendations for the ASF to modernize the
installation of SA for the foundation.

We have some givens:

Using Ubuntu
Using Postfix
Need to stick with maintainable packages
Likely needs to stay away from lots of tweaks and heavy customization
such as using MIMEDefang (unfortunate).

So I'd like any input you might have, on or off list.



Axb wrote:
| Although I'd suggest Fuglu, the obvious choice should probably be 
amavisd-new

| considering Mark is also highly involved in SA dev work.
| It's also distributed by Ubuntu so it would be one package less to 
maintain

| outside the distro. We'd get the best of both worlds.
| Axb

Thanks, Amavis would be my choice too :)))


back to Kevin:

Here's some questions I believe will help guide things:

Q1 - What is the best glue for SA for Postfix that does the following:

- uses spamc calls so that spamd's can be distributed and load 
balanced?


Amavis uses a standard protocol SMTP for communication with an MTA
instead of the proprietary spamc/spamd protocol. Other than that,
interfacing to the SpamAssassin is pretty much the same as in spamd,
i.e. uses pre-forked set of processes which use the SpamAssassin 
library.

For this reason the performance is pretty much the same - the bottleneck
is processing rules in the SpamAssassin.


can be distributed and load balanced?


Yes, can be distributed and load balanced. Two approaches are most
apparent:
- the classical approach is to run multiple postfix+amavis
combos on several hosts, and let MX dns record distribute the load
across them. If a single IP address is desired, an SMTP proxy (such
as nginx) can do the task of load sharing in front of Postfix.
- if a single MTA is preferred with multiple content filters on
multiple hosts, then traffic from Postfix to amavisd instances
can be spread using HAProxy (or some other load balancer).

Note that it is beneficial to feed outgoing mail through amavis too
for the following reasons:
- the PenPals feature keeps track of ongoing conversations and
contributes negative score points to such, preventing some false
positives on marginal mail content (a requirement is a common
database for all amavis instances, preferably redis, possibly SQL);
- when SpamAssassin autolearning is enabled, outgoing mail
contributes its valuable share of ham samples;
- when an internal machine or a user mail account gets compromised
and starts spewing malware or spam, it will get blocked and detected.
- not to forget: to DKIM-sign outbound mail it needs to pass
through a signer. Amavisd can do DKIM signing (and verification).



- can implement clamav before SA call


Yes.

Also, considering that some of the third-party ClamAV rulesets
are prone to false positives, or intentionally target spam (not
viruses and other malware), amavis can be configured to reclassify
certain malware (by name) as spam, contributing to SpamAssassin score
and not blocking as malware right away.



- should silently discard emails if a virus is detected


Configurable, but you don't want to do that, and (as Reindl Harald
noted) may even be violating law. Unwanted mail must be rejected
at an SMTP level (or delivered to a dedicated folder or quarantine),
it must not be lost. Amavis is nowadays typically deployed as a
before-queue Postfix content filter so that it can reject mail
while the original session is still open.

Keep in mind that antivirus software does occasionally produce
false positives, ClamAV with third party rules even more so.
A legitimated sender must be notified is this happens.



- must use clamdscan but ideally can utilize some sort of socket
solution for clamd to run distributed and load balanced


Can do.  Amavisd cam interface with clamd either through
clamdscan, or (preferably) by directly talking to it over
the clamd protocol (thus eliminating clamdscan from the setup).
As this is a normal TCP connection, it can be load balanced
using HAProxy, although it probably makes more sense to keep
amavis+clamd pairs on each host.



- should bound email over a certain threshold (let's say 5) and
silently discard email over a certain threshold for SA (let's say 10)


Possible. There are a couple of configurable spam score levels,
each with its configurable action:

  tag level  - adds X-Spam-* headers (ham or spam)
  tag2 level - adds X-Spam-* headers, claims it is spam
  tag3 level - adds X-Spam-* headers, claims it is blatant spam
  kill level - (typically) rejects mail (or can discard or deliver)

Quarantining at each spam level is configurable independently.



- Might use a few RBLs to decline connections to start


Yes. That belongs to Postfix.



- Implements a good implementation of greylisting


That belongs to Postfix.
I tend to shy away from greylisting, it is much less effective
as it used to be initially. In my opinion it does more harm than good.



- Temporary failure for scanning (virus or spam) failures


Yes. Any 

Re: SA 3.3, Debian and *BL...

[Mark Martinec]

dns_available yes


Marco Gaiarin wrote:

Ok, i have:
dns_available test: 172.0.0.1


That's wrong. The documentation tells that you need to specify
a domain name there, and that SpamAssassin will test for a NS
record of that domain name:

| dns_available { yes | no | test[: domain1 domain2...] }
| [...]
| When the option value is a test (with or without arguments),
| SpamAssassin will query some domain names
| [...]
| The test queries for NS records of these domain

Does your *domain* name '172.0.0.1' have a NS record?
I bet it doesn't.

Just set it to:
  dns_available yes
as Axb suggested.

  Mark

Re: whitelist_from_rcvd not working, WAIDW

[Mark Martinec]

http://pastebin.com/EV6g15aN

I have this in user_prefs:
  trusted_networks 198.1.2.3/32
  [...lots snipped...]
  whitelist_from_rcvd *@wetransfer.com *.wetransfer.com

Why is the whitelist not firing?



It seems the:

Received: (from itz@localhost)
by myalias.trusted.mx (8.14.4/8.14.4/Submit) id t1N7YK8O020727
for i...@my.post.office; Sun, 22 Feb 2015 23:34:20 -0800

is breaking a trust chain.

  Mark

Re: Bayes expiration with Redis backend

[Mark Martinec]

Matteo Dessalvi wrote:

I am using a centralized Redis instance to
host the bayesian data for a bunch of MTAs.

AFAICS the SA filter is working quite well
and the BAYES_* rules are triggered correctly,
no false positive so far.

But I am concerned about the expiration of the
bayesian data. sa-learn reports the following:

0.000  0  3  0  non-token data: bayes db 
version

0.000  0   8437  0  non-token data: nspam
0.000  0 495000  0  non-token data: nham

As stated here:

search.cpan.org/dist/Mail-SpamAssassin/lib/Mail/SpamAssassin/BayesStore/Redis.pm

Expiry is done internally in Redis using *_ttl settings (...)
This is why --force-expire etc does nothing, and token counts
and atime values are shown as zero in statistics.

So, why the nham tokens have grown so much? It looks
like it was never 'pruned'.


When redis automatically expires tokens internally based on their
TTL, this operation does not affect nspam and nham counts. These
counts just grow all the time (as there is no explicit expiration
that SpamAssassin would know about), reflecting the count of
(auto)learning operations.

Don't worry about large nspam and/or nham counts when redis is
in use, all that matters is that these counts are above 200
(otherwise bayes is disabled).

You may get the number of tokens that are actually in the redis
database (not expired) by counting the number of lines produced
on stdout by 'sa-learn --backup' or 'sa-learn --dump data'.

The format of fields produced by --dump data is:

  probability  spam_count  ham_count  atime  token

The --backup format is similar, but does not provide
probabilities, just spam and ham counts.

To get some estimate on the number of hammy vs. spammy tokens
(not messages) currently in a database, try something like:

  sa-learn --dump data' | \
awk '$10.1 {h++}; $10.9 {s++}; END{printf(h=%d, s=%d\n,h,s)}'


(caveat: sa-learn --backup or --dump data may not work on a huge
database, as they need all the tokens (redis keys) to fit into memory)


  Mark

Re: Lots of Polish spam

[Mark Martinec]

Axb wrote:

didn't need a password to extract but... whatever format those .eml
are in, none of text editors was able to handle them so that  didn't
help.


$ mkdir Spam; cd Spam
$ 7z e -pspam ../Spam.zip


Sophos reports it as Troj/Tinba-O, like most others on virustotal.com
ClamAV does not detect anything suspicious.

  Mark

Re: DKIM dependency issues

[Mark Martinec]

Alex Regan wrote:


Feb 15 18:44:41.383 [16434] dbg: spf: [...] Compilation failed in
require at 
/usr/share/perl5/vendor_perl/Mail/SpamAssassin/Plugin/SPF.pm

line 500.


Looks to me like the same issue (but a different symptom) as reported
my mls mid January 2015 on the SA users mailing list:
Mail::SpamAssassin::Plugin::Check::_head_tests_0_1, skipping ...

I can now reproduce it here now (only with SPF, not with DKIM
or in compiling header tests).


configuration issue, but it only happens on mbox files.
The ones I've tested have only one message.


The only happens on mbox files was a key, thanks!
It does not seem to be a misconfiguration, more like a genuine bug
somewhere in the @INC or 'require' logic.


Awesome. it really looked like a configuration problem, but there
wasn't nothing in my mail logs along these lines - it was only when
run from the command-line.

I was also pretty sure that mail-spf-query has long been deprecated in
favor of mail-spf so I didn't want to go that route, and I know my
config has been working well on multiple machines for quite a while.

Is there now a bug report submitted for this?


Not yet. I'm trying to boil it down to a manageable test case.
So far it seems it will end up as a perl bug report.

Please open a SA problem report so that the issue can be tracked.
There may be a workaround, but first the problem must be understood.

  Mark

Re: DKIM dependency issues

[Mark Martinec]

On 16. feb. 2015 00.59.42 Alex Regan mysqlstud...@gmail.com wrote:

I've done a little more testing, and it certainly sounds like a local
configuration issue, but it only happens on mbox files. The ones I've
tested have only one message.

[...]
Tests on a few other mbox messages have produced similar errors for 
SPF

as well as DKIM.



Feb 15 18:44:41.383 [16434] dbg: spf: cannot load Mail::SPF module or
create Mail::SPF::Server object: Insecure dependency in require while
running with -T switch at
/usr/share/perl5/vendor_perl/Mail/SPF/Server.pm line 28.
Feb 15 18:44:41.383 [16434] dbg: spf: [...] BEGIN failed--compilation
aborted at /usr/share/perl5/vendor_perl/Mail/SPF/Server.pm line 28.
Feb 15 18:44:41.383 [16434] dbg: spf: [...] Compilation failed in
require at /usr/share/perl5/vendor_perl/Mail/SPF.pm line 29.
Feb 15 18:44:41.383 [16434] dbg: spf: [...] BEGIN failed--compilation
aborted at /usr/share/perl5/vendor_perl/Mail/SPF.pm line 29.
Feb 15 18:44:41.383 [16434] dbg: spf: [...] Compilation failed in
require at 
/usr/share/perl5/vendor_perl/Mail/SpamAssassin/Plugin/SPF.pm

line 500.


Looks to me like the same issue (but a different symptom) as reported
my mls mid January 2015 on the SA users mailing list:
Mail::SpamAssassin::Plugin::Check::_head_tests_0_1, skipping ...

I can now reproduce it here now (only with SPF, not with DKIM
or in compiling header tests).


configuration issue, but it only happens on mbox files.
The ones I've tested have only one message.


The only happens on mbox files was a key, thanks!
It does not seem to be a misconfiguration, more like a genuine bug
somewhere in the @INC or 'require' logic.

  Mark

Re: sa-update failing: SHA1 verification failed

[Mark Martinec]

Tim Taylor wrote:
For the last few days (from 31 Jan 2015), sa-update has been failing on 
my
server. I've checked and can't see that any package updates happened 
around
then, so I'm puzzled by why this is happening (it's been working fine 
for

over a year beforehand).

I'm running sa-update version svn1475932 (with Perl version 5.18.2) on 
a

64-bit Ubuntu 14.04 box.

Running sa-update -D shows the following output:

[...]
So the update is not getting the expected sha1 verification. There are 
4

mirrors in MIRRORED.BY, and I've tested it with each one selected
individually and the others removed, but I get the same result for each
mirror. I also reinstalled Digest::SHA1 on CPAN, but still no joy.

Has anyone else noticed this problem, or is it just me? Any ideas how I 
can

fix this? Is it a problem with my installation or with the mirrors?


Perhaps you have an existing (old or corrupted) file 1655961.tar.gz
with a timestamp younger than the timestamp of this file on the
web server, and the -z option in curl is preventing a new file to
be downloaded.

  Mark

Re: Mail::SpamAssassin::Plugin::Check::_head_tests_0_1, skipping: Jan 17 08:00:12.091 [2891] warn: (Insecure dependency in require while running with -T switch at (eval 1037) line 9

[Mark Martinec]

mls wrote:

The Rule2XSBody plugin is loaded on ubuntu in sa-compile.pre.
After commenting it out the issue no longer happened.


Great, that is an important finding!


Benny Pedersen wrote:

warn dont use cpan direct in ubuntu, you will break dependice, and
later ask why does it not work

if you like to use cpan, then remove perl complete first then install
perl with make, make install, then when perl is installed in
/usr/local tree then begin installing all needed perl modules you need
with a cpan shell as documented

dont mix repo managers, if ubuntu miss something report to lunchpad


An alternative is to try it under perlbrew

  http://perlbrew.pl/

  http://search.cpan.org/~gugod/App-perlbrew-0.72/bin/perlbrew

It installs any chosen version of perl locally in your home directory,
not touching anything on the system. You need to install necessary
SpamAssassin dependencies inside the perlbrew context as well (e.g.
using the cpan installation command, which will automatically track
the necessary dependencies). All the 'installed' modules will be
located inside the perlbrew subdirectory.

  Mark

Re: Problem installing Spammassassin 3.4.0

[Mark Martinec]

Carlo Filippetto wrote:

I have a CentOS release 5.5 (Final)
SpamAssassin version 3.3.1
  running on Perl version 5.8.8

I would like to upgrade it to 3.4.0 so I upgrade cpan module and 
install

all the module on INSTALL document in the default root of the packages.

When I tried to install it with
perl Makefile.PL PREFIX=/opt/MailSpamassassin-3.4.0
CONFDIR=/etc/mail/spamassassin

I have this error:

optional module out of date: IO::Socket::IP
optional binary missing or nonfunctional: fetch

warning: some functionality may not be available,
please read the above report before continuing!


Just warnings. If you don't require a functional IPv6,
then you don't really need the IO::Socket::IP .


After some search on google I tried to install with cpan several 
module,

tried to install e remove spamassassin

but nothing

I tried to forse install IO::Socket::IP and now I have:
cpan install
IO::Socket::IP

CPAN: Storable loaded ok
Going to read /root/.cpan/Metadata
  Database was generated on Tue, 20 Jan 2015 02:29:02 GMT
IO::Socket::IP is up to date.

If I made a test I have:
Warning: prerequisite Socket 1.97 not found. We have 1.78.


Your perl is very old. Instead of investing potentially great effort
to bring the Socket module up to date too, it's probably easier
just to deinstall the new IO::Socket::IP .

  Mark

Re: Mail::SpamAssassin::Plugin::Check::_head_tests_0_1, skipping: Jan 17 08:00:12.091 [2891] warn: (Insecure dependency in require while running with -T switch at (eval 1037) line 9

[Mark Martinec]

mls wrote:

Search for 'rules: INC:' in the resulting debug output.
Is any of the reported elements reported as tainted (with an 'Y') ?



I don't see tainted elements. But please check here:
http://paste2.org/HzINJUwL


Thanks. Strange...


Does the problem go away if you disable the Rule2XSBody plugin,
i.e. by commenting-out a line:
   loadplugin Mail::SpamAssassin::Plugin::Rule2XSBody
in one of the .pre config files (probably in v320.pre)  ?


My mailserver runs on ubuntu 14.04 and Rule2XSBody is commented out by 
default

in /etc/spamassassin/v320.pre


If not in v320.pre then it must be in some other .pre file.
As Benny Pedersen noted your log shows that this plugin is enabled.
Perhaps in /etc/spamassassin/sa-compile.pre .

  Mark

Re: Mail::SpamAssassin::Plugin::Check::_head_tests_0_1, skipping: Jan 17 08:00:12.091 [2891] warn: (Insecure dependency in require while running with -T switch at (eval 1037) line 9

[Mark Martinec]

Jan 20 08:16:11.048 [18848] dbg: bayes: found bayes db version 3
Jan 20 08:16:11.049 [18848] warn: plugin: eval failed: Insecure 
dependency in sprintf
  while running with -T switch at 
/usr/share/perl5/Mail/SpamAssassin/Logger.pm line 241.

Jan 20 08:16:11.050 [18848] dbg: config: score set 1 chosen.

[...]
Jan 20 08:16:11.068 [18848] dbg: plugin: 
Mail::SpamAssassin::Plugin::DNSEval=HASH(0xae42840)

   implements 'check_start', priority 0
Jan 20 08:16:11.081 [18848] warn: plugin: eval failed: Insecure 
dependency in sprintf
  while running with -T switch at 
/usr/share/perl5/Mail/SpamAssassin/Logger.pm line 241.


Btw, the fix for your first two cases of 'Insecure dependency in 
sprintf'

for SpamAssassin 3.4.0 is in:

  https://issues.apache.org/SpamAssassin/show_bug.cgi?id=7065#c1

It is unrelated to the 'Insecure dependency in require' further down,
which is the topic of this thread.


The RedHat bug report as pointed out by Kevin A. McGrail is intriguing,
as I don't see how it was resolved (just says 'upgraded').

It may be worth trying with the current trunk version from SVN
(a would-be-3.4.1), as Kevin suggested, so that we'd be on the same 
page.



http://wiki.apache.org/spamassassin/DevelopmentStuff :

  The code itself is maintained in a Subversion repository
$ svn checkout https://svn.apache.org/repos/asf/spamassassin/trunk

followed by the usual CPAN install:
  perl Makefile.PL; make; make test; make install
  sa-update


Mark

Re: Mail::SpamAssassin::Plugin::Check::_head_tests_0_1, skipping: Jan 17 08:00:12.091 [2891] warn: (Insecure dependency in require while running with -T switch at (eval 1037) line 9

[Mark Martinec]

Please find the output here:
http://paste2.org/KNtnHHbv


Thanks. Seems like the array @INC contains a tainted element,
don't know why.

Try inserting the following two lines:

  use Scalar::Util qw(tainted);
  dbg(rules: INC: %s, join(, , map($_: .(tainted($_)?'Y':'n'), 
@INC)));


right before the debug line which you commented out previously.
(that debug line is no longer needed, it may be commented-out again).

Search for 'rules: INC:' in the resulting debug output.
Is any of the reported elements reported as tainted (with an 'Y') ?


Does the problem go away if you disable the Rule2XSBody plugin,
i.e. by commenting-out a line:
  loadplugin Mail::SpamAssassin::Plugin::Rule2XSBody
in one of the .pre config files (probably in v320.pre)  ?

  Mark

Re: Mail::SpamAssassin::Plugin::Check::_head_tests_0_1, skipping: Jan 17 08:00:12.091 [2891] warn: (Insecure dependency in require while running with -T switch at (eval 1037) line 9

[Mark Martinec]

Klaus wrote:

Thanks. I am running SA version 3.4.0.



Jan 17 08:00:12.091 [2891] warn: rules: failed to compile
  Mail::SpamAssassin::Plugin::Check::_head_tests_0_1, skipping:
Jan 17 08:00:12.091 [2891] warn:  (Insecure dependency in require
  while running with -T switch at (eval 1037) line 9.


In Mail/SpamAssassin/Plugin/Check.pm near line 453 (version 3.4.0)
(i.e. in sub flush_evalstr) there is a commented-out debug line:

  # dbg(rules: eval code(2): %s, $self-{evalstr});

Try uncommenting it and enable debugging (option '-D rules'
in spamd or in a command-line spamassassin, or '-d rules'
in amavisd). Let's see what was the code that the eval()
was trying to compile.

  Mark

Re: possible bug in Mail::DKIM when keysize is under 1024 bits

[Mark Martinec]

On Jan 11, 2015, at 3:40 PM, Kevin A. McGrail kmcgr...@pccc.com 
wrote:
I disagree as well. You can't cherry pick your quotes and you are 
missing
the long-lived caveat as well as the next sentence: Verifiers MUST be 
able

to validate signatures with keys ranging from 512 bits to 2048 bits

If it is 512 to 2048, I think the rfc is clear for recipients.


Gmail and a few others have decided to behave like if there was no
DKIM signature if the key 1024. Because today nearly anyone can crack
a 512bits DKIM key and just for a few dollars.


The only value of a DKIM signature is for a signer to prove
that some message originated from his domain. If some ignorant but
important signer choses a short signing key, it is risking that
recipients will ignore the signature and treat mail as unsigned,
and it is risking that someone could fake their mail. It's their
choice, and the risk is theirs too. By now, practically no important
sending domain is using keys shorter than 1024 bits.

Such mail signed with a short key may still carry a valid DKIM
signature. It would be inappropriate *not* to trigger a DKIM_VALID
rule, its score is almost zero. An entirely different question is
whether a recipient wants to trust such short key for whitelisting
or not.

The default DKIM-based whitelist as it comes with SpamAssassin
distribution (or with rule updates) does not contain any domain
that signs their mail with a short key (last time I checked).
So the original concern with this thread is weak.

If a local site decides to whitelist additional domain based
on their short DKIM signing key, its entirely their decision.
If whitelisting is really needed, it's probably still better
to whitelist based on a weak DKIM signature, then whitelist
based on some other weaker information (like a From address,
SPF with a too-wide set of allowed addresses, or guessing on
an IP address in a Received header field).

In my opinion it would be wrong to prevent a local site
administrator from doing that. The RFC only says the key must
be at least 1024 bits long. but does not tell a recipient
how to treat a message which breaks that requirement.

Franck Martin wrote:

spamassassin could add positive points if the key 1024


A message with a short but valid DKIM key is no worse than an
unsigned message. It would be wrong to assign score to short keys.

What may be useful is to update the DKIM plugin so that it disregards
signatures with a short key for whitelisting purposes. It may also
be useful to add additional rules (score zero) to indicate ranges
of a key size, which could then be used in meta rules when desired.

  Mark

Re: possible bug in Mail::DKIM when keysize is under 1024 bits

[Mark Martinec]

On January 12, 2015 8:06:00 AM EST, Mark Martinec

It would be wrong to assign score to short keys.


Kevin A. McGrail wrote:

Actually the rfc specifies that keys 512 to 2048 bits must be verified
so I think there is a grey area and there is this long-lived key
caveat as well.



I think if we can make a rule that fires on 1024 bits it's would be
good.


Fine with me.


The score may not be much but it could be helpful.


A message with a valid signature but a short DKIM key cannot be
scored more severely than an unsigned message, or a message with
an invalid signature - none these are currently assigned
any score.

  Mark

Re: spamassassin bayes rules

[Mark Martinec]

Filip Havlíček wrote:


Anybody can help with this? I still cannot find some helpful
information, thanks.



Dne 10.12.2014 v 14:52 Christian Grunfeld napsal(a):
when you run bayes in SQL and does sa-learn --username it will not try 
to setuid to that user (in a real system user scenario it will fail 
for non existent users). Instead it uses that username to save and 
recall data from database. Due to forged addresses your system treat 
any originating address as yours and then try to interact with the DB.


**-u* /username/, *--username*=/username/*
If specified this username will override the username taken from
the runtime environment. You can use this option to specify users
in a virtual user configuration. NOTE: This option will not
change to the given /username/, it will
only attempt to act on behalf of that user. Because of this you
will need to have proper permissions to be able to change files
owned by /username/. In the case of SQL this generally is not a
problem.

A lot of time ago I came with the same problem to Marc Martinec and he
implemented some sort of checks of addreses to see if they are local
to you or notbut I dont remeber


This last statement probably refers to a 2007 thread regarding per-user 
bayes

in amavisd-new. It is probably unrelated to the issue here.



2014-12-10 10:22 GMT-03:00 Filip Havlíček filip.havli...@pro-com.cz:
I have configured spamasssin with bayes user rules with this
configuration:
http://pastebin.com/KWW78DJx

I would like to ask you, if everything is correct, because I found
in table bayes_vars lot of (thousands) unknown email addresses 
like:

a...@hotmail.com mailto:a...@hotmail.com
ablewi...@hotmail.com mailto:ablewi...@hotmail.com
abl...@hotmail.com mailto:abl...@hotmail.com

My table bayes_token is also 350MB large!



That pastebin document is no longer there, so I'm only guessing.

If you are calling SpamAssassin via spamc/spamd combo, the recipient
usernames you specify with the option -u in spamc are what ends up
in an SQL field bayes_vars.username . Make sure you only allow
legitimate usernames or email addresses of you existing users there.
How you accomplish that depends on how spamc/spamd is integrated
with your mailer.

  Mark

Re: spamd: spf: lookup failed: addr is not a string at /usr/share/perl5/IO/Socket/IP.pm line 646.

[Mark Martinec]

Michael Grant wrote:


I'm getting this message in my mail.log:



spamd: spf: lookup failed: addr is not a string at
/usr/share/perl5/IO/Socket/IP.pm line 646.

I'm running debian stable with Spamassassin from backports:

SpamAssassin Server version 3.4.0
  running on Perl 5.14.2

and libmail-dkim-perl version 0.39 from stable (wheezy)

I see from this thread on gossamer that this may have something to do 
with

libmail-dkim-perl:

http://www.gossamer-threads.com/lists/spamassassin/users/188895

Has anyone made any progress on this?  Anyone found a workaround?


Yes, it's the same issue. The bug is in perl, fixed in 5.18 or later.
A minimal test program triggering the bug was posted in the mentioned
thread by me on 2014-10-29.

A workaround is setting your recursive DNS resolver to return
minimal responses, as suggested by alessio in that thread.
The fix is to upgrade perl.  I considered reporting the issue
to perl, but I doubt anyone would care to backport the fix
to old versions, so I didn't.

  Mark

Re: Help with Disk quota exceeded for a user depending on the mail quota

[Mark Martinec]

Almond wrote:

3) ...spamd[28040]: plugin: eval failed: error writing to 
/tmp/.spamassassin28040V31F7ftmp:
 Disk quota exceeded at 
/usr/share/perl5/vendor_perl/Mail/SpamAssassin/Message.pm

line 965, GEN840 line 1575.



this was the situation on /tmp before tmpwatch installation:
[root@myhostname ~]# du -lh /tmp



Then, I set up tmpwatch to do its job on /tmp every 12 hours...
BUT really, is not yet clear why that error about Disk quota
exceeded happened... and this is a problem if I don't understand,
'cause it could be repeated.

tmp folder issue and so why? mail quota? Do you really think that is
not because of user mail quota and temp files produced by Spamassassin
and ClamAV ? are you sure?


The reported error message Disk quota exceeded comes directly from
system's errno. So it is about a user disk quota on a /tmp file system.
It has nothing to do with a file system being full, or other quotas.
Check your user quotas at the /tmp file system.

  Mark

Re: fixing errors with --debug --lint

[Mark Martinec]

The From header is supposed to have a space between the display name
and the address.


It's just a convention for better readability.

RFC 5322 section 3.4.:
   address =   mailbox / group
   mailbox =   name-addr / addr-spec
   name-addr   =   [display-name] angle-addr
   angle-addr  =   [CFWS]  addr-spec  [CFWS] /
   obs-angle-addr


Omission of that space is a fair spam indicator.


I guess so.

  Mark

Re: different results when using --debug

[Mark Martinec]

On Dec 08, 2014, at 19.28, Mark Martinec mark.martinec...@ijs.si 
wrote:

Actually, looking at a diff of DBM.pm between 3.4.0 and 3.4.1
I can see the taint bug has already been fixed by r1608413:

@@ -814,3 +816,3 @@
  my @vars = $self-get_storage_variables();
-  dbg(bayes: DB journal sync: last sync: .$vars[7],'bayes','-1');
+  dbg(bayes: DB journal sync: last sync: %s, $vars[7]);

The extra parameters shouldn't have been in that dbg call.

See:
 Bug 7065 - Debug Mode breaks Bayes but only if DBM storage is used
 https://issues.apache.org/SpamAssassin/show_bug.cgi?id=7065



Ben wrote:

i've manually grafted that patch onto my 3.4.0, and it seems to do the
trick, thanks. i now see bayes results [and more consistent results
overall] when using --debug.


Good, thanks for confirming!



there is still a slight variation in scoring, however:

without --debug:
Content analysis details:   (19.6 points, 5.0 required)
[...]
-3.8 AWLAWL: adjust score towards average for this 
sender


with --debug:
Content analysis details:   (19.5 points, 5.0 required)
[...]
-3.7 AWLAWL: adjust score towards average for this 
sender


all other scoring is consistent.  it's a trivial variation in this
instance, but does it mean something additional may not be working as
intended?  or just something else i need to learn?


That's normal, AWL scores are dynamic. Especially if you use the same
test message multiple times the AWL score would slowly shift its value.

  Mark

Re: different results when using --debug

[Mark Martinec]

btb wrote:

i hope it's not too soon to ask about this again.  i'm not quite sure
how to debug something like this when when it's --debug that is what
behaves differently :)  i've put the commands and output into a
pastebin this time.


Possibly this PR is related:

  https://issues.apache.org/SpamAssassin/show_bug.cgi?id=7110


Mark

Re: different results when using --debug

[Mark Martinec]

btb wrote:

i hope it's not too soon to ask about this again.  i'm not quite sure
how to debug something like this when when it's --debug that is what
behaves differently :)  i've put the commands and output into a
pastebin this time.


Please show the debug output from:
  spamassassin --test-mode --debug bayes message3.txt

(or paste the full debug from:
  spamassassin --test-mode --debug message3.txt
)

Somehow in the debug case the use of bayes is disabled.

  Mark

Re: different results when using --debug

[Mark Martinec]

Ben wrote:

On 2014.12.08 10.13, Mark Martinec wrote:

btb wrote:

i hope it's not too soon to ask about this again.  i'm not quite sure
how to debug something like this when when it's --debug that is what
behaves differently :)  i've put the commands and output into a
pastebin this time.



Please show the debug output from:
   spamassassin --test-mode --debug bayes message3.txt


http://dpaste.com/3N49S6P.txt


(or paste the full debug from:
   spamassassin --test-mode --debug message3.txt


http://dpaste.com/3189XPC.txt

i did both just in case.

-ben



Great, thanks!

Does the following patch help?

--- lib/Mail/SpamAssassin/BayesStore/DBM.pm (revision 1643879)
+++ lib/Mail/SpamAssassin/BayesStore/DBM.pm (working copy)
@@ -1348,7 +1348,7 @@
 if ($opts-{verbose}) {
   print $msg,\n;
 } else {
-  dbg($msg);
+  dbg(%s, $msg);
 }
   }


Mark

Re: different results when using --debug

[Mark Martinec]

Does the following patch help?

--- lib/Mail/SpamAssassin/BayesStore/DBM.pm (revision 1643879)
+++ lib/Mail/SpamAssassin/BayesStore/DBM.pm (working copy)
@@ -1348,7 +1348,7 @@
  if ($opts-{verbose}) {
print $msg,\n;
  } else {
-  dbg($msg);
+  dbg(%s, $msg);
  }
}


Ben wrote:

it doesn't appear to have made a difference.  however, i am using
version 3.4.0, not revision 1643879.  is that a prerequisite for this
particular patch?


Thanks for testing.


have i applied the patch properly?:

[...]

  #dbg($msg);
  dbg(%s, $msg);


Yes, correctly.


Actually, looking at a diff of DBM.pm between 3.4.0 and 3.4.1
I can see the taint bug has already been fixed by r1608413:

@@ -814,3 +816,3 @@
   my @vars = $self-get_storage_variables();
-  dbg(bayes: DB journal sync: last sync: .$vars[7],'bayes','-1');
+  dbg(bayes: DB journal sync: last sync: %s, $vars[7]);

The extra parameters shouldn't have been in that dbg call.

See:
  Bug 7065 - Debug Mode breaks Bayes but only if DBM storage is used
  https://issues.apache.org/SpamAssassin/show_bug.cgi?id=7065


My today's patch can't hurt, it avoids a potential trouble,
but as it happens it may not necessary.

  Mark

Re: different results when using --debug

[Mark Martinec]

listsb-spamassas...@bitrate.net wrote:

i was testing with a sample message, and noticed that when running
manually with --debug, there seem to be numerous differences in the
results, such as scores for the same tests differing, visual ordering
of results differing [is this significant?], and bayes not being
listed when using --debug.  am i doing something wrong?  are my
expectations misguided?  i'm doing these tests as the user named
amavis, which the amavis software runs as.


spamassassin --test-mode --debug  message3.txt

 1.6 RCVD_IN_BRBL_LASTEXT   RBL: No description available.

[...]


spamassassin --test-mode  message3.txt

 1.4 RCVD_IN_BRBL_LASTEXT   RBL: No description available.
[94.73.46.5 listed in 
bb.barracudacentral.org]

-1.9 BAYES_00   BODY: Bayes spam probability is 0 to 1%
[score: 0.]



Apparently in the first case a score set 1 was chosen, and in the second
case a score set 3. Availability of a bayes scanner choses between the 
two.


Could it be that you have a fresh bayes database which had less than 200
spam and 200 ham entries in the first attempt, but became populated
and functional by the time of the second attempt?

  Mark

Re: SA dns_server option

[Mark Martinec]

Matteo Dessalvi wrote:

I have a short question about the dns_server option of SA.
Is this option used when SA is called from Amavis and there
isn't any spamd process running?


Yes it is.


To be more clear: should I also be forced to add the IP
address of the caching DNS server to /etc/resolv.conf
or the option would be sufficient?


The dns_server only affects SpamAssassin. If you want other
applications on that host to also use the same recursive
name server, its address needs to be in /etc/resolv.conf.
For example DKIM validation is done by amavisd calling
Net::DNS directly, which has no idea about SpamAssassin
settings. Similarly a milter or MTA.


Yes, I have read the docs but I was not sure if SA,
when used through Amavis, would use such option.

Nevermind, I pushed up the log verbosity of my DNS
caching service and it looks like SA is using it.
So, problem solved :-).



  Mark

Re: SA dns_server option

[Mark Martinec]

For example DKIM validation is done by amavisd calling
Net::DNS directly


A nitpick:  Actually, amavisd is calling Mail::DKIM when DKIM
validation is enabled, which in turn calls Net::DNS. The validation
result is then passed to SpamAssassin's DKIM plugin, so that it
doesn't need to do the validation again.

  Mark

Re: Confused by new version of spamassassin

[Mark Martinec]

2014-11-25 13:52, Paul Gardiner wrote:

On 25/11/2014 12:39, Paul Gardiner wrote:

On 25/11/2014 12:13, Axb wrote:

On 11/25/2014 12:50 PM, Paul Gardiner wrote:

I'm using spampd as follows:

spampd --port=10025 --relayhost=127.0.0.1:25 --user=paul 
--group=users
--homedir=/home/paul/.spamassassin --tagall --children=5 
--maxsize=512

--log-rules-hit


This is hardly a SA issue...

Suggest you check the spampd docs...

http://www.worlddesign.com/Content/rd/mta/spampd/spampd.html

I'd bet you're missing options in your command line...


Ok thanks. I was thinking you guys produce spampd as well because
it was part of the same opensuse package, but I guess it's just
the packager put them together for convenience.

Thanks for your time. It helps a lot just to know that the change
of spampd version is the more likely cause of my problems. Probably
spampd has become more configurable and as you say I need another
option on the command line, although I didn't before.


Actually, just so that I'm sure of what you are saying: is that it
is spampd's responsibility to add X-Spam-Status that I'm now not
seeing or is it spamassassin's responsibility, but spampd is not
requesting it, or is maybe suppressing it in some way.


Glancing at its code, spampd pulls a rewritten message from
SpamAssassin, which should include the X-Spam-Status header field
by default. The X-Spam-Status in this setup is added by the rule:

  add_header all Status _YESNO_, score=_SCORE_ required=_REQD_ 
tests=_TESTS_ autolearn=_AUTOLEARN_ version=_VERSION_


in a default SpamAssassin config file 10_default_prefs.cf.
The 'add_header' option can be overruled by your local configuration
(by a 'clear_headers') option. Missing 10_default_prefs.cf would
have the same effect (like being installed in a directory where
SpamAssassin/spampd isn't using), so make sure the spampd is
configuring SpamAssassin to use correct directories for rules.

  Mark

Re: could not create IO::Socket::IP socket on [127.0.0.1]:783: Address already in use

[Mark Martinec]

Tom Schulz wrote:

Here is my debug output. It looks like the socket is created for
127.0.0.1 and then an attempt is also made for ::1. That fails as
expected. But in my case that failure causes a retry that includes
retrying for 127.0.0.1. This now fails because the port has already
been created (as noted in the error message). Retries then continue
until the retry limit is reached.

dbg: spamd: socket module of choice: IO::Socket::IP 0.32, Socket 2.013,
have PF_INET, have PF_INET6, using Socket::getaddrinfo, AI_ADDRCONFIG 
is

supported
dbg: spamd: socket specification: localhost, IP address: localhost,
port: 783
dbg: spamd: attempting to listen on IP addresses: 127.0.0.1, ::1, port
783
dbg: spamd: creating IO::Socket::IP socket: Listen: 128, LocalAddr:
127.0.0.1, LocalPort: 783, Proto: tcp, ReuseAddr: 1, Type: 2, V6Only: 1
dbg: spamd: creating IO::Socket::IP socket: Listen: 128, LocalAddr: 
::1,

LocalPort: 783, Proto: tcp, ReuseAddr: 1, Type: 2, V6Only: 1
server socket setup failed, retry 1: spamd: could not create
IO::Socket::IP socket on [::1]:783: Cannot assign requested address
dbg: spamd: attempting to listen on IP addresses: 127.0.0.1, ::1, port
783
dbg: spamd: creating IO::Socket::IP socket: Listen: 128, LocalAddr:
127.0.0.1, LocalPort: 783, Proto: tcp, ReuseAddr: 1, Type: 2, V6Only: 1
server socket setup failed, retry 2: spamd: could not create
IO::Socket::IP socket on [127.0.0.1]:783: Address already in use
dbg: spamd: attempting to listen on IP addresses: 127.0.0.1, ::1, port
783
dbg: spamd: creating IO::Socket::IP socket: Listen: 128, LocalAddr:
127.0.0.1, LocalPort: 783, Proto: tcp, ReuseAddr: 1, Type: 2, V6Only: 1
server socket setup failed, retry 3: spamd: could not create
IO::Socket::IP socket on [127.0.0.1]:783: Address already in use


Thanks, that explains how we get to 'Address already in use'.

This opens up a couple of questions:

- creating a socket listening on '::' apparently does work
(it is tested in the BEGIN phase, and reported by 'have PF_INET6'),
but creating a socket listening on '::1' (i.e. a loopback address)
fails. Are you running spamd in jail? Does a loopback interface
have an IPv6 address assigned? Is it '::1'? Or if you think the
IPv6 is not supported on that host, why then does creating a
socket listening on '::' succeed?

- why did it work with older versions of perl and modules

- this chain of events shows that the logic in server_sock_setup_inet()
and/or its caller is flawed. When multiple sockets are to be created
and not all succeed, we'd need to undo the successfully created
sockets before bailing out on error - or alternatively, make do
with partially fulfilling the request.

Please open up a ticket on the bugzilla.

  Mark

Re: amavis question

[Mark Martinec]

Robert Moskowitz wrote:


I am running a new mailserver virtual domain environment with postfix
and mysql for the virtual domain definitions.  amavis-new seems to be
rejecting all emails for my one domain not a sub domain of my main
one.  That is, my domain (as you can see from my email addr) is
htt-consult.com and I have not problems processing emails for
labs.htt-consult.com addresses.  However I am seeing in the logwatch
email in the amavis-new section:

 **Unmatched Entries**
1   (26010-02-3) Open relay? Nonlocal recips but not
originating:res...@communaljob.com
1   (27713-04-4) Open relay? Nonlocal recips but not
originating:res...@communaljob.com
1   (24487-05-4) Open relay? Nonlocal recips but not
originating:res...@communaljob.com
1   (26339-01-2) Open relay? Nonlocal recips but not
originating:res...@communaljob.com
1   (27916-03-3) Open relay? Nonlocal recips but not
originating:res...@communaljob.com



Messages a plenty.


These are warnings, not rejects.

Assuming that domain communaljob.com is not one of your local domains,
the problem is that mail was not considered to be originating from
your site. This means that you either forgot to list all your internal
IP address spaces in the @mynetworks list, or in a more difficult setup
where your users are submitting authenticated mail to your MSA from
remote IP addresses, in which case you need to configure a dedicated
policy bank with 'originating=1' property for such mail submissions.

A background story:

  A combination of msgs.originating and msgrcpt.is_local tells a
  direction a message is traveling:

  originating is_local
  NNopen relay (probably misconfigured 
@local_domains_maps

or $originating flag not set)
  NYinbound message
  YNoutbound message
  YYinternal message (inside or authenticated - inside)

  Note that a direction is a per-recipient property, a multi-recipient
  message can be outbound for some recipients and internal for others
  at the same time.



No spamassasin section in logwatch, interestingly. Where do I allow
this to work?


Perhaps you have SpamAssassin disabled by @bypass_spam_checks_maps,
or maybe by a non-default @spam_scanners.

  Mark

Re: MISSING_SUBJECT versus empty subject

[Mark Martinec]

Matus UHLAR - fantomas wrote:

I remember I have asked the same some years ago.
http://www.gossamer-threads.com/lists/spamassassin/users/104646

the explanation was that the current rule detects empty subject the 
same as

no subject at all. IIRC it was in fact the same rule as it is now:

header __HAS_SUBJECT   exists:Subject


In a 2007 posting:
| The header exists:header_name test really converts into:
|  header_data =~ /./

It was true years ago, but is no longer the case.
The 'exists:' now tests for a presence of a header field
(regardless of its header body - empty or not).

A test like:
  header L_EMPTY_SUBJECT  Subject !~ /\S/

triggers on both cases: missing or empty Subject header field.

Its counterpart:
  header L_NONEMPTY_SUBJECT  Subject =~ /\S/


Mark

Re: could not create IO::Socket::IP socket on [127.0.0.1]:783: Address already in use

[Mark Martinec]

Tom Schulz wrote:

I am not sure if this is a bug or a usage problem.

I have been running spamd on Solaris 10 and Perl v5.8.9 for years, 
starting

it with the command:
spamd -d -c -m 18 --min-spare=1 --max-spare=4 --syslog-socket=inet

I upgraded Perl to v5.20.1 and installed the current modules. When I
try to start spamd with that command, I get the following errors:

setlogsock(): type='tcp': TCP service unavailable at
/opt/local/lib/perl5/site_perl/5.20.1/Mail/SpamAssassin/Logger/Syslog.pm
line 100.


This one is normal if your syslogd listens only on an UDP socket.

The setlogsock(inet) tries TCP first, and if that fails it tries UDP.
When warnings are enabled (like in spamd), you see the TCP attempt
failure as a warning, but then the UDP is used.

So if you know the syslogd only listens on UDP, you should
specify this explicitly: --syslog-socket=udp

I don't know if this worked differently previously. Possibly the
version of Sys::Syslog brought some changes in warnings issued.



server socket setup failed, retry 1: spamd: could not create
IO::Socket::IP socket on [::1]:783: Cannot assign requested address
server socket setup failed, retry 2: spamd: could not create
IO::Socket::IP socket on [127.0.0.1]:783: Address already in use

And spamd does not start. If I add '-4' to the above command [...]
And spamd does start and I do have logging.



I conclude that the failed attempt to create an IPv6 socket blocks
the ability to later create IPv4 sockets to the same port.


Yes, so it seems.

Please start spamd without option '-d' (daemonize) and with '-D spamd',
and note the logged order of socket bind attempts. Mine shows:

dbg: spamd: socket module of choice: IO::Socket::IP 0.32, Socket 2.015, 
have PF_INET, have PF_INET6, using Socket::getaddrinfo, AI_ADDRCONFIG is 
supported
dbg: spamd: socket specification: localhost, IP address: localhost, 
port: 783
dbg: spamd: attempting to listen on IP addresses: 127.0.0.1, ::1, port 
783
dbg: spamd: creating IO::Socket::IP socket: Listen: 128, LocalAddr: 
127.0.0.1, LocalPort: 783, Proto: tcp, ReuseAddr: 1, Type: 1, V6Only: 1
dbg: spamd: creating IO::Socket::IP socket: Listen: 128, LocalAddr: ::1, 
LocalPort: 783, Proto: tcp, ReuseAddr: 1, Type: 1, V6Only: 1


Note the order: tries 127.0.0.1 first, only then the ::1.
Note also the reported 'V6Only: 1' and a version of IO::Socket::IP
(or maybe it's IO::Socket::INET6 in your case???).


But note that with
Perl v5.8.9 and the module versions available in Dec 2013, there was no
problem without the '-4' switch. This sounds like it could be a bug
somewhere.




And why does setlogsock complain about type='tcp now when
it did not with the older Perl?


Probably unrelated to a version of perl, but has to do with
a version of Sys::Syslog or a version of the 'warnings' pragma.

  Mark

Re: .co.at

[Mark Martinec]

snowshoe... high fashion
May cause FPs  co.at is a subdomain hoster/pseudo tld
uri AXB_URI_WHATCOAT/http\:\/\/\w+\.co\.at\//


uri  AXB_URI_WHATCOAT  m{http://\w+\.co\.at/}

http://en.wikipedia.org/wiki/Leaning_toothpick_syndrome

:)

  Mark

Re: DMARC policy check with AskDNS posible?

[Mark Martinec]

After a couple of iterations and re-reading the policy syntax in a
DMARC draft, I ended up with the following set of rules ( based on
https://issues.apache.org/SpamAssassin/show_bug.cgi?id=7099 ):


ifplugin Mail::SpamAssassin::Plugin::AskDNS

askdns  __DMARC_POLICY_NONE   _dmarc._AUTHORDOMAIN_ TXT /^v\s*=DMARC1 
(?=\s*;) .* ;\s* p\s*=\s*none   \s*(?:;|\z)/x
askdns  __DMARC_POLICY_QUAR   _dmarc._AUTHORDOMAIN_ TXT /^v\s*=DMARC1 
(?=\s*;) .* ;\s* p\s*=\s*quarantine \s*(?:;|\z)/x
askdns  __DMARC_POLICY_REJECT _dmarc._AUTHORDOMAIN_ TXT /^v\s*=DMARC1 
(?=\s*;) .* ;\s* p\s*=\s*reject \s*(?:;|\z)/x


meta__DMARC_REJECT !(DKIM_VALID_AU || SPF_PASS)  
__DMARC_POLICY_REJECT
meta__DMARC_QUAR   !(DKIM_VALID_AU || SPF_PASS)  
__DMARC_POLICY_QUAR
meta__DMARC_NONE   !(DKIM_VALID_AU || SPF_PASS)  
__DMARC_POLICY_NONE


metaDMARC_REJECT__DMARC_REJECT  !__VIA_ML
score   DMARC_REJECT2.1
metaDMARC_REJECT_ML __DMARC_REJECT   __VIA_ML
score   DMARC_REJECT_ML 0.8

metaDMARC_QUAR  __DMARC_QUAR  !__VIA_ML
score   DMARC_QUAR  1.8
metaDMARC_QUAR_ML   __DMARC_QUAR   __VIA_ML
score   DMARC_QUAR_ML   0.7

endif



Mark

Re: DNS checks not being performed-

[Mark Martinec]

Niamh Holding wrote:

spamd[26862]: spamd: connection from 127.0.0.1 [127.0.0.1]:46489 to 
port 783, fd 5

spamd[26862]: spamd: setuid to spamtest succeeded
spamd[26862]: spamd: processing message 
2014121508.3321875.277...@sailthru.com for spamtest:1028
spamd[26862]: plugin: eval failed: IO::Socket: Cannot configure a 
generic socket at 
/usr/lib/perl5/site_perl/5.8.8/Mail/SpamAssassin/DnsResolver.pm line 
414

spamd[26862]: rules: failed to run NO_DNS_FOR_FROM RBL test, skipping:
spamd[26862]:  (IO: [...]:Socket: Cannot configure a generic socket at 
/usr/lib/perl5/site_perl/5.8.8/Mail/SpamAssassin/DnsResolver.pm line 
414)


NH /usr/lib/perl5/site_perl/5.8.8/Mail/SpamAssassin/DnsResolver.pm 
line 414.

$sock = $io_socket_module_name-new(%args);
And a quick test script suggests that
$io_socket_module_name = IO::Socket::INET6


Start spamd with option  -D dns  and compare its dns debuging
to what you obtained by running spamassassin -D dns .

The 'Cannot configure a generic socket' in IO::Socket means
that it could not determine which 'domain' (i.e. protocol family)
it should use.

  Mark

Re: ***UNCHECKED(Encrypted)*** Re: DNS checks not being performed-

[Mark Martinec]

Is your IO::Socket::IP at version 0.21 or older?

$ perl -le 'use IO::Socket::IP; print IO::Socket::IP-VERSION'


0.21 CHANGES:
  * Ensure that IO::Socket-new( Domain = ... ) definitely returns a
socket in the right family


Mark

Re: ***UNCHECKED(Encrypted)*** Re: DNS checks not being performed-

[Mark Martinec]

Niamh Holding wrote:


MUf ... as it was already said: do NOT mix manually installed CPAN 
modules with

MUf packages of any kind. use either one or the other.

so don't use yum and cpan?


Well, you probably can - if you are careful.
But using an 8 years old version of perl you'd be safer with
pre-packaged modules.

  Mark

Re: DNS checks not being performed-

[Mark Martinec]

Reindl Harald wrote:


BTW: what is the Re: ***UNCHECKED(Encrypted)*** in your subjects?\


Sorry, my fault. Forgot to remove this from a Subject when replying.

It was added here because the posting includes a PGP signature
and the file(1) utility cannot distinguish a PGP/GPG encrypted
message MIME part from a standalone signature, so 'just in case'
the message is labeled as potentially unchecked.

Just started to write an enhancement report to the file(1) development
list ...

  Mark

Re: DMARC policy check with AskDNS posible?

[Mark Martinec]

2014-06-03 09:43, Christian Laußat wrote:

I'm trying to improve my rules for DMARC policy checking. For now I
only use the Authentication-Results header from the OpenDMARC milter
as described here:
https://kvm.laussat.info/2014/05/19/using-dmarc-in-spamassassin/

To get ride of this dependency, I looked at 
Mail::SpamAssassin::Plugin::AskDNS.
It seems it would be easy to write a DMARC policy check with these 
rules, e.g.:

[...]
My problem now is how to get the _SENDERDOMAIN_ tag for the AskDNS 
check?

If the message is DKIM signed I could use _DKIMDOMAIN_, but what if
it's not signed but has a DMARC policy on the domain?

Any ideas how to do this without writing a plugin?


Done, added SENDERDOMAIN and AUTHORDOMAIN tags:

  https://issues.apache.org/SpamAssassin/show_bug.cgi?id=7099


Mark

Re: Subroutine IO::Socket::INET6::AF_INET6

[Mark Martinec]

Niamh Holding wrote:


After updating to  SpamAssassin 3.4.0 (2014-02-07) I'm getting the
following-

/usr/lib/perl5/5.8.8/Exporter.pm line 65.
 at /usr/lib/perl5/vendor_perl/5.8.8/IO/Socket/INET6.pm line 16

rpm -q  perl-IO-Socket-INET6
perl-IO-Socket-INET6-2.51-2.fc6


Upgrade or deinstall IO::Socket::INET6 :

  http://marc.info/?t=12863774364r=1w=2


Mark

Re: spf: lookup failed: addr is not a string

[Mark Martinec]

Benny Pedersen wrote:

Spampd 2.30 does not work with perl 5.18, spampd 2.42 does, seem lot
is changed in perl, so is it not just mail::dkim that needs updates
for perl 5.18 ?

Or is it unrelated ?



me wrote:

Don't know about Spampd, it's probably unrelated.
[...]
I'm not aware of any noteworthy compatibility differences
between 5.14 / 5.16 and 5.18 / 5.20. I don't think anything
needed to be changed either in SpamAssassin or in Amavis
to make it run under 5.18 or 5.20.



Seems that the change in Spampd between 2.30 and 2.42
was needed because of a change in a module Getopt::Long :

  Changes in version 2.39
  * Retain taintedness of command line option values.

and is unrelated to a version of perl.


  Mark

Re: shellshock via SMTP?

[Mark Martinec]

2014-10-29 16:26, Joe Acquisto-j4 wrote:

Comments on the ZD net article that claims shellshock exploit via
crafty SMTP headers?   Just asking, that's all . . .

I attached a link to it below, please excuse if that is improper 
behavior.

  http://www.zdnet.com/shellshock-attacks-mail-servers-735094/


I have seen one such sample. Must be a really dumb mail delivery agent
or a content filter or a MUA that lets a mail header touch a shell.

No matter whether bash is patched or not, tainted data from a mail
message must never be handed over to shell.

  Mark

Re: spf: lookup failed: addr is not a string

[Mark Martinec]

Thomas Preißler wrote:

Hey Mark,
thanks for your explanation!


I'm beginning to understand what is going on here.
Because you have a older version of Mail::DKIM, spamassassin is
unable to provide it with its own resolver, so Mail::DKIM does
it by directly calling Net::DNS, which uses IO::Socket::INET,
while the rest of the SpamAssassin's DNS resolving goes through
IO::Socket::IP. For some reason a TCP DNS request by Net::DNS
affects the socket used by IO::Socket::IP, making a variable
holding a string also get a numerical component, and moreover
it becomes tainted. In the end the getnameinfo() falls into
the snag. Weird...


The only way to fix this issue is to update Mail::DKIM? I’m gonna ask
the maintainer to upload an update to wheezy-backports.


Can't hurt to have an up-to-date version of Mail::DKIM.
There may be other workarounds, I'm still looking into the matter.

Here is a short indicator of the problem:

  perl -Te 'use Socket; use Devel::Peek; $s=$ENV{PATH};
$s=0; $s=\2\0\0005\301\2\4\367\0\0\0\0\0\0\0\0;
Devel::Peek::Dump($s); Socket::getnameinfo($s,0)'

It stumbles across 'addr is not a string' in getnameinfo
with perl 5.14 and 5.16.

The problem is solved with perl 5.18, 5.20, 5.21.5,
which deal with pPOK vs. POK flags somewhat differently.

  Mark

Re: spf: lookup failed: addr is not a string

[Mark Martinec]

Benny Pedersen wrote:

The problem is solved with perl 5.18, 5.20, 5.21.5,
which deal with pPOK vs. POK flags somewhat differently.


Spampd 2.30 does not work with perl 5.18, spampd 2.42 does, seem lot
is changed in perl, so is it not just mail::dkim that needs updates
for perl 5.18 ?

Or is it unrelated ?


Don't know about Spampd, it's probably unrelated.

The upgrade of Mail::DKIM in this context was mentioned because
it avoids an issue in an unrelated piece of code, as SpamAssassin
can use a feature offered by a newer Mail::DKIM, which happened to
provide a workaround for the getnameinfo() problem. The Mail::DKIM
itself does not have a problem with running under any of the last
five or so major versions of perl.

I'm not aware of any noteworthy compatibility differences
between 5.14 / 5.16 and 5.18 / 5.20. I don't think anything
needed to be changed either in SpamAssassin or in Amavis
to make it run under 5.18 or 5.20.

The main benefit of using a newer version of perl is avoiding
several old bugs in perl core or modules. Some of these bugs
required workarounds in our code (like several taint-related bugs
in Encode, some utf8-related bugs in regexp evaluation).

There are a couple of stricter warnings issued with each newer
version of perl, but these just reveal some hidden bugs in our
application code and needed to be fixed anyway, perl made them
easier to find. Just recently I needed to fix/avoid two such
warnings under perl 5.21.5 in SpamAssassin (trunk), as well as
in Amavisd (2.10.1).

Other benefits are: updated some essential core modules
(like Encode, Digest::SHA, Socket, and IO::Socket::IP), as well
as some speedups and improved Unicode support.

  Mark

Re: spf: lookup failed: addr is not a string

[Mark Martinec]

On 2014-10-28 13:25, Thomas Preißler wrote:

Finally, I’m able to reproduce this issue on a plain debian wheezy system:
- install debian wheezy
- enable backports and run apt-get update
- apt-get -t wheezy-backports install spamassassin
- apt-get install libmail-dkim-perl
- set 156.154.70.1 as the only nameserver in /etc/resolv.conf
- run spamassassin -D  mail.eml

But removing libmail-dkim-perl is not really a solution. This package
provides the Mail::DKIM module which is required to check the DKIM
signature.


Thanks, I can now reproduce this.

I'm beginning to understand what is going on here.
Because you have a older version of Mail::DKIM, spamassassin is
unable to provide it with its own resolver, so Mail::DKIM does
it by directly calling Net::DNS, which uses IO::Socket::INET,
while the rest of the SpamAssassin's DNS resolving goes through
IO::Socket::IP. For some reason a TCP DNS request by Net::DNS
affects the socket used by IO::Socket::IP, making a variable
holding a string also get a numerical component, and moreover
it becomes tainted. In the end the getnameinfo() falls into
the snag. Weird...

  Mark

Re: spf: lookup failed: addr is not a string

[Mark Martinec]

Thomas Preißler wrote:

Already tried that. When using unbound as a local caching nameserver
and 156.154.70.1 as the resolver, spamassassin produces the same error
message. The same happens when unbound accesses the root nameservers
directly and acts as a local resolver. But when unbound caches 8.8.8.8
there is no error message from spamassassin.


Do you have a firewall in place that tries to do a deep packet 
inspection

on DNS UDP packets but does not understand EDNS0 (the OPT RR) ?

Try capturing a query  response with 156.154.70.1 on the wire
(e.g.:  tcpdump -s 0 -w 0.log 'host 156.154.70.1 or icmp' )
while running your mail sample through spamassassin, and see if
there is anything funny there.

  Mark

Re: tag DKIMDOMAIN is still blocking action 0

[Mark Martinec]

Chris,


Ran some spam and ham through 'spamassassin -D -t' today mainly looking
to see if there were any mention of dns issues as I had reported
earlier. At the end of the run I see this whether it's ham or spam:

Oct 21 19:30:09.086 [31076] dbg: check: tagrun - tag DKIMDOMAIN is 
still

blocking action 0


If a message does not contain a *valid* DKIM signature, then the
tag DKIMDOMAIN won't be set, so any rules that depend on this tag
will not be activated. So this is a normal situation for unsigned
or forged mail.

The rules in question are probably
  DKIMDOMAIN_IN_DWL and __DKIMDOMAIN_IN_DWL_ANY


20_dnsbl_tests.cf : (wrapped for clarity):

  askdns   DKIMDOMAIN_IN_DWL
_DKIMDOMAIN_._vouch.dwl.spamhaus.org TXT
 /^([a-z]+ )*(transaction|list|all)( [a-z]+)*$/

  askdns   __DKIMDOMAIN_IN_DWL_ANY
_DKIMDOMAIN_._vouch.dwl.spamhaus.org TXT

So these rules would launch a DNS query against _vouch.dwl.spamhaus.org
if and only if a message would contain a valid DKIM signature.

  Mark

Re: tag DKIMDOMAIN is still blocking action 0

[Mark Martinec]

Chris,


Mark, now I'm confused. As you can see the 'action 0 .' takes place
before the DKIM lookup

Oct 22 09:16:14.220 [8459] dbg: check: tagrun - action 0 blocking on
tags DKIMDOMAIN


Yes, that's normal. It happens immediately after basic information
has been extracted from a mail header ('extract_metadata' plugins hook).

In case of this dependency on DKIMDOMAIN it is a direct consequence
of having rules DKIMDOMAIN_IN_DWL and __DKIMDOMAIN_IN_DWL_ANY,
regardless of the actual message.

The 'check: tagrun - action ... blocking on ...' log message
just says that a callback routine has been provided, which is
to be called at some point later if/when a tag value for a
tag DKIMDOMAIN becomes available.


Oct 22 09:16:14.623 [8459] dbg: dkim: signature verification result:
FAIL (BODY HAS BEEN ALTERED)
Oct 22 09:16:14.623 [8459] dbg: dkim: FAILED signature by
shop.identitydirect.com, author en...@shop.identitydirect.com, no valid
matches
Oct 22 09:16:14.624 [8459] dbg: dkim: FAILED signature by
shop.identitydirect.com, author en...@shop.identitydirect.com, no valid
matches
Oct 22 09:16:14.624 [8459] dbg: dkim: author
en...@shop.identitydirect.com, not in any dkim whitelist

The tests show that the DKIM test failed



Right. So the DKIM signature was not valid, so the tag
DKIMDOMAIN never got its value assigned, so a callback routine
attached to DKIMDOMAIN tag was never called, which yields the:

  dbg: check: tagrun - tag DKIMDOMAIN is still blocking action 0

at the end of mail processing. Normal.



yet the SA headers show AFAICT it's good.

,DKIM_SIGNED=0.1, DKIM_VALID=-0.1,DKIM_VALID_AU=-0.1


You are probably looking at the report header from a previous
spamassassin run. The messages that you provided in your test run
was somehow clobbered, invalidating a DKIM signature.

  Mark

Re: dns: bad dns reply: Connection refused

[Mark Martinec]

2014-10-20 20:11, Reindl Harald wrote:

[...]

sorry, no, but what i face repeatly are messages like below
in fact only if the machine has more than 1 dns in resolv.conf
configure it to just use 127.0.0.1 and that won't happen

Oct 19 09:04:42 caladan spamd[20546]: dns: no callback for id
40563/IN/A/uwc.org.dbl.spamhaus.org, ignored; packet: ;; Answer
received from 10.0.0.6 (53 bytes)
Oct 19 09:04:42 caladan spamd[20546]: dns: [...] ;; HEADER SECTION
Oct 19 09:04:42 caladan spamd[20546]: dns: [...] ;; id = 40563
Oct 19 09:04:42 caladan spamd[20546]: dns: [...] ;; qr = 1 aa = 0 tc =
0 rd = 1 opcode = QUERY
Oct 19 09:04:42 caladan spamd[20546]: dns: [...] ;; ra = 1 z = 0 ad =
0 cd = 0 rcode = SERVFAIL
Oct 19 09:04:42 caladan spamd[20546]: dns: [...] ;; qdcount = 1
ancount = 0 nscount = 0 arcount = 1
Oct 19 09:04:42 caladan spamd[20546]: dns: [...] ;; do = 0
Oct 19 09:04:42 caladan spamd[20546]: dns: [...] ;; EDNS version 0
Oct 19 09:04:42 caladan spamd[20546]: dns: [...] ;; flags: 
Oct 19 09:04:42 caladan spamd[20546]: dns: [...] ;; rcode: NOERROR
Oct 19 09:04:42 caladan spamd[20546]: dns: [...] ;; size: 1024
Oct 19 09:04:42 caladan spamd[20546]: dns: [...] ;; option:
Oct 19 09:04:42 caladan spamd[20546]: dns: [...]
Oct 19 09:04:42 caladan spamd[20546]: dns: [...]
Oct 19 09:04:42 caladan spamd[20546]: dns: [...] ;; QUESTION SECTION (1 
record)

Oct 19 09:04:42 caladan spamd[20546]: dns: [...] ;;
uwc.org.dbl.spamhaus.org. IN A
Oct 19 09:04:42 caladan spamd[20546]: dns: [...]
Oct 19 09:04:42 caladan spamd[20546]: dns: [...] ;; ANSWER SECTION (0 
records)

Oct 19 09:04:42 caladan spamd[20546]: dns: [...]
Oct 19 09:04:42 caladan spamd[20546]: dns: [...] ;; AUTHORITY SECTION
(0 records)
Oct 19 09:04:42 caladan spamd[20546]: dns: [...]
Oct 19 09:04:42 caladan spamd[20546]: dns: [...] ;; ADDITIONAL SECTION
(1 record)
Oct 19 09:04:42 caladan spamd[20546]: dns: [...] ;; EDNS version 0
Oct 19 09:04:42 caladan spamd[20546]: dns: [...] ;; flags: 
Oct 19 09:04:42 caladan spamd[20546]: dns: [...] ;; rcode: NOERROR
Oct 19 09:04:42 caladan spamd[20546]: dns: [...] ;; size: 1024
Oct 19 09:04:42 caladan spamd[20546]: dns: [...] ;; option:
Oct 19 09:04:42 caladan spamd[20546]: dns: no likely matching queries
for id 40563


This happens when a DNS response comes late and ALARM signal
interrupts its decoding. They call it a 'design feature',
I call it bug:

  https://rt.cpan.org/Ticket/Display.html?id=83451

Mark

Re: rejected Null-Senders

[Mark Martinec]

On Tue, 2014-10-07 at 17:46 +0200, Reindl Harald wrote:

can somebody comment in what context null-senders and
so bounces and probably autorepsonders are blocked
by DKIM_ADSP_NXDOMAIN,USER_IN_BLACKLIST


DKIM_ADSP_NXDOMAIN is checking a domain in a From header field,
not the envelope sender address.

  Mark

Re: Program failure (69) of spamc

[Mark Martinec]

2014-09-10 23:25 Geoff Soper wrote:

Hi,
I'm calling spamc (3.3.2) from procmail as suggested at
https://wiki.apache.org/spamassassin/UsedViaProcmail

Having made a change to not use the lockfile (as suggsted in another
thread) I'm now getting the following error in my procmail log:

procmail: Program failure (69) of spamc
procmail: Rescue of unfiltered data succeeded
From   Wed Sep 10 22:10:47 2014
 Subject: 
  Folder: /var/qmail/mailnames/*

This persists even after SA has been restarted and the procmailrc file
restored to its original state.

Can anyone suggest what the issue may be?


According to spamc/libspamc.c the EX_UNAVAILABLE (code 69)
can be a result of any of the following conditions:

/*
 * translate_connect_errno()
 *
 *  Given a UNIX error number obtained (probably) from connect(2),
 *  translate this to a failure code. This module is shared by both
 *  transport modules - UNIX and TCP.
 *
 *  This should ONLY be called when there is an error.
 */
static int _translate_connect_errno(int err)
{
switch (err) {

[...]

case ECONNREFUSED:
case ETIMEDOUT:
case ENETUNREACH:
return EX_UNAVAILABLE;



Mark

Re: Dealing with suspicious unicode in domains

[Mark Martinec]

Alex writes:

Just came across this article about measures Google is taking to block
domains using suspicious unicode characters:

http://threatpost.com/google-tweaks-gmail-to-help-limit-spam/107732

Does SA yet have similar measures? I seem to recall some discussion 
about

this probably a year ago. What are some example domains?


Nontrivial, but probably worth doing eventually, especially now that
SMTPUTF8 is getting some traction. I hope there already exists some
Perl module to help with Unicode normalization and matching.

Please open a problem report so that this does not go by forgotten.

  Mark

Re: Google, IPv6 and SPF (was Re: No SPF/DKIM/DMARC rule)

[Mark Martinec]

David F. Skoll wrote:

Google must be using some secret algorithm to decide whether or not to
be strict.


https://support.google.com/mail/answer/81126?p=ipv6_authentication_errorrd=1#authentication
(September 2013)

- Sign messages with DKIM. We do not authenticate messages signed
with keys using fewer than 1024 bits.

Additional guidelines for IPv6

- The sending IP must have a PTR record (i.e., a reverse DNS of the
sending IP) and it should match the IP obtained via the forward
DNS resolution of the hostname specified in the PTR record.
Otherwise, mail will be marked as spam or possibly rejected.
- The sending domain should pass either SPF check or DKIM check.
Otherwise, mail might be marked as spam.


2014-07-25 20:05 Robert Schetterer wrote:

its simply a bug, but they dont care, spam tagging was reported
with all settings good ipv6 SPF/DKIM/DMARC/PTR
however it might fixed recent or will fixed some day, i never retested
it ,now using ipv4 transport only ,for them


Our mail to google over IPv6 is still being accepted
(have it all: SPF/DKIM/DMARC/PTR).

Consider enabling DMARC daily reports, google and some other
big players are sending them, can be insightful.

Here is our DNS record:
  $ host -t txt _dmarc.ijs.si
  _dmarc.ijs.si  v=DMARC1\; p=none\; 
rua=mailto:mailauth-repo...@ijs.si;


Mark

Re: Bayer Filter Not Working

[Mark Martinec]

On 2014-07-05 14:18, Herbert J. Skuhra wrote:


Jun 24 13:47:53.167 [3245] warn: plugin: eval failed: Insecure
dependency in sprintf while running with -T switch at
/usr/local/share/perl/5.14.2/Mail/SpamAssassin/Logger.pm line 241.
Jun 24 13:47:53.168 [3245] dbg: config: score set 0 chosen.

That seems to be the last time Bayes is referenced in a spamassassin
-D ―lint



So, I can only reproduce this issue on FreeBSD (and not on Fedora
20, Arch Linux) because only FreeBSD runs spamassassin in taint mode.

Is this a bug in Logger.pm? Or simply switch off taint mode? Any comments?


The bug is not in the logger, but is a routine that calls it.

Which Bayes database backend are you using? The bug must be specific
to that one.

 I could resolve this issue by setting $sa_debug from 1- 0 in
 amavisd.conf.
 This is abviously a workaround and not a proper fix.

For the time being please stick to that workaround until the
problem is investigated.

 Or simply switch off taint mode?

Please don't, that would not be a right thing to do.

Looks like the kind of a thing appropriate to open a
problem report in the SpamAssassin's bugzilla.

  Mark

Re: Installing IO::Socket::IP

[Mark Martinec]

Chris,


Looking for another pair of eyes here. I've been trying to install this
module for a few hours now and having no luck. When testing on the
module begins the output for each test is:

Socket version 1.97 required--this is only version 1.82

So, I found Socket V1.97 and tried to install it from source. The 
output

is here:

http://pastebin.com/XGZGAq2f

Any idea of where I'm going astray? Perl version is 5.10.1


Looks like gcc is not happy:
  const-xs.inc:834: error: format not a string literal and no format 
arguments


The current version of module Socket is 2.014,
try that instead of the oldish 1.97 from 2012.

  Mark

Re: Installing IO::Socket::IP

[Mark Martinec]

Chris,


Looks like gcc is not happy:
   const-xs.inc:834: error: format not a string literal and no format
arguments



http://pastebin.com/CYgt0Uq7


Remove the
  -Werror=format-security
from your compiler options.

  Mark