Re: Welcome/unwelcome list not working correctly.

[Tom Hendrikx]

On 22-07-2023 13:31, Henrik K via users wrote:

On Sat, Jul 22, 2023 at 10:13:42AM +0200, Benny Pedersen wrote:

Henrik K via users skrev den 2023-07-22 06:50:


| gvk  | unwhitelist_from|
grant.kel...@sonic.com   | 7421538 |
| gvk  | whitelist_from  | *@sonic.com   | 7526210 |


user_prefs in sql/ldap can not do unwhitelist, it
missing priority field in sql/ldap for this to work,
only thing that is possible as now is to remove the
whitelist not add unwhitelist, would need feature
request for priority field


No need for feature request, already exists:

user_scores_sql_custom_query

Create own SQL that sorts as you want, for example MySQL ORDER BY
FIELD(...).

Then again why even write unneeded stuff in DB.  Fix it in the backend or
create triggers etc to cleanup redundant stuff.



I do agree with Benny that if the order of directives matterbecause of 
the way it is implemented in SA, then the default/example implementation 
should enforce a clear (if not useful) order.


There is nothing mentioned regarding order in 
https://cwiki.apache.org/confluence/display/SPAMASSASSIN/usingsql, or in 
https://svn.apache.org/repos/asf/spamassassin/tags/spamassassin_current_release_4.0.x/sql/README



Kind reagards,
Tom

Re: increase Pyzor weight

[Tom Hendrikx]

On 28-06-2023 10:46, Richard Lucassen wrote:

Hello list,

Is it possible to increase the weight of PYZOR_CHECK score?

R.



Yes,

This works the same as a regular score (it is a regular score of 
course). See 
https://spamassassin.apache.org/full/4.0.x/doc/Mail_SpamAssassin_Conf.html#SCORING-OPTIONS


Kind regards,
Tom

Re: SA build from cpan fails under certain conditions

[Tom Hendrikx]

On 22-12-2022 01:48, Shawn Iverson wrote:
> I will not engage in furthering this conversation.  Sad there seems to
> be some toxicity here.
>

Hi Shawn,

Please ignore comments from Reindl Harald, he has been banned from 
several mailing lists for sending negative, abusive or outright 
aggressive comments in response to questions. He doesn't post to these 
lists anymore, but apparently he still sends responses directly to 
subscribers. He does not represent the regular sentiment on this mailing 
list.


I'm afraid that I can't help you with your issue, I don't use RHEL, RPMs 
or CPAN. But I want to let you know that your questions on this list are 
useful, so please continue.


Kind regards,

  Tom

Re: FMBLA_NDBLOCKED and DKIMWL_BLOCKED

[Tom Hendrikx]

On 18-11-2022 04:20, Alex wrote:

Hi,

I just noticed I've apparently hit the regular limits of use for fmbla 
and dkimwl for my few domains and honeypots. I believe this is a service 
provided by Paul Stead - does anyone know if there's a "pro" version or 
how I might be able increase the permissible capacity allowed?


Given it's integrated into SA now, it would be nice to be able to 
benefit from it. There's nothing on the fmb.la  website 
to indicate how I might be able to do that.


I'm using a personal resolver, not a public DNS server.


In the terms there is some talk about accounts etc. Did you try to sign up?

Kind regards
Tom

Re: Question about whitelisting of naadac.org

[Tom Hendrikx]

Hi Lukasz,

The Spamassassin score looks reasonable. If mail-tester uses anything
similar to a stock Spamassassin setup, then you should be safe and
spamassassin will not be the cause of your delivery problems.
Whitelisting a somewhat arbitrary URL will not solve your problem.

Of course, it could be that certain recipients of your customer have
setup additional Spamassasin rules, tuned their setup to raise some
penalties, or added additional filtering (outside of SA) to their
mailstack that results in a different conclusion. You cannot be sure
unless you ask the mail-admin of those customers.

So you need to get in touch with them, not with the SA community (but as
you can see, we're happy to point you in the correct direction ;-> ).

Kind regards,
Tom

On 12-08-2021 22:16, Lukasz Maik wrote:

Dear John,

Sure, please find full tests results here:
https://www.mail-tester.com/test-bw02eaxrt

We've lost a point for not having DKIM/DMARC authentication, which is
unfortunately not supported by our hosted exchange. We also lost 0.5
point for not having alt attribute in the images, so we will add it. 
Total is 7.8/10.


The problem, when user is sending normal work e-mails, recipients are
finding those messages in the Junk Email folder. Even people with who
he was previously working before.

Kind Regards Lukas

-Original Message- From: John Hardin  
Sent: Thursday, August 12, 2021 5:43 AM To:

users@spamassassin.apache.org Subject: Re: Question about
whitelisting of naadac.org

This message was sent from an external source. Please be careful
opening attachments/links or replying to sources you don't know.

On Wed, 11 Aug 2021, Lukasz Maik wrote:


Hi All,

The company naadac.org is experiencing problems with their e-mails 
being marked as SPAM, when they are putting link to their domain 
https://eur02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.naadac.org%2Fdata=04%7C01%7CLukasz.Maik%40ricoh-europe.com%7Cd9ba04e2fffa42bd4b1b08d95d435fec%7Cdd29478d624e429eb453fffc969ac768%7C0%7C0%7C637643367114945933%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000sdata=IkcJvzYcpJvlUWr3l%2FzGbvD3IbSSaeia66LNwTjOj60%3Dreserved=0

in the signature of their mails.

Is it possible to whitelist this domain/link in your SPAM
filtering? Results from the mail-tester.com tool are available
below:

[cid:image001.png@01D78EFB.CD78CAE0]


0.644 points is not sufficient to mark a message as spam using the
default scoring, and isn't worth hitting the panic button. If it's
being marked as spam by some recipients, there are other reason(s).
Is this analysis the only thing you are basing your analysis on?

As Kenneth said, contact Spamhaus regarding why that domain is
listed.

In order to offer more advice, we would have to see the results from
a site that is actually marking such a message as spam (i.e. where
it's scoring 5 or more points).

-- John Hardin KA7OHZ
https://eur02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.impsec.org%2F~jhardin%2Fdata=04%7C01%7CLukasz.Maik%40ricoh-europe.com%7Cd9ba04e2fffa42bd4b1b08d95d435fec%7Cdd29478d624e429eb453fffc969ac768%7C0%7C0%7C637643367114945933%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000sdata=99khbdmpdLV%2BpMuWur8MkrCcd2dzn5qr02xBSWC7GH8%3Dreserved=0



jhar...@impsec.org pgpk -a jhar...@impsec.org

key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C  AF76 D822 E6E6 B873
2E79 
---




The difference between ignorance and stupidity is that the stupid
desire to remain ignorant. -- Jim Bacon 
---




Tomorrow: the 900th anniversary of the muslim Seljuq defeat at Didgori

Ricoh Europe Holdings PLC is a company registered in England, under
company number 06273215, with a registered office at 20 Triton
Street, London, NW1 3BF. The UK business of Ricoh Europe Holdings PLC
is operated by: (i) Ricoh Europe PLC, a company registered in England
under company number 00720944, with a registered office at 20 Triton
Street, London, NW1 3BF; (ii) Ricoh UK Limited, a company registered
in England under company number 01271033, with a registered office at
Ricoh House, 800 Pavilion Drive, Northampton, NN4 7YL; and (iii)
Ricoh Capital Limited, a company registered in England under company
number 03001351, with a registered office at 20 Triton Street,
London, NW1 3BF Please consider the environment before printing this
e-mail

Re: Detect Emoticons in Subject

[Tom Hendrikx]

On 20-05-2021 18:19, RW wrote:

On Thu, 20 May 2021 11:42:59 -0400
Clive Jacques wrote:


Hi,

I've been using SA a long time.  Lately, I'm getting more and more
spam with emoticons in the subject line.  I'd say about 90% of my
emails with emoticons in the subject are spam.  I'd like to create a
local rule which scores email with emoticons in the subject.



# Local Rule for Emoticons in subject
subjectEMOTICON_IN_SUBJECT  Subject =~ /\p{Emoticons}/


The rule should start with "header", that's what's causing the lint
failure.

However, AFAIK, the rule still won't work because \p{Emoticons}
isn't supported in spamassassin, which works on byte sequences. You
need to rewrite it to match UTF-8 bytes.



I'm not a real fan of very complex regular expressions, as they tend to 
get hard to read/understand very quickly. This thread is a perfect 
example: the syntax that the OP proposed (/\p{Emoticons}/) seems 
perfectly readable, and all the actually working alternatives are, with 
all respect to the authors, a nightmare to decipher. Especially for 
users not really proficient in regular expressions, the OP's syntax is 
perfectly understandable and all the alternatives aren't.


I'm not really into the regex engine of perl/SA, so please correct if 
I'm wrong. The /\p{Emoticons}/ syntax seems to me a builtin feature of 
the regex spec/perl (as opposed to pseudo-code, displaying something 
that actually doesn't exist).


Can someone explain why SA cannot support this type of syntax, or what 
would be needed to get it supported? IMHO it makes it a lot easier for 
end-users to understand a rule, and for rule developers to write or even 
contribute new UTF-8-related rules, so it might be worth the effort to 
get it supported?


Thanks in advance,
Tom

Re: Using spamassassin modules from a git repo

[Tom Hendrikx]

How about cloning outside your etc directory, for instance in 
/usr/local? And then adding the correct paths to local.cf, as usual.


Kind regards,
Tom

On 08-04-2021 11:05, Michael Grant wrote:

I'm running debian on my mail server.  I use etckeeper to track
changes in /etc.

Often I run across modules such as spamassassin-esp and maybe I would
consider playing with Jared Hall's CHAOS module.

I'm curious what the recommended best practice is to install such
modules from a git repo.

For spamassassin-esp, I cloned the repo into my /etc/spamassassin/
directory and then added this to my local.cf:

loadplugin Mail::SpamAssassin::Plugin::Esp spamassassin-esp/Esp.pm
include spamassassin-esp/Esp.cf

This allows me to 'git pull' from this repository from time to time to
update it.  But it's not perfect, especially as I have local changes
to Esp.cf.  It's actually worse since I forked it to give back some
changes but I'd say that's perhaps less usual.

Furthermore, as I said, I use etckeeper and when I 'apt upgrade', I get
constant warnings:

modified:   spamassassin/spamassassin-esp (modified content, untracked content)

So clearly it's not ideal to clone a spamassassin module into
/etc/spamassassin!

I'm curious if someone has a clean solution here that allows updating
the module from time to time from git.

I realize this may be more a debian question and I may post it on the
debian-users list if I don't get any decent replies here.

Michael Grant

Re: adding AV scanning to working Postfix/SA system

[Tom Hendrikx]

On 02-12-2020 16:18, Joe Acquisto-j4 wrote:

X-Spam-Virus: _CLAMAVRESULT


I never integrated Clam using this plugin, but this seems a config typo 
to be: there should be a Yes/No in there, and optionally a virus name.


Kind regards,

Tom

Re: Trusted network mail spam detection

[Tom Hendrikx]

On 16-10-19 12:19, Simon Wilson wrote:
Hi, I have a Horde system submitting to a 
postfix/amavisd-new/spamassassin server for spam detection (different 
servers, same subnet). I *do* consciously run SA over internally 
submitted emails to catch compromised accounts (it happened once to me 
when a family member's email password was compromised and a bunch of 
spam got sent out).


I'm having occasional issues with mail sent by some users from their 
home ISP connections (i.e. Chrome client on ISP dynamic IP -> Horde 
server/postfix etc). Email validly sent through the trusted host Horde 
server gets a bonus (ALL_TRUSTED = -2) which SA is triggering fine when 
appropriate, but some emails are still triggering thresholds, so I was 
wondering what others do for configuring for traffic that is *mostly* 
trusted but should still be checked for obvious spam?


This is not a new system, it's well trained with thousands of ham and 
spam over several years. This email was genuine ham, and was discarded 
(Amavis threshold 6.0 -> discard).


Content analysis details:   (6.0 points, 6.2 required)

  pts rule name  description
 -- 
--

-2.0 ALL_TRUSTED    Passed through trusted hosts only via SMTP
  0.8 BAYES_50   BODY: Bayes spam probability is 40 to 60%
     [score: 0.5000]
  0.0 SPF_HELO_NONE  SPF: HELO does not publish an SPF Record
  0.0 HTML_MESSAGE   BODY: HTML included in message
  3.6 HELO_DYNAMIC_IPADDR2   Relay HELO'd using suspicious hostname (IP 
addr

     2)
  1.0 RDNS_DYNAMIC   Delivered to internal network by host with
     dynamic-looking rDNS
  2.1 TO_NO_BRKTS_DYNIP  To: lacks brackets and dynamic rDNS
  0.5 NO_FM_NAME_IP_HOSTN    No From name + hostname using IP address

It feels like the dynamic IP rules are killing it here - what do others 
do for valid dynamic IP emails inbound from a web client email through 
trusted hosts? Just give ALL_TRUSTED more of a boost? Or anything more 
scientific??


The default rule scores obviously don't apply for your use case here: 
dynamic RDNS is to be expected for the relayed emails you are scanning.


Also it is not an indicator that the sender is abusing a (hacked) 
end-user host. So you should adjust the scores of the rules that are not 
applicable for your use case:


score RDNS_DYNAMIC 0.001
score HELO_DYNAMIC_IPADDR2 0.001

Something to note: RDNS_DYNAMIC tries to exclude authenticated email. 
Are you accepting email from senders without authentication? Or maybe 
your trusted_networks/internal_networks are misconfigured, so the 
authentication is not properly detected?


Kind regards,
Tom

Re: Rule for detecting two email addresses in From: field.

[Tom Hendrikx]

On 04-10-19 04:31, Bill Cole wrote:

On 3 Oct 2019, at 20:01, Rick Cooper wrote:


Philip wrote:

Morning List,

Lately I'm getting a bunch of emails that are showing up with two
email addresses in the From: field.

From: "Persons Name " 

When you look in your mail client (Outlook, Thunderbird) it's showing
only "Persons Name "

Is there a way I can mark From: that has 2 email addresses in it as
spam? Pro's Cons?

Phil


From: =~ /^.*?<.+?\@.+?>.*?<.+\@.+?>/g

Can't imagine the circumstance where such a from: format would be 
required


I've seen it used as a perfectly reasonable workaround for the 
misfeature described above of many MUAs of hiding the address field in 
To/From/CC headers. Because many people actually want to know what the 
actual address is.





I would disagree on the "reasonable" here. People using a mailclient 
should configure it as they wish. My client hides email addresses for 
everyone in my address book, but not for 'unknown' addresses.


That is how I like it, and I don't think senders should try to enforce a 
workaround for this because their recipients are too stupid to configure 
their email client (or switch to a decent one).


Anyway, the main harm is done when the email adresses in the 'addr' 
field and the 'name' are different, and that's detectable.


Kind regards,
Tom

Open source (WAS: Spam rule for HTTP/HTTPS request to sender's root domain)

[Tom Hendrikx]

On 20-03-19 19:56, Mike Marynowski wrote:
> 
> A couple people asked about me posting the code/service so they could
> run it on their own systems but I'm currently leaning away from that. I
> don't think there is any benefit to doing that instead of just utilizing
> the centralized service. The whole thing works better if everyone using
> it queries a central service and helps avoid people easily making bad
> mistakes like the one above and then spending hours scrambling to try to
> find non-existent botnet infections on their network while mail bounces
> because they are on a blocklisted :( If someone has a good reason for
> making the service locally installable let me know though, haha.

When people are interested in seeing the code, their main incentive for
such a request is probably not that they want to run it themselves. They
might, in no particular order:

- would like to learn from what you're doing
- would like to see how you're treating their contributed data
- would like to verify the listing policy that you're proposing
- would like to study if there could be better criteria for
listing/unlisting than the ones currently available
- change things to the software and contribute that back for the
benefit of everyone
- squash bugs that you're currently might be missing
- help out on further development of the service if or when your time is
limited
- don't be depending on a single person to maintain a service they like

This is called open source, and it's a good thing. For details on the
philosophy behind it,
http://www.catb.org/~esr/writings/cathedral-bazaar/cathedral-bazaar/ is
a good read.

In short: if you like your project to prosper, put it on github for
everyone to see.

Kind regards,

Tom



signature.asc
Description: OpenPGP digital signature

Re: New type of SPAM aggression

[Tom Hendrikx]

Hi,

Anyone can start a DNSBL and list IP space of people they don't like, as 
you surely know. As long as no one uses such a DNSBL to block traffic, 
no harm is done.


The interesting part is which "engines" (I guess that you mean antispam 
software or antispam saas providers) think that such a DNSBL should be 
actually used. Can you disclose which parties you found?


Kind regards,

Tom

On 06-02-19 14:40, Rupert Gallagher wrote:
The spammers at gremlin.ru have just created a homepage, with no 
information on how to delist an IP.


Their fake dnsbl is listed as genuine in at least two antispam engines.


On Wed, Feb 6, 2019 at 12:55, Rupert Gallagher > wrote:

This is to inform about a new type of SPAM aggression.

We received from Russia, for months, and redirected them automatically 
to an administrative address for manual inspection. All emails were 
spam with links. From the standpoint of the attacker(s), all emails 
were delivered, but none turned into exploits.


Today, we learned that "gremlin.ru" included our IPs in their DNSBL. 
We followed the address to de-list, but gremlin.ru does not exist.


So, if you are successful against Russian spam, you will be ... 
blacklisted by an unknown gremlin.

Re: RBL

[Tom Hendrikx]

On 10-10-18 21:51, Tom Hendrikx wrote:
> On 10-10-18 21:05, Gokan Atmaca wrote:
>> Hello
>>
>> I'm using Postfix and Dovecot. I use Spamassassin as an antispam
>> service. I don't know how to do RBL checks.
>> How do I control RBL? (I don't want to do it with Postfix, because I
>> don't want to do two different whitelists.)
>>
>> Thanks.
>>
> 
> Hi,
> 
> See: http://lmgtfy.com/?q=postfix+rbl
> 
> Please ask further questions on the Postfix mailinglist, as you have a
> postfix question, and this is the spamassassin list.
> 

Excuse me, that was read and replied too quick. You can find more
information about using DNSBLs in spamassasin at:
https://wiki.apache.org/spamassassin/DnsBlocklists.

However, in general it's better to use DNSBLs at the MTA level, which
uses a lot less resources than implementing them in Spamassassin. So try
and set them up in postfix first.


Kind regards,
Tom

Re: RBL

[Tom Hendrikx]

On 10-10-18 21:05, Gokan Atmaca wrote:
> Hello
> 
> I'm using Postfix and Dovecot. I use Spamassassin as an antispam
> service. I don't know how to do RBL checks.
> How do I control RBL? (I don't want to do it with Postfix, because I
> don't want to do two different whitelists.)
> 
> Thanks.
> 

Hi,

See: http://lmgtfy.com/?q=postfix+rbl

Please ask further questions on the Postfix mailinglist, as you have a
postfix question, and this is the spamassassin list.

Kind regards,

Tom

Re: No rule updates since 1/1/17

[Tom Hendrikx]

Hi David,

I'm already running masschecks since feb 2017, results labeled
'thendrikx' are mine. :)

I'm not adding massive volumes though, mostly because I'm running a
small '3 men and a dog' setup. But I think it's important that I can
contribute sample data in my locale (nl_NL), so I would invite others to
set it up too: It's not a lot of work and it mostly runs without any
manual intervention (I was already manually sorting ham and spam).

To give a bit of an idea of how I do it: I run a postfix server on
ubuntu, with spamassassin as a milter. I redirect all possible spam into
my Junk folder, and check that daily.

The masscheck is run using a simple wrapper script that takes the
following steps (from daily cron):
- Copy all spam in $workdir from Spamtraps and Junk folders (only
IMAP-seen emails) and not older than 2 months
- Copy all ham into $workdir from several IMAP folders that are known to
be sorted by hand, and not older than 6 years
- Run masscheck on the copied messages
- Print a list of the subjects of the lowest scoring spam samples, and
the highest scoring ham samples
- Cleanup all copied email
- Mail all output to myself

I spent less than a day in setting this up, and it has been running
without issues ever since. When you're interested, read up on
https://wiki.apache.org/spamassassin/NightlyMassCheck and try to set it
up. If you run into issues, other masscheckers can probably help you out.

Kind regards,
Tom

On 25-08-18 16:12, David Jones wrote:
> Tom,
> 
> Let me know if you are still interested in setting up a masschecker. 
> That goes for anyone on this list as well.  I have worked out the
> sorting issue pretty well now and my ena-weekX masscheckers are now the
> largest contributions to the RuleQA corpus keeping the nightly rule
> scoring updating regularly the past year.
> 
> http://ruleqa.spamassassin.org/  (see the ena-weekX in the green box)
> 
> New/more masscheckers are always welcome and will help you learn the
> best way to tune your SA platform to get every last drop of accuracy
> from your local meta rules.  We could really use masscheckers with
> primary languages not English to add/improve core SA rules.
> 
> Here's my setup:
> 
> - I have an iRedmail server that I split copies of most of my email to
> an internal-only email domain "sa.ena.net."
> 
> - The iRedmail server has Sieve rules (easily managed by RoundCube)
> based on certain rule hits and scores from my main Internet edge
> MailScanner filtering that move them into Ham and Spam folders as
> unread.  Mail scoring in the middle -- not high enough for obvious Spam
> or low enough for obvious Ham are left in the main Inbox.
> 
> - I spend a few minutes each day visually scanning the Subjects of the
> unread email then mark them as Read.
> 
> - If I find a zero-hour email in the main Inbox, then I move it to a
> SpamCop folder.  A script that runs every 5 minutes to check the SpamCop
> folder, strips of some extra Received headers from my internal hops,
> then submits it as an attachment to my SpamCop account.
> 
> - A script moves the Maildir email to 4 other masschecker VMs to split
> out the load so they will be able to submit their results quickly. 
> Ena-week0 is the last week of ham/spam that is still on the iRedMail
> server.  Ena-week1-4 are running on the other 4 masschecker VMs to give
> a total of 5 weeks of recent corpus.  I currently have 100,939 Ham and
> 292,001 Spam in ena-week0-4.
> 
> - I run a local Bayesian train on the ena-week0 Ham and Spam folder to
> my Redis-based Bayes storage shared across my 8 MailScanner nodes and my
> iRedMail/amavis server.  This method has shown to keep my Bayes scores
> very accurate.
> 
> Hope someone finds this information helpful.
> 
> Dave
> 
> 
> On 01/20/2017 01:02 PM, Tom Hendrikx wrote:
>> On 20-01-17 19:46, David Jones wrote:
>>>> From: Kevin Golding 
>>>> Sent: Friday, January 20, 2017 11:59 AM
>>>> To: users@spamassassin.apache.org
>>>> Subject: Re: No rule updates since 1/1/17
>>> 
>>>> On Fri, 20 Jan 2017 17:26:01 -, Bill Keenan  
>>>>  wrote:
>>>>> What is the fix needed so /usr/bin/sa-update starts getting updates? I  
>>>>> too have not received an update from updates.spamassassin.org  
>>>>> <http://updates.spamassassin.org/> since 1-Jan-17.
>>>>>
>>>>> Besides updates.spamassassin.org <http://updates.spamassassin.org/>, 
>>>>> what other rule sets are commonly used? Hundreds of spam messages are  
>>>>> getting through with only updates.spamassassin.org  
>>>>> <http://updates.spamassassin.org/> rules.
>>>> This seems

Re: Remove SA tagging when learning as ham

[Tom Hendrikx]

Hi,

"Moving out of the Junk folder" definitely sounds like IMAP. In the IMAP
standard, messages can't be changed after delivery. To alter the message
(change subject, remove headers), you'll need to delete the old message,
and create a new, altered message. This is bad for caching, and could
mess up your MUA because you might delete a message serverside when the
client is interacting with that same message.

When you don't want to see the result of Junk filtering in you MUA,
don't tag the subject, and do everything based on the SA headers. The
message is Spam when it ends up in your Spam folder, otherwise it's not.
And when moving around the message, you don't end up with non-spam
messages that have a spam tag in the subject (because you never added one).

Kind regards,

Tom

On 18-06-18 14:22, Kevin A. McGrail wrote:
> I'd look
> at https://serverfault.com/questions/817928/procmailrc-change-email-subject
> 
> --
> Kevin A. McGrail
> VP Fundraising, Apache Software Foundation
> Chair Emeritus Apache SpamAssassin Project
> https://www.linkedin.com/in/kmcgrail - 703.798.0171
> 
> On Mon, Jun 18, 2018 at 8:13 AM, @lbutlr  > wrote:
> 
> I have a script that runs when a mail is moved out of the Junk
> folder to pass the mail through sa-learn --ham, but it doesn’t
> removed the subject tagging (Spam: 05.5) nor does it remove the
> X-Spam-Flag header.
> 
> What would I need to do in the script to remove the SA tags on
> messages that are processed by this script?
> 
> -- 
> Stone circles were common enough everywhere in the mountains. Druids
> built them as weather computers, and since it was always cheaper to
> build a new 33-Megalith circle than to upgrade an old slow one, there
> were generally plenty of ancient ones around --Lords and Ladies
> 
> 

Re: how to remove T_RP_MATCHES_RCVD

[Tom Hendrikx]

On 05-04-18 18:40, Motty Cruz wrote:
> Thanks for your prompt reply John,
> 
> X-Spam-Status: No, score=5.27 tagged_above=-999.9 required=5.7
>     tests=[BAYES_50=4.3, FROM_EXCESS_BASE64=0.979, HTML_MESSAGE=0.001,
>     T_RP_MATCHES_RCVD=-0.01] autolearn=no autolearn_force=no
> 

BAYES_00 means 'pretty sure it's ham'.
BAYES_99 means 'pretty sure it's spam'.
BAYES_50 means 'no idea'.

Scoring BAYES_50 at 4.3 is your scoring issue, nothing's wrong with
T_RP_MATCHES_RCVD.

Kind regards,
Tom


> always the score is -0.01 regardless; I will take your suggestion and
> set it to 0.01, will report back shortly.
> 
> Thanks,
> 
> 
> On 04/05/2018 09:32 AM, John Hardin wrote:
>> On Thu, 5 Apr 2018, Motty Cruz wrote:
>>
>>> Hello, T_RP_MATCHES_RCVD  this rule is allowing spammy emails past
>>> through. Is there a way to disable in local.cf?
>>
>> The best way to disable it without breaking any meta-rules that may be
>> using it is to set its score to 0.001 in your local config file.
>>
>> I don't see a score for it in the latest rules update, so it should by
>> default be *adding* one point to scores, which won't contribute to FNs.
>>
>> What is it currently scored in your environment?
>>
>> It is, however, used as a suppressor subrule in some spam meta-rules.
>> Is that why it's causing FNs for you?
>>
> 




signature.asc
Description: OpenPGP digital signature

Re: bypass milter but not the test.

[Tom Hendrikx]

Hi,

Sounds like a mimedefang question, not a spamassassin one. But did you
restart mimedefang after adding the rule?

Kind regards,
Tom

On 04-04-18 15:16, saqariden wrote:
> Hello everybody,
> 
> I'm using spamassassin with mimedefang, i have some custom rulesets, one
> of those match when i test the positifs mail through "spamassassin -t",
> but the mail still bypassing the milter.
> 
> the rule:
> header  FR_SHORT_SPAM_H Subject =~
> /((\(|\[)\d{3,6}(\)|\]))/
> describe    FR_SHORT_SPAM_H Subject with digits
> score   FR_SHORT_SPAM_H 3
> 
> --
> result of spamassassin -t:
>  3.0 FR_SHORT_SPAM_H    Subject with digits]
>  1.5 FREEMAIL_FROM  Sender email is commonly abused enduser mail
> provider
>  0.7 SPF_SOFTFAIL   SPF: sender does not match SPF record
> (softfail)
>  0.0 DKIM_ADSP_CUSTOM_MED   No valid author signature, adsp_override is
>     CUSTOM_MED
> -0.2 RCVD_IN_MSPIKE_H2  RBL: Average reputation (+2)
> --
> 
> mimedefang log:
> Apr  4 14:39:55 vml224antispam-03 mimedefang.pl[14378]: w34CdrxJ026146:
> MDLOG,w34CdrxJ026146,mail_in,,,,,(3750)
> test
> 
> the others custom rules work fine, what can be the problem please ?
> 
> Signature Academique

Re: Junk mixed in with ham on whitelists

[Tom Hendrikx]

On 21-02-18 14:54, David Jones wrote:
> On 02/21/2018 07:44 AM, Kevin A. McGrail wrote:
>> On 2/21/2018 8:42 AM, David Jones wrote:
>>> Do we need to open a bug to get SA's DKIM code to check for a minimum
>>> key size? 
>>
>> When in doubt, open a bug.
>>
> 
> Well. Ummm.  I found this when starting to create the bug:
> 
> https://bz.apache.org/SpamAssassin/show_bug.cgi?id=7124
> 

Note that the patch in that PR sets the default to 512 bits, and not to
1024 as proposed in the bug, and which is what would be a current sane
default. Not sure about the actual default in current SA code though.

Kind regards,

Tom

Re: Report AmazonSES spam?

[Tom Hendrikx]

On 21-02-18 13:34, @lbutlr wrote:
> I've been trying to find a way to report a spammer to Amazon SES (Simple 
> Email Service), but I haven't found anywhere to report this spam.
> 
> (SA is tagging the messages, but I'm tired of Amazon allowing this company to 
> continue doing this).
> 
> X-Spam-Status: Yes, score=7.3 required=5.0 tests=BAYES_95,DKIM_SIGNED,
>   HEADER_FROM_DIFFERENT_DOMAINS,HTML_FONT_LOW_CONTRAST,HTML_MESSAGE,
>   
> KAM_LOTSOFHASH,MIME_HEADER_CTYPE_ONLY,RAZOR2_CF_RANGE_51_100,RAZOR2_CHECK,
>   RCVD_IN_DNSWL_NONE,RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL,SPF_PASS,
>   T_DKIM_INVALID,T_RP_MATCHES_RCVD,URIBL_BLACK autolearn=no 
> autolearn_force=no
>   version=3.4.1

How about: https://aws.amazon.com/forms/report-abuse

Kind regards,

Tom

Re: Email filtering theory and the definition of spam

[Tom Hendrikx]

On 08-02-18 16:33, Giovanni Bechis wrote:
> On 02/08/18 16:23, David Jones wrote:
>> On 02/07/2018 06:28 PM, Dave Warren wrote:
>>> On Wed, Feb 7, 2018, at 15:52, Martin Gregorie wrote:
> Technically, you asked for the email and they have a valid opt-out
> process that will stop sending you email.  Yes, the site has scummy
> practices but that is not spam by my definition.
>
 Yes, under EU/UK that counts as spam because the regulations say that
 the signer-upper must explicitly choose to receive e-mail from the
 site, and by-default sign-in doesn't count as 'informed sign-in'.
>>>
>>> Canadian law is the same, this is absolutely spam without any ambiguity.
>>>
>>
>> But how can you tell the difference based on content then?  You can't. Two 
>> different senders could send the exact same email and one could be spam from 
>> tricking the recipient to opt-in and another could be ham the recipient 
>> consciously opted into.
>>
>> This would have to be blocked or allowed based on reputation.  One would 
>> train the message as spam in their Bayes database and allow trusted senders 
>> via something like a domain whitelist, URI whitelist, or a whitelist_auth 
>> entry.
>>
>> We are back to needing a curated WL based on something like DKIM.  Alex just 
>> made me aware of http://dkimwl.org/ which looks brilliant.  Exactly lines up 
>> with how I filter and what I have been wanted to do for a couple of years 
>> now.  A community-driven clearing house for trusted senders.
>>
> dkimwl.org looks promising, but tell them their https cert has expired.
>  Giovanni 
> 

Also, they refer to the TOU for acceptable usage, but both /terms and
/license have a 404.

Kind regards,

Tom



signature.asc
Description: OpenPGP digital signature

Re: dns-blocklist aren't used but should be

[Tom Hendrikx]

On 07-01-18 16:26, Jan Klein wrote:
> Hi.
> 
> For work I am investigating an issue where none of the dns blacklists
> are used.
> We are using the current spamassassin version and also current version
> of Net::DNS.

We can't say a thing about version 'curent'. Please state full versions
of spamassassin, Net::DNS and debian. Also, tell us how you installed
spamassassin: are these distro packages or a manual install (if so: tell
us how you did that).

> 
> It is installed on a current version debian system.
> We run a local nameserver using bind.
> We invoke spamassassin via "spamassassin -t < testmail" where testmail
> is a spam mail.

Please post full output of  'spamassassin -t -D < testmail'

> 
> The weird thing is that a "dig" command works fine on the debian system,
> so name resolving is actually working outside of spamassassin. And after
> using the dig command to check the origin of the mail: dig
> xxx.xxx.xxx.xxx.zen.spamhaus.org
> Then after using that command, spamassassin will then consider spamhaus
> when checking the testmail. Probably because the dns entry is cached for
> a while or something. It will work for some minutes. Same thing with
> other blacklists. After a dig command spamassassin will start using the
> respective rule.
> 
> What is going on? It seems to be DNS related. I've read that Net::DNS is
> responsible for dns resolving for spamassassin. How can I check if it is
> working correctly? In my /etc/resolv.conf there is only one entry:
> 127.0.0.1 since we are running a local nameserver (again: dig or host
> command work just fine for name resolving ).
> 

What dns related settings do you have configured in /etc/spamassassin?

Please show values for:
- dns_available
- dns_server
- clear_dns_servers
- dns_local_ports_permit
- dns_local_ports_avoid
- dns_local_ports_none
- dns_options
- dns_query_restriction
- clear_dns_query_restriction
- rbl_timeout

You can find the meaning of these (and many other) configuration options
at:
https://spamassassin.apache.org/full/3.4.x/doc/Mail_SpamAssassin_Conf.html

Kind regards,

Tom



signature.asc
Description: OpenPGP digital signature

Re: help with phishing email?

[Tom Hendrikx]

On 08-12-17 19:09, AJ Weber wrote:
> I'm trying to decide the best way to detect something like this.
> 
> https://pastebin.com/hCX9MWNg
> 
> Looking at the raw headers and body it's pretty easy to tell this is a
> spoof, but when it shows-up in an inbox, it looks pretty good.
> 
> Something specific to Amazon (where this is purported to come from)
> would be to check if their domain is in the From and Reply-To and at
> least score that relatively high if it's not correct - but compared to
> what?  Maybe if From text contains amazon/i and from-address does not
> end with amazon.com (for me in the US at least)?
> 
> That feels forced.  Does anyone have any suggestions to help me out on
> this fine Friday?
> 

Actual Amazon email is always sent with passing SPF, DKIM and DMARC. SO
you can easily whitelist anything from amazon based on that, and then
subtract some points for everything that has '\bAmazon\b' is the
from:name. Header.

Kind regards,
Tom




signature.asc
Description: OpenPGP digital signature

Re: Rule to match when multiple FROM addresses exist

[Tom Hendrikx]

On 01-12-17 14:15, RW wrote:
> On Fri, 1 Dec 2017 12:01:35 +0100
> Simeon Ott wrote:
> 
>> Hi
>>
>> Occasionally I get spam mails with non-quoted display names like 
>>
>> John, Doe, Lastname > >
>>
>> My MTA (Postfix) thinks this are multiple FROM addresses and adds my
>> local servername to John and Doe. Spamassassin gets the forwarded
>> Mail with a From Header like:
>>
>> From: John@localservername, Doe@localservername, Example
>> >
>>
>> Any suggestion how-to match this kind of From-Headers? 
> 
> Does Postfix keep the the original From header with a rewritten
> header name? If there's an Original-From or similar it would be better
> to detect the original problem rather than a side effect.
> 
> If that mailto thing repeats, I'd go after that too. Maybe 
> 
> header   ...   From =~ />\s*>\s*$/
> 

You're mistaken about postfix. It does not rewrite the From headers in
the way you describe, unless you explicitly configured it to. You should
change your postfix configuration, or verify the input data that postfix
receives (maybe the addresses were already malformed before they entered
your system?

The easy way to catch these messages is to set your machine name to some
domain that never receives mail.

email addresses: john@company.tld
server domain: mailserver.company.tld

Automatic generated addresses would look like
john@mailserver.company.tld, these are easily recogized because
nobody uses them.

Kind regards,
Tom

Re: Ruleset updates via nightly masscheck status

[Tom Hendrikx]

On 28-10-17 15:20, David Jones wrote:
> On 10/27/2017 03:02 AM, Merijn van den Kroonenberg wrote:
>>

 Please provide feedback in the next 48 hours -- positive or negative so
 I know we are good to enable DNS updates again on Sunday.

>>>
>>> After installing these rules, I'm seeing one warning in my log during
>>> spamassassin reload:
>>>
>>> Oct 27 09:48:24 myhostname spamd[16256]: rules: failed to run
>>> DKIM_VALID_EF test, skipping:
>>> Oct 27 09:48:24 myhostname spamd[16256]:  (Can't locate object method
>>> "check_dkim_valid_envelopefrom" via package "Mail:
>>> [...]:SpamAssassin::PerMsgStatus" at (eval 1369) line 305.
>>> Oct 27 09:48:24 myhostname spamd[16256]: )
>>
>> The DKIM_VALID_EF rule should not be published yet as it depends on a
>> change in a Plugin.
>>
> 
> Tom, thank you for testing and providing feedback.  I didn't notice this
> error because I had patched my DKIM.pm plugin for testing the new
> DKIM_VALID_EF rule (intended to be used in meta rules).  I confirmed
> what you found on my default Fedora 26 installation.
> 
> I have fixed the rulesets, specifically 25_dkim.cf and 50_scores.cf, to
> check for the SA version to remove this error and tested it.  Monday's
> ruleset should have this fix after tomorrow's masscheck validates it.
> 
> I will confirm Monday's ruleset has fixed this DKIM_VALID_EF error and
> let sa-update start updating again via DNS on Tuesday.
> 
> If anyone else is testing the latest rulesets from the past couple of
> days, please provide feedback in the next 48 hours.  And thank you for
> testing.

Hi,

I noticed that rule updates are still not live in DNS. Can I get an
updated ruleset for additional testing somewhere, or are we going live?

Regards,
Tom

Re: Ruleset updates via nightly masscheck status

[Tom Hendrikx]

On 26-10-17 20:33, David Jones wrote:
> On 10/26/2017 01:09 PM, David Jones wrote:
>> On 10/25/2017 06:15 AM, David Jones wrote:
>>> cd /tmp
>>> wget http://sa-update.ena.com/1813149.tar.gz
>>> wget http://sa-update.ena.com/1813149.tar.gz.sha1
>>> wget http://sa-update.ena.com/1813149.tar.gz.asc
>>> sa-update -v --install 1813149.tar.gz
> 
> Last night's run also successfully put the last known good 72_scores.cf
> from March into the ruleset.
> 
> Steps to manually installing last night's ruleset:
> 
> cd /tmp
> wget http://sa-update.ena.com/1813258.tar.gz
> wget http://sa-update.ena.com/1813258.tar.gz.sha1
> wget http://sa-update.ena.com/1813258.tar.gz.asc
> sa-update -v --install 1813258.tar.gz
> 
> restart spamd, MailScanner, amavisd, mimedefang, etc.
> 
> Please provide feedback in the next 48 hours -- positive or negative so
> I know we are good to enable DNS updates again on Sunday.
> 

After installing these rules, I'm seeing one warning in my log during
spamassassin reload:

Oct 27 09:48:24 myhostname spamd[16256]: rules: failed to run
DKIM_VALID_EF test, skipping:
Oct 27 09:48:24 myhostname spamd[16256]:  (Can't locate object method
"check_dkim_valid_envelopefrom" via package "Mail:
[...]:SpamAssassin::PerMsgStatus" at (eval 1369) line 305.
Oct 27 09:48:24 myhostname spamd[16256]: )

Any idea? This is ubuntu 16.04, latest ubuntu package (3.4.1-3) for
spammassassin.


Kind regards,
Tom

Re: URIBL_BLOCKED - which one?

[Tom Hendrikx]

Hi,

Note that on at least Ubuntu from some time ago, unbound was
automatically configured to take the dns servers that were received from
an upstream server during DHCP, and configure those as forwarders.

Can you show us output of: unbound-control list_forwards

Kind regards,
Tom

On 13-10-17 18:59, John Hardin wrote:
> 
> I just want to call this out as the critical detail in all the
> back-and-forth:
> 
>> The main thing with setting up a DNS server for DNSBL lookups is not
>> "caching", it is "non-forwarding".  Take a look at your unbound
>> settings and make sure it is doing all of the lookups itself and not
>> forwarding to another server.
> 




signature.asc
Description: OpenPGP digital signature

Re: Bayes auto-learn - not happening, tentative success....

[Tom Hendrikx]

xOn 11-08-17 17:05, Scott wrote:
> I'm going to go back and look at my build notes but I think that directory
> got created for me. It's just as possible i followed some "guide".  I am
> positive i did not think it up on my own LOL.   I remember more than set of
> instructions one with that path setting, and it very well could be the
> related Centos7 package.  Glad i found the casue though. Regardless of the
> source. 
> 
> In the FWIW department, as shown above, I still don't have it in the default
> location (I know, risks...), but why it is happy there and not under /etc I
> don't know.  And really don't care at this point.   
> 

I had to go way back in thread to look it up, but I noticed you're
running Centos, which has selinux.

Maybe your custom path is disallowed under the amavis/spamd/whatever
role? And manual testing when su'ing from the root role will not have
the same impact as running amavis using an init system.

Kind regards,
Tom



signature.asc
Description: OpenPGP digital signature

Re: reason why sendmail w/ SA3.4.1 scantime=15.0, delay=00:01:06 w/ SquirrelMail?

[Tom Hendrikx]

On 17-07-17 16:39, Robert Kudyba wrote:
> 
>> On Jul 17, 2017, at 10:28 AM, Tom Hendrikx <t...@whyscream.net
>> <mailto:t...@whyscream.net>> wrote:
>>
>> On 17-07-17 16:00, Robert Kudyba wrote:
>>>
>>>> On Jul 17, 2017, at 9:39 AM, Antony Stone
>>>> <antony.st...@spamassassin.open.source.it
>>>> <mailto:antony.st...@spamassassin.open.source.it>
>>>> <mailto:antony.st...@spamassassin.open.source.it>> wrote:
>>>>
>>>> On Monday 17 July 2017 at 14:25:17, Robert Kudyba wrote:
>>>>
>>>>>> On Jul 14, 2017, at 4:00 AM, Matus UHLAR - fantomas
>>>>>> <uh...@fantomas.sk <mailto:uh...@fantomas.sk>
>>>>>> <mailto:uh...@fantomas.sk>>
>>>> wrote:
>>>>>>> Robert Kudyba <rkud...@fordham.edu <mailto:rkud...@fordham.edu>
>>>>>>> <mailto:rkud...@fordham.edu>> wrote:
>>>>>>>> Over the past few days sending mail via SquirrelMail has become
>>>>>>>> glacial. The load on the server is under 1. I've restarted the SA,
>>>>>>>> sendmail and dovecot processes several times. Here are some logs
>>>>>>>> I can
>>>>>>>> provide any settings if desired.
>>>>>>
>>>>>> tried to run a message through "spamassassin -D" ?
>>>>>> that should give you debug/timing info.
>>>>>
>>>>> OK here is the pastebin of spamassassin -D < gtube.txt:
>>>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__pastebin.com_iZtm2hhy=DwIFAw=aqMfXOEvEJQh2iQMCb7Wy8l0sPnURkcqADc2guUW8IM=X0jL9y0sL4r4iU_qVtR3lLNo4tOL1ry_m7-psV3GejY=wV3-oZ_3m8NtSuw_6UTtdU1WptL8Pl1vNOok-EXrcZo=802-414zeT59KVCIFVa_uxfSq0XezT7e4OVZibWbIwc=
>>>>>
>>>>
>>>>
>>>> Jul 16 09:01:42.796 [29903] dbg: dns: entering helper-app run mode
>>>> Jul 16 09:01:47.806 [29903] dbg: dns: leaving helper-app run mode
>>>> Jul 16 09:01:47.806 [29903] dbg: razor2: razor2 check timed out after 5
>>>> seconds
>>>
>>> OK so I ran: /var/spool/amavisd/.razor
>>>
>>> ls -l /var/spool/amavisd/.razor
>>> total 100
>>> -rw-r- 1 amavis amavis 72420 Dec 22  2014 razor-agent.log
>>> -rw-r- 1 amavis amavis   998 Jul 17 09:49
>>> server.c301.cloudmark.com.conf
>>> -rw-r- 1 amavis amavis   998 Jul 17 09:46
>>> server.c302.cloudmark.com.conf
>>> -rw-r- 1 amavis amavis   995 Dec 20  2014
>>> server.c303.cloudmark.com.conf
>>> -rw-r- 1 amavis amavis57 Jul 17 09:49 servers.catalogue.lst
>>> -rw-r- 1 amavis amavis30 May 23  2013 servers.discovery.lst
>>> -rw-r- 1 amavis amavis76 Jul 17 09:49 servers.nomination.lst
>>>
>>> New pastebin:
>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__pastebin.com_9RWEYuSt=DwIC-g=aqMfXOEvEJQh2iQMCb7Wy8l0sPnURkcqADc2guUW8IM=X0jL9y0sL4r4iU_qVtR3lLNo4tOL1ry_m7-psV3GejY=QspHQBi1X_n1ZQylsERsyborPsWRSy3cHQlXJ8FUf7c=DZ63JGDSr9nTI6HaajZtLRvUf0ao4tBA4dKtq_77Xlg=
>>>
>>>
>>> Still taking 15 seconds.
>>>
>>> Jul 17 09:55:28 storm spamd[28111]: spamd: clean message (-103.4/5.0)
>>> for spamd:1001 in 15.0 seconds, 1843 bytes.
>>> Jul 17 09:55:28 storm spamd[28111]: spamd: result: . -103 -
>>> ALL_TRUSTED,BAYES_00,FROM_IS_TO,USER_IN_WHITELIST
>>> scantime=15.0,size=1843,user=spamd,uid=1001,required_score=5.0,rhost=localhost,raddr=::1,rport=53074,mid=<32889a456ed9c9911ff0034513796858.squirrel@ourdomain>,bayes=0.00,autolearn=no
>>> autolearn_force=no
>>> Jul 17 09:55:28 storm spamd[28041]: prefork: child states: II
>>>
>>
>> The error is still the same. Do you even have access to those cloudmark
>> razor servers? Does razor work outside of spamassassin/amavisd?
> 
> Is that supposed to be a paid service? This test seems successful. 
> 
> razor-check -d <   /usr/share/doc/spamassassin/sample-spam.txt
>  Razor-Log: Computed razorhome from env: /root/.razor
>  Razor-Log: Found razorhome: /root/.razor
>  Razor-Log: read_file: 15 items read from /root/.razor/razor-agent.conf

Note that the paths are different when you run this as root. Does it
also work when you run it as user amavisd? And don't bother posting the
full log unless you're unsure about the actual outcome. You can read
them yourself.

Re: reason why sendmail w/ SA3.4.1 scantime=15.0, delay=00:01:06 w/ SquirrelMail?

[Tom Hendrikx]

On 17-07-17 16:00, Robert Kudyba wrote:
> 
>> On Jul 17, 2017, at 9:39 AM, Antony Stone
>> > > wrote:
>>
>> On Monday 17 July 2017 at 14:25:17, Robert Kudyba wrote:
>>
 On Jul 14, 2017, at 4:00 AM, Matus UHLAR - fantomas
 >
>> wrote:
> Robert Kudyba > wrote:
>> Over the past few days sending mail via SquirrelMail has become
>> glacial. The load on the server is under 1. I've restarted the SA,
>> sendmail and dovecot processes several times. Here are some logs I can
>> provide any settings if desired.

 tried to run a message through "spamassassin -D" ?
 that should give you debug/timing info.
>>>
>>> OK here is the pastebin of spamassassin -D < gtube.txt:
>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__pastebin.com_iZtm2hhy=DwIFAw=aqMfXOEvEJQh2iQMCb7Wy8l0sPnURkcqADc2guUW8IM=X0jL9y0sL4r4iU_qVtR3lLNo4tOL1ry_m7-psV3GejY=wV3-oZ_3m8NtSuw_6UTtdU1WptL8Pl1vNOok-EXrcZo=802-414zeT59KVCIFVa_uxfSq0XezT7e4OVZibWbIwc=
>>>
>>
>>
>> Jul 16 09:01:42.796 [29903] dbg: dns: entering helper-app run mode
>> Jul 16 09:01:47.806 [29903] dbg: dns: leaving helper-app run mode
>> Jul 16 09:01:47.806 [29903] dbg: razor2: razor2 check timed out after 5
>> seconds
> 
> OK so I ran: /var/spool/amavisd/.razor
> 
> ls -l /var/spool/amavisd/.razor
> total 100
> -rw-r- 1 amavis amavis 72420 Dec 22  2014 razor-agent.log
> -rw-r- 1 amavis amavis   998 Jul 17 09:49 server.c301.cloudmark.com.conf
> -rw-r- 1 amavis amavis   998 Jul 17 09:46 server.c302.cloudmark.com.conf
> -rw-r- 1 amavis amavis   995 Dec 20  2014 server.c303.cloudmark.com.conf
> -rw-r- 1 amavis amavis57 Jul 17 09:49 servers.catalogue.lst
> -rw-r- 1 amavis amavis30 May 23  2013 servers.discovery.lst
> -rw-r- 1 amavis amavis76 Jul 17 09:49 servers.nomination.lst
> 
> New pastebin: https://pastebin.com/9RWEYuSt
> 
> Still taking 15 seconds.
> 
> Jul 17 09:55:28 storm spamd[28111]: spamd: clean message (-103.4/5.0)
> for spamd:1001 in 15.0 seconds, 1843 bytes.
> Jul 17 09:55:28 storm spamd[28111]: spamd: result: . -103 -
> ALL_TRUSTED,BAYES_00,FROM_IS_TO,USER_IN_WHITELIST
> scantime=15.0,size=1843,user=spamd,uid=1001,required_score=5.0,rhost=localhost,raddr=::1,rport=53074,mid=<32889a456ed9c9911ff0034513796858.squirrel@ourdomain>,bayes=0.00,autolearn=no
> autolearn_force=no
> Jul 17 09:55:28 storm spamd[28041]: prefork: child states: II
> 

The error is still the same. Do you even have access to those cloudmark
razor servers? Does razor work outside of spamassassin/amavisd?

Tom

Re: Score maths

[Tom Hendrikx]

Hoi Geoff,

The scores actually have a precision of 3 numerals after the dot. The
actual score of NO_RELAYS = -0.001. While rounding would still give you
3.0 as final score for this message, the actual score is below 3.

When you would have a ham/spam threshold at exactly 3, and the final
score would say '3.0', you would be asking why a message with score 3
wasn't blocked. So the 2.9 indicates that it's not 3 ;)

Kind regards,

Tom

On 25-04-17 10:27, Geoff Soper wrote:
> X-Spam-Status: No, Score=2.9
> 
> X-Spam-Report:
> 
> * -0.0 NO_RELAYS Informational: message was not relayed via SMTP
> 
> * 3.0 GS_NO_RLYS_PHP No description available.
> 
> X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on
> 
> server.alphaworks.co.uk 
> 
> 
> Can anyone explain why this isn't scoring 3.0?
> 
> :)
> 

Re: Problem with massive log files

[Tom Hendrikx]

Hi,

The thing that immediately caught my eye was the fact that in a line
such as:

Apr  2 10:31:26 oss2 spamfilter: Sat Oct 15 15:20:22 2016 [2758] info:
spamd: connection from ip6-localhost [::1]:55708 to port 783, fd 5

There are 2 timestamps, far away from each other. After some pondering,
my guess is that spamfilter.sh is writing away log lines to a temporary
file for each delivery, and them spewing them again when handling is
complete. But there is a bug  where spamfilter.sh does not cleanup after
itself, and new lines are appended to the existing temporary file, and
then the complete contents of the file are sent to syslog. Thus for each
single delivery the logging for all messages in the past half year (Oct
15 -> Apr x) is sent to syslog.

Please post the full contents of the spamfilter.sh, and examine the
contents of any temporary files that is it using.

Kind regards,

Tom


On 04-04-17 23:09, Jim McLachlan wrote:
> Hi,
> 
> I have a problem with the huge amount of messages being logged by
> spamassassin.   I have around 10 active e-mail users on the system, none
> of whom have any unusual e-mail usage.  This is what I've seen in the
> last 2 hours:
> 
> $ date
> Mon  3 Apr 08:00:50 UTC 2017
> 
> $ ls -l /var/log/mail.log
> -rw-r- 1 syslog adm  86370829860 Apr  3 08:00 /var/log/mail.log
> -rw-r- 1 syslog adm 331608479025 Apr  2 09:20 /var/log/mail.log.1
> 
> 
> $ spamassassin --version
> SpamAssassin version 3.4.1
>running on Perl version 5.22.1
> 
> My set up consists of Postfix, Postgrey, Spamassassin, Clam-AV,
> Amavis-new and Dovecot.
> 
> When I send an e-mail through the system, it immediately starts
> churning out a long list of log messages that implies it's checking
> messages from last October (when I set up the server).  It goes through
> thousands of messages like this and then settles down again until
> another e-mail is processed.
> 
> My initial e-mail with attachment didn't work, so an excerpt from
> the log file can be found here:
> 
> http://pasted.co/5e546e7a
> 
> Can someone please explain to me why it's repeating all this work
> and all these messages for every e-mail that gets processed and what I
> can do to fix this.  I reduced the problem slightly yesterday by
> preventing all these messages getting logged to syslog at the same time.
> 
> Kind regards.
> 
> Jim.

Re: Fastest listing RBL ?

[Tom Hendrikx]

On 16-02-17 06:22, Ian Zimmerman wrote:
> On 2017-02-15 16:30, Tom Hendrikx wrote:
> 
>> Note that the period that you describe as 'seen by SA a bit later' is
>> typically less than a second.
> 
> Not in my case.  I have a custom Exim configuration where I
> intentionally wait for a period of time (currently 4 minutes) between
> SMTP acceptance and delivery (SA runs at delivery time), precisely
> because I want to give all the collaborative mechanisms the maximum
> chance to kick in.

Why are you keeping mail in your queue, when you could also use
greylisting and achieve roughly the same delivery delay? Except that
lots of spambots don't understand greylisting and will never return for
the second delivery? And that you don't get a full queue when something
weird happens?

To be honest, I never heard of this kind of setup before. Is this a
typical Exim trick?:)

> 
> When I wrote my OP, 4 minutes was shorter than my BIND max-ncache-ttl
> parameter.  I have since set that to 180 (3 minutes), so that angle
> shouldn't matter any more.  Still the balance between bouncing the most
> junk outright and the risk of false positives means it's something to
> think about.
> 
>> Which RBLs to use, depends on the typical spam you receive, and the
>> policies that you wish to apply. IMHO, the trust you put in RBLs (and
>> their listing policies) should be more important in making decisions
>> than their typical response time to new (types of) spam and their
>> TTLs.
> 
> Agreed.
> 

Re: Fastest listing RBL ?

[Tom Hendrikx]

On 15-02-17 15:19, Bowie Bailey wrote:
> On 2/14/2017 11:04 PM, Ian Zimmerman wrote:
>> Given a piece of horrible spam, on which RBL is the sending IP address
>> likely to appear first?
>>
>> I want to rationally decide which RBL/s to consult at SMTP time.  Afraid
>> to use all of them, not just due to false positives, but also due to
>> negative caching in DNS, which could affect the result when the spam is
>> seen by SA a bit later.
> 
> I find zen.spamhaus.org to be the most reliable RBL to use for
> blacklisting.
> 
> I wouldn't worry too much about negative caching.  It looks like the TTL
> for negative results with Spamhaus is 10 seconds.
> 

Naturally, blocklists decide based on their data (f.i. removal times for
typical listings, the way they publish updates, etc) the best ttl for
their data. You should probably just use them.

Note that the period that you describe as 'seen by SA a bit later' is
typically less than a second. In the rare case that postfix sees other
values than spamassassin for the same delivery, many people on this list
will (on first sight) assume your setup is broken when you see
differences in that timeframe, in stead of 'very smart with RBLs and TTLs'.

Which RBLs to use, depends on the typical spam you receive, and the
policies that you wish to apply. IMHO, the trust you put in RBLs (and
their listing policies) should be more important in making decisions
than their typical response time to new (types of) spam and their TTLs.

Kind regards,
Tom

Re: No rule updates since 1/1/17

[Tom Hendrikx]

On 20-01-17 19:46, David Jones wrote:
>> From: Kevin Golding 
>> Sent: Friday, January 20, 2017 11:59 AM
>> To: users@spamassassin.apache.org
>> Subject: Re: No rule updates since 1/1/17
> 
>> On Fri, 20 Jan 2017 17:26:01 -, Bill Keenan  
>>  wrote:
> 
>>> What is the fix needed so /usr/bin/sa-update starts getting updates? I  
>>> too have not received an update from updates.spamassassin.org  
>>>  since 1-Jan-17.
>>>
>>> Besides updates.spamassassin.org , 
>>> what other rule sets are commonly used? Hundreds of spam messages are  
>>> getting through with only updates.spamassassin.org  
>>>  rules.
> 
>> This seems like a good time to mention  
>> https://wiki.apache.org/spamassassin/NightlyMassCheck
> 
>> If more people can contribute, even just a small corpora of mail, then  
>> updates will be published more frequently. At the moment a very small  
>> number of people provide data, meaning there is very little margin for  
>> error.
> 
> I would like to help with the nightly masscheck but I don't have the
> resources to manually check ham and spam.  This also gets into the
> grey area of how people define spam.  I also have a very good MTA
> setup with RBLs and DNS checks that block most of the spam before
> it reaches SA in MailScanner.  My SA only has to block a very small
> percentage of my definition of spam so I am not sure how helpful
> my mail filtering platform can be even though it's very accurate.
> 
> Dave
> 

I think I can say the same about my platform, but since this issue keeps
popping up I just applied for an account just to find out if my
contribution could help. I can't speculate so I'm just gonna try if it
helps :)

Kind regards,
Tom



signature.asc
Description: OpenPGP digital signature

Re: Increase BAYES_99 score?

[Tom Hendrikx]

On 10-01-17 07:07, Michael B Allen wrote:
> If I understand correctly, the BAYES_X tags add a value corresponding
> to the X value. So BAYES_99 is basically adding 0.99 to the spam
> score?

This is incorrect. The number in the tag only corresponds with the
result of the bayesian classification. The score of the rule is static
and is set in one of the config files managed by sa-update.

> 
> Ideally I feel it should be possible to scale this value such as by
> using simple multiplication or even exponentially.
> 
> Is it possible to increase the score associated with the BAYES_99 and
> BAYES_999 tags?

Yes. Just define a custom score in local.cf for the rule:

score BAYES_99 3.0

Just be aware that the defaults are chosen wisely, and if you think some
score should be higher, then discuss that problem here. Maybe there's a
flaw in your setup that makes other rules perform less then optimal (DNS
issues f.i.).

> 
> PS: Is it possible to see what values are associated with all tags for
> debugging purposes? Meaning can I run a command that dumps a list of
> all tags and their associated values so that I can decide which tags
> could have their scores adjusted?

You can grep for the scores in /var/lib/spamassassin. Note that the name
of a rule doesn't say everything: before tinkering with the score you
should verify that the rule is doing what you expect from it by reading
the regex.

> 
> PS2: Is there a tag that indicates that the message contains a large
> amount of non-latin1 text? I do get a lot of legitimate non-ISO-8859-1
> messages but usually it's just a name or at most an address. So less
> than 100 bytes.
> 

Please start a new thread and show us a sample of such a message, and
the scores you are seeing with your setup.

Kind regards,

Tom

Re: T_DKIM_INVALID from yahoo.com

[Tom Hendrikx]

On 29-12-16 19:40, Marc Stürmer wrote:
> Zitat von Tom Hendrikx <t...@whyscream.net>:
> 
>> Did you file a ticket with them? I'm curious as to what they are saying
>> about it.
> 
> Actually I got this info by their phone support, and the info was back
> then it's not supported and unlikely will be supported very soon.
> 
> Just like DNSSEC, they also don't offer it and say don't count on it
> that it will happen soon.

Sounds like you should vote with your wallet. If they're not supporting
current accepted internet standards or actively working on their support
for (i.e. have some kind of timeline), there are plenty of other parties
in Germany that offer equal hosting plans, but with decent features. Be
sure to tell them why you're leaving them. ;-)

Kind regards,
Tom



signature.asc
Description: OpenPGP digital signature

Re: T_DKIM_INVALID from yahoo.com

[Tom Hendrikx]

On 29-12-16 11:35, Marc Stürmer wrote:
> Zitat von RW :
> 
>> Are there really resolvers that can't handle it? My understanding is
>> that the relevant limit here is on the length of a string, 255 bytes.
>> Yahoo have broken their DKIM TXT record into multiple short strings to
>> keep within the limit.
> 
> There are still enough resolvers around which cannot handle 2048bit DKIM
> keys. Sad, but true.
> 
> For example the well known registrar Hetzner from Germany cannot handle
> more than 1024 bit on their DNS software - tried it myself.
> 

Did you file a ticket with them? I'm curious as to what they are saying
about it.

Kind regards,
Tom

Re: Penalizing code not working?: Don't mix company and user email domains.

[Tom Hendrikx]

On 15-06-16 00:13, Linda A. Walsh wrote:
> 
> 
> spamassas...@linkcheck.co.uk wrote:
>> The code below is found in several places online and for some months I
>> have been trying to get it to work, but whatever I do it flags up Fail
>> even if the source is good. Typically I have been concentrating on
>> gmail: from known good contacts I always get NOTVALID_GMAIL (I have
>> reduced the scores to 0.01 to avoid false rejections). Is this code
>> known to fail or is it something I'm doing wrong?
>>
>> Spamassassin version: 3.3.2
>> Perl version: 5.14.2
>> OS: Linux Mint 13
>> =
>> The section header for the code runs...
>>
>> "penalize mail claiming to be from PayPal, eBay, Yahoo or Gmail but
>> was not signed by their official mailers:"
> ---
> Someone is mixing apples and oranges in that rule.

I think you are :)

> Mail from PayPal and eBay would be coming from those companies, AFAIK,
> not end users.  Same with email from "google.com". But 'yahoo.com' and
> gmail.com are both *end-user* services.  I don't know if yahoo mixes
> it's official email sendings with user-email sendings, so it might be
> an odd case,
> But on the above list, "gmail" should be replaced with "google".

The bulk of mail coming from paypal and ebat is likely transactional
mail, and the bulk of mail coming from gmail and yahoo is likely end
user mail. Mail from the company itself, or from employees, the odd
case, should have its own (sub)domain (f.i. corp.yahoo.com), which might
or might not be covered by the SA rules from OP.

I think the rules from the OP (a poor man's DMARC check) are targeted
for the bulk case, i.e. transactional and end-user mail, and employee
mail is left out of the equation (maybe not intentionally, but who
cares). The only thing that could be criticized, is the rule description
that fails to indicate that the mail "claims to be from the Yahoo!
platform" and not from the "Yahoo! company".

But in all situations, the mails are expected to be sent by the mailers
that belong to that service: if you have a gmail account, you are
expected to send mail from that account through the google smtp relay
service, which will make sure that the mail is OK with SPF, DKIM, DMARC
checks.

If you expect to have any useful mail contact by relaying mail with a
gmail/aol/yahoo sender envelope through your own mail platform, (i.e.
delivering without using their SMTP relay), you're dreaming. The
freemail providers are making that harder and harder by deploying DMARC
p=reject.

The question arises: why would you use a Gmail address at all if you
don't want to use their services?

Regards,
Tom

Re: local uribl is not called

[Tom Hendrikx]

On 14-06-16 11:47, Reindl Harald wrote:
> 
> Am 13.06.2016 um 22:53 schrieb Reindl Harald:
>> Am 13.06.2016 um 22:10 schrieb Axb:
>>> HA! take a look into list and first thing you find is the moaner needing
>>> help coz he so smart he looks at ANCIENT /3.2.x/doc instead of
>>
>>> https://spamassassin.apache.org/full/3.4.x/doc/Mail_SpamAssassin_Plugin_URIDNSBL.html
>>>
>>> use
>>>
>>> urirhsbl BLAH uribl.thelounge.net. A
>>> or
>>> urirhssub BLAH uribl.thelounge.net. A 127.0.0.2
>>>
>>> instead of
>>> uridnsbl
>>>
>>> so no "as said the syntax seems to be correct" it is NOT
> 
> again: what about fix
> https://spamassassin.apache.org/full/3.4.x/doc/Mail_SpamAssassin_Plugin_URIDNSBL.html
> which says still:
> 

Please suggest an proved documentation text so the devs can add it.

Kind regards,
Tom

Re: spamassassin --lint errors like Subroutine File::Spec::Unix::canonpath

[Tom Hendrikx]

On 09-06-16 22:04, kud...@netzero.com wrote:
> I installed Pyzor from source now getting the below. Fedora 22 with sendmail 
> and procmail, SA 3.4.1
> 
> spamassassin --lint
> Subroutine File::Spec::Unix::canonpath redefined at 
> /usr/share/perl5/XSLoader.pm line 92.
> Subroutine File::Spec::Unix::catdir redefined at /usr/share/perl5/XSLoader.pm 
> line 92.
> Subroutine File::Spec::Unix::catfile redefined at 
> /usr/share/perl5/XSLoader.pm line 92.

Don't know what this is about, but it's not related to pyzor, as far as
I can see.

> pyzor check
> ^CTraceback (most recent call last):
>   File "/usr/bin/pyzor", line 8, in 
> pyzor.client.run()
>   File "/usr/lib/python2.7/site-packages/pyzor/client.py", line 1022, in run
> ExecCall().run()
>   File "/usr/lib/python2.7/site-packages/pyzor/client.py", line 205, in run
> if not apply(dispatch, (self, args)):
>   File "/usr/lib/python2.7/site-packages/pyzor/client.py", line 281, in check
> for digest in get_input_handler(sys.stdin, self.digest_spec, do_mbox):
>   File "/usr/lib/python2.7/site-packages/pyzor/client.py", line 648, in 
> get_input_handler
> (DataDigester(rfc822BodyCleaner(fp),
>   File "/usr/lib/python2.7/site-packages/pyzor/client.py", line 702, in 
> __init__
> msg= mimetools.Message(fp, seekable=0)
>   File "/usr/lib64/python2.7/mimetools.py", line 25, in __init__
> rfc822.Message.__init__(self, fp, seekable)
>   File "/usr/lib64/python2.7/rfc822.py", line 108, in __init__
> self.readheaders()
>   File "/usr/lib64/python2.7/rfc822.py", line 155, in readheaders
> line = self.fp.readline()
> KeyboardInterrupt
> 

This happens when you CTRL-C the pyzor client (KeyboardInterrupt) that
is waiting for data at standard input. See 'man pyzor' for details.

What were you trying to achieve?

Kind regards,
Tom



signature.asc
Description: OpenPGP digital signature

Re: DNS again

[Tom Hendrikx]

On 03-06-16 18:19, jpff wrote:
> X-Originating-<%= hostname %>-IP: [217.155.197.248]
> 
> OK I expect to get flamed but anyway
> 
> I run a couple of mailers, one of which is small with ~5 users.  For
> years I ran dnsmasq which was easy to set up and only gave occasional
> troubles with the RBL lookups being rejected from my ISP (hi Zen!).  I
> knew why but it did not seem to cause much problem in stopping spam.
> But with the latest outbreak of discussion and some spare time I
> changed to use unbound which was suggested by someone.  Apart from one
> semi-error in the instructions it was easy to deploy
> 
> BUT
> 
> I as still seeing the occasional URIBL_BLOCKED
> 
>   0.0 URIBL_BLOCKED  ADMINISTRATOR NOTICE: The query to URIBL was 
> blocked.
>  See
>  
> http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block
>   for more information.
>  [URIs: zakofr.top]
> 
> I thought the recursive caching dns system was supposed to remove
> this.  Just seeking enlightenment.
> ==John ffitch
> 

Which OS is this? The default setting on ubuntu 14.04 for unbound was
unfortunately that the init script automatically added upstream dns
servers as forwarders, which effectively mimics the dnsmasq behaviour
that gives troubles for spamassassin.

To fix that behaviour, set RESOLVCONF_FORWARDERS=false in
/etc/default/unbound, and restart unbound.

Regards,
Tom



signature.asc
Description: OpenPGP digital signature

Re: Way to set user-prefs without a database?

[Tom Hendrikx]

On 19-05-16 05:06, Dan Mahoney, System Admin wrote:
> Hey there,
> 
> We have a couple of user accounts (really, role aliases) that need a
> different required_score from our global defaults.  Since they're role
> accounts, they don't have a homedir.  We're using a milter that passes
> the whole username (including domain name) along, anyway.
> 
> Is there a dead-simple way to make this work using only the config
> files, or do I have to go to the trouble of setting up all of mysql just
> to make this happen?
> 
> Best,
> 
> -Dan Mahoney
> 

How about using "whitelist_to r...@example.tld", possibly with an
adapted negative score for the associated rule (which is not really
clear from the documentation)?

Kind regards,
Tom



signature.asc
Description: OpenPGP digital signature

Re: FSL_HELO_HOME: deep headers again

[Tom Hendrikx]

On 13-05-16 18:29, Reindl Harald wrote:
> 
> Am 13.05.2016 um 18:11 schrieb John Hardin:
>> On Fri, 13 May 2016, Reindl Harald wrote:
>>
>>> the problem is blowing out such rules with such scores at all with a
>>> non working auto-QA (non-working in: no correction for days as well as
>>> dangerous scoring of new rules from the start)
>>>
>>> 02-Mai-2016 00:12:34: SpamAssassin: No update available
>>> 03-Mai-2016 01:55:05: SpamAssassin: No update available
>>> 04-Mai-2016 00:43:33: SpamAssassin: No update available
>>> 05-Mai-2016 01:48:15: SpamAssassin: Update processed successfully
>>> 06-Mai-2016 00:53:17: SpamAssassin: No update available
>>> 07-Mai-2016 01:21:23: SpamAssassin: No update available
>>> 08-Mai-2016 01:38:23: SpamAssassin: No update available
>>> 09-Mai-2016 00:02:56: SpamAssassin: No update available
>>> 10-Mai-2016 01:10:29: SpamAssassin: No update available
>>> 11-Mai-2016 00:55:46: SpamAssassin: No update available
>>> 12-Mai-2016 00:21:17: SpamAssassin: Update processed successfully
>>> 13-Mai-2016 00:33:31: SpamAssassin: No update available
>>
>> Perhaps you could help with that by participating in masscheck. You seem
>> to get a lot of FPs on base rules; contributing masscheck results on
>> your ham would reduce those
> 
> i can't rsync customer mails to a 3rd party

That is not necessary for masscheck.
> 
> if that would be based on some webervice where you just feed local
> samples and only give the rules which hitted and spam/ham flag out it
> would be somehow possible

The process is clearly documented on the wiki:
https://wiki.apache.org/spamassassin/MassCheck
> 
> especially you would not have much from the bayes-samples because they
> would trigger all sort of wrong rules after strip most headers and and a
> generic received header (which seems to be needed by the bayes-engine
> for whatever reason since it otherwise scores samples completly different)

This is an assumption: you can't know what your data would contribute to
the masscheck process.
> 
> in any case: such a rule with 3.7 must not happen at all, even if it has
> no such bad impact - 3.7 is very high and only deserved when you are
> certain that a mail is spam which is *not* backed by a single header,
> deep inspection or not
> 
That is true, but I think you should put your money where your mouth is:
just run the masscheck on your corpus and send the results to the devs
for inspection. If it's not working, you lost nothing. If the data *is*
useful, we all win from your work by getting better scores.

Just my 2 cents.
Regards,
Tom



signature.asc
Description: OpenPGP digital signature

Re: What do I do to fix this? bayes db update ignored: Permission denied

[Tom Hendrikx]

Hi,

you probably messed up the permissions by running sa-learn or any other
tool that messes with the bayes files directly (i.e. not via spamd) as
root.

Your changes work because they allow read/write access to anyone on the
system, which is not very secure. Best would be to do something like:

chown spamuser:wheel 
chmod 0660 

Then restart spamd and see of it doesn't complain. This should allow
access for spamd and for users in the wheel group (administrative accounts).

Regards,
Tom

On 03-03-16 16:35, Robert Chalmers wrote:
> ok, I can see that. Interesting I missed it on the set up
> 
> So, I’m running on OSX, and have to use plist files to start processes.
> The spamd owner is ‘spamuser’ - ( just because I did…. and as it’s not
> used outside that, I may as well leave it as such.)
> 
> 
> 
> /opt/local/bin/daemondo
> --label=spamd
> --start-cmd
> /opt/local/libexec/perl5.22/spamd
> -l
> -u
> spamuser
> ;
> --pid=exec
> 
> 
> So anyway, on  spamd restart, it all still appears to be working ok.
> Although I’m fully expecting something to come along and bite me.
> 
> So what exactly is the “kludge” - given that mostly I followed the Wiki
> and various other setup guidelines? I’m not doing per user configs, but
> site wide.
> 
> 
> 
> 
> 
>> On 3 Mar 2016, at 15:09, RW > > wrote:
>>
>> On Thu, 3 Mar 2016 14:46:33 +
>> Robert Chalmers wrote:
>>
>>
>>>
>>> /var/spamassassin/bayes_db
>>>
>>> drwxr-xr-x   3 root  wheel  102  3 Mar 14:37 .
>>> drwxr-xr-x  28 root  wheel  952 23 Jan 15:58 ..
>>> drwxr-xr-x   5 root  wheel  170  3 Mar 14:37 bayes_db
>>>
>>>
>>> -rw-rw-rw-  1 root  wheel 2304  3 Mar 14:39 bayes_journal
>>> -rw-rw-rw-  1 root  wheel   176128  3 Mar 14:32 bayes_seen
>>> -rw-rw-rw-  1 root  wheel  3112960  3 Mar 14:32 bayes_toks
>>
>> If spamd is running as user spamd (i.e. started as spamd -u spamd) the
>> files should be own by spamd.
>>
>> Don't run spamd without "-u" less you absolutely need to read per user
>> config from unix home directories. In that case use an sql database or
>> or leave the db files under ~/.spamassassin  What you have there is a
>> dreadful kludge.
>>
>> And yes, I do know that it's suggested on the wiki.
> 
> Robert Chalmers
> rob...@chalmers.com .au  Quantum Radio:
> http://tinyurl.com/lwwddov
> Mac mini 6.2 - 2012, Intel Core i7,2.3 GHz, Memory:16 GB. El-Capitan
> 10.11.  XCode 7.2.1
> 2TB: Drive 0:HGST HTS721010A9E630. Upper bay. Drive 1:ST1000LM024
> HN-M101MBB. Lower Bay
> 
> 
> 
> 




signature.asc
Description: OpenPGP digital signature

Re: Missed spam, suggestions?

[Tom Hendrikx]

On 29-02-16 06:24, Charles Sprickman wrote:
> Hi all,
> 
> Recently I occasionally get bursts of spam that slips through Postfix
> (postscreen BL checks, protocol checks) and SpamAssassin.  I just had
> another big jump in the last week.  This was mostly spam touting Oil
> Changes, SUV sales and Lawyer Finders.
> 
> What I just did was go through a collection of missed spam and re-ran
> it through spamassassin. All of it jumped from originally scoring
> around 2-3 to a minimum of 6.5 with most hitting around 12.  The
> biggest difference I see is that DNSBL and URIBL services had started
> hitting. When originally received, these emails all originated from
> very clean IPs.
> 
> I have TXREP enabled as well, but that doesn’t seem to be having
> either a positive or negative impact.
> 
> What are my options to try to catch this junk before it hits the
> various *BLs?
> 
> I’ve not had much luck with Bayes - when I had it enabled recently on
> a per-user basis it was just hitting the master DB server too hard
> with udpates.  I’m considering enabling it again with a shared db for
> all users, which I hope might work better.  It would only be auto
> trained, perhaps with some manual training by me.
> 
> Here’s a few samples, hosted elsewhere so as not to trip anyone’s
> filters:
> 
> https://gist.github.com/anonymous/0fcaf481875959c9151f (2.7 on
> Friday, 14 tonight)
> 
> https://gist.github.com/anonymous/a5396f68699392808988 (3.4 earlier
> tonight, 6.5 just now)
> 
> I have more samples, I can dig them up if that’s helpful.
> 
> Sometimes I wonder how much this has to do with the age of our domain
> and the fact that it begins with “b”. :)
> 
> The only thing I’ve been contemplating is a local spamtrap and DNSBL.
> We have a site that’s regularly trawled for email addresses, so
> seeding it should not be too difficult…
> 

Hi,

You want to give the RBLs a bit more time to kick in, you could consider
greylisting (or postscreen after-220 checks which also cause a delay and
a retry).

Regards,
Tom

Re: Removing markup

[Tom Hendrikx]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

On 02-02-16 18:20, @lbutlr wrote:
> So it seems that no one uses spamassassin -d to remove markup for
> spam messages reclassified as ham?
> 
> OK, I can work with that.
> 
> The trouble with using formail/procmail is that the "mailbox
> timestamp" for the message will change, but i’ll cobble together a
> procmailrc to feed to formail and see how it all works. Also, since
> amavisd runs spamassassin, there is the issue of keeping the mail
> in the right user’s maildirs.
> 
> find $H_PATH -type f -mtime -${AGED} | grep -v dovecot | \ xargs
> /usr/local/bin/sa-learn --ham -u ${AMUSER} >/dev/null 2>&1
> 
> Is what runs currently
> 
> find $H_PATH -type f -mtime -${AGED} | grep -v dovecot | \ xargs
> /usr/local/bin/procmail -m /path/to/despammingrc
> 
> should be a start, though how to manage multiple users… Well, we’ll
> work on that.
> 

There is a different problem here: you shouldn't alter a message that
is stored in IMAP. You should create a new message (with markup
removed) in the same folder, and then delete the old one.

This does mean that the MUA will have to download the message again.
Which should happen anyway after you alter the message server-side,
since otherwise the MUA shows the locally cached message (without
altered headers) to the user.

For details on what you can and cannot do, please seek for advice on
the dovecot mailing list.

IMHO, removing the markup is simply not a good idea: probably easier
to use headers only (i.e. no subject tagging), and then use IMAP flags
to highlight spam messages for the user. These can be unset later
without 'nasty' tricks.

Regards,
Tom

-BEGIN PGP SIGNATURE-
Version: GnuPG v2

iQIcBAEBCAAGBQJWsS1nAAoJEJPfMZ19VO/1BaYQAN4xKvDhh6XBgywnt7LQCcVk
tWKNeC5ECMNCYfk4RYAkEIwK3TtnDfA+c2PM6QIfES4r7op7YYSYc0c8stOOV8DR
3njgAJP7WLU5S/pKJzItZPUICQEj1OqsLiXVEWYYhR5uNHJA8JmCDlM0giZWX2xM
531Tn+6OeOpjHqgA2JOizdTb2IaWbAf8JAmXApq6ZtiFQ4tnR3N8q0H7ECRNTZEM
ySJbjX4ZxLYGNGfOSsv8/l6/nN+pWJasNIG1GriRzbNIYlG5bxAggDC9vtmdycmq
lL68Oo8D47+9b7W9p6GtwMK+qARk+bJmlCRJbhmDrrz9silPLT6/1WQruClDpJMa
pEczH1F0jKesVABE9PbPq2YEELq7U5hi1+N4IBKzHbbYeIXboMnt/lEEk32cZOVD
SbIKAh2pxzKh/qJMwWFg3Ap60PtSgYpkHX10HmDPqTTXGDiRgX98yHCaBjdZyMwF
rRTMjDekadJBtBhuPbmgPN7GmACGv7pQcpLfy1NxwALtFEPzzkpwyyz1I6CKbsN/
7O1SjJ38OXw5x9zxizFSoncfsmTpql+XPODeaVkMT8d8ldMj/XnnW5pgbSyyh3fQ
+MoK39MuoJ7a32n5nkmil/uLG6zsnwU4DrGZ8Y3gIkSZWHvKni/jmrTVTvAM4lLu
IpdKG1iKY7OnTYKwpCrr
=fkvv
-END PGP SIGNATURE-

Re: FSL_HELO_BARE_IP_2 fires on wrong header

[Tom Hendrikx]

On 26-01-16 10:33, Reindl Harald wrote:
> 
> 
> Am 26.01.2016 um 09:45 schrieb Tom Hendrikx:
>> On 25-01-16 16:38, Reindl Harald wrote:
>>>
>>> Am 25.01.2016 um 16:22 schrieb Matus UHLAR - fantomas:
>>>> On 25.01.16 15:17, Reindl Harald wrote:
>>>>> not worth an argument when it's simply wrong and hits mostly clear ham
>>>>> and is broken by definition looking at *random* headers?
>>>>>
>>>>> cat maillog | grep FSL_HELO_BARE_IP_2 | grep "result: Y" | wc -l
>>>>> 21
>>>>>
>>>>> cat maillog | grep FSL_HELO_BARE_IP_2 | wc -l
>>>>> 130
>>>>>
>>>>> cat maillog | grep FSL_HELO_BARE_IP_2 | grep BAYES_00 | wc -l
>>>>> 93
>>>>
>>>> excuse me, did you get a FP?
>>>> Together with BAYES_00?
>>>
>>> excuse but the point of a rule hit is not "did it end in a complete FP"
>>> but "if the rule bahvior is reasonable and hits more spam than ham"
>>>
>>> yet talked with another sysadmin
>>>
>>> same numbers, all spam-hits between 10-36, so without the rule a sure
>>> mitler-reject and most hits where clear ham, a few only rescued with
>>> BAYES_00 and otherwise tagged
>>
>> The way this rule works, sounds to me like it catches a lot of crappy
>> mailers that send through a legitimate relay. I've also seen issues with
>> this and its score is lowered to 0.001 @here too.
> 
> i would not call 21 in 26 days "a lot" given they would have been
> rejected anyways but i call 93 wong hits a lot
> 
> when only 5 of them would become a real FP because no bayes rescue them
> hell goes on for support calls when it hits registration confirmations
> and like mails which are often driven by phpmailer
> 
>> However the main cause for FPs seems to me internal mail (reporting
>> scripts sending mail to sysadmin or BI people) which is semi-whitelisted
>> anyway. This means it only hurts when the client is not able to
>> whitelist (maybe even before mail hits SA).
> 
> depends on your number of users / domains
> 
>> @Reindl: maybe you could check with your client(s) what type of mail it
>> is? Especially if you see the hits popping at regular (cron-like)
>> intervals. And then inform them that their phpmailer (or whatever crap
>> mailer they use) could need an upgrade.
> 
> as mailprovider you are not in the position to educate your customers
> how they have to educate anybody who is sending them email - that don't
> scale and is not your job when you are responsible for incoming mail
> which includes minimize false positives

But you do need to investigate anomalies in the mail setup. If you don't
trust the rule, you need to see why it's triggered on ham mail. I had
the opportunity to work with a few friendly customers in order to do
these kind of things. Maybe you can improve the rule based on your
findings. Or add some meta rules with other non-spam signs (see below).

If you can't investigate, you either need to put your trust in bayes to
compensate, or just drop the score. Whining because the rule does not
match *your* type of mail does not make sense.

In another case, I had a large group of customers being hit by mail from
a legit freemail company whose webmail generated really crappy HTML
messages. The rules triggered (MPART_ALT_DIFF, MIME_HTML_ONLY,
HTML_MIME_NO_HTML_TAG, etc) set the baseline for all webmail generated
messages somewhere at 3.x, putting a lot of legit mail into quarantine
when other rules added up. In the end, I meta'd a lot of stuff in order
to compensate for this webmailer (names and scores made up):

header __CRAPPY_ENVELOPE Return-Path:addr =~ /\@crappy\.tld$/i
meta __FROM_CRAPPY (SPF_PASS && __RETURNPATH_CRAPPY)
# (above check uses more msg features but you get the idea)

meta CRAPPY_COMPENSATE_MPART_ALT_DIFF (__FROM_CRAPPY && MPART_ALT_DIFF)
score CRAPPY_COMPENSATE_MPART_ALT_DIFF -0.7

In order to do this kind of stuff, you need to be able to inspect the
messages.

> 
>> In any way, it would be interesting to see what type of ham mail
>> triggers such a rule when masscheck allows such a high score, before
>> starting an argument about it
> 
> the mail reported to me was some board-notification of a website
> 
> have fun to educate your customers how they have to educate random
> website owners where they receive mail from
> 

I didn't educate, I adapted the score after I looked at the specific FP
messages. Or meta'd to compensate the FPs. The board notifications sound
like a good candidate for the latter.

Re: FSL_HELO_BARE_IP_2 fires on wrong header

[Tom Hendrikx]

On 25-01-16 16:38, Reindl Harald wrote:
> 
> Am 25.01.2016 um 16:22 schrieb Matus UHLAR - fantomas:
>> On 25.01.16 15:17, Reindl Harald wrote:
>>> not worth an argument when it's simply wrong and hits mostly clear ham
>>> and is broken by definition looking at *random* headers?
>>>
>>> cat maillog | grep FSL_HELO_BARE_IP_2 | grep "result: Y" | wc -l
>>> 21
>>>
>>> cat maillog | grep FSL_HELO_BARE_IP_2 | wc -l
>>> 130
>>>
>>> cat maillog | grep FSL_HELO_BARE_IP_2 | grep BAYES_00 | wc -l
>>> 93
>>
>> excuse me, did you get a FP?
>> Together with BAYES_00?
> 
> excuse but the point of a rule hit is not "did it end in a complete FP"
> but "if the rule bahvior is reasonable and hits more spam than ham"
> 
> yet talked with another sysadmin
> 
> same numbers, all spam-hits between 10-36, so without the rule a sure
> mitler-reject and most hits where clear ham, a few only rescued with
> BAYES_00 and otherwise tagged
> 

The way this rule works, sounds to me like it catches a lot of crappy
mailers that send through a legitimate relay. I've also seen issues with
this and its score is lowered to 0.001 @here too.

However the main cause for FPs seems to me internal mail (reporting
scripts sending mail to sysadmin or BI people) which is semi-whitelisted
anyway. This means it only hurts when the client is not able to
whitelist (maybe even before mail hits SA).

@Reindl: maybe you could check with your client(s) what type of mail it
is? Especially if you see the hits popping at regular (cron-like)
intervals. And then inform them that their phpmailer (or whatever crap
mailer they use) could need an upgrade.

In any way, it would be interesting to see what type of ham mail
triggers such a rule when masscheck allows such a high score, before
starting an argument about it.

Regards,
Tom

Re: My new method for blocking spam - example

[Tom Hendrikx]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

On 20-01-16 21:01, Dianne Skoll wrote:
> On Wed, 20 Jan 2016 11:52:35 -0800 Marc Perkel
>  wrote:
> 
>> Again - Bayes compares what matches. My filter compares what
>> doesn't match.
> 
> Your filter is exactly equivalent to Bayes if you do the following 
> things:
> 
> 1) Use combinations of up to four words as tokens, instead of just 
> single tokens.
> 
> 2) Throw out any tokens whose probability is not either 100% spam
> or 100% ham.
> 
> Idea (1) is probably good.  We use words and word-pairs.  I'm not
> sure the extra storage for more than pairs is justifiable.
> 

Dspam implements up to 5 words, including wildcards for intermediate
words. See OSB and SBPH methods explained at
http://wiki.linuxwall.info/doku.php/en:ressources:dossiers:dspam#content
_tokenizing

In general, OSB gave the best results IIRC.

Regards,
Tom
-BEGIN PGP SIGNATURE-
Version: GnuPG v2

iQIcBAEBCAAGBQJWn/kZAAoJEJPfMZ19VO/1JnMP/j7DCm1frjbJZy/uhyEZzskQ
+o010elBUdzubXgW0QIaHdxvkbbPgyu6yyEZHe/lj04h/vMgRmFHjbAWo+ZF1DZX
rjt2Aa85fLtoAxKojZjh/dfL0XaPfkTL3kAm8KQupe/2QRRAdVAfiC4iXKovvMSw
hBtj1Hjrsfj6j4h4k+JjtrSY+R4K8ugCTK004x+uNxUzrWUVt/WWhVet4DkEJrHR
P9JTy7tdh5qQjm0cY93689a5pKf7WBDQ1skAsKek0LWb3BhwUEuWclBR+PLhjtqR
0DJzdKh35I4Gl1k2zYb56xE1DkjiBoyQzR6FwvIeq8+9WZzMlDbxgTaz3vwOMsJ3
AcWyO1mG/FaNzzGLrcY0A586aeb9Gn+x+otFkx85vR3EBGvZMIrKbMq8V16uSldt
P87wM3aHsiR0OnqebytbFCmWQ4E+QG7jH+eIEqL2utWsyKKmrzXR/d4dd0w0Gkhl
65s5nm3/7UK6UI73fvnuY5/PFy5L7pFIcZGGeHiMeQlxyrduJvkmC8uCPEarMJ2D
X0xhOahUnglfxAPnjG31umB/9aJ1Z68f6pOEqw8IwmgeH5TxSZNEcuLexAFvSFYH
naB3A7Csv7Km46xdJ92XnH3yuD03zrxIOfBNymaUea6IYSZpCB3tQw95B7t1+7U+
IzOeONJNye+fdmHu5GBk
=M3sm
-END PGP SIGNATURE-

Re: SPF rules and my domain

[Tom Hendrikx]

On 10-12-15 03:42, Alex wrote:
> Hi,
> 
>>> Yes, understood. This was always about my own MTA receiving a message
>>> appearing to be "FROM" my own domain, and my own SPF record would be
>>> used to check the IP of the remote system to determine if it was
>>> permitted. I may have made that especially clear at one point.
>>>
>>> Does this make sense now? I'm trying to use my SPF record to verify
>>> mail FROM our domain being received by our MX is not spoofed.
>>
>> Right, that was understood.
>>
>> My response was based on how you worded your question, which has been
>> removed from the thread now:
>>
> Please help me understand why SPF_FAIL would not be triggered when > >
> an incoming email using my domain is received by a server that is > > not 
> in
> my SPF record.
>>
>> I was addressing the apparent assumption within that question that the
>> recipient MTA matters to SPF validation.
> 
> I'm not sure if there's a question there, or I'm still confused. It
> matters because the recipient MTA is my own.
> 
> Spamassassin is just going to record a generic SPF_FAIL, regardless of
> whether it's my SPF record or an email from some other domain.
> 
> If I wanted to use SPF in spamassassin to block spoofing attempts
> against my domain, how would I do that?
> 
> Can I create a meta that combines SPF_FAIL with the From header for my
> domain to do this?
> 

This all sounds like:

I (Alex) want to use SPF for incoming email, and score mail that fails
SPF policy. But I (Alex) know for sure that my own SPF record is
correct, so for messages that fail my own SPF record, I want a stricter
policy (i.e. a higher score, or a plain reject).

If this is what you mean, then a meta rule that combines your envelope
sender domain with SPF_FAIL would be a correct solution.

Or you could add something like below, which adds a penalty for *all*
messages using your domain, and SPF saves the real ones.

whitelist_from_spf: *@example.tld (your domain)
header Return-Path =~ example.tld


Regards,
Tom

Re: question re/ RDNS_NONE

[Tom Hendrikx]

Thank you both, please stop this pissing contest.

On 24-11-15 12:35, Reindl Harald wrote:
> 
> 
> Am 24.11.2015 um 12:29 schrieb Benny Pedersen:
>> Reindl Harald skrev den 2015-11-24 11:56:
>>
>>> it's the exim of the ISP
>>
>> with old version of exim
> 
> it's still the exim of the ISP
> 
>>> it's the exim of the ISP
>>
>> with old version of exim
> 
> it's still the exim of the ISP
> 
 again disable of rdns_none is not the solution, so why fokus on that?
>>>
>>> because *it is* the solution damned when "make spamassassin exceptions
>>> for the faulty isp headers" would do exactly the same, not fire the
>>> rule and since it affects *every* mail, well disable it
>>
>> no isp might recieve mail from more then one single ip / domains, so
>> solve rdns_none with disable is incorrect with score 0
> 
> you really don't understand it - it don't matter from where the ISP
> receives mail when the Received-header don't contain the reverse DNS in
> general
> 
>> if the 3 header exeption is added it would still work on other isps spam
>> sources breaking rfc rules on how headers should be
> 
> and how does that matter for a local setup using fetchmail from *that* ISP?
> 
 spamassasssin is not a single rule spam scanner / tagger
>>> and hence you disable the rules which don't match your environment
>>
>> bah, upgrade exim is not that hard is it ?
> 
> it's still the exim of the ISP - DAMNED - go out and update it, i wish
> you success
> 

Re: URIDNSBL but with full URL

[Tom Hendrikx]

On 02-09-15 10:44, Reindl Harald wrote:
> 
> 
> Am 02.09.2015 um 10:23 schrieb Axb:
>> On 09/02/15 09:51, Olivier Nicole wrote:
>>> Hi,
>>>
>>> I am looking at malware patrol, but they offer a list of over 300,000
>>> rules, that is way too big.
>>>
>>> So I was considering using it in a URIDNSBL type of way, but including
>>> the full URL, not only the host part. It should be able to accept things
>>> like foo.example.com:81/directory/foo?something
>>>
>>> Does that exist already?
>>
>> that doesn't exist, publicly...
>>
>> There are many reasons why running this isn't trivial either.
>>
>> - tracking IDs/unique identifiers in URLs
>> - *can* cause massive scanning overhead
>> - depending on special cases, DNS spec limitations.
>> etc, etc..
>>
>> What problem are you trying to solve which cannot be solved with "known"
>> methods?
> 
> on example would be a URL for masshosting / freehosting in the way of
> http://hosterdomain/username/ where URIBL over the whole domain is not
> correct just because one user account was hacked and malware placed there
> 
> in general it would make sense at least be specific to subdomains and
> the first folder in some cases on the listing side, drawback would be
> more URIBL requests for each possible variant
> 

I implemented this at my previous $workjob, using a webservice in stead
of a DNSBL, and limiting the amount of requests by selecting a max of 10
URLs picked randomly from the message.

Client-side performance was OK, but the dataset in the BL was meager,
making the test results not very useful.
Using a DNSBL in stead of a webservice should be fairly easy, by using a
hash values of the URIs in stead of the URI itself; I've seen
implementations of that technique before.

Regards,
Tom

Re: phishing rules

[Tom Hendrikx]

On 24-08-15 18:34, Joseph Brennan wrote:
 
 Nick Edwards nick.z.edwa...@gmail.com wrote:
 
 example
 the displayed version in mail might be www.example.com, but the actual
 URI when you highlight or click on it, is foobar.example.net
 
 
 The most common case is that the text shows the real web page, but the
 link goes to a click counter page that redirects to the real web page.
 This is usually not spam but wanted list mail from Mail Chimp, Constant
 Contact, and friends.

That is why all those messages actually don't use a URL in the text, but
a regular textual description:

BAD: a href=http://redirector.tld?go=acme.com;acme.com/a

GOOD: a href=http://redirector.tld?go=acme.com;Visit ACME website/a

Basically every MUA I know will label the message as a possible scam
when you use the BAD version, which why you actually never see it in
non-spam mail, unless the editor was a real noob. I have no recent
experience with MailChimp and friends, but I hope they're educating
users to use the GOOD version.

So a clear spam indicator for me.

Regards,
Tom

Re: spamassassin detailed logging

[Tom Hendrikx]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

On 19-06-15 16:19, Axb wrote:
 On 19.06.2015 16:01, Reindl Harald wrote:
 
 Am 19.06.2015 um 15:56 schrieb Reindl Harald:
 envelope=_SENDERDOMAIN_, from=_AUTHORDOMAIN_
 
 syslog to SQL and you can xref all the info you need
 
 that's a workaround and not a solution, there's a reason why
 the spamfirewall is the *only* machine not logging to mysql
 because you really don't want a dozen millions of sql-inserts
 each month
 
 *YOU* don't want them. If you have a cluster of more than the one 
 spamfirewall it can be very practical to have central SQL
 logging.
 
 and for messages without a MID you have currently no way at all
 to xfer anything since you only see the rules and the result with
 *nothing* to grep for
 
 Postfix/MTA/Glue Session IDs, etc... having the data in a DB also
 allows all kinds of stats.
 
 Of course you are free to hack SA's logging and write out all you
 want and then filter it out via syslog/regex.. now if that is more
 efficient I/O-wise
 

I send all my logs to logstash + elasticsearch. Works like a charm,
but when there's nothing (message-id, sender recipient(s)) to search
on you're still screwed...

Tom
-BEGIN PGP SIGNATURE-
Version: GnuPG v2

iQIcBAEBCAAGBQJVhCXVAAoJEJPfMZ19VO/1O6sP/3tBqFPS9KhuUOH89cqXGb7W
bOdomyNJLwulqhdCrDlTjgRK6Nn9GEcIAG+BNoN4CA+naEa3n7VIL89aFKbZ0Nb2
3y/QrQ61RY5rmkgmd5Fr/xi98n1sQriygJP/+ckBcE0wbex6A7LmKESE6X2WMCAZ
DhjPvcfEAYhzAMdhbS7dNzNFNlblwsq+PdGr0BTbf+wf/baVA/UeBCVOXemp/ltC
7hmTtdapwpL+LGWRd6Vr53oJR+yvJpq00cG7qQuTr9RJa1U/qCVG22VyemvuwlzN
Mr5sqOvPy7yiuGbuqQKLK43qiZ9jcvfKgFt6CFl8OBt51Hc9+Jpdai/nuzlZSbpl
DC0EWBfXQJBqdbXAQiDeCgfS375cieGFDyn62Mq73nZK2EPnaYw2MM95v7jqGXV1
q9B7FoEG2esGBU6+qpYW8Dw96HOSisU3hGIlJ20NJ2rFYa0V3hdxxYknsiOMFcfT
QlpoAonhFeKfrkThHeBuCLVGY9j13fwNYnLsTOalfHAAApPzwv41Aisphevn4W2u
h3391Q5KMSfKMXkmnzEHqjRfIfcfjtrS2n9VEKSc3xw1z0o5F72n6UVxkeM28JiK
yxNzR1bB/jzG36042QTu4HW2y4U2S41hbzXlziiaQRZ6Ou5+SyMYEOY0enrX5AC/
JgQMiCs5092wBQUppba7
=rthH
-END PGP SIGNATURE-

Re: SA bayes filter learns ham but no spam

[Tom Hendrikx]

On 17-06-15 20:00, Dieter Scholz wrote:
 Hello,
 
 My problem is: The bayes filter does (auto-)learn ham mails but no
 spam mails. In my logs I found spam mails that have a very high score
 and should be autolearned. I think my bayes setup is correct, because
 ham mails are learned as expected.
 Autolearning is based on a different score to the one used for
 classification. It uses the rule scores that would be used if Bayes
 were disabled and ignores some type of rule altogether. For spam you
 need 3 points from *both* the headers and the body.

 It probably is working correctly, as far as I can tell you haven't
 actually established that no spam is being autolearned.
 
 Ok, there's a different score - I understand. But what worries me is,
 that I found mails in the log with a really high spam score and lots of
 rule matches. For all the (SA) marked spam mails I do not get a line in
 the logs with 'autolearn=no' as is the case for all ham mails. My mail
 server is now running for a week with hundreds of ham mails in the
 filter but no spams. It's a company server with 300 users and a long
 established MX record. So for me it is hard to imagine that there is no
 spam mail suitable for filter learning.
 
 I suspect the milter style of integration reject leads to a situation in
 which the spam learning part is suppressed. So there's no message about
 the mail being learned or not in the logs.
 
 What do you think? Any chance to debug this?
 
 Dieter


As far as I can see, you don't really know whether your setup is
actually learning spam or not. The logs just don't mention anything
useful about spam email. Could you check the bayes stats so we actually
know if there is a problem?

Please show us output of sa-learn --dump magic, probably ran as the
user amavisd user.

Regards,
Tom

Re: DMARC validation failed

[Tom Hendrikx]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

On 22-05-15 22:45, Alex Regan wrote:
 Hi,
 
 Can someone help me understand the DMARC_FAIL_REJECT rule? I have
 an emailfrom aol.com that was quarantined as a result of this
 rule.
 
 May 22 16:21:32.695 [23166] dbg: async: calling callback on key 
 askdns:TXT:_dmarc.aol.com May 22 16:21:32.695 [23166] dbg: askdns:
 answer received, rcode NOERROR, query IN/TXT/_dmarc.aol.com, answer
 has 1 records May 22 16:21:32.695 [23166] dbg: askdns: domain
 _dmarc.aol.com listed (__DMARC_POLICY_REJECT): v=DMARC1;
 p=reject; pct=100; rua=mailto:d...@rua.agari.com;
 ruf=mailto:d...@ruf.agari.com; May 22 16:21:32.696 [23166] dbg: dns:
 __DMARC_POLICY_REJECT lookup finished
 
 I've put a copy of the headers here:
 
 http://pastebin.com/HcbD2FJj
 
 This is from rules posted to the list in Feb by Christian Laußat:
 
 http://spamassassin.1065346.n5.nabble.com/Amazon-phishing-spam-td11442
9.html

 
 
 It seems there are quite a few in the quarantine from this rule, so
 it's worth re-evaulating.
 
 Thanks, Alex


I assume that T_SPF_TEMPERROR SPF: test of record failed (temperror)
says enough. The rules in the archive post you provided only checks
for valid results, and has no safety net for temporary errors...

Tom
-BEGIN PGP SIGNATURE-
Version: GnuPG v2

iQIcBAEBCAAGBQJVX7BLAAoJEJPfMZ19VO/1uBgQALspECJCWAHO6YGuwtFnKtDv
GGEYlIT2E5KNH07HlIk70BzLFlhqgCa4ymKcSdSBkVtVHQK7FlEYjxkFJ0yOyhhj
lm8sd2f0tVJvlfRK/HtEKCw6sE4vnsWMgLX2ScQmrnIFf64hUWzmrbmMYqJpDlFs
5faqH2yaenSO8AT5xSr5fbI2vsnhIzWTZzetWaEI3PIjUslpzVEc0IvfJ8RQ7ZVc
wMVZ+u2sKpWFNmzpSMT7eYbl1LJ/gbEJAnLFrt4uuLCgbuHAwo0iYIEbocUpiOo5
rk8A06GTkuh2lOFdrDdpbNCua7K7Bbl9Oi18CBiKJG3f5pHG8SN1bstG+RVfPZqt
Om8dXoFvCUesS8+ds5L+pXFI2tarxBusGsY2+eyg+ZwxvJ9XYKGarSIQ7TO+J4Jb
m1yFpWI5kdPcYDDkSt1fPLtc5Dlhc21T7g712H4eA1VKgyjU/Jn9Ig+VHv66tS75
OismVS9RNLJiFVGPiXM78w+G/7bdDK8ArwkVXf4tWsjdN6LFmUs3HwV4Qhdv+rbl
oVpJTEUPkTUODZT0Wd6/64CMWE6bCoxjW0JcLO3gsN/ZxORWxx1qwmS+mpyrE1sU
QrudVMtbyQg6RMFopOV+nv8YEBzY7ycckz9Umf1LyzE92wxLv8GxnTzJ6neVC2rv
FWQOg81IVsv2E+WW/ay1
=sHSO
-END PGP SIGNATURE-

Re: Rejecting without backscatter (was Re: Spamassassin not catching spam (Follow-up))

[Tom Hendrikx]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

On 26-03-15 17:28, Steve Freegard wrote:
 On 26/03/15 13:47, Reindl Harald wrote:
 
 that below was *one* message with two different recipients
 
 X-Spam-Status: No, score=-10.1, tag-level=5.5, block-level=8.0 
 X-Spam-Status: No, score=-8.1, tag-level=5.5, block-level=8.0
 
 
 I hate to piss on your parade, but your example here is totally 
 flawed; this mail from from Gmail right?
 
 X-Local-Envelope-From: reindl.har...@gmail.com 
 X-Local-Envelope-To: h.rei...@thelounge.net Received: from 
 mail-ig0-f171.google.com Message-ID: 
 caacbkvp4dpczlhodtuvugcfq9pat10yozsaum_7k9ositbo...@mail.gmail.com



 
X-Local-Envelope-From: reindl.har...@gmail.com
 X-Local-Envelope-To: ha...@rhsoft.net Received: from 
 mail-ie0-f177.google.com Message-ID: 
 caacbkvp4dpczlhodtuvugcfq9pat10yozsaum_7k9ositbo...@mail.gmail.com


 
 Gmail splits multi-recipient mail into separate deliveries, so 
 whilst you sent a single message to multiple recipients at your 
 domain from Gmail, what the big Goog does is turn that into two 
 separate messages that are delivered separately.
 
 Whilst the messages have identical Message-ID headers - you missed 
 this bit:
 
 Received: from mail-ig0-f171.google.com Received: from 
 mail-ie0-f177.google.com
 
 Your single message was delivered by two different hosts, with a 
 single recipient in each.
 

This is actually very logical because the recipients don't share the
same MX hosts or IP addresses. But as Harald shows in his logs that
the mail ends up at the same machine, and I'm really interested how it
actually works, I did some old-fashioned telnet:

- 8-

$ telnet mail-gw.thelounge.net. 25
Trying 91.118.73.19...
Connected to mail-gw.thelounge.net.
Escape character is '^]'.
220-mail-gw.thelounge.net ESMTP Spamfirewall (Enforcing
SMTP-Compliance, PTR/HELO/RBL-Checks, SPF-Policies and
Sender-Verification)
220 mail-gw.thelounge.net ESMTP Spamfirewall (Enforcing
SMTP-Compliance, PTR/HELO/RBL-Checks, SPF-Policies and
Sender-Verification)
helo valerie.whyscream.net
250 mail-gw.thelounge.net
mail from:tom+testing-...@whyscream.net
250 2.1.0 Ok
rcpt to:ha...@rhsoft.net
250 2.1.5 Ok
rcpt to:h.rei...@thelounge.net
250 2.1.5 Ok
data
354 End data with CRLF.CRLF
Subject: test message for spamassassin user mailing list

This is the gtube:
actual gtube string stripped

.
550 5.7.1 Blocked by Spamfilter, please forward this to YOUR
tech-support first, time: Mar 26 23:06:06, client: 89.105.204.244,
server: mail-gw.thelounge.net, contact: postmas...@thelounge.net
+4315953999
quit
221 2.0.0 Bye
Connection closed by foreign host.

- 8-

Ok, so the machine accepts both addresses, but rejects at end-of-data.
Harald, if one of the used recipient addresses accepts all spam
messages (all_spam_to), you should have one copy of the message,
right? Could you share the result of my test with us?

Kind regards,
Tom

-BEGIN PGP SIGNATURE-
Version: GnuPG v2

iQIcBAEBCAAGBQJVFIbYAAoJEJPfMZ19VO/1n3IP+waEs4/mbCA0PP9VB4oHmEGL
GTvGom+7Qmr1OknbNSHVnKdcN5+zSkNQEcd2poGBH/iU36I1aWlLNUfpstH0XeFI
iY1JognVE67LSqHA6Y1Q0bGdDrJSLg8QeYB961n7biBDnWphfUwOevC9F2cP/10k
KaLHg+MRlMUF/1MHqZlDJqhF0XM0dPapKReBAmQ4/PiCZnQ/xHG05qY37ruL/xz0
xBjWU2Dfkri6lC3MAH+p1X/2YyLVx9pi+rUzMUXiYJbR6ihKFHarOrd0Z9hEacPr
0Eyaryv89qPRUpV2cuPudpFOb3Twk2mXXH6IFQKSFyKBM33WyC/iCkKVrTYR28Sp
J5pg2O8V1PSsnkjHZzj1sEA/GeKOTNGyl6jh1XC2ofctKmGrXtOxrIg2ubImHRzK
sIfKizbjRP5/NJum/y0xTYGo+huT1wIxBS0ntSaHtCHS6gwMovHCHR8T84UdVO+S
Q/xDu8+2mpt5h4XNqcOQMePoyL2hSW/Yywh78+jFuUmesLrornsk8pz6TdgIb7Lq
aDxQ8xnuoUyBASnzTjfGhhuKmiIGCzv0dEMu1NAuL7X/w/P1dRjYx79bIHzWLYR0
wkj3vCeqxmAbOzujNY3zsh9LKUfNDcqY7Bj2hPgM1QSDQRVGegkF1bR4bNo0OpG2
tdE13QR2C1SrW2UYrsD5
=C7pe
-END PGP SIGNATURE-

Re: spamass filter blocked yahoo, but why?

[Tom Hendrikx]

On 12-03-15 21:55, @lbutlr wrote:
 
 Can you show us the actual message that you received (headers and
 all)?  Post it to pastebin and give us the link.
 
 Since the message was rejected, no, I do not have the actual message.
 I am relying, at this point, on my bother having given me correct
 information. Like all bothers, this is a risky assumption.
 

Change the milter decision to HOLD the message in stead of rejecting,
and inspect it after arrival.

Tom

Re: Lots of Polish spam

[Tom Hendrikx]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

On 24-02-15 22:56, Yves Goergen wrote:
 Am 24.02.2015 um 22:00 schrieb Axb:
 On 02/24/2015 09:28 PM, Yves Goergen wrote:
 https://drive.google.com/file/d/0B8CN0ghdY1SdSzBqdkswRUdOb0U/view


 
ZIP password: spam
 (Google thinks there's a virus in it so I needed to encrypt
 it.)
 
 didn't need a password to extract but... whatever format those
 .eml are in, none of text editors was able to handle them so that
 didn't help.
 
 If you weren't asked for a password, then the files were not
 decrypted. If you can decrypt them (I used 7-Zip to create the
 archive, but ZIP encryption seems incompatible between programs,
 could create a .7z archive as well, but these seem to be
 unsupported and unwanted by most, despite their highly superiour
 performance), then you'll have plain text files as Thunderbird
 received and exported them. Nothing unusual.
 
 - What plugins are you using? (pls specify: Razor, Pyzor,
 DCC, etc)
 
 neither of thsoe are installed by default so you ma want to look
 into them.
 
 RAZR/PYZOR DCCC will make a huge difference.
 
 Okay, so I'll take a look into what they are and how to install
 and configure them.
 
 I'd definitely suggest you enable the Spamhaus  SURBL rules.
 
 They have strange TOS that actually forbid using them for more than
 a single mailbox. Otherwise you need to pay for it. My data centre 
 provider wrote an interesting posting about the current situation
 in their closed customer forums. They're in a bad position as long
 as customers still access Spamhaus services from their network.
 Nobody should support them anymore, really. They're evil.
 
 Last but not least, get your Bayes setup running and it will give
 you the extra edge.
 
 I once had Bayes enabled, but since it's an unattended server
 system, it can only learn from itself. And that had worked really
 bad in the past. So I disabled it completely last time I set it up.
 How should Bayes work if nobody gives feedback about the messages
 from their Thunderbird clients? And I've tried creating rules for
 those Polish words, but it's different words all the time. I wonder
 whether they actually mean something. And it's only very few words
 per messages, many even with corrupt encoding including HTML
 entities. Again, how could Bayes help here?
 

The problem here that you're stating it's an unattended server
system. E-mail and spam change all the time, you cannot have great
filtering without adjusting to new trends and threats. Using bayesian
filtering is an easy way to improve detection, because you only need
to decide whether mail is ham or spam, and the bayes engine does most
of the other hard work for you.

If you're not going to put in some effort to either train a bayesian
filter for your users, of enable them to train it themselves (this has
some risks you should be aware of), your filtering won't improve. But
on the other hand: trying to write your own SA rules in order to block
mails in a language you don't even understand is a lot harder.

Tom

-BEGIN PGP SIGNATURE-
Version: GnuPG v1

iQIcBAEBCAAGBQJU7PebAAoJEJPfMZ19VO/14HAP/A57vNLDgHjmt0JvBi98RAnY
Jez138atfRynhLIFxy2w8MDwWy2HWSi6v/NpnXMVjZ2DYmyMBrJNC1Tn2aYrM3Mu
uxPxfOTVNlyG2FjY3iw8XwvlSJ28shcuU8WrbTrvvD81Mv2UyBVhsaNcFNNss5Gg
PATgOtU8IIn/sqb85uLu0DkyWR7LbM3SMfmNLQ1O5uRV+9212vFfHSS6Bi8/XdJ0
KCwxBlQrI8wb1hMPzPUGFa39ke307V6HugOZtSX/JkM4Ub8vSpscvvi14Up3Cde3
WlCod8SPf7cox1jwt5aahLNhESFp9fAeXHA+QCpfBAq2wiGTVFnOhm3EqE93JMw8
Btt51cTzeaIW58ho6mTAU56IBEb6phPI7mCXtcIDbtJ6WtHNWv9ozYdyp/3aQB6Z
gfe2DxzuF9Nx9g3jimyCHSeh7/ZOHw5i3U5sp77lvQ1a+B0UEf5cW6DkG/flXnRg
o6j2yWfId0elYU+h5K8EMeWzOCK8fv02W5PIsBzVRMv9VR15/22IdOdger1gAYWZ
uyttaqOVJkN8FREO4jo1JO4Si6BJFTu3c4bWLeFbYcx2s/7RY0jQeE59B8RWBa+l
CHYLOU8IhXSBKlMyvn2A6cnDPr/lyk1aeZzL0eSVGHXXyU7b0VdEYAH7K9LzEpXK
EHRkis0DOJAomQrjNuvj
=TuOX
-END PGP SIGNATURE-

Re: Amazon phishing spam

[Tom Hendrikx]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

On 15-02-15 01:24, LuKreme wrote:
 On 12 Feb 2015, at 17:58 , Dave Pooser dave...@pooserville.com
 wrote:
 Also, I score blacklist_from at 80 points so an address that's
 both blacklisted and whitelisted will be effectively whitelisted,
 thanks to a net -20 score.
 
 Quick stupid question:
 
 Is this the right syntax in local.cf to change the scores for
 blacklist_from and whitelist_auth:
 
 score blacklist_from 5.0 score whitelist_auth -10.0
 

You need to use the rule name that is triggered by
(black|white)list_from. For whitelists I have USER_IN_WHITELIST in my
logs. No blacklist entries here, so grep your own logs or headers.

Tom
-BEGIN PGP SIGNATURE-
Version: GnuPG v1

iQIcBAEBCAAGBQJU4Q/oAAoJEJPfMZ19VO/1duYP/0dvfCGym4EKenWGNnD+7ry9
5M0OEp/b21xKZKc7bDMhP0mKWCEy8XM2jWP8IdDV8E2pgkkQ2ml0ClSX5uh02BVn
I73Rt9a8GE940VVq6ZMo7WEdaIS+H4xgpOi+E4ytHy11z+XI/1dqOjvKqdCgeVSr
WIupJ0EPbhyytm13LHLas64ntPR4Ouk2OxZO+9mf/ZjfyuNa5TdlaH/Sm10M19W2
MJ7tu64juyq7xIe9CObiFh6kQlqnZp/LaSkWZxsdjEc1Di2NaP9cHE2vIAxJT4Wh
jJpsLEEwmkO9W463jayT9fprIVqr5uSV5D1v2b0ULlTKHedB5wtOxtj5C5zAbq/Y
WgJ6TTfWbM8N+Unga8IbWsHaxhIaTO3s3hx59XY/lX9vTFfITWyvl96KLeGzhQ52
ealBlWFuI3fjQMoAGgIG0eBGE0vA0ySSMaq//er4DJqbmHSoJuuqbg7HENPiSwbN
ns0BQJl6ujW3a6PjW6aRBJ6SO9sU93PeFQuYtQLoX4TI5io1Qc0mzZcWdklUXPRu
rpZWjWMeSyJxPxbBilJYKY2iJsUL6GfoVGzDlgI7iqc0z89tIFdkjZVwGf18pUEI
uk9xuLk3JhomDa11vlmYG1LQOsMBnQASWs1HRhgLJg3nQD2NvHoTSmZUQbyhEy9w
a/IT69HEve1kTEbKpKIg
=k/QI
-END PGP SIGNATURE-

Re: regex: chars to escape bsides @

[Tom Hendrikx]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

On 04-01-15 11:03, Reindl Harald wrote:
 
 
 Am 04.01.2015 um 09:44 schrieb Henrik K:
 On Sat, Jan 03, 2015 at 10:43:49PM -0700, Bob Proulx wrote:
 
 Maybe someone else will come up with a better documentation
 pointer for variables expanded inside Perl strings.
 
 Umm.. (sorry) for once Reindl is somewhat correct. We are writing
 rules using _SpamAssassin_, not coding Perl.  What low-level
 regex/variables do in any language is meaningless in this context
 as SpamAssassin might manipulate things in any number of ways.
 Quoting requirements and other strange things should be
 documented in SpamAssassin, but at a quick glance nothing is 
 mentioned about @, only # is referred as needing quoting.  So 
 documentation could use an update.
 
 and h ebiggest issue is that the testmail from gmail hitted 
 MISSING_HEADERS and MISSING_SUBJECT while both where present
 and so it looks the whole rule enigne is going crazy because one
 unescaped @
 

If you add custom rule that don't pass a lint test, you pretty much
screwed it up yourself. You can't blame spamassassin for that.

Regards,

Tom
-BEGIN PGP SIGNATURE-
Version: GnuPG v1

iQIcBAEBCAAGBQJUqRRCAAoJEJPfMZ19VO/1UjYP/Ay0lEpNoLw2WgVVJwW+x5j5
wJW3WpFBHiENUUT1tFtUX/a6YjDN5V0kV5gXy7zvOt9VDD3fkuRLxgjQ6koWKgDK
DA3eF7D6nr+nlU7FIZqVb6LO7GtPHMoi+3MEcHwNTysyQTjz3XS6Xe71U5NtRHfG
xv+/w9t+St2TTII1mWBPn+GSH0BH2dlQm8V1cIYHBsI9VmAZj96/wWQVx18jLOOc
ZKpFUnHhRiC9HDud0jeOm4nMg0dY5eZh1BNUtz0bkzDB6u/rkmYkDIy88y752jAQ
DThue9V1nNosEmqkMNA665QHprNg3NewwysErprEak5v+0C3+qL3/hYtA4niGomN
N7t9Dgvh7JvFG/Va70RO2Cd0VyY3LEgnOi2qyqQnwAlbkFMyEH9UMzVrH8snf03D
RbGAId5Ja/Gkuw0LDt2CTnWshpKkyd+Jb1Vo0sktDnH/BtMUzUVpNMdxXbT95/DG
eQG9G1Z/I7O9h1ZGGmJl3r4jfZMiKNhlkLVZItBnngtG0QoiGk9/+CUGD8WCFlso
tH1l3Z5Nyo9abYGG0fcxmPKQENthVG64oqqO3hOKsBPb6hJiVQd7s7P7QXdAZ8Sy
Ebf9OAURabv2ibwTheDLTiPHVnDSXNU4kz4jJrVabgevC+tWgwDtK/7CquXyUINt
Y7S79bkM4zbdZ6kyo+5u
=SpVP
-END PGP SIGNATURE-

Re: Can't change SpamAssassin score without enabling the Spam Auto-Delete function

[Tom Hendrikx]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

On 15-12-14 19:16, Herbert Eppel wrote:
 
 On 15.12.2014 18:03 UK Time, Joe Quinn wrote:
 On 12/15/2014 12:34 PM, Herbert Eppel wrote:
 On 15.12.2014 17:27 UK Time, Joe Quinn wrote:
 On 12/15/2014 12:20 PM, Herbert Eppel wrote:
 I use QiQ.co.uk for web hosting and associated e-mail
 services. QiQ offer SpamAssassin as an integrated feature
 that is accessible via cPanel – see screenshot below.
 
 In view of the fact that some of my domains are
 increasingly inundated with spam, I would like to reduce
 the SpamAssassin score from the default value of 5 to a
 lower value, in order to make SpamAssassin more
 'aggressive'. However, I don't really want to enable Spam
 Auto-Delete because I want to be able to check the spam 
 folder for false positives, but apparently there is no way
 to change the setting without simultaneously enabling the
 Spam Auto-Delete function.
 
 Am I missing something?
 
 Thank you
 
 Herbert Eppel www.HETranslation.co.uk
 
 You'll need to ask QiQ. SA is only a classifier. It takes an
 email as input and outputs score results, plus a modified
 email with spam headers.
 
 I would also caution against lowering the score threshold
 more than 0.5 or so. There's a point where it goes from
 catching more spam to catching more of everything.
 
 Thanks for your reply.
 
 I already asked QiQ support. They asked me to ask SA :-)
 
 Here is their reply: /I will take a look at the filter but
 would imagine if that is how Spam Assassin made the program
 that way, then there is nothing we can do to change it, though
 contacting Spam Assassin for advice may be the way forward, as
 we do not provide Spam Assassin, it is a part of cPanel - both
 third party programs./
 
 The question remains: How can I change the SA score without 
 simultaneously enabling the Spam Auto-Delete function?
 
 Perhaps I need to ask cPanel support?
 
 Herbert Eppel www.HETranslation.co.uk
 (bringing this back on list)
 
 Quite likely. I am not familiar enough with cPanel to know the
 level of configuration it gives, but my educated guess would be
 that it's up to QiQ to configure their install of cPanel. It
 shouldn't hurt to ask cPanel people anyway, even if it's just for
 them to say exactly what options QiQ should set that opens the
 option to classify without discard.
 
 Ooops, I didn't realise I had replied privately. Thanks for your
 further reply.
 
 Is there another way to access the SA score setting, outside
 cPanel?
 

I think this is more a cPanel question than a spamassassin one. A very
quick google pointed me towards
https://documentation.cpanel.net/display/ALD/Apache+SpamAssassin

After skimming that, it seems to me that your hosting provider (QIQ)
needs to configure cPanel for you in order to have mail labeled as
spam delivered (to your spam folder or otherwise). Also, in the
default settings, mail labeled as spam by spamassassin is already
dropped (auto-delete spam): you can only change the score threshold in
the gui.

So RTFM, try out some cPanel settings, and contact QIQ when you have a
better understanding of what you can do yourself, and what cPanel
should be able to provide given that QIQ does some tweaking for you.

As far as the level of technical insights you've shown in your
messages until now, I don't think it's useful (or fair) to get into
the gory details of spamassassin config with you. I'm not being
unfriendly but I'm guessing that you use cPanel for a reason: you're
not the seasoned (e-mail) admin that is comfortable tinkering with
command line and manual config file editing.

Kind regards,
Tom
-BEGIN PGP SIGNATURE-
Version: GnuPG v1

iQIcBAEBCAAGBQJUj0ErAAoJEJPfMZ19VO/1UBkQAMH9iUJBMnoU30UmH142IrYQ
Hdh/8CzS77kNHJ/P4vslFDumHca6vLVhEEc8tivLa0OlPl5gs7d0jOEmyDP5XWdq
+B6OUZIiwhsDfmKzT2FiO8xxw2JK4qwC+CDae0K+ul+SLOz9mbhuDWg+GamvuqmK
yoVu4tWkKQ4Kl+5iDczBnxjWJLDckt/ut97VsbJOK1giNGu1NLA+Mhswn+aKznYP
dIAEIP0UacgKfQJj5+I744CPwJcTUz0eErPzqWsBvyo5CuVOjBpiTG/qhBBEsJ4P
94bHSX0R9WzrI33HTHqYR1WCQ1ZTY21ETUOzEjo/XypNqtTPkwiGLvFGtngwp1vk
NOzNBPrA6/+bBCrmKgv7VKnBXEK1/1j3fXW5SpPHhNBusbStQ4Tr+83EMlPkNqc/
hNtztSDbRTBbsB4igTXwKXPx4sYnSvSOc1rwjAgjMJuYkENPv7TDGjHo8K7Wsyvq
aF0v9ekxOYSYSD31P70hmJ1iqKFPYVjaxH3skE5EH+bHJRtht7x6IMeVjUhiUxeT
wdUkpttgUH/TUjDGu6nsK1p+PC5N2JfvTC2nuk2qDB62UqcXmbh6XFCynodXe4cb
hWsGfFY0tWsVBxZm+iMv9aqqpX0a+QuGl+mVd6UEGROfzM0l71KqPvBL5koI+hr6
qAx9HxCzuzZ79/uWEO8c
=wVtz
-END PGP SIGNATURE-

Re: 23_bayes_ignore_header.cf

[Tom Hendrikx]

On 10/14/2014 11:54 PM, Axb wrote:
 On 10/14/2014 05:07 PM, RW wrote:
 On Tue, 14 Oct 2014 13:58:27 +0200
 Axb wrote:

 On 10/14/2014 01:51 PM, RW wrote:
 On Tue, 14 Oct 2014 10:44:51 +0200
 Axb wrote:


 have you verified that some of these are not included?

 X-Originating-IP will not be included as it can be used to help
 detect ham or spam

 It's really no different to other headers you are ignoring.

 for example, if you get a flood of 419s from the same source, you may
 want it to be tokenized...


 As I do with, for example:

X-AntiAbuse: Originator/Caller UID/GID - [514 32007] / [47 12]

 in this spam Bayes found

0.999-4--HX-AntiAbuse:32007

 These numbers seem to be very good indicators for me.


 Most of the headers in the file have never appeared in my ham, so
 they'll be pure spam indicators if they are ever faked. In general
 it's difficult for a spammer to gain an overall advantage against
 an average per user database using faked headers.

 Whatever the merits of this on system-wide Bayes (if any beyond
 reducing token count), I think it would have a negative effect on
 per user Bayes.

 
 ok..
 now here's a suprise (it's all in the code :)
 
 the Bayes.pm plugin alreafy includes:
 
 
 # Which headers should we scan for tokens?  Don't use all of them, as
 it's easy
 # to pick up spurious clues from some.  What we now do is use all of them
 # *less* these well-known headers; that way we can pick up spammers'
 tracking
 # headers (which are obviously not well-known in advance!).
 
 # Received is handled specially
 $IGNORED_HDRS = qr{(?: (?:X-)?Sender# misc noise
   |Delivered-To |Delivery-Date
   |(?:X-)?Envelope-To
   |X-MIME-Auto[Cc]onverted |X-Converted-To-Plain-Text
 
   |Subject  # not worth a tiny gain vs. to db size increase
 
   # Date: can provide invalid cues if your spam corpus is
   # older/newer than ham
   |Date
 
   # List headers: ignore. a spamfiltering mailing list will
   # become a nonspam sign.
   |X-List|(?:X-)?Mailing-List
   |(?:X-)?List-(?:Archive|Help|Id|Owner|Post|Subscribe
 |Unsubscribe|Host|Id|Manager|Admin|Comment
 |Name|Url)
   |X-Unsub(?:scribe)?
   |X-Mailman-Version |X-Been[Tt]here |X-Loop
   |Mail-Followup-To
   |X-eGroups-(?:Return|From)
   |X-MDMailing-List
   |X-XEmacs-List
 
   # gatewayed through mailing list (thanks to Allen Smith)
   |(?:X-)?Resent-(?:From|To|Date)
   |(?:X-)?Original-(?:From|To|Date)
 
   # Spamfilter/virus-scanner headers: too easy to chain from
   # these
   |X-MailScanner(?:-SpamCheck)?
   |X-Spam(?:-(?:Status|Level|Flag|Report|Hits|Score|Checker-Version))?
   |X-Antispam |X-RBL-Warning |X-Mailscanner
   |X-MDaemon-Deliver-To |X-Virus-Scanned
   |X-Mass-Check-Id
   |X-Pyzor |X-DCC-\S{2,25}-Metrics
   |X-Filtered-B[Yy] |X-Scanned-By |X-Scanner
   |X-AP-Spam-(?:Score|Status) |X-RIPE-Spam-Status
   |X-SpamCop-[^:]+
   |X-SMTPD |(?:X-)?Spam-Apparently-To
   |SPAM |X-Perlmx-Spam
   |X-Bogosity
 
   # some noisy Outlook headers that add no good clues:
   |Content-Class |Thread-(?:Index|Topic)
   |X-Original[Aa]rrival[Tt]ime
 
   # Annotations from IMAP, POP, and MH:
   |(?:X-)?Status |X-Flags |X-Keywords |Replied |Forwarded
   |Lines |Content-Length
   |X-UIDL? |X-IMAPbase
 
   # Annotations from Bugzilla
   |X-Bugzilla-[^:]+
 
   # Annotations from VM: (thanks to Allen Smith)
   |X-VM-(?:Bookmark|(?:POP|IMAP)-Retrieved|Labels|Last-Modified
 |Summary-Format|VHeader|v\d-Data|Message-Order)
 
   # Annotations from Gnus:
   | X-Gnus-Mail-Source
   | Xref
 
 )}x;
 
 # Note only the presence of these headers, in order to reduce the
 # hapaxen they generate.
 $MARK_PRESENCE_ONLY_HDRS = qr{(?: X-Face
   |X-(?:Gnu-?PG|PGP|GPG)(?:-Key)?-Fingerprint
   |D(?:KIM|omainKey)-Signature
 )}ix;
 
 funny...
 

Doing this in code has some drawbacks, just like the tld listing: it's
not visible to most people (like this thread nicely illustrates), and
you actually want to have it configurable. This one actually is
configurable, so now there are 2 tuneables for this problem: the code
(mostly static, hidden from view and unreachable for 99% of the users),
and the config file.

I propose to simplify, and move the code-wise exclusion to a config file
too: one tuneable (and one location to look at) is better than two.
Besides, the config file is far easier to read for the not so
regex-capable admin :)

Regards,
Tom



signature.asc
Description: OpenPGP digital signature

Re: 23_bayes_ignore_header.cf

[Tom Hendrikx]

On 10/14/2014 02:02 PM, Reindl Harald wrote:
 
 Am 14.10.2014 um 13:58 schrieb Axb:
 On 10/14/2014 01:51 PM, RW wrote:
 On Tue, 14 Oct 2014 10:44:51 +0200
 Axb wrote:

 have you verified that some of these are not included?

 X-Originating-IP will not be included as it can be used to help
 detect ham or spam

 It's really no different to other headers you are ignoring.

 for example, if you get a flood of 419s from the same source, you may
 want it to be tokenized... or not?
 or if it only sends ham
 
 but are those IP's not mostly dynamic ones from botnets and so you end
 in a lot of tokens over the time?
 

Or it is a good machine that is rooted, and then cleaned up and restored
to business by its whitehat admin. But it still is blocked by your
self-inflicted 'bayes poison' :)

Use rbls for ip-based reputation, not bayes.

Tom



signature.asc
Description: OpenPGP digital signature

Re: Output of sa-learn --dump magic

[Tom Hendrikx]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

On 02-10-14 12:38, Axb wrote:
 On 10/02/2014 11:13 AM, Tom Hendrikx wrote:
 Hi,
 
 I am using dspam besides spamassassin, and am interested in
 comparing the bayesian data between the two. Dspam reports
 statistics that include somewhat standardised metrics for spam
 filtering: Spam Hit Rate, Ham Strike Rate and Positive Predictive
 Value.
 
 I would like to calculate these for spamassassin bayes too, but I
 need to know the number of re-learned messages for that, i.e.
 false positives and false negatives. The output of sa-learn does
 not show these, it just changes the numbers of ham and spam. Are
 these numbers available to the spamassassin internals? If so,
 would it be possible to show these in the sa-learn output?
 
 Kind regards, Tom
 
 
 probably not what you want: run  sa-learn -D bayes .you
 may be able to grep out some info

Not really. I wrote a small wrapper script that writes to syslog when
FPs or FNs are retrained.

Now that I have data, I can give the statistics a go ;)

- --
Tom
-BEGIN PGP SIGNATURE-
Version: GnuPG v1

iQIcBAEBCAAGBQJUMaGoAAoJEJPfMZ19VO/1ht0P/RVSGvUIgH4TVx1v08yGmMq6
cJqOd7QkHCUTcRO0K+7UR3WVI6H/3cVEnftX8oYiYirNh6jKPDpqeklXYIYyFOn8
x0d+CUycYS4FOXiqih5P1Olw9l0Nan0TVjbG60EWyjAfq53BL6I8mvVsuF+aozPU
FQhgK3LgaLwr/6plxaF5KkyFXCqHhfYI14VAdV5klVgZ3ADIgjLTSLBX2v1OIqy4
hFmS1FAfA0RL7N9KVkuW027vvzNjPqyDkrcUgnQWnFs21WUihgDibvD0dBmV5X87
bBJU2uso+ZTdmQtUuAv9v2oBaN37IsUdxY+3m/ZsnEmSNhjy1NGuyolhf3LWvmEo
+QToj1TDtuB3eXIqbPnNoX9rm7qMDooEE0aClo7FZ1Upyu2aXfH+y5jWXd69Oc0b
zZDhZKPc1VVRJXbz1dSSh5BQerIm3K8TNzmPm+1LVLLG4Xf9ItuPyJ6Gm7CMrRWc
0RuQ2ul5p4Q06EEM8nlqCqDe+dPF8oTflTWl+15EjaTDewrNDSpWmaoEuXMAxaUk
8mGOaREkh+UQsCk40ROXHZGF7RilG+iABQIUfP0tGZiHOnfc6VhaqqiwKtwkbqhe
UdWYm8niH7N+6yA3g7uUH8I1awF+D7ZBioFDnn7Bkj1fPM1I+BDVdFdxlW7Bve+l
oVAodF2fc/adKXySZKQW
=T4OC
-END PGP SIGNATURE-

Output of sa-learn --dump magic

[Tom Hendrikx]

Hi,

I am using dspam besides spamassassin, and am interested in comparing
the bayesian data between the two. Dspam reports statistics that include
somewhat standardised metrics for spam filtering: Spam Hit Rate, Ham
Strike Rate and Positive Predictive Value.

I would like to calculate these for spamassassin bayes too, but I need
to know the number of re-learned messages for that, i.e. false positives
and false negatives. The output of sa-learn does not show these, it just
changes the numbers of ham and spam. Are these numbers available to the
spamassassin internals? If so, would it be possible to show these in the
sa-learn output?

Kind regards,
Tom



signature.asc
Description: OpenPGP digital signature

Re: How to report spam to mailspike

[Tom Hendrikx]

On 09/09/2014 11:39 AM, Marcin Mirosław wrote:
 W dniu 29.08.2014 o 23:36, Dave Warren pisze:
 On 2014-08-29 02:38, Marcin Mirosław wrote:
 So what should I do in your opinion? I'm getting spam to my private
 spamtrap so I can't fill fields about company - it doesn't matter where
 I'm hired for reporting spam. What if I would be unemployed? Then I
 would have to lie about company? IMHO it is the way to hinder sending
 complaints from users.

 If you're not willing to provide the information they request, and they
 won't accept an inquiry without it, then you're left with a different
 choice: 1) Do nothing, 2) Cease using the service.

 From their perspective, either the policy will increase the quality of
 reports they get by reducing the noise, allowing them to focus on real
 queries, and ultimately increasing the quality of the list, or it will
 discourage people from reporting, decreasing the quality of the list,
 resulting in less users and less relevance.

 They've made their choice, now you get to make yours. Personally, I'm
 quite pleased with their performance, and I have no problem identifying
 myself when I contact a company. If I'm acting on my own behalf, I'd put
 Personal or None or N/A into a form, and if it's not accepted, oh
 well.
 
 Hi!
 In a half of past week I asked them about how should I report spam to
 them. I didn't get any answer yet. I don't expect to get it in future.
 For me they are unreliable as a RBL provider.
 

There are a lot of DNSBL/DNSWL providers. They all use different inputs.
Apparently Mailspike is not interested in end users submitted a few
messages a day. Spamcop does, so you can submit to them.

You should look into the listing and delisting criteria for a list. For
instance, I don't trust any list provider that accepts money to get
delisted.

If you decide that you don't like a list because it doesn't accept user
submissions, that's your opinion of course. But not very strong one,
IMHO: Spamhaus also doesn't accept user submissions, and they run lists
that are used by virtually everyone.

Regards,
Tom



signature.asc
Description: OpenPGP digital signature

Re: Large commented out body HTML causing SA to timeout/give up/allow spam

[Tom Hendrikx]

On 09/05/2014 09:16 PM, Jari Fredriksson wrote:
 05.09.2014, 21:56, Karsten Bräckelmann kirjoitti:
 On Fri, 2014-09-05 at 11:55 -0400, Justin Edmands wrote:
 We are seeing a few emails that are about a 1MB and [...]
 dbg: timing: total 46640 ms
 BUT, because the live test likely took 46 seconds, I think SA is
 giving up or something similar. The actual email run through the live
 SA instance shows no score at all.
 If SA timed out, this would be reflected in your logs. Your guessing
 suggests you did not check logs.

 How are you passing messages to SA? Using spamc/d? With spamc the size
 limit of messages it will process is 500 kByte by default. Other methods
 and glue are likely to have a size limit, too.

 Odds are, that message simply has not been passed to SA.


 I have 5MB limit with spamc and have not encountered any timeouts. My
 spamd server is quite obsolete with only a 64 bit Pentium 4 processor
 and 3 gigs of RAM.
 
 Small load though, not an ESP, but my personal mail for mailing lists
 and also my 12 person company uses this as a filter.
 

We saw more big spam messages in the last half year, and experimented
with increasing max message size (we use spamc). In the end we settled
with a 4M limit. Never seen any timeout (or other) issues with larger
files, but it didn't matter: there was no significant traffic above that
limit.

Tom



signature.asc
Description: OpenPGP digital signature

Re: Spam relayed through trendmicro?

[Tom Hendrikx]

On 08/25/2014 04:51 AM, Alex wrote:
 Hi all,
 
 I'm having difficulty understanding this one:
 
 http://pastebin.com/LYJVas5e
 
 It looks like a host in Japan relayed this message through a few systems
 within trendmicro.com http://trendmicro.com, then on to our system
 before being tagged as obvious spam. The part I don't understand is, why
 is Trend involved with this? Is one of their systems compromised?
 

you should ask them :)

 I'm also wondering why out13.sjc.mx.trendmicro.com
 http://out13.sjc.mx.trendmicro.com in the one Received header shows as
 an invalid fqdn when it resolves fine here to the IP in the header.
 
 Received: from out13.sjc.mx.trendmicro.com
 http://out13.sjc.mx.trendmicro.com (unknown [216.99.131.50])

Maybe they disabled dns lookups on their relayhosts since they don't
care about the data: they own the hosts so they are already known. As
for the hostname of the customer that sent the message: they probably
use some non-visible way to easily relate the message to a customer
(f.i. smtp auth).

 
 # host out13.sjc.mx.trendmicro.com http://out13.sjc.mx.trendmicro.com
 out13.sjc.mx.trendmicro.com http://out13.sjc.mx.trendmicro.com has
 address 216.99.131.50
 # host 216.99.131.50
 50.131.99.216.in-addr.arpa domain name pointer
 out13.sjc.mx.trendmicro.com http://out13.sjc.mx.trendmicro.com.
 
 
 Thanks,
 Alex
 




signature.asc
Description: OpenPGP digital signature

Re: Spam Assassin - does it work or not?

[Tom Hendrikx]

On 08/10/2014 04:30 PM, Andy wrote:
 Hello it's the toymaker with the spam problem again.
 
 I am just wondering if I could get a second opinion on a response I just
 received from Lunarpages tech support (albeit the first level, and
 probably a canned response). It would be helpful to present other
 viewpoints, if any, to the higher level techs and executives that I'm in
 touch with there. They are promising to come out with a fix in 60 days,
 but aren't exactly saying what it is.
 
 I received two near identical pieces of spam. I understand, in advance,
 that their being sent from two different locations/servers/IP addresses
 can certainly mean something when it comes to scoring, but just the same,
 they are both still full of BS.
 
 Here are the headers:
 
 http://pastebin.com/bMi7Ewju

When looking at the header, the message sure is filtered by some spam
filtering engine, and it was marked as spam. Maybe Lunarpages does only
that, and leaves moving the message to the spam folder to the end
customer (by means of creating a subject sorting rule in the local MUA).
If the spam filtering technology used is spamassassin (it looks a bit
customized), then it is actually working, for all I can see.

That does not say much about Lunarpages support, who only are trying to
dodge your questions in stead of pointing you to the documentation on
how to configure your MUA.

Tom



signature.asc
Description: OpenPGP digital signature

Re: SPAM from a registrar

[Tom Hendrikx]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

On 05-06-14 20:54, Andreas Schulze wrote:
 Tom Hendrikx:
 but postfix has a feature that can check the MX and NS records of
 the envelope sender or hostname of the connecting ip.
 I know and use that.
 
 
 If these are all the same, you could block connections based on
 those.
  that's intersting, no idea how to compare something in
 postfix. Could you post an example?
 

It's a manual process: you'll need to check the whois data of the
domains that pass your spam controls, and block the NS hosts if you
find consistency, and the OP saw with Enom.

Checking whois data could be automated, but is discouraged by whois
services (and applying a blanket block based on NS records should not
be done without operator review, imho, since the possible huge impact).

Postfix cannot compare since it has no concept of multiple messages
arriving at the same time: it happens, but the smtpd processes
handling them have no knowledge of each other (or their data strcutures).

Tom
-BEGIN PGP SIGNATURE-
Version: GnuPG v1
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQIcBAEBCAAGBQJTkwvPAAoJEJPfMZ19VO/13W0QAJni1zj4W2C6F6Til1os/AJh
7uzHlfr0ucb0wxt5oLZSJXWHvV2okX9lrsJSHLK/ajzLDXfYtgoejBuGwX3/g7tG
Ppl460NyL0ok7r0E73VspeGcpRRQNlB8+l3q31eHSuScawhLEaZQczF0W6AFlF7X
+cP8YJWYGyaXxPB8MHHuELT+/ak2AIa9OueEwTTmRhiVhFtpWotBPtWDP5LJrfbB
pu0JF9jqglqyw1qeRQl6ppkNDuLpG7CqiIBzse7maFHTUweiabEd55rS5K0TpruK
TWTc1KnKtNHRJ5ykPp+2MPM6bKAAxykGWfkSxZ7o6rctNMO4Xb1gtDCkCVPfTUAr
ATwIKaDLLV6/FeNAjEIqN/z2/HBZxPF6XGWYqEl20CeFUoc4pMZ8FVFpYa3QFGAc
hjZpLXc9UJeOUTU/uqhOxBKeOIRmVBTF13cYy9G3l4vLYclv1JhoGsMVl/E7i4eF
Ub5g7ZLSp3nLJGg5YGcvolQ4VT0T9tttx8Xr88oPJ8bi2Z93wOUjyLIGunC1cbJ5
BRN8q4YvhVh+mxSZVOy9yra3JOS5yf2l29xTKgObBVQYa+6wvBEa5YNEBge6ZNaD
KgbgyhpQpv7iBp+YKaop62lljo2KuCTyaQQAhyv3ih+uEwRaFLmGapIR1qN2mnTa
6VAghLYDO3HyxGeVtJpH
=dUJ1
-END PGP SIGNATURE-

Rule updates?

[Tom Hendrikx]

Hi,

After checking the results of sa-update and doing some manual dns
queries, it seems that last rule updates were done more than a month
ago. This used to be an almost daily process, even when there were only
score changes due to masschecks.

Any specific reason for no new updates? Something we can assist with?

Regards,
Tom



signature.asc
Description: OpenPGP digital signature

Re: Rule updates?

[Tom Hendrikx]

On 05/22/2014 03:36 PM, Kevin A. McGrail wrote:
 On 5/22/2014 9:04 AM, Tom Hendrikx wrote:
 After checking the results of sa-update and doing some manual dns
 queries, it seems that last rule updates were done more than a month
 ago. This used to be an almost daily process, even when there were only
 score changes due to masschecks.

 Any specific reason for no new updates? Something we can assist with?
 
 Hi Tom,
 
 The system running the update processing failed catastrophically and
 backups were insufficient.

Ah, bugger ;

 
 I've been rebuilding the box as time allows.

Fair enough :)
Thanks fr the insight.

Kind regards,
Tom



signature.asc
Description: OpenPGP digital signature

Re: SPAM from a registrar

[Tom Hendrikx]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

On 15-05-14 16:31, James B. Byrne wrote:
 
 On Thu, May 15, 2014 09:08, David Jones wrote:
 We use the fresh15.spameatingmonkey.net RBL.
 
 http://spameatingmonkey.com/lists.html
 
 
 
 I checked three domain names used by the spam messages received
 yesterday. All of the domains were registered yesterday as well.
 None of them report as being in any of the fresh lists at
 spameatingmonkey.com.   Nor are they listed in DOB at
 support-intelligence.net.  I have to wonder how soon after
 creation new domains are added to the fresh lists.  Over 20% of the
 coverage period is already over for fresh.spameatingmonkey.net and
 I suspect that the domain used yesterday has already been
 abandoned.  At least we are getting the exact same messages today
 from a bunch of different domains all registered with the same 
 registrar: enom.com.
 
 At this point I would be willing to implement a rule to block all
 domains registered with that registrar and be done with it.  Is
 there a spamassassin whois plug-in that can parse and check the
 registrar and the domain creation date?
 

This depends on the actual domains you're seeing, and your setup
ofcourse, but postfix has a feature that can check the MX and NS
records of the envelope sender or hostname of the connecting ip. If
these are all the same, you could block connections based on those.

See http://www.postfix.org/postconf.5.html#smtpd_client_restrictions
and www.postfix.org/postconf.5.html#smtpd_client_restrictions,
especially the check_*_mx_access and check_*_ns_access directives.

Tom
-BEGIN PGP SIGNATURE-
Version: GnuPG v1
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQIcBAEBCAAGBQJTdlmZAAoJEJPfMZ19VO/1OCwQAN1yS5CAyOJAeYnzbiNAwbmP
4SagnWD6pB3nZysRigqWp3ISchi8L4DAn6Y9ZzI4m0iowdf3UEXmyYDWOADSUu9P
amDfRB0Rw1jq/yolMJvzEhLWeFSo0/9WnRipk/v8yaIEae67henSK1TdPxxXIqwO
Xzuxs30EYbTuzUgvV9PXvLxHZavhktFT6r+sLOI3rqUTmRLzhwsFEHVb58PwhreF
YBtvSOkQqk6z22nUA5lx45gNrZfhrVGhaLhYJ5iNUsqwPSWcIy08hPHt3qMHnZ0R
BZDwRbM/1hCglJlhpkuSEExB9+CCxPGBreiMtUGDBMOJ8olACXCidKgjKugZJEJ1
nDA6JhrcF81yBVqDxiM6oHXvobWALEWwd7pLoBK9Y08ukleA+KFJYMsefV5cTykP
d56bTkzJTy7kmQGIDCpb0dDYFo9pHzpOmfTMI+D2ZlmEvQf1IDNV0c2X3IjNyrnh
84ek+jyK8qmI3R13RiXKovWFpI9AZnmoc5XsiJYRnkVO3DQtZ1frixoI+MO2IENE
QcA6gFsEYJm410a2mIpuqmoYZgi15oDfzaw1qslXBo7dSEyrp1oXW52exSO5tUAc
5nJaI6k18Mhh6aJueRWThLmS5GaKV0FTp36LFQnzn/2ZRXTFVqQtjFkURHw6YxRT
IsggecowyypYTacGNK9w
=J6v+
-END PGP SIGNATURE-

Re: Are messages bypassing Spamassassin checks? Why?

[Tom Hendrikx]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

Hi,


If the message is supposed to get SA headers always, but they're not
there, your mail routing is borked or misconfigured. Please find all
related logging for this message from the moment it entered your mail
stack until the moment it was stored by cyrus. You seem to use 2
hosts, be sure to get full logging from both
inet08.hamilton.harte-lyne.ca and inet07.hamilton.harte-lyne.ca

Tom

On 11-05-14 21:26, James B. Byrne wrote:
 
 CentOS-6.5 Postfix-2.6.6 Amavisd-new-2.8.0 Spamassassin-3.3.1 
 OpenDKIM-2.9.0 pypolicyd-spf-1.2
 
 We use Spamassassin through Amavisd-new with Postfix. Our Postfix
 / Amavisd-new / Spamassassin setup has worked reliably for the past
 18 months or so. Recently we made changes to Postfix to enable SPF
 policy checking and to have DKIM sign outgoing messages.  Since
 then we have noticed a considerable decline in spam but we have
 also noticed that incoming mail no longer has any SPAM headers
 applied by Spamassassin at all.
 
 This might be because the messages are in fact not triggering any
 spam checks but in our experience even legitimate mail usually
 trips at least one rule. In consequence we are concerned that
 something is allowing messages to bypass SA and we need help in
 determining if this is true and what can be done to correct the
 problem.
 
 We have this in /etc/amavisd.amavisd.conf which previously ensured
 that every check gets listed in the delivered headers.
 
 $sa_tag_level_deflt  = -;  # add spam info headers to
 everything
 
 We added SPF policy checking to Postfix in master.cf in this
 manner:
 
 # SPF policy check policyd-spf unix  y   n   n   -
 -   spawn user=nobody argv=/usr/libexec/postfix/policyd-spf # #
 After-queue amavis spam/malware filter setup - #   but see
 before-queue setup options on smtp above # smtp-amavis unix  -
 -   n - 6   smtp -o
 smtp_data_done_timeout=1200 -o smtp_send_xforward_command=yes -o
 disable_dns_lookups=yes -o max_use=20
 
 Spamassassin appears to be working as we are still getting entries
 in the quarantine directory.  But the spam headers and scores are
 not showing up in any messages that pass through our filters.  This
 is now an issue because we are getting the occasional spam message
 delivered with no indication that they have been looked at by
 Spamassasin at all.  However, we see that they have been virus
 scanned by amavisd so the absence of spam headers is somewhat 
 mystifying.
 
 This is an example of a phishing spam message that got through:
 
 Return-Path: alert-boun...@aossystems.com Received: from
 inet07.hamilton.harte-lyne.ca ([unix socket]) by
 inet07.hamilton.harte-lyne.ca (Cyrus 
 v2.3.16-Fedora-RPM-2.3.16-6.el6_2.5) with LMTPA; Fri, 09 May 2014
 20:19:26 -0400 X-Sieve: CMU Sieve 2.3 Received: from
 inet08.hamilton.harte-lyne.ca (inet08.hamilton.harte-lyne.ca 
 [216.185.71.28]) by inet07.hamilton.harte-lyne.ca (Postfix) with
 ESMTP id 280278B2C6 for byrnej...@harte-lyne.ca; Fri, 9 May 2014
 20:19:26 -0400 (EDT) Received: from localhost (localhost
 [127.0.0.1]) by inet08.hamilton.harte-lyne.ca (Postfix) with ESMTP
 id DC99E60EF3 for byrnej...@harte-lyne.ca; Fri, 9 May 2014
 20:19:25 -0400 (EDT) X-Virus-Scanned: amavisd-new at harte-lyne.ca 
 Received: from inet08.hamilton.harte-lyne.ca ([127.0.0.1]) by
 localhost (inet08.hamilton.harte-lyne.ca [127.0.0.1])
 (amavisd-new, port 10024) with ESMTP id R7F5tnPcOww4 for
 byrnej...@harte-lyne.ca; Fri, 9 May 2014 20:19:24 -0400 (EDT) 
 Received-SPF: Pass (sender SPF authorized) identity=helo; 
 client-ip=198.57.229.237; helo=sof.softech.in; 
 envelope-from=alert-boun...@aossystems.com;
 receiver=byrnej...@harte-lyne.ca Received: from sof.softech.in
 (sof.softech.in [198.57.229.237]) (using TLSv1 with cipher
 DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a
 certificate) by inet08.hamilton.harte-lyne.ca (Postfix) with
 ESMTPS for byrnej...@harte-lyne.ca; Fri, 9 May 2014 20:19:22
 -0400 (EDT) Received: from localhost.localdomain ([127.0.0.1]:45354
 helo=sof.softech.in) by sof.softech.in with esmtp (Exim 4.80.1) 
 (envelope-from alert-boun...@aossystems.com) id 1WhHE3-Of-0L;
 Mon, 05 May 2014 06:38:27 -0500 Received: from
 bigbear.arvixevps.com ([108.175.145.28]:58594) by sof.softech.in
 with esmtps (TLSv1:DHE-RSA-AES256-SHA:256) (Exim 4.80.1)
 (envelope-from ca...@bigbear.arvixevps.com) id 1WhH4p-Mb-1H 
 for al...@aossystems.com; Mon, 05 May 2014 06:28:55 -0500 Received:
 from cable by bigbear.arvixevps.com with local (Exim 4.82) 
 (envelope-from ca...@bigbear.arvixevps.com) id 1WhH4m-0008Cb-63
 for al...@aossystems.com; Mon, 05 May 2014 04:28:52 -0700 To:
 al...@aossystems.com Subject: Changes to the Electronic Access
 Agreements From: CIBC nore...@cibc.net MIME-Version: 1.0 
 Content-Type: text/html Content-Transfer-Encoding: 8bit Message-Id:
 e1whh4m-0008cb...@bigbear.arvixevps.com Date: Mon, 05 May 2014
 04:28:52 -0700 X-AntiAbuse: This header was 

Re: Plans for a DMARC plugin ???

[Tom Hendrikx]

On 04/30/2014 11:00 AM, Axb wrote:
 On 04/30/2014 10:30 AM, Michael Storz wrote:
 Am 2014-04-30 10:23, schrieb Axb:
 On 04/30/2014 10:10 AM, Michael Storz wrote:
 Are there any plans for a DMARC plugin for SpamAssassin? Reacting to a
 DMARC policy of reject (AOL/Yahoo) seems only feasible with
 SpamAssassin
 because so many exceptions are needed for software which destryes DKIM
 signatures:

 - mailing lists
 - MS Exchange
 - Novell GroupWise
 - Lotus Domino Server ???
 - web form emails
 - ESPs
 ...

 exceptions, which could be configured via SpamAssassin rules.


 How could a SA plugin help?
 Isn't this something that should be handled at MTA level?

 Well, we are using amavisd-new in prequeue filtering mode. In our
 configuration a score of 5 will quarantine an email, a score of 10 will
 reject the email.
 
 You can submit a feature request in SA's bugzilla
 
 and in the meantime may want to look at
 http://sourceforge.net/projects/opendmarc/
 

I proposed a DMARC plugin for spamassassin on the dmarc mailing list
last year, to make it easier for people to give DMARC a spin. They
didn't really like the idea (I still do), because a simple plugin
wouldn't do the report sending, which is an important part of DMARC.

Regards,
Tom



signature.asc
Description: OpenPGP digital signature

Re: Plans for a DMARC plugin ???

[Tom Hendrikx]

On 04/30/2014 01:36 PM, Kevin A. McGrail wrote:
 On 4/30/2014 7:15 AM, Michael Storz wrote:

 Thanks, your answers are very helpful for solving the problems we are
 facing. 
 On a related note, if you need, I did implement a modification routine
 for mailman in mimedefang.  Code published at
 http://lists.roaringpenguin.com/pipermail/mimedefang/2014-April/037324.html
 
 As for an SA plugin, I think it will be needed but I believe it is just
 an overlay on top of existing DKIM and SPF information.
 
 If you open a bug for the plugin with your list of desired features,
 that would be good.
 
 Otherwise, to me, I think the features are:
 
 - Make sure SPF and DKIM are enabled
 - Check those results
 - Check the DMARC policy
 - If policy is reject or quarantine and the SPF/DKIM fails, give a
 fairly high score to a rule
 
 Beyond that, I doubt I would support a reporting mechanism.  Like
 reporting viruses, the likelihood of causing a problem and not notifying
 the correct person is far higher.
 

The reporting mechanism is working fine in DMARC, you don't have to
guess based on message headers or SMTP envelope. The report recipient is
well defined, so no reason not to send reports. Making it easy to send
(aggregated) reports is probably a bit harder. I did not take a look at
Matt Simersons code but I might take a stab at it.

Tom



signature.asc
Description: OpenPGP digital signature

Re: Missing header when skipping mail

[Tom Hendrikx]

On 04/18/2014 11:31 AM, Erik Logtenberg wrote:
 Hi,
 
 I noticed that SA has a safety feature that causes it to skip messages
 that are too large:
 
 spampd[29159]: skipped large message (68.9130859375KB)
 
 I agree very much with the reasoning behind this feature: it avoids
 certain types of denial of service attacks, prevents SA from eating too
 many resources for a single mail, and such a large mail probably isn't
 going to be spam anyway, but is usually ham with some large attachment.
 
 Having said that, I would very much appreciate it if SA would be kind
 enough to nevertheless add a header so that users know that SA has at
 least taken a look at it, and preferably also telling them the reason
 for skipping this specific email.
 
 I am now receiving questions by customers if our spam scanning is
 unavailable for some reason, and the only way to conclusively tell them
 what's going on is checking out all the maillogs...
 
 Is it possible to configure (or change) SA to add a header always?
 

The tool that hands the message to spamasassin (spampd in your case)
imposes the size limit. The message is never seen by spamassassin.
You're barking up the wrong tree ;)

Tom



signature.asc
Description: OpenPGP digital signature

Re: Disable awl when some other rule hit

[Tom Hendrikx]

On 03/24/2014 12:14 PM, Nuno Fernandes wrote:
 On Thursday 20 March 2014 07:50:50 Matt Kettler wrote:
 
 Does this do it?

 score AWL 0
 meta LOCAL_SCORE_AWL AWL!URIBL_DBL_SPAM
 score LOCAL_SCORE_AWL-10

 where -10 is whatever score AWL usually has (I forget)

 AWL has a variable score, so you can't negate it by a fixed-score rule..
 Fundamentally, it is a past-history based score averaging system, hence
 the scores for it constantly change.
 
 That's why i would like to remove it in certain scenarios. If it the mail 
 hits 
 my local RBL i would like to remove AWL all together.
 
 Guess i will have to hack AWL.pm :(
 
 Thanks,
 Nuno Fernandes
 

If the rbl gets preference above spamassassin scoring, then why don't
you just implement the rbl as a separate check in your mta, and make the
mta reject/quarantine the message without consulting spamassassin at
all? Just an idea;)

Tom



signature.asc
Description: OpenPGP digital signature

Re: false positive: KHOP_BIG_TO_CC

[Tom Hendrikx]

Hi,

Raising an old thread again, I'm also seeing FPs on this one. No real
changes have been made as far as I can see: a high score and no increase
of number of recipients (nor anything else)...

Regards,
Tom

On 10/02/2013 01:37 PM, Daniel McDonald wrote:
 On 10/2/13 6:30 AM, Tony Finch d...@dotat.at wrote:
 
 We've had a report from a user about a false positive involving
 KHOP_BIG_TO_CC which has a score of 3.4. This seems like an excessive
 penalty for perfectly reasonable behaviour.
 
 I've also seen false positives on this.  I was going to change it to 25
 addresses locally, but haven't gotten around to it yet.
 

 header   KHOP_BIG_TO_CC  ToCc =~ /(?:[^,\@]{1,60}\@[^,]{4,25},){10}/
 describe KHOP_BIG_TO_CC  Sent to 10+ recipients instaed of Bcc or a list
 scoreKHOP_BIG_TO_CC  3.199 3.399 3.199 3.399

 Tony.
 




signature.asc
Description: OpenPGP digital signature

Re: How to get removed from spamcop?

[Tom Hendrikx]

On 10/29/2013 05:21 AM, Marc Perkel wrote:
 
 
 What's odd is that all my inbound servers are listed. 

This sounds like a typical backscatter problem to me...


Kind regards,
Tom




signature.asc
Description: OpenPGP digital signature

Bare addresses alternative for __MANY_RECIPS?

[Tom Hendrikx]

Hi,

I have been using __MANY_RECIPS in some meta rules for some time now,
and noticed a weird FP today. The rule seems to count the number of '@'s
in the To and CC header. Someone sent a mail to using the (albeit silly)
format, probably by using reply-to-all in a braindead MUA:

To The foo mailing list f...@lists.domain.tld
CC: f...@lists.domain.tld f...@lists.domain.tld

This triggers the __MANY_RECIPS rule as the @ occurs (at least?) 3 times.

Is there any alternative to this rule, that only lists the addresses
(i.e. excludes the name part in the To/CC)? Or maybe even removes the
duplicates (that would probably be an eval rule)?

Regards,
Tom



signature.asc
Description: OpenPGP digital signature

Re: Strange URIBL_SBL false positive?

[Tom Hendrikx]

On 10/17/2013 12:25 PM, Marco wrote:
 Hello,
 
  If I submit this to Spamassassin 3.3.2:
 
   divbDa:/b lt;a
 href=mailto:ziop...@errebian.it;ziop...@errebian.it/agt;br;
bCc:/b Alice lt;a
 href=mailto:al...@errebian.it;al...@errebian.it/agt;,
Bob lt;a href=mailto:b...@errebian.it;b...@errebian.it/agt;br;
 
 I see:
 
  7.0 URIBL_SBL  Contains an URL listed in the SBL blocklist
 [URIs: errebian.it]
 
 ...but errebian.it IPs are not in SBL..!
 
 Could you help me to understand?
 Thank you very much!!
 
 Marco
 

We had this too for one of our customers. Your problem is that one of
the nameservers of the domain is listed:

http://www.spamhaus.org/query/ip/151.1.141.150

I'm not really sure whether it's a feature or a bug that the rule/plugin
goes that deep while searching for possible wrongdoing ip addresses...

Regards,
Tom



signature.asc
Description: OpenPGP digital signature

Re: Strange URIBL_SBL false positive?

[Tom Hendrikx]

On 10/17/2013 02:08 PM, Axb wrote:
 On 10/17/2013 02:00 PM, Tom Hendrikx wrote:
 On 10/17/2013 12:25 PM, Marco wrote:
 Hello,

   If I submit this to Spamassassin 3.3.2:

divbDa:/b lt;a
 href=mailto:ziop...@errebian.it;ziop...@errebian.it/agt;br;
 bCc:/b Alice lt;a
 href=mailto:al...@errebian.it;al...@errebian.it/agt;,
 Bob lt;a
 href=mailto:b...@errebian.it;b...@errebian.it/agt;br;

 I see:

   7.0 URIBL_SBL  Contains an URL listed in the SBL blocklist
  [URIs: errebian.it]

 ...but errebian.it IPs are not in SBL..!

 Could you help me to understand?
 Thank you very much!!

 Marco


 We had this too for one of our customers. Your problem is that one of
 the nameservers of the domain is listed:

 http://www.spamhaus.org/query/ip/151.1.141.150

 I'm not really sure whether it's a feature or a bug that the rule/plugin
 goes that deep while searching for possible wrongdoing ip addresses...
 
 Why would this be a bug? The rule performs as expected.
 the original score is low enough not to push it over the top on its
 own.. and if you have your domain on a dirty NS or A  IP neighbourhood,
 you may want to move to a more adequate gate community :)

Basicly the description Contains an URL listed in the SBL blocklist
[URIs: example.com] is false, since the domain or any of the ip
addresses linked directly to it aren't listed.

Maybe it would be nice have a split between 'direct' hits (A/ record
of hostname) and 'secondaries' (ip addresses extracted from DNS
'metadata' such as MX or NS records), so the rule description can be
more informative.

First time when I ran into this, we spent quite some time on finding
which ip was actually listed, and what relation it had to the customer
domain.

 
 the unreal score this person is using 7.0 URIBL_SBL
 means he's screaming for trouble

Totally agree.




signature.asc
Description: OpenPGP digital signature

Re: When/How to train bayes from user mail?

[Tom Hendrikx]

On 10/15/2013 09:03 PM, Florian Lindner wrote:
 Am Dienstag, 15. Oktober 2013, 07:19:01 schrieb Andreas Schulze:
 Zitat von Florian Lindner mailingli...@xgm.de:
 Since we move our server (and upgrade from oldstabe to stable) I want to
 reconsider how I organize mails serverside.

 Debian, MTA is postfix, MDA maildrop (like procmail), IMAP was
 courier, will be dovecot.

 if you use dovecot, maildrop is obsolete.
 deliver your mail via LMTP (or dovecot-lda) to dovecot and let
 dovecot-sieve do the filtering to subfolders.
 
 I object.
 AFAIK when using dovecot an LDA there is a 1:1 relation of mail adress and 
 mailbox. When using maildrop I can deliver multiple adresses to a single 
 maildir or one adress to multiple maildirs.

I would really keep those requirements at the MTA level using aliasing
and whatnot. Delivering a message to multiple folders within a single
maildir (duplication) can be one using sieve.

 Additionally sieve can not call 
 external programms.

Wrong. http://wiki2.dovecot.org/Pigeonhole/Sieve lists several (albeit
experimental) extensions that can run external programs. Setup is not as
easy as with procmail (sorry, no maildrop experience here) but it gives
the administrator control over what disasters can be triggered when
email triggers execution of external tools set up by users.

  
 Also consider using amavisd-new + clamav + spamassassin to REJECT
 mails. (not accept + delete)
 You may connect amavisd-new as SMTPD_PROXY or using amavisd-milter to
 your postfix MTA.

 My biggest open question is how to integrate the SA bayes filter,
 esp. when and on what folders to do training.

 I train sa only using the autolearn feature.
 

dovecot has the antispam plugin, which can be used to call sa-learn (or
any other tool) for each message that is moved in/out of the spam
folder. See http://wiki2.dovecot.org/Plugins/Antispam for details

Regards,
Tom



signature.asc
Description: OpenPGP digital signature

Re: New rule for HTML spam, using comments?

[Tom Hendrikx]

On 06/20/2013 01:34 AM, Amir 'CG' Caspi wrote:
 On Wed, June 19, 2013 3:47 pm, Axb wrote:
 SA's URIBL plugin doesn't and shouldn't look in the alt attribute.
 
 Why not, exactly?  I wouldn't look at it for _all_ img tags, only for ones
 that are clearly MailScanner-munged.  That is, one would look for the
 patterns that MailScanner uses for munging, and if detected, pull out the
 original URI from the alt attribute.  I admit to being new to the SA game
 but I'm not understanding why that shouldn't happen, i.e. why it's bad,
 against form, insecure, etc.
 
 Now, MailScanner's munging format is, IIRC, user-configurable.  Therefore,
 there may not be a fully universal munged format (although there is
 certainly a default format).  So, one way to glue this to MailScanner is
 to have SA load the MailScanner config, figure out what the munged format
 is from that, and use that as the rule for whether or not to look in the
 alt attribute.  If MailScanner is not installed or one does not want to
 glue them together, then one would use the default format.  And, of
 course, this could be completely user-toggleable, i.e. one could choose
 whether to unmunge MailScanner tags, or leave them as-is (i.e. what
 currently happens).
 
 Also, I should clarify that I wasn't advocating for a modification to the
 URIBL plugin, but rather the creation of a NEW plugin that would unmunge
 MailScanner URIs.  This plugin would pre-process the mail prior to the
 URIBL and Bayes analysis, to return the mail to its original state
 before MailScanner munged it.  If that's not possible due to how SA
 plugins work (i.e. if you can't specify the order of plugins being run)
 then it could simply run alongside URIBL as a Mailscanner-unmunged URIBL
 ...
 
 In any case, I guess I don't see why this isn't possible or not
 recommended.  I only see that nobody has done it, but I don't see that it
 shouldn't be done.
 

Since mailscanner already has support for integrating spamassassin [1],
why would you ever want to put work in reversing some of mailscanners
'protection'? Why don't you try the integration docs first, change the
processing order (i.e. process the mail with spamassassin first, then
with mailscanner), or disable the url munging in mailscanner?

For the result that you want to achieve (get protection from both
filters), your proposed solution seems to be the hardest way to success.
Not to mention probably the most error prone, or involving large amounts
of labor. Your proposal is not per definition impossible or plain stupid
(can't judge on both on both of those), it's just that there are many
reasons to try other routes before going down that road...

[1] http://www.mailscanner.info/spamassassin.html

Regards,
Tom

Disclaimer: I have never used mailscanner, so I don't claim any
knowledge beyond anything a 2 minute googling session wouldn't turn up.




signature.asc
Description: OpenPGP digital signature

Re: .pw / Palau URL domains in spam

[Tom Hendrikx]

On 06-05-13 19:55, Neil Schwartzman wrote:
 
 
 On May 6, 2013, at 10:39 AM, Matus UHLAR - fantomas uh...@fantomas.sk
 mailto:uh...@fantomas.sk wrote:
 
 On May 6, 2013, at 9:08 AM, John Hardin jhar...@impsec.org
 mailto:jhar...@impsec.org wrote:
 If there is a working abuse@ address that *isn't being ignored*, they're
 compliant.

 On 06.05.13 09:55, Neil Schwartzman wrote:
 heh, i don't think 'don't ignore' is part of the RFC, but yeah.

 well, if it clearly is not working, it's not compliant. if it's visibly
 ignored, trashed, dropped, it violates the RFC
 
 
 At risk of being pedantic, but this is, after all an RFC discussion,
 where do you see that in 2142? So long as someone receives a report,
 there is no specification against ignoring it, visibly or not.
 
 http://www.ietf.org/rfc/rfc2142.txt
 
The purpose of this memo is to aggregate and specify the basic set of
mailbox names which organizations need to support.  Most
organizations do not need to support the full set of mailbox names
defined here, since not every organization will implement the all of
the associated services.  However, if a given service is offerred, (sic)
then the associated mailbox name(es) must be supported, resulting in
delivery to a recipient appropriate for the referenced service or
role.

Chiming in here, the 'abstract' of the same RFC clearly states:

   This specification enumerates and describes Internet mail addresses
   (mailbox name @ host reference) to be used when contacting personnel
   at an organization.

To me, that sounds as if you should be able to reach an actual human
being ('personnel') by sending to the specified addresses. Ignoring
messages that get sent there which are valid within the context for the
addressee seems a clear violation. I.e. ignoring marketing mails sent to
abuse@ would be ok, but  ignoring abuse complaints isn't.

--
Tom

Re: Calling spamassassin directly yields very different results than calling spamassassin via amavis-new

[Tom Hendrikx]

On 17-04-13 21:40, Ben Johnson wrote:
 Ideally, using the above directives will tell us whether we're
 experiencing timeouts, or these spam messages are simply not in the
 Pyzor or Razor2 databases.
 
 Off the top of your head, do you happen to know what will happen if one
 or both of the Pyzor/Razor2 tests timeout? Will some indication that the
 tests were at least *started* still be added to the SA header?

The razor client (don't know about pyzor) logs its activity to some
logfile in ~razor. There you can see what (or what not) is happening.

It's also possible to raise logfile verbosity by changing the razor
config file. See the man page for details.

--
Tom

Re: Weird test names?

[Tom Hendrikx]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

On 03/12/2013 06:38 PM, Axb wrote:
 On 03/12/2013 02:20 PM, Tom Hendrikx wrote:
 -BEGIN PGP SIGNED MESSAGE- Hash: SHA1
 
 Hi,
 
 I just noticed 2 tests named 
 __HS_SUBJ_RE_FW_rulesrc_sandbox_jm_20_basic_cf and 
 __HS_QUOTE_rulesrc_sandbox_jm_20_basic_cf in 72_active.cf, one
 of which is used in FROM_12LTRDOM. They seem to have been
 introduced a few days ago.
 
 Not sure if their names are intentional, but they don't comply
 with the 'informal convention' of test names in the
 documentation (uppercased, max 22 chars).
 
 iirc, this limit is obsolete and docs should be corrected. in the
 days of SA 2.x --lint -D used to complain if rule names were longer
 than 22 chars, does it now?
 

I don't see anything coming by while linting, using 3.3.1 and 3.3.2.
Never saw these long names before though ;)

-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQIcBAEBAgAGBQJRQIZiAAoJEJPfMZ19VO/15WAQAM0W3oqwF1EpiLw+9irIFf1/
9Xb7Learpzsy2LUFaEhxMRaaQUdZRJJncwSxI9CSbEDMKRsprEt+8Bns+knAuQsF
W4lWLA4sIGCKjat3F6FTwZKwQtDG1up6viBlcaHPlMjPHFK5s7yRDd1R77HDo+X/
cCe5VXfCo4Q95w7sdFSpq2+waYRdvFYugi63KgB+6n/qTK17I+ewoTDKj7EuHZbB
kgNhXbQdyjkON+gTJi4GUS7gvM4nkQK6dX92zQHiGR8eWYrC6PbnSohGCP6rAwvE
1jeJEpk/TkXrCrjWHdZyAN3k3od97ZzMHFMaF7i/2KazwE8COB4lcp+uqpiRH5ox
Adj5bVFPResUR0PokHPNLlQisYNbijmAJytjw2kGiGtssJ/XCxMGlN+RlAmC2LXh
uLBbhCE1O9yPcwRNIvMVnx5/Pdc7ql/R3Vkux8LNi6IOK/RyhRFXnX7Vcjz5sTMe
v6F6+SrtH67/95swEIdOUjuYZpne1iMFOdteeZP5syD5HJSHOm01WG8lifPoc+6C
a2oAgF6tZ7txrppvna8W9oKD12BQcdhtOM3z1Lf75ajVpqTrGeNkj4iA+HBCrihi
dmwKrIysnRsf8CyC4YLCYuVfgp7jgq6o/oIyazwefV3m8ebBfmLR5hmctksZ/pMH
P8zXPNFBE91Hqy+IGnLD
=5hlk
-END PGP SIGNATURE-

Weird test names?

[Tom Hendrikx]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Hi,

I just noticed 2 tests named
__HS_SUBJ_RE_FW_rulesrc_sandbox_jm_20_basic_cf and
__HS_QUOTE_rulesrc_sandbox_jm_20_basic_cf in 72_active.cf, one of
which is used in FROM_12LTRDOM. They seem to have been introduced a
few days ago.

Not sure if their names are intentional, but they don't comply with
the 'informal convention' of test names in the documentation
(uppercased, max 22 chars).

Kind regards,
Tom
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQIcBAEBAgAGBQJRPyuVAAoJEJPfMZ19VO/11CYQAOQffVvoa2fbiIY2tRiSpdV5
SMAgo+YVSjBNwQswzqzXCAf/3nWXLunn7/HL5HgTc3Aw+EGbhH6X61woIpN+YNLY
H4iaLqCZHFAAW9b/jcL28rf2z3AOJ0EIHp8mq2nfCysltODI+bAWi0zF0eEwuOzp
xyxc0e6vsFFGxi3MykjY6tPT/UdrFah4S0RVIq9nmdWLD7Y+Udi62QruudnjHUqp
S82433Oy45ggKhLyVIKEqE+Eb7Nq95yWtFGYj1e+t6rtoqTTQA+A0XzKw4pOMeiE
hqVQCAf6wFiCd35GXjf3bZ3/SNv3BN/cb9PzDbNZ5QOYCamHqfRgowrNWcbuoiSc
f0Qz21bwniOjQLx9U4rjZa+67oqu29tukycr7tYt4keaUcs9ptB3ICw7gllw6dbw
c0/gPyhZN+gpNs6BRJ/1f1lqXuJAc/zt/Kemm1gchx2d3vZ3XzgpA0v0B1lyAjpt
158wDO7ncLDZJJD9v1Eqxe8e77/EHxxoz45LMS8AlWUinHdppaI6BGDTkSAGtQ7W
Dxa8Jlfzwe/foAoqo3ZAo2bkyRdnONX/Ielp8/sGGtbPlEMOmNHkLV892SFlbdjQ
2+RjkqPueALwu7hjKRFetH3TLwozJroCtUC/rnAh9aLZjegjykLuc6cA/wjzZmmE
FAC9stxu5YxmBTtXJIfH
=Zcef
-END PGP SIGNATURE-

Re: Calling spamassassin directly yields very different results than calling spamassassin via amavis-new

[Tom Hendrikx]

On 1/15/13 5:26 PM, Ben Johnson wrote:

 
 In postfix's main.cf:

snip
 
 Hmm, very interesting. No, I have no greylisting in place as yet, and
 no, my userbase doesn't demand immediate delivery. I will look into
 greylisting further.

If you're running postfix, consider using postscreen. It's a recent
addition to postfix that also can behave in a greylisting alike way, and
much more.

Read: http://www.postfix.org/POSTSCREEN_README.html

--
Tom

Re: spamc exit code for exceeding max size

[Tom Hendrikx]

On 11-01-13 19:45, Kevin A. McGrail wrote:
 On 1/11/2013 1:10 PM, John Hardin wrote:
 On Fri, 11 Jan 2013, Kevin A. McGrail wrote:

 On 1/10/2013 8:46 PM, jdow wrote:
  I'd suggest an option similar to the header option.

  pass_errors5,18,21,2,6
  ignore_errors23,3,19

 Spamc currently has no options file currently so this is a big change
 that someone will need to open a bug and likely spearhead with some
 draft patches. Otherwise, I haven't yet seen the value of this overhaul.

 how about:

spamc --pass_errors 5,18,21,2,6 -ignore_errors 23,3,19 ...

 ?


 Perhaps but shouldn't it tie into EX_TOOBIG and not a number?
 
 I think this has merit especially if we can say:
 
 used to use -x?  Use --pass_errors=EX_.., EX_...  and give examples of
 old to new.

While this allows for maximum flexibility (great), I'd still nitpick on
the fact that anyone technically advanced enough to know which errors
can be safely configured to be passed to the caller (and handle them
there), would also be capable to write a simple shell wrapper doing the
same. Ergo, a simple option that passes everything would do the same.

Risking the option I'm sounding a bit like a broken record, I'm just
trying to advocate the KISS principle here: all the extra options look
like feature creep to me when a clean -x implementation is available.

I started on a patch for trunk yesterday, but it's not working yet. As
my C skills are not very good, adding something like the --pass-errors=
would be out of my league.

--
Tom

spamc exit code for exceeding max size

[Tom Hendrikx]

Hi,

I was trying to detect various error conditions during spamc execution
based on its exit code, which, according to its manpage, should be easy
using something -x / --no-safe-fallback.

$ spamc --full -no-safe-fallback --port=12345  message.eml; echo $?

This nicely reports an exit code of 69 (where nothing is actually
listing on that port).

However, checking for maxsize issues does not work:

$ spamc --full --no-safe-fallback --max-size=10  message.eml; echo $?

This reports an exit code of 0. Checking the source code [1] tells me
that the exit code is suppressed on purpose, but the bugreport
referenced in the source code [2] doesn't explain why EX_TOOBIG should
be suppressed.

So either the code has an issue, or the man page (as EX_TOOBIG can never
happen but it is documented). I'd rather see the code fixed, unless
there is a compelling reason not too which I failed to understand.

Kind regards,
Tom

[1]
http://svn.apache.org/viewvc/spamassassin/trunk/spamc/spamc.c?view=markup (line
1050)
[2] https://issues.apache.org/SpamAssassin/show_bug.cgi?id=5412

Re: spamc exit code for exceeding max size

[Tom Hendrikx]

On 1/10/13 2:49 PM, Kevin A. McGrail wrote:
 On 1/10/2013 6:41 AM, Tom Hendrikx wrote:
 Hi,

 I was trying to detect various error conditions during spamc execution
 based on its exit code, which, according to its manpage, should be easy
 using something -x / --no-safe-fallback.

 $ spamc --full -no-safe-fallback --port=12345  message.eml; echo $?

 This nicely reports an exit code of 69 (where nothing is actually
 listing on that port).

 However, checking for maxsize issues does not work:

 $ spamc --full --no-safe-fallback --max-size=10  message.eml; echo $?

 This reports an exit code of 0. Checking the source code [1] tells me
 that the exit code is suppressed on purpose, but the bugreport
 referenced in the source code [2] doesn't explain why EX_TOOBIG should
 be suppressed.

 So either the code has an issue, or the man page (as EX_TOOBIG can never
 happen but it is documented). I'd rather see the code fixed, unless
 there is a compelling reason not too which I failed to understand.

 Kind regards,
 Tom

 [1]
 http://svn.apache.org/viewvc/spamassassin/trunk/spamc/spamc.c?view=markup
 (line
 1050)
 [2] https://issues.apache.org/SpamAssassin/show_bug.cgi?id=5412
 My best guess, it exits that way so that the mail that is larger than
 the scan limit is still accepted by MTAs and continues along the process.
 
 Suggest you look at bug 6717 perhaps and see if you can work up a patch
 to give the behavior you would like.

Since EX_TOOBIG is not really a temporary condition, I'm not sure if
that condition and the semantics of -X from the patch actually helps.

I'm thinking that it might be better to have a switch with the semantics
'Change all temporary errors to EX_TEMPFAIL' which would change most of
the named exit codes in the man page into EX_TEMPFAIL, except when the
message should be simply passed through (effectively only EX_TOOBIG).

This would do for integration as documented in the bug:
- spamc without special switch: exitcode indicates succes (0) or failure (1)
- spamc -new: exitcode indicates succes (0), failure (1) or retry (75)
- spamc -x: always raw exitcode

But I guess it depends largely on the setup and the sysadmins opinion
whether 'addressee unknown' is a temporary condition, so there is
another exception to be handled...

--
Tom

Re: Calling spamassassin directly yields very different results than calling spamassassin via amavis-new

[Tom Hendrikx]

On 10-01-13 19:55, Ben Johnson wrote:
 
 
 On 1/10/2013 1:06 PM, RW wrote:
 On Thu, 10 Jan 2013 12:48:07 -0500
 Ben Johnson wrote:
 pon further consideration, this behavior makes perfect sense if the
 mailbox user has moved the message from Inbox to Junk between scans;
 Dovecot's Antispam filter is in use on this server. This action would
 cause the message tokens to be added to the Bayes database, which
 explains why the SA score is higher on subsequent scans, even with
 network tests disabled.

 Also by turning-off network tests you switch to a different score set so
 the score for RDNS_NONE rose.

 
 Ahh; I didn't realize that disabling network tests changes the score set
 entirely. Thanks for the clarification there.
 
 So, at this point, I'm struggling to understand how the following happened.
 
 Over the course of 15 minutes, I received the same exact message four
 times. Each time, the message was sent to the same recipient mailbox.
 The From and Return-Path headers changed slightly each time, but the
 message bodies appear to be identical.
 
 Here are the X-Spam-Status headers for each message:
 
 1:28 PM
 
 Yes, score=7.008 tagged_above=-999 required=2 tests=[BAYES_00=-1.9,
 HTML_MESSAGE=0.001, MIME_HTML_ONLY=0.723, RCVD_IN_BRBL_LASTEXT=1.449,
 RCVD_IN_CSS=1, RCVD_IN_XBL=0.375, RDNS_NONE=0.793, SPF_PASS=-0.001,
 T_LOTS_OF_MONEY=0.01, URIBL_DBL_SPAM=1.7, URIBL_JP_SURBL=1.25,
 URIBL_WS_SURBL=1.608] autolearn=disabled
 
 1:35 PM
 
 No, score=-0.374 tagged_above=-999 required=2 tests=[BAYES_00=-1.9,
 HTML_MESSAGE=0.001, MIME_HTML_ONLY=0.723, RDNS_NONE=0.793,
 SPF_PASS=-0.001, T_LOTS_OF_MONEY=0.01] autolearn=disabled
 
 1:36 PM
 
 Yes, score=7.008 tagged_above=-999 required=2 tests=[BAYES_00=-1.9,
 HTML_MESSAGE=0.001, MIME_HTML_ONLY=0.723, RCVD_IN_BRBL_LASTEXT=1.449,
 RCVD_IN_CSS=1, RCVD_IN_XBL=0.375, RDNS_NONE=0.793, SPF_PASS=-0.001,
 T_LOTS_OF_MONEY=0.01, URIBL_DBL_SPAM=1.7, URIBL_JP_SURBL=1.25,
 URIBL_WS_SURBL=1.608] autolearn=disabled
 
 1:41 PM
 
 Yes, score=7.008 tagged_above=-999 required=2 tests=[BAYES_00=-1.9,
 HTML_MESSAGE=0.001, MIME_HTML_ONLY=0.723, RCVD_IN_BRBL_LASTEXT=1.449,
 RCVD_IN_CSS=1, RCVD_IN_XBL=0.375, RDNS_NONE=0.793, SPF_PASS=-0.001,
 T_LOTS_OF_MONEY=0.01, URIBL_DBL_SPAM=1.7, URIBL_JP_SURBL=1.25,
 URIBL_WS_SURBL=1.608] autolearn=disabled
 
 Questions:
 
 1.) I have a fairly well-trained Bayes DB; why on earth does a message
 with the subject Cash Quick? Get up to 1500 Now, and an equally
 nefarious body, trigger BAYES_00?

This will solely depend on the contents of your bayes db. Is this shared
between users, etc etc. No good answer ready without looking at it.

 2.) Why weren't network tests performed on message 2 of 4? This seems to
 be evidence of the fact that network tests are not being performed some
 percentage of the time, which could very well be at the root of this
 whole problem.

The fact that not a single network test was triggered, is indeed
suspicious. The DNSBL tests are of course sender sender dependent, but
if the body is the same the URIBL stuff should fire. Maybe you DNS
queries timed because your DNS setup is borked? Maybe you should
temporarily enable debug logging for dns lookups in spamassassin?

--
Tom

Re: spamc exit code for exceeding max size

[Tom Hendrikx]

On 10-01-13 17:51, Kevin A. McGrail wrote:
 On 1/10/2013 11:26 AM, Martin Gregorie wrote:
 On Thu, 2013-01-10 at 15:59 +0100, Tom Hendrikx wrote:

 Since EX_TOOBIG is not really a temporary condition, I'm not sure if
 that condition and the semantics of -X from the patch actually helps.

 I'm thinking that it might be better to have a switch with the semantics
 'Change all temporary errors to EX_TEMPFAIL' which would change most of
 the named exit codes in the man page into EX_TEMPFAIL, except when the
 message should be simply passed through (effectively only EX_TOOBIG).

 This would do for integration as documented in the bug:
 - spamc without special switch: exitcode indicates succes (0) or
 failure (1)
 - spamc -new: exitcode indicates succes (0), failure (1) or retry (75)
 - spamc -x: always raw exitcode

 But I guess it depends largely on the setup and the sysadmins opinion
 whether 'addressee unknown' is a temporary condition, so there is
 another exception to be handled...

 It seems to me that the set of replies as currently linked to spamc
 options would do a near-prefect job with two minor tweaks:

 (1) remove the EX_TOOBIG kludge from --no-safe-fallback so it does
  what it says on ythe tin

 (2) use the --exitcode behavior as the default in place of
  --no-safe-fallback

 The manpage needs to say that --no-safe-fallback should be used during
 testing and explain that, although  it catches more errors, it should
 not be used for production because it treats EX_TOOBIG as an error
 rather than a warning.

 Personal opinion: the best approach would be to stick with
 --no-safe-fallback after removing the EX_TOOBIG kludge and for the
 calling script/code to make more detailed exit code checks. However, I
 also realise this change would break a lot of installations, which is
 why I'm suggesting making --exitcode the default in place of
 --no-safe-fallback.
 Ignore breaking things because with a major version like 3.4.0, if this
 is the right thing to do, it's completely feasible and best to do
 right now with no delay.
 
 Overall, what we almost need is Usage scenarios and appropriate
 parameters.  Then we can identify scenarios with missing parameters to
 support.
 
 Or perhaps spamc needs a configuration file to set any and all of the
 possible exit levels which would alleviate the need for all these
 switches perhaps?

The reason why I proposed a new blanket option was because there are
already too many exit code switches to my liking, so I'd rather not
introduce a new single use option: my proposal would fit most
implementations and would be extensible when new errors should be added
(but I did not spend hours to get all usage scenarios imagined). I know
see that I already missed the default 'always exit with EX_OK' scenario...

Since I wrap spamc with a different programming language, I have all the
tools available to handle any error condition: detecting EX_TOOBIG is
however not possible. For a short term solution I added my own kludge to
fix the existing EX_TOOBIG kludge in spamc: I added '-l' to my spamc
arguments, and apply regexp on stderr to catch the 'size exceeded'
condition now. This is my usage scenario: if someone wants all error
conditions, give him the tools without kludges (and the ability to shoot
himself in the foot if he screws up :).

--
Tom

Re: spamc exit code for exceeding max size

[Tom Hendrikx]

On 10-01-13 22:43, Kevin A. McGrail wrote:
 On 1/10/2013 3:16 PM, Tom Hendrikx wrote:
 Since I wrap spamc with a different programming language, I have all
 the tools available to handle any error condition: detecting EX_TOOBIG
 is however not possible. 
 
 I don't understand this as I use MD to call spamc.  Why can't you just
 check the file size prior to calling sa?

Because I don't want to maintain the maxsize in two places? Because I
still want to catch the error when someone lowers only the maxsize
settings because he did not RTFM?

I want to handle the error as spamc throws it at me, implementing a
second safety net is a waste. Given enough development time and cpu
cycles, I could safeguard against all possible errors that spamc is able
to return before I actually execute it, but it would be easier to just
build a new version of spamc in stead.

 
 Otherwise, like I said, now's the time to open a bug and work on a patch
 if you are interested in this.

Yes, that is why I was discussing the different options available.
Adding another 17 switches for different scenarios is ugly, the existing
6(!) already look disappointingly overcomplicated to me. So I'd be happy
to contribute a patch that contains an elegant solution, but not another
kludge that fixes only my stupid little issue but makes matters worse in
the long run. As said, creating a kludge in the surrounding code is just
as ugly, but much faster.

--
Tom

Re: spamc exit code for exceeding max size

[Tom Hendrikx]

On 10-01-13 17:26, Martin Gregorie wrote:
 On Thu, 2013-01-10 at 15:59 +0100, Tom Hendrikx wrote:
 
 Since EX_TOOBIG is not really a temporary condition, I'm not sure if
 that condition and the semantics of -X from the patch actually helps.

 I'm thinking that it might be better to have a switch with the semantics
 'Change all temporary errors to EX_TEMPFAIL' which would change most of
 the named exit codes in the man page into EX_TEMPFAIL, except when the
 message should be simply passed through (effectively only EX_TOOBIG).

 This would do for integration as documented in the bug:
 - spamc without special switch: exitcode indicates succes (0) or failure (1)
 - spamc -new: exitcode indicates succes (0), failure (1) or retry (75)
 - spamc -x: always raw exitcode

 But I guess it depends largely on the setup and the sysadmins opinion
 whether 'addressee unknown' is a temporary condition, so there is
 another exception to be handled...

 It seems to me that the set of replies as currently linked to spamc
 options would do a near-prefect job with two minor tweaks:
 
 (1) remove the EX_TOOBIG kludge from --no-safe-fallback so it does 
 what it says on ythe tin
 
 (2) use the --exitcode behavior as the default in place of
 --no-safe-fallback
 
 The manpage needs to say that --no-safe-fallback should be used during
 testing and explain that, although  it catches more errors, it should
 not be used for production because it treats EX_TOOBIG as an error
 rather than a warning.
 
 Personal opinion: the best approach would be to stick with
 --no-safe-fallback after removing the EX_TOOBIG kludge and for the
 calling script/code to make more detailed exit code checks. However, I
 also realise this change would break a lot of installations, which is
 why I'm suggesting making --exitcode the default in place of
 --no-safe-fallback.
 

Reviewing my previous suggestion, I mostly agree with the above, and
meant this too (but with wrong words). I meant to provide the user with
consistent behaviour for:

1) always exit with EX_OK, disregarding actual processing outcome or
errors (current default behaviour)

2) indicate ham/spam difference with EX_OK/EX_FAILURE (current
--exitcode behaviour)

3) all power (and responsibility) to the user (current
--no-safe-fallback behaviour, but with kludges removed)

In my original proposal:
4) indicate ham/spam/failure with EX_OK/EX_FAILURE/EX_TEMPFAIL, use case
based on introduction of the -X switch, but generalised for all other
tempfail-alike exit codes too (i.e. nearly all error conditions except
EX_TOOBIG).

The first 3 seem very common, the last one is a more exotic scenario.
The question is: where is the line you draw between common and exotic,
and do you introduce extra complexity for that? In fact all exotic
scenarios could be implemented easily by wrapping 'spamc
--no-safe-fallback' with a 4-line shell script that evaluates and
manipulates the exitcode, but only if spamc would consistently expose
all exit codes.

Exit code manipulation for -c should ultimately be removed for
simplicity: the user can easily combine it with -E (as documented in the
man page), and exit code manipulation is then separated from presented
output (i.e. you can easily make all of the above switches
non-combinable, avoiding the crap from
https://issues.apache.org/SpamAssassin/show_bug.cgi?id=5412).

The recently added -X switch would also be too exotic for me, and be
removed again. I would also not implement a variant as described in 4)
above, also discarding as too exotic. Final result: simple, no
unexpected kludges, common scenarios facilitated out of the box, all
exotic scenarios available to the brave.

Kind regards,
Tom

Re: How to report a spam botnet

[Tom Hendrikx]

On 11/20/12 1:29 PM, Jason Ede wrote:
 However, ISP's blocking smtp ports for suspected spammers would help... 
 Ideally they'd block all traffic on port 25 or 587 not sent through their 
 SMTP engine which would do some basic spam checks...
 
 -Original Message-
 From: Martin Gregorie [mailto:mar...@gregorie.org]
 Sent: 20 November 2012 11:29
 To: users@spamassassin.apache.org
 Subject: Re: How to report a spam botnet

 On Tue, 2012-11-20 at 01:26 +, Chih-Cherng wrote:

 Notification help raise victims' security awareness, and motivate them
 to fix vulnerabilites within their computers.

 I have my doubts about this. I have friends who help at retiree's computer
 clubs and with disinfecting their friend's computers.

 The message I hear from them is that there are significant numbers of users
 who refuse to help themselves: they don't/won't update their system or
 their AV software, will click on anything, open any and all mail and who 
 won't
 learn that this is stupid behaviour. The reinfection time for such gentry is
 about two weeks: it takes about that long before they show up whining that
 their computer has become very slow again so please do something about it.

 I'm not sure what, if anything, can be done about such computer owners
 apart from repairing their machine with a 5 kg lump hammer, though a
 general ISP agreement to auto-disconnect infected computers may well
 help. Fat chance of that, though.


At my previous $dayjob I handled incoming abuse complaints for
consumer-grade DSL/fiber customers. Problematic lines would get their
SMTP traffic cut (outgoing port 25) along with educational e-mails/phone
calls.

Repeating offenders without any clue ended up on this list long-term,
simply because they didn't understand our messages to them, and they
never cared because they used crappy free webmail provider in stead of
desktop MUAs anyway. This way, the problem ended under the carpet.

Not very satisfactory, but ending contracts was indeed no real option,
if only because the customer simply does not understand the problem he's
being accused of (no matter how much time you spend on educating).

--
Tom