Re: short circuit content_filters

[Michael Munger]

As you have suggested, so I have done. For each of these, I have added
the whitelist.cf and shortcircuit.cf files via check_sender_access as
the /first thing/ these checks do so that domains on the whitelist are
shortcircuited and routed without prejudice.

smtpd_helo_restrictions = check_sender_access
mysql:/etc/postfix/whitelist.cf,
check_sender_access mysql:/etc/postfix/shortcircuit.cf,
permit_mynetworks,
permit_sasl_authenticated,
reject_invalid_helo_hostname,
reject_non_fqdn_helo_hostname,
reject_unknown_helo_hostname

smtpd_recipient_restrictions =  check_sender_access
mysql:/etc/postfix/whitelist.cf,
check_sender_access mysql:/etc/postfix/shortcircuit.cf,
reject_invalid_hostname,
reject_unauth_pipelining,
reject_unknown_recipient_domain,
reject_unauth_destination,
permit_mynetworks,
permit_sasl_authenticated,
permit

smtpd_client_restrictions = check_sender_access
mysql:/etc/postfix/whitelist.cf,
check_sender_access
mysql:/etc/postfix/shortcircuit.cf,
reject_rbl_client dnsbl.sorbs.net,
reject_rbl_client b.barracudacentral.org,
reject_rbl_client cbl.abuseat.org,
reject_rbl_client bl.spamcop.net,
reject_rbl_client zen.spamhaus.org,
reject_rbl_client dnsbl-1.uceprotect.net,
reject_rbl_client dnsrbl.org,
reject_rbl_client bad.psky.me,
check_sender_access mysql:/etc/postfix/blacklist.cf

Michael Munger, dCAP, MCPS, MCNPS, MBSS
High Powered Help, Inc.
Microsoft Certified Professional
Microsoft Certified Small Business Specialist
Digium Certified Asterisk Professional
mich...@highpoweredhelp.com
On 11/28/2016 11:55 PM, Noel Jones wrote:
> Don't forget to put your shortcircuit.cf in smtpd_{helo, sender,
> recipient, data}_restrictions.
>
> Or even better, to prevent any unintentional open relay have your
> shortcircuit.cf return "permit_auth_destination" rather than simply OK.
>
>
>   -- Noel Jones
>
>
> On 11/28/2016 9:22 PM, Michael Munger wrote:
>> Good timing. I was just trying to figure out why a domain on the
>> whitelist was still caught by an rbl when you sent this. Thank you.
>>
>> To fix the issue, I have added an additional check_sender_access
>>
>> smtpd_client_restrictions = check_sender_access
>> mysql:/etc/postfix/whitelist.cf,
>> check_sender_access
>> mysql:/etc/postfix/shortcircuit.cf,
>> reject_rbl_client dnsbl.sorbs.net,
>> reject_rbl_client b.barracudacentral.org,
>> reject_rbl_client cbl.abuseat.org,
>> reject_rbl_client bl.spamcop.net,
>> reject_rbl_client zen.spamhaus.org,
>> reject_rbl_client dnsbl-1.uceprotect.net,
>> reject_rbl_client dnsrbl.org,
>> reject_rbl_client bad.psky.me,
>> check_sender_access
>> mysql:/etc/postfix/blacklist.cf
>>
>> Contents of shortcircuit.cf:
>>
>> user = postfix
>> password = foopass
>> dbname = postfix
>> query = SELECT 'OK' AS whitelist_action FROM whitelist WHERE
>> whitelist_domain = '%s'
>> hosts = 127.0.0.1
>>
>> Michael Munger, dCAP, MCPS, MCNPS, MBSS
>> High Powered Help, Inc.
>> Microsoft Certified Professional
>> Microsoft Certified Small Business Specialist
>> Digium Certified Asterisk Professional
>> mich...@highpoweredhelp.com
>> On 11/28/2016 10:12 PM, Noel Jones wrote:
>>> On 11/28/2016 8:14 PM, Michael Munger wrote:
>>>
>>> ...
 we simply changed the action
 from "OK" to  "FILTER relay:192.168.10.81". This means that inbound
 mail, once it is discovered to be on a whitelist, is immediately
 relayed to our Exchange server.
>>> No, the mail is not immediately relayed. It goes through the rest of
>>> the configured restrictions, is queued, and is then relayed to the
>>> next-hop specified by your FILTER statement. Of particular interest
>>> is that FILTER -- similar to DUNNO -- does not skip further checking
>>> such as rbl lookups or hostname restrictions.
>>>
>>> To both set the next-hop AND skip further postfix restriction
>>> processing, you need two check_sender_access tables; the first to
>>> return FILTER and the second to return OK to skip further checks.
>>>
>>>
>>>
>>>   -- Noel Jones

Re: short circuit content_filters

[Noel Jones]

Don't forget to put your shortcircuit.cf in smtpd_{helo, sender,
recipient, data}_restrictions.

Or even better, to prevent any unintentional open relay have your
shortcircuit.cf return "permit_auth_destination" rather than simply OK.


  -- Noel Jones


On 11/28/2016 9:22 PM, Michael Munger wrote:
> Good timing. I was just trying to figure out why a domain on the
> whitelist was still caught by an rbl when you sent this. Thank you.
> 
> To fix the issue, I have added an additional check_sender_access
> 
> smtpd_client_restrictions = check_sender_access
> mysql:/etc/postfix/whitelist.cf,
> check_sender_access
> mysql:/etc/postfix/shortcircuit.cf,
> reject_rbl_client dnsbl.sorbs.net,
> reject_rbl_client b.barracudacentral.org,
> reject_rbl_client cbl.abuseat.org,
> reject_rbl_client bl.spamcop.net,
> reject_rbl_client zen.spamhaus.org,
> reject_rbl_client dnsbl-1.uceprotect.net,
> reject_rbl_client dnsrbl.org,
> reject_rbl_client bad.psky.me,
> check_sender_access
> mysql:/etc/postfix/blacklist.cf
> 
> Contents of shortcircuit.cf:
> 
> user = postfix
> password = foopass
> dbname = postfix
> query = SELECT 'OK' AS whitelist_action FROM whitelist WHERE
> whitelist_domain = '%s'
> hosts = 127.0.0.1
> 
> Michael Munger, dCAP, MCPS, MCNPS, MBSS
> High Powered Help, Inc.
> Microsoft Certified Professional
> Microsoft Certified Small Business Specialist
> Digium Certified Asterisk Professional
> mich...@highpoweredhelp.com
> On 11/28/2016 10:12 PM, Noel Jones wrote:
>> On 11/28/2016 8:14 PM, Michael Munger wrote:
>>
>> ...
>>> we simply changed the action
>>> from "OK" to  "FILTER relay:192.168.10.81". This means that inbound
>>> mail, once it is discovered to be on a whitelist, is immediately
>>> relayed to our Exchange server.
>> No, the mail is not immediately relayed. It goes through the rest of
>> the configured restrictions, is queued, and is then relayed to the
>> next-hop specified by your FILTER statement. Of particular interest
>> is that FILTER -- similar to DUNNO -- does not skip further checking
>> such as rbl lookups or hostname restrictions.
>>
>> To both set the next-hop AND skip further postfix restriction
>> processing, you need two check_sender_access tables; the first to
>> return FILTER and the second to return OK to skip further checks.
>>
>>
>>
>>   -- Noel Jones
> 

Re: short circuit content_filters

[Michael Munger]

Good timing. I was just trying to figure out why a domain on the
whitelist was still caught by an rbl when you sent this. Thank you.

To fix the issue, I have added an additional check_sender_access

smtpd_client_restrictions = check_sender_access
mysql:/etc/postfix/whitelist.cf,
check_sender_access
mysql:/etc/postfix/shortcircuit.cf,
reject_rbl_client dnsbl.sorbs.net,
reject_rbl_client b.barracudacentral.org,
reject_rbl_client cbl.abuseat.org,
reject_rbl_client bl.spamcop.net,
reject_rbl_client zen.spamhaus.org,
reject_rbl_client dnsbl-1.uceprotect.net,
reject_rbl_client dnsrbl.org,
reject_rbl_client bad.psky.me,
check_sender_access mysql:/etc/postfix/blacklist.cf

Contents of shortcircuit.cf:

user = postfix
password = foopass
dbname = postfix
query = SELECT 'OK' AS whitelist_action FROM whitelist WHERE
whitelist_domain = '%s'
hosts = 127.0.0.1

Michael Munger, dCAP, MCPS, MCNPS, MBSS
High Powered Help, Inc.
Microsoft Certified Professional
Microsoft Certified Small Business Specialist
Digium Certified Asterisk Professional
mich...@highpoweredhelp.com
On 11/28/2016 10:12 PM, Noel Jones wrote:
> On 11/28/2016 8:14 PM, Michael Munger wrote:
>
> ...
>> we simply changed the action
>> from "OK" to  "FILTER relay:192.168.10.81". This means that inbound
>> mail, once it is discovered to be on a whitelist, is immediately
>> relayed to our Exchange server.
> No, the mail is not immediately relayed. It goes through the rest of
> the configured restrictions, is queued, and is then relayed to the
> next-hop specified by your FILTER statement. Of particular interest
> is that FILTER -- similar to DUNNO -- does not skip further checking
> such as rbl lookups or hostname restrictions.
>
> To both set the next-hop AND skip further postfix restriction
> processing, you need two check_sender_access tables; the first to
> return FILTER and the second to return OK to skip further checks.
>
>
>
>   -- Noel Jones

Re: short circuit content_filters

[Noel Jones]

On 11/28/2016 8:14 PM, Michael Munger wrote:

...
> we simply changed the action
> from "OK" to  "FILTER relay:192.168.10.81". This means that inbound
> mail, once it is discovered to be on a whitelist, is immediately
> relayed to our Exchange server.

No, the mail is not immediately relayed. It goes through the rest of
the configured restrictions, is queued, and is then relayed to the
next-hop specified by your FILTER statement. Of particular interest
is that FILTER -- similar to DUNNO -- does not skip further checking
such as rbl lookups or hostname restrictions.

To both set the next-hop AND skip further postfix restriction
processing, you need two check_sender_access tables; the first to
return FILTER and the second to return OK to skip further checks.



  -- Noel Jones

Re: short circuit content_filters

[Michael Munger]

>
>> How do I configure Postfix to NOT use spamassassin to scan the email if
>> it is coming from one of these whitelisted domains?
>
> Make the check_sender_access queries return a FILTER result
> (specifying a suitable transport, likely 'local:$myhostname') which
> will override your content_filter directive. See access(5) 

This is exactly what I wanted. For future searchers, here was the answer:

Since FILTER (see: access(5)) must be the name of a service in
master.cf, AND this is a mail gateway, we simply changed the action from
"OK" to  "FILTER relay:192.168.10.81". This means that inbound mail,
once it is discovered to be on a whitelist, is immediately relayed to
our Exchange server.

Michael Munger, dCAP, MCPS, MCNPS, MBSS
High Powered Help, Inc.
Microsoft Certified Professional
Microsoft Certified Small Business Specialist
Digium Certified Asterisk Professional
mich...@highpoweredhelp.com

Re: short circuit content_filters

[Bill Cole]

On 28 Nov 2016, at 17:37, Michael Munger wrote:


TLDR:

How do I configure Postfix to NOT use spamassassin to scan the email 
if
it is coming from a domain that is whitelisted via 
check_sender_access?


You *COULD* do this inside SA, but for that you'd be better off asking 
in the SA mailing list rather than here. You can also do at the Postfix 
level more cleanly.



DETAILS:


We maintain a white-list of company domains that are aggregated from 
all

our employee's address books.

We want all email that claims to be from those domains (even if they 
are

spam, forged, contain cryptolocker, etc...). It's full-on, highway to
the danger zone, unbridled email acceptance if your domain is on this
whitelist.


At least you understand how crazy that is...

If you did this in SA, you could do that with whitelist_from entries and 
narrow that down by using whitelist_from_rcvd or whitelist_auth (or the 
older whitelist_from_{spf,dk,dkim}) but those would require some sort of 
export/transformation from your MySQL DB to put in place all the 
whitelisting rules.



For the most part, the configs below work. check_sender_access queries
our MySQL database and gets an "OK" for these domains. And, (I 
believe)
once this is returned, the remainder of the checking stops for that 
step

in the process. (Please correct me if I am wrong here, but that IS the
desired behavior).

We still have a few automated emails (these are order confirmations)
that are getting caught by spamassassin DESPITE being on the whitelist
because they are HTML only, and they talk about money if you fill out 
a

form. And, Spamassassin is executing post-queue, where these rules do
not apply post-queue, and only upon receipt.

Example of a good message caught in the bad filter:

Content analysis details: (2.8 points, 2.0 required) pts rule name


[Switching hats]

Do not do this. The default SA ruleset is re-scored daily by the "Rule 
QA" process with changes to rules and their scores distributed to users 
who run the 'sa-update' program. That process assumes that 5.0 is the 
line between ham and spam. You can safely adjust the threshold a little 
bit (best done with data to support your number) but going as low as 2.0 
essentially ruins the assumptions of Rule QA and guarantees that you 
will see false positives. As you are.



How do I configure Postfix to NOT use spamassassin to scan the email 
if

it is coming from one of these whitelisted domains?


Make the check_sender_access queries return a FILTER result (specifying 
a suitable transport, likely 'local:$myhostname') which will override 
your content_filter directive. See access(5)



Or, in the
alternative, can I configure Postfix to add a header to emails that 
are

on the whitelist, and then configure spam assassin to not check emails
with that header?


Make the check_sender_access queries return a PREPEND result with an 
obscure header name and value of your choosing, then create a local SA 
rule that looks for that header with that value, scores it with an 
absurdly big negative value, and make it a "shortcircuit" rule so as to 
not waste resources figuring out whether to score messages it hits at 
-1001 or -999.


NOTE: If you do this, you should only do it in 
smtpd_recipient_restrictions, and have the earlier restriction lists use 
a query that still just returns OK. i.e. add a new DB field and use it 
for the select_field in the smtpd_recipient_restrictions check.



Relevant sections of
master.cf:**


[snip]

Also possibly relevant: your other enabled transports.


*Relevant sections for main.cf:*


[snip]

Also possibly relevant: smtpd_delay_reject, mailbox_transport, 
default_transport, local_transport.


This is why the welcome message and the DEBUG_README advice sending 
postconf -nf and postconf -Mf output rather than what you think is 
relevant from config files.

Re: short circuit content_filters

[Patrick Ben Koetter]

* Michael Munger :
> 
> > Fix it in spamassassin. Use whitelist_from or better whitelist_from_dkim.
> > See "perldoc Mail::SpamAssassin::Conf" for config instructions.
> 
> Seems legit. How do I configure SpamAssassin to look up the domain in MySQL?

There may be a way to do that. But I think it would be better to ask this on
the Spamassassin Mailing list.

p@rick

-- 
[*] sys4 AG
 
https://sys4.de, +49 (89) 30 90 46 64
Schleißheimer Straße 26/MG,80333 München
 
Sitz der Gesellschaft: München, Amtsgericht München: HRB 199263
Vorstand: Patrick Ben Koetter, Marc Schiffbauer
Aufsichtsratsvorsitzender: Florian Kirstein
 

Re: short circuit content_filters

[Michael Munger]

> Fix it in spamassassin. Use whitelist_from or better whitelist_from_dkim.
> See "perldoc Mail::SpamAssassin::Conf" for config instructions.

Seems legit. How do I configure SpamAssassin to look up the domain in MySQL?

Michael Munger, dCAP, MCPS, MCNPS, MBSS
High Powered Help, Inc.
Microsoft Certified Professional
Microsoft Certified Small Business Specialist
Digium Certified Asterisk Professional
mich...@highpoweredhelp.com

Re: short circuit content_filters

[Patrick Ben Koetter]

* Michael Munger :
> TLDR:
> 
> How do I configure Postfix to NOT use spamassassin to scan the email if
> it is coming from a domain that is whitelisted via check_sender_access?

Fix it in spamassassin. Use whitelist_from or better whitelist_from_dkim.
See "perldoc Mail::SpamAssassin::Conf" for config instructions.

Alternatively put amavis in between Postfix and Spamassassin and exempt the
domains from spam filtering.

p@rick

-- 
[*] sys4 AG
 
https://sys4.de, +49 (89) 30 90 46 64
Schleißheimer Straße 26/MG,80333 München
 
Sitz der Gesellschaft: München, Amtsgericht München: HRB 199263
Vorstand: Patrick Ben Koetter, Marc Schiffbauer
Aufsichtsratsvorsitzender: Florian Kirstein
 

short circuit content_filters

[Michael Munger]

TLDR:

How do I configure Postfix to NOT use spamassassin to scan the email if
it is coming from a domain that is whitelisted via check_sender_access?

DETAILS:


We maintain a white-list of company domains that are aggregated from all
our employee's address books.

We want all email that claims to be from those domains (even if they are
spam, forged, contain cryptolocker, etc...). It's full-on, highway to
the danger zone, unbridled email acceptance if your domain is on this
whitelist.

For the most part, the configs below work. check_sender_access queries
our MySQL database and gets an "OK" for these domains. And, (I believe)
once this is returned, the remainder of the checking stops for that step
in the process. (Please correct me if I am wrong here, but that IS the
desired behavior).

We still have a few automated emails (these are order confirmations)
that are getting caught by spamassassin DESPITE being on the whitelist
because they are HTML only, and they talk about money if you fill out a
form. And, Spamassassin is executing post-queue, where these rules do
not apply post-queue, and only upon receipt.

Example of a good message caught in the bad filter:

Content analysis details: (2.8 points, 2.0 required) pts rule name
description  --
-- -0.0
RCVD_IN_MSPIKE_H2 RBL: Average reputation (+2) [67.231.157.48 listed in
wl.mailspike.net] 1.1 MIME_HTML_ONLY BODY: Message only has text/html
MIME parts 0.0 HTML_MESSAGE BODY: HTML included in message 0.0
LOTS_OF_MONEY Huge... sums of money 0.6 HTML_MIME_NO_HTML_TAG HTML-only
message, but there is no HTML tag 0.0 T_FILL_THIS_FORM_SHORT Fill in a
short form with personal information 1.0 MONEY_FORM_SHORT Lots of money
if you fill out a short form

How do I configure Postfix to NOT use spamassassin to scan the email if
it is coming from one of these whitelisted domains? Or, in the
alternative, can I configure Postfix to add a header to emails that are
on the whitelist, and then configure spam assassin to not check emails
with that header?

Relevant sections of
master.cf:**

# ==
# service type  private unpriv  chroot  wakeup  maxproc command + args
#   (yes)   (yes)   (yes)   (never) (100)
# ==
smtp  inet  n   -   -   -   -   smtpd
-o content_filter=spamassassin

# ... other stuff ... #

spamassassin unix - n   n   -   -   pipe
  user=debian-spamd argv=/usr/bin/spamc -f -e /usr/sbin/sendmail -oi -f
${sender} ${recipient}

*Relevant sections for main.cf:*

smtpd_recipient_restrictions = reject_invalid_hostname,
reject_unauth_pipelining,
reject_unknown_recipient_domain,
reject_unauth_destination,
permit_mynetworks,
permit_sasl_authenticated,
check_sender_access mysql:/etc/postfix/whitelist.cf,
check_policy_service unix:private/policy-spf
permit

smtpd_helo_restrictions = permit_mynetworks,
check_sender_access mysql:/etc/postfix/whitelist.cf,
permit_sasl_authenticated,
reject_invalid_helo_hostname,
reject_non_fqdn_helo_hostname,
reject_unknown_helo_hostname

# 1. We whitelist customers as best we can regardless of what a cesspool
their email servers, IP address, or network neighborhood might be.
# 2. Then, we let the RBLs do their job to lessen the load on the MySQL
server.
# 3. If the email passes all those checks, it's time to see if it is on
our custom blacklists. If so, we reject, otherwise, we'll let it
continue down the chain.

smtpd_client_restrictions = check_sender_access
mysql:/etc/postfix/whitelist.cf,
reject_rbl_client dnsbl.sorbs.net,
reject_rbl_client b.barracudacentral.org,
reject_rbl_client cbl.abuseat.org,
reject_rbl_client bl.spamcop.net,
reject_rbl_client zen.spamhaus.org,
reject_rbl_client dnsbl-1.uceprotect.net,
reject_rbl_client ubl.unsubscore.com,
reject_rbl_client dnsrbl.org,
reject_rbl_client bad.psky.me,
reject_rbl_client dnsbl-2.uceprotect.net
check_sender_access mysql:/etc/postfix/blacklist.cf

Incidentally, blacklist.cf rejects an email if it is coming from a
domain that NONE of our customers have ever sent from before. This
eliminates day-old-bread TLDs that are just spam factories. (.top for
example is spewing forth untold mountains of spam right now based on our
logs.).
-- 
Michael Munger, dCAP, MCPS, MCNPS, MBSS
High Powered Help, Inc.
Microsoft Certified Professional
Microsoft Certified Small Business Specialist