Re: How to create my personal RBL
Thank you for this interesting list. Unfortunately my external MTA is based on exim. I think that a lot of the checks you list are already performed by exim and by the rules we added to that MTA. I will check one by one anyway. Perhaps the most interesting idea, something we were already thinking about, is to move spamassassin away from zimbra and give it a standalone server. We will lose something in integration but we can be free from zimbra release cycles. On Thu, Jun 27, 2019 at 2:38 PM David Jones wrote: > On 6/26/19 3:43 AM, hg user wrote: > > Thank you everybody for your really interesting answers. In this moment > > I'm just collecting informations. > > > > I have one main problem: one of the engines used by our commercial > > antispam solution returns too many FPs. I'm gradually introducing > > spamassassin (included in zimbra) and I'd like to mitigate the FPs with > > some other checks... using a proven, well-known technology like AskDNS > > seems a quick and viable solution to me. > > > > Unfortunately a personal RBL may not cover all the use cases I'm > > thinking about and looking at the source code of a plugin that queries a > > sql or redis server can be interesting. > > Before you start working on a custom plugin, have you tuned out your MTA > and SpamAssasin? From my personal experience, I setup an edge MTA as > the MX and sent filtered mail to Zimbra and smarthosted from Zimbra back > to the edge MTA. This provides the most flexibility to upgrade perl and > SpamAssassin to the latest version along with many other benefits. > > Tuning out the MTA: > - Setup Postfix with Postscreen > - Enable weighted RBLs in Postscreen, lots of them. See the SA mailing > list archives for "postscreen_dnsbl_sites". >__This will block 80% or more of spam/junk alone.__ > - Setup postfwd to give extra control to add headers based on SMTP > conversation time so SA can use those headers later. For example, I set > headers based on the number of recipients which is very useful when > email has been BCC'd. > - Setup sqlgrey and slowly phase it in where users won't even know it. > - Setup policyd-spf, OpenDMARC, and OpenDKIM > - Setup fail2ban for repeat spammers/bots > - Setup Postwhite to whitelist trusted senders by their SPF record. > This allows for turning up other Postfix config settings > - Setup TLS with a Letsencrypt certificate > - Setup rate limiting then put exceptions in > smtpd_client_event_limit_exceptions. > - Postfix header_checks, body_checks, smtpd_client_restrictions, > smtpd_helo_restrictions, smtpd_sender_restrictions, > smtpd_relay_restrictions, smtpd_recipient_restrictions, > smtpd_data_restrictions in the main.cf can be tuned over time. > - Enable reject_unverified_recipient in smtpd_recipient_restrictions so > Postfix will "look ahead" to Zimbra and not accept invalid recipients. > - > > Tuning out SpamAssassin: > - Make sure your internal_networks and trusted_networks are correct so > RBL checks will happen correctly for the last external IP. I have > extended this out to Google, Office 365, and other major platforms to > detect the X-Originating-IP of the web/mail client. > - Install KAM.cf and KAMonly.cf > - Install DCC, Razor, Pyzor > - Install ClamAV unofficial (extra) signatures > - Add local rules to use the headers from OpenDMARC > - Enable extra RBLs that aren't in the stock SA > - I use the ShortCircuit plugin heavily, disable the ALL_TRUSTED > shortcircuit, and enable shortcircuit on a number of the USER_IN_* rules. > - I have created a massive list of whitelist_auth entries that are > mostly subdomain senders from trusted senders. > - Setup a way to train your Bayes easily by dragging email into a Spam > and Ham folder as things are misclassified to keep the Bayesian DB tuned > correctly. > - Get on the latest version of perl even if you have to compile it > because your OS might be older. > - Install the latest stable version of SpamAssassin. > - Many more things covered on this list over the years. > - I setup local DBLs and DWLs for brand new Office 365 senders and other > common sources of spam like secureserver.net, unifiedlayer.com, > websitewelcome.com, myregisteredsite.com, etc to add a couple of points > for new senders. Then I add good senders on those bad hosting platforms > to a DWL that subtracts a couple of points and excludes them from other > meta rules that amplifies certain scores for the spam. > > Note that a lot of this can be found by setting up a quick VM and > installing iRedMail to check out the Postfix configuration for the > milters mentioned above and the TLS configuration. It uses Amavisnew so > that might be different from how you want to "glue" SpamAssassin into > the MTA. > > I use MailScanner which has a few extra features of it's own in addition > to processing emails in batches for high volume mail flow. > > After I did all of that work above over many years, my mail filtering > accuracy is very good for about 80,000 mailboxes.
Re: How to create my personal RBL
On 6/26/19 3:43 AM, hg user wrote: > Thank you everybody for your really interesting answers. In this moment > I'm just collecting informations. > > I have one main problem: one of the engines used by our commercial > antispam solution returns too many FPs. I'm gradually introducing > spamassassin (included in zimbra) and I'd like to mitigate the FPs with > some other checks... using a proven, well-known technology like AskDNS > seems a quick and viable solution to me. > > Unfortunately a personal RBL may not cover all the use cases I'm > thinking about and looking at the source code of a plugin that queries a > sql or redis server can be interesting. Before you start working on a custom plugin, have you tuned out your MTA and SpamAssasin? From my personal experience, I setup an edge MTA as the MX and sent filtered mail to Zimbra and smarthosted from Zimbra back to the edge MTA. This provides the most flexibility to upgrade perl and SpamAssassin to the latest version along with many other benefits. Tuning out the MTA: - Setup Postfix with Postscreen - Enable weighted RBLs in Postscreen, lots of them. See the SA mailing list archives for "postscreen_dnsbl_sites". __This will block 80% or more of spam/junk alone.__ - Setup postfwd to give extra control to add headers based on SMTP conversation time so SA can use those headers later. For example, I set headers based on the number of recipients which is very useful when email has been BCC'd. - Setup sqlgrey and slowly phase it in where users won't even know it. - Setup policyd-spf, OpenDMARC, and OpenDKIM - Setup fail2ban for repeat spammers/bots - Setup Postwhite to whitelist trusted senders by their SPF record. This allows for turning up other Postfix config settings - Setup TLS with a Letsencrypt certificate - Setup rate limiting then put exceptions in smtpd_client_event_limit_exceptions. - Postfix header_checks, body_checks, smtpd_client_restrictions, smtpd_helo_restrictions, smtpd_sender_restrictions, smtpd_relay_restrictions, smtpd_recipient_restrictions, smtpd_data_restrictions in the main.cf can be tuned over time. - Enable reject_unverified_recipient in smtpd_recipient_restrictions so Postfix will "look ahead" to Zimbra and not accept invalid recipients. - Tuning out SpamAssassin: - Make sure your internal_networks and trusted_networks are correct so RBL checks will happen correctly for the last external IP. I have extended this out to Google, Office 365, and other major platforms to detect the X-Originating-IP of the web/mail client. - Install KAM.cf and KAMonly.cf - Install DCC, Razor, Pyzor - Install ClamAV unofficial (extra) signatures - Add local rules to use the headers from OpenDMARC - Enable extra RBLs that aren't in the stock SA - I use the ShortCircuit plugin heavily, disable the ALL_TRUSTED shortcircuit, and enable shortcircuit on a number of the USER_IN_* rules. - I have created a massive list of whitelist_auth entries that are mostly subdomain senders from trusted senders. - Setup a way to train your Bayes easily by dragging email into a Spam and Ham folder as things are misclassified to keep the Bayesian DB tuned correctly. - Get on the latest version of perl even if you have to compile it because your OS might be older. - Install the latest stable version of SpamAssassin. - Many more things covered on this list over the years. - I setup local DBLs and DWLs for brand new Office 365 senders and other common sources of spam like secureserver.net, unifiedlayer.com, websitewelcome.com, myregisteredsite.com, etc to add a couple of points for new senders. Then I add good senders on those bad hosting platforms to a DWL that subtracts a couple of points and excludes them from other meta rules that amplifies certain scores for the spam. Note that a lot of this can be found by setting up a quick VM and installing iRedMail to check out the Postfix configuration for the milters mentioned above and the TLS configuration. It uses Amavisnew so that might be different from how you want to "glue" SpamAssassin into the MTA. I use MailScanner which has a few extra features of it's own in addition to processing emails in batches for high volume mail flow. After I did all of that work above over many years, my mail filtering accuracy is very good for about 80,000 mailboxes. The more mailboxes and domains you filter, the more time it takes to tune everything properly. > > Thank you > Francesco > > On Tue, Jun 25, 2019 at 10:20 PM Matus UHLAR - fantomas > mailto:uh...@fantomas.sk>> wrote: > > >On Tue, 2019-06-25 at 11:09 -0500, David B Funk wrote: > >> that's way overthinking it. > > On 25.06.19 17:55, Martin Gregorie wrote: > >I agree, now that there's a configurable OSS dnsbl server available, > >that using it is the obvious choice for dealing with a standalone > list, > >but the OP did ask specifically about using database queries to > >implement a
Re: How to create my personal RBL
On Wed, 2019-06-26 at 10:43 +0200, hg user wrote:
> Unfortunately a personal RBL may not cover all the use cases I'm
> thinking about and looking at the source code of a plugin that queries
> a sql or redis server can be interesting.
>
If you can't find source for an SQL plugin, contact me off list and I'll
let you have a copy of mine together with the rule that triggers it. Its
more or less documented and should be fairly easy to adapt if you have a
basic knowledge of Perl and a copy of the "Camel book" ('Programming
Perl' by Larry Wall, Tom Christiansen & Jon Orwant, pub. O'Reily).
My plugin is not entirely original code because I modified an SQL plugin
I found on the 'net to query a view in my PostgreSQL database and
expanded the comments a bit. To install it in SA, you just put both
files in the same directory as local.cf
Martin
Re: How to create my personal RBL
Thank you everybody for your really interesting answers. In this moment I'm just collecting informations. I have one main problem: one of the engines used by our commercial antispam solution returns too many FPs. I'm gradually introducing spamassassin (included in zimbra) and I'd like to mitigate the FPs with some other checks... using a proven, well-known technology like AskDNS seems a quick and viable solution to me. Unfortunately a personal RBL may not cover all the use cases I'm thinking about and looking at the source code of a plugin that queries a sql or redis server can be interesting. Thank you Francesco On Tue, Jun 25, 2019 at 10:20 PM Matus UHLAR - fantomas wrote: > >On Tue, 2019-06-25 at 11:09 -0500, David B Funk wrote: > >> that's way overthinking it. > > On 25.06.19 17:55, Martin Gregorie wrote: > >I agree, now that there's a configurable OSS dnsbl server available, > >that using it is the obvious choice for dealing with a standalone list, > >but the OP did ask specifically about using database queries to > >implement a blacklist, so I thought it was worthwhile to tell him what's > >involved in doing that. > > No. The OP wanted to store data in DB to avoid restarting SA, not > mentioning > any other specific reason to use DB. > > using DNSBL does avoid restarting SA and does not require any plugin, which > is a great advantage. > > we are trying to provide described requirements, while avoiding proposed > complicated solutions. > > >For all I know the OP either has a similar archive or is intending to > >implement one: searching for a specific message with a database tool is > >a *lot* faster than ferreting through a set of very large mail folders > >with your MUA, though of course the effort of creating and maintaining > >the database, mail loader, query tools and SA plugin is non trivial. > > well, if THIS is the real reason... > > -- > Matus UHLAR - fantomas, uh...@fantomas.sk ; http://www.fantomas.sk/ > Warning: I wish NOT to receive e-mail advertising to this address. > Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu. > Saving Private Ryan... > Private Ryan exists. Overwrite? (Y/N) >
Re: How to create my personal RBL
On 6/25/19 10:11 AM, David Jones wrote: I use PowerDNS Recursor but Unbound or BIND would work fine. BIND has an option to load zone data from a database. Check out BIND's Dynamically Loadable Zones support. -- Grant. . . . unix || die smime.p7s Description: S/MIME Cryptographic Signature
Re: How to create my personal RBL
On Tue, 2019-06-25 at 11:09 -0500, David B Funk wrote: that's way overthinking it. On 25.06.19 17:55, Martin Gregorie wrote: I agree, now that there's a configurable OSS dnsbl server available, that using it is the obvious choice for dealing with a standalone list, but the OP did ask specifically about using database queries to implement a blacklist, so I thought it was worthwhile to tell him what's involved in doing that. No. The OP wanted to store data in DB to avoid restarting SA, not mentioning any other specific reason to use DB. using DNSBL does avoid restarting SA and does not require any plugin, which is a great advantage. we are trying to provide described requirements, while avoiding proposed complicated solutions. For all I know the OP either has a similar archive or is intending to implement one: searching for a specific message with a database tool is a *lot* faster than ferreting through a set of very large mail folders with your MUA, though of course the effort of creating and maintaining the database, mail loader, query tools and SA plugin is non trivial. well, if THIS is the real reason... -- Matus UHLAR - fantomas, uh...@fantomas.sk ; http://www.fantomas.sk/ Warning: I wish NOT to receive e-mail advertising to this address. Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu. Saving Private Ryan... Private Ryan exists. Overwrite? (Y/N)
Re: How to create my personal RBL
On Tue, 2019-06-25 at 11:09 -0500, David B Funk wrote: > that's way overthinking it. > David & David, I agree, now that there's a configurable OSS dnsbl server available, that using it is the obvious choice for dealing with a standalone list, but the OP did ask specifically about using database queries to implement a blacklist, so I thought it was worthwhile to tell him what's involved in doing that. I've been running a whitelist off my mail archive for around a decade now. My archive contains both incoming and outgoing messages and is held in a PostgreSQL database. My SA plugin whitelists anybody who I've sent mail to, so provides a very low maintenance whitelist since it automatically sees new outgoing messages as well as the effect of archive maintenance. For all I know the OP either has a similar archive or is intending to implement one: searching for a specific message with a database tool is a *lot* faster than ferreting through a set of very large mail folders with your MUA, though of course the effort of creating and maintaining the database, mail loader, query tools and SA plugin is non trivial. Martin
Re: How to create my personal RBL
On 6/25/19 10:20 AM, Martin Gregorie wrote: > On Tue, 2019-06-25 at 16:11 +0200, hg user wrote: >> I'd like to create my own RBL that answers queries about IP, domain or >> address reputation. >> Data should be stored in a database (mysql, postgres, redis, etc) so >> that information can be added/modified/removed without the need to >> restart spamassassin (I think the simpler solution would be a list in >> SA...) >> >> How can I create this setup? >> > You need to build a Perl plugin for Spamassassin that connects to, and > queries the database together with at least one SA rule that triggers > the plugin via an eval:plugin_query() call where plugin_query() is a > plugin function that runs the database query using data extracted from > the message by SA and returns either 1 (the query found a match in the > database) or zero (no matches found). > > > Martin > > Actually the SA part is very simple. Use the AskDNS SA plugin to do the DNS lookup: askdns MYRBL_ENV _SENDERDOMAIN_.dbl.example.com A /^127\.0\.0\.2$/ tflags MYRBL_ENV nice net describeMYRBL_ENV Sender's envelope domain listed in my RBL. score MYRBL_ENV askdns MYRBL_FROM _SENDERDOMAIN_.dbl.example.com A /^127\.0\.0\.2$/ tflags MYRBL_FROM nice net describeMYRBL_FROM Sender's From domain listed in my RBL. score MYRBL_FROM 0.001 The trickier part is to setup the DNS side. If you have a single SA host, you should already have a local caching DNS server and the /etc/resolv.conf and/or the SA DNS setting pointed to 127.0.0.1. I use PowerDNS Recursor but Unbound or BIND would work fine. Install rbldnsd for your distro and get it listening on an alternate port like 127.0.0.1:530. https://rbldnsd.io/ Create a text file with domains to block. This can come from a database with a web front-end or whatever you want. I have a database that I push records into from sources of spam and entries by a web interface. Then a script does a simple SELECT of the domains to a text file, then rsync's it to my 2 DNS servers that my 8 SA servers point to. Cron this for every 2-3 minutes and rbldnsd will gladly detect changes to the files without needing to be restarted/reloaded. I recommend putting a "test" entry at the top of the rbldnsd file so you can query test.dbl.example.com from a monitoring system to make sure it answers with the expected value. Then you setup your local caching DNS server to forward the dbl.example.com to 127.0.0.1:530. Note that this "dbl.example.com" doesn't have to be a real DNS zone. It could be "dbl.local" or whatever since it's only known by the local DNS server(s) that your SA server(s) are pointed to. These DNS servers should not be accessible by the Internet so they should be separate DNS caches dedicated to the SA server(s). If it's only one, then it could all be setup on 127.0.0.1. If it's a few, you could put rbldnsd on all of them and still use 127.0.0.1 and rsync the rbldnsd files to all of them locally. -- David Jones
Re: How to create my personal RBL
On Tue, 25 Jun 2019, Martin Gregorie wrote:
On Tue, 2019-06-25 at 16:11 +0200, hg user wrote:
I'd like to create my own RBL that answers queries about IP, domain or
address reputation.
Data should be stored in a database (mysql, postgres, redis, etc) so
that information can be added/modified/removed without the need to
restart spamassassin (I think the simpler solution would be a list in
SA...)
How can I create this setup?
You need to build a Perl plugin for Spamassassin that connects to, and
queries the database together with at least one SA rule that triggers
the plugin via an eval:plugin_query() call where plugin_query() is a
plugin function that runs the database query using data extracted from
the message by SA and returns either 1 (the query found a match in the
database) or zero (no matches found).
that's way overthinking it.
SA already has perfectly good DNS query tools built in, why not use those.
It's pretty simple to set up your own local private DNS zones using rbldnsd.
Adding/updating those kinds of zones is simple as adding or editing lines in
a text file (as simple as echo ".this.bad.domain :127.0.0.2:" >> my-zone-file ).
No muss no fuss, not server restart, etc.
I run two private zones for this purpose, one a IP address RBL list and one a
URIBL list.
--
Dave Funk University of Iowa
College of Engineering
319/335-5751 FAX: 319/384-0549 1256 Seamans Center
Sys_admin/Postmaster/cell_adminIowa City, IA 52242-1527
#include
Better is not better, 'standard' is better. B{
Re: How to create my personal RBL
On Tue, 2019-06-25 at 16:11 +0200, hg user wrote: > I'd like to create my own RBL that answers queries about IP, domain or > address reputation. > Data should be stored in a database (mysql, postgres, redis, etc) so > that information can be added/modified/removed without the need to > restart spamassassin (I think the simpler solution would be a list in > SA...) > > How can I create this setup? > You need to build a Perl plugin for Spamassassin that connects to, and queries the database together with at least one SA rule that triggers the plugin via an eval:plugin_query() call where plugin_query() is a plugin function that runs the database query using data extracted from the message by SA and returns either 1 (the query found a match in the database) or zero (no matches found). Martin
Re: How to create my personal RBL
On 25.06.19 16:11, hg user wrote: I'd like to create my own RBL that answers queries about IP, domain or address reputation. Data should be stored in a database (mysql, postgres, redis, etc) so that information can be added/modified/removed without the need to restart spamassassin (I think the simpler solution would be a list in SA...) you don't need restart SA to refresh RBL data. rbldnsd stores data in simple files but is able to reload them automatically. -- Matus UHLAR - fantomas, uh...@fantomas.sk ; http://www.fantomas.sk/ Warning: I wish NOT to receive e-mail advertising to this address. Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu. (R)etry, (A)bort, (C)ancer
How to create my personal RBL
I'd like to create my own RBL that answers queries about IP, domain or address reputation. Data should be stored in a database (mysql, postgres, redis, etc) so that information can be added/modified/removed without the need to restart spamassassin (I think the simpler solution would be a list in SA...) How can I create this setup? Thank you Francesco
