Re: spamd - blacklists

[Peter Hessler]

On 2016 Mar 16 (Wed) at 10:53:36 +1100 (+1100), Damien Miller wrote:
:On Tue, 15 Mar 2016, li...@wrant.com wrote:
:
:> What's going on with the BGP as a transport then, is it available to
:> the general public?  Must be much better than the fubar DNS.  Nackts
:> thing and we'd be attempting carping on tunnelled over DNS syndrome.
:
:Years ago I added the pftable keyword to bgpd.conf for this very
:reason. Assuming it hasn't bitrotted, it's trivial to use bgpd
:to fill a PF table that can be used to block or tarpit spammers.
:

This feature is used quite heavily by my bgp-spamd.net system, and has
certainly not bit-rotted :).


-- 
The past always looks better than it was.  It's only pleasant because
it isn't here.
-- Finley Peter Dunne (Mr. Dooley)

Re: spamd - blacklists

[Stuart Henderson]

On 2016/03/16 10:53, Damien Miller wrote:
> On Tue, 15 Mar 2016, li...@wrant.com wrote:
> 
> > What's going on with the BGP as a transport then, is it available to
> > the general public?  Must be much better than the fubar DNS.  Nackts
> > thing and we'd be attempting carping on tunnelled over DNS syndrome.
> 
> Years ago I added the pftable keyword to bgpd.conf for this very
> reason. Assuming it hasn't bitrotted, it's trivial to use bgpd
> to fill a PF table that can be used to block or tarpit spammers.

The default setup with spamd is in greylisting mode, this mode doesn't use
a big PF table for blacklist but instead the blacklist-or-greylist decision
is done inside spamd itself, the only PF table involved is the (usually much
smaller) one containing hosts that have previously made it through greylisting.

It is possible to use BGP to distribute addresses for spamd but the usual
setup for this involves 'bgpctl sh rib' to write out a file for spamdb to
read so it's not realtime.

There is another way to use spamd, in blacklist-only mode (-b flag), which
does use a PF table to hold the blacklist. In that case I think the table
could be populated directly from BGP but I don't know if that combines
well with also loading a blacklist via spamd-setup. Additionally for
large blacklists this can end up using a bunch of kvm, some people did
run into problems with this in the past before spamd changed to using
the greylisting mode by default.

Re: spamd - blacklists

[Damien Miller]

On Tue, 15 Mar 2016, li...@wrant.com wrote:

> What's going on with the BGP as a transport then, is it available to
> the general public?  Must be much better than the fubar DNS.  Nackts
> thing and we'd be attempting carping on tunnelled over DNS syndrome.

Years ago I added the pftable keyword to bgpd.conf for this very
reason. Assuming it hasn't bitrotted, it's trivial to use bgpd
to fill a PF table that can be used to block or tarpit spammers.

-d

Re: spamd - blacklists

[lists]

Tue, 15 Mar 2016 12:52:06 -0400 Michael McConville 
> Stuart Henderson wrote:
> > On 2016/03/15 12:55, Craig Skinner wrote:  
> > > There are a few more paid rsync lists here:
> > > http://en.wikipedia.org/wiki/Comparison_of_DNS_blacklists  
> > 
> > Ah that is a useful page. Maybe we could list it, e.g.
> > 
> > Index: spamd.conf
> > ===
> > RCS file: /cvs/src/etc/mail/spamd.conf,v
> > retrieving revision 1.5
> > diff -u -p -r1.5 spamd.conf
> > --- spamd.conf  14 Mar 2016 21:36:52 -  1.5
> > +++ spamd.conf  15 Mar 2016 13:27:04 -
> > @@ -13,8 +13,10 @@
> >  # Lists specified with the :white: capability apply to the previous
> >  # list with a :black: capability.
> >  #
> > -# As of November 2004, a place to search for blacklists is
> > -# http://spamlinks.net/filter-bl.htm
> > +# As of March 2016, a place to search for blacklists is
> > +# http://en.wikipedia.org/wiki/Comparison_of_DNS_blacklists
> > +# - most of these are DNS-based only and cannot be used with spamd(8),
> > +# but some of the lists also provide access to text files via rsync.
> >  
> >  all:\
> > :uatraps:nixspam:  
> 
> ok mmcc@
> 
> > > Generally, everything has changed from file feeds to DNS.  
> > 
> > Yep, because for the more actively maintained ones 1) new entries show
> > up more quickly than any sane rsync interval, this is quite important
> > for good blocking these days 2) DNS is less resource intensive and more
> > easily distributed than rsync, and 3) importantly for the rbl providers,
> > it gives additional input to them about new mail sources (if an rbl
> > suddenly starts seeing queries from all over the world for a previously
> > unseen address, it's probably worth investigation - I am sure this is
> > why some of the commercial antispam operators provide free DNS-based
> > lookups for smaller orgs).
> > 
> > A more flexible approach would be to skip the PF table integration
> > completely and do DNS lookups in spamd (or, uh, relayd, or something
> > new) and based on that it could choose whether to tarpit, greylist or
> > transparent-forward the connection to the real mail server. This
> > would also give a way to use dnswl.org's whitelist to avoid greylisting
> > for those hosts where it just doesn't work well (gmail, office365 etc).  
> 
> Interesting, I didn't even know that rsync blacklists existed. That was
> the cause for confusion about Spamhaus's price earlier.

A list is a flat data file, which is text, word / line oriented.  The
DNS transport for this is incomprehensible even on hard attempt at it.
Records, routes, filters, zones, regions, are text, should be carried
by all means using text interfaces.  No matter word, line or region at
a time.

What's going on with the BGP as a transport then, is it available to
the general public?  Must be much better than the fubar DNS.  Nackts
thing and we'd be attempting carping on tunnelled over DNS syndrome.

Constructively, it's all about money extortion when it comes to email
malpractice, nobody wants to stop spam except a handful of idealists,
so resource lists are another financial abstraction to allow only the
good much more worthier spam you actually need to reach your mailbox.

> Would it make sense to enable a blacklist or two by default in spamd?
> They seem to be an effectively necessary part of a sane mail server
> configuration these days.

Commercial interests, are best avoided by peer review.  So, why not
enable OpenBSD own block list then?  It would just take a few mythic
man years to maintain.  Hm, never thought of it this way, scratch it.

Re: spamd - blacklists

[Bob Beck]

Absolutely not. do not enable a blacklist by default


On Tue, Mar 15, 2016 at 10:52 AM, Michael McConville  wrote:
> Stuart Henderson wrote:
>> On 2016/03/15 12:55, Craig Skinner wrote:
>> > There are a few more paid rsync lists here:
>> > http://en.wikipedia.org/wiki/Comparison_of_DNS_blacklists
>>
>> Ah that is a useful page. Maybe we could list it, e.g.
>>
>> Index: spamd.conf
>> ===
>> RCS file: /cvs/src/etc/mail/spamd.conf,v
>> retrieving revision 1.5
>> diff -u -p -r1.5 spamd.conf
>> --- spamd.conf14 Mar 2016 21:36:52 -  1.5
>> +++ spamd.conf15 Mar 2016 13:27:04 -
>> @@ -13,8 +13,10 @@
>>  # Lists specified with the :white: capability apply to the previous
>>  # list with a :black: capability.
>>  #
>> -# As of November 2004, a place to search for blacklists is
>> -# http://spamlinks.net/filter-bl.htm
>> +# As of March 2016, a place to search for blacklists is
>> +# http://en.wikipedia.org/wiki/Comparison_of_DNS_blacklists
>> +# - most of these are DNS-based only and cannot be used with spamd(8),
>> +# but some of the lists also provide access to text files via rsync.
>>
>>  all:\
>>   :uatraps:nixspam:
>
> ok mmcc@
>
>> > Generally, everything has changed from file feeds to DNS.
>>
>> Yep, because for the more actively maintained ones 1) new entries show
>> up more quickly than any sane rsync interval, this is quite important
>> for good blocking these days 2) DNS is less resource intensive and more
>> easily distributed than rsync, and 3) importantly for the rbl providers,
>> it gives additional input to them about new mail sources (if an rbl
>> suddenly starts seeing queries from all over the world for a previously
>> unseen address, it's probably worth investigation - I am sure this is
>> why some of the commercial antispam operators provide free DNS-based
>> lookups for smaller orgs).
>>
>> A more flexible approach would be to skip the PF table integration
>> completely and do DNS lookups in spamd (or, uh, relayd, or something
>> new) and based on that it could choose whether to tarpit, greylist or
>> transparent-forward the connection to the real mail server. This
>> would also give a way to use dnswl.org's whitelist to avoid greylisting
>> for those hosts where it just doesn't work well (gmail, office365 etc).
>
> Interesting, I didn't even know that rsync blacklists existed. That was
> the cause for confusion about Spamhaus's price earlier.
>
> Would it make sense to enable a blacklist or two by default in spamd?
> They seem to be an effectively necessary part of a sane mail server
> configuration these days.
>

Re: spamd - blacklists

[Theo de Raadt]

> > Generally, everything has changed from file feeds to DNS.
> 
> Yep, because for the more actively maintained ones 1) new entries show
> up more quickly than any sane rsync interval, this is quite important
> for good blocking these days 2) DNS is less resource intensive and more
> easily distributed than rsync, and 3) importantly for the rbl providers,
> it gives additional input to them about new mail sources (if an rbl
> suddenly starts seeing queries from all over the world for a previously
> unseen address, it's probably worth investigation - I am sure this is
> why some of the commercial antispam operators provide free DNS-based
> lookups for smaller orgs).
> 
> A more flexible approach would be to skip the PF table integration
> completely and do DNS lookups in spamd (or, uh, relayd, or something
> new) and based on that it could choose whether to tarpit, greylist or
> transparent-forward the connection to the real mail server. This
> would also give a way to use dnswl.org's whitelist to avoid greylisting
> for those hosts where it just doesn't work well (gmail, office365 etc).

The argument does not make sense.

There are numerous tools that can do DNS lookup in userland.

spamd was the only one which could do it via radix tables on the
kernel edge, which even allows you to do management in a bridge.

pf integration is the basis of spamd, that is how it steps out of
the accepted address list, so that userland can make the decision.

Your proposal can be summed up as: delete spamd.

Re: spamd - blacklists

[Michael McConville]

Stuart Henderson wrote:
> On 2016/03/15 12:55, Craig Skinner wrote:
> > There are a few more paid rsync lists here:
> > http://en.wikipedia.org/wiki/Comparison_of_DNS_blacklists
> 
> Ah that is a useful page. Maybe we could list it, e.g.
> 
> Index: spamd.conf
> ===
> RCS file: /cvs/src/etc/mail/spamd.conf,v
> retrieving revision 1.5
> diff -u -p -r1.5 spamd.conf
> --- spamd.conf14 Mar 2016 21:36:52 -  1.5
> +++ spamd.conf15 Mar 2016 13:27:04 -
> @@ -13,8 +13,10 @@
>  # Lists specified with the :white: capability apply to the previous
>  # list with a :black: capability.
>  #
> -# As of November 2004, a place to search for blacklists is
> -# http://spamlinks.net/filter-bl.htm
> +# As of March 2016, a place to search for blacklists is
> +# http://en.wikipedia.org/wiki/Comparison_of_DNS_blacklists
> +# - most of these are DNS-based only and cannot be used with spamd(8),
> +# but some of the lists also provide access to text files via rsync.
>  
>  all:\
>   :uatraps:nixspam:

ok mmcc@

> > Generally, everything has changed from file feeds to DNS.
> 
> Yep, because for the more actively maintained ones 1) new entries show
> up more quickly than any sane rsync interval, this is quite important
> for good blocking these days 2) DNS is less resource intensive and more
> easily distributed than rsync, and 3) importantly for the rbl providers,
> it gives additional input to them about new mail sources (if an rbl
> suddenly starts seeing queries from all over the world for a previously
> unseen address, it's probably worth investigation - I am sure this is
> why some of the commercial antispam operators provide free DNS-based
> lookups for smaller orgs).
> 
> A more flexible approach would be to skip the PF table integration
> completely and do DNS lookups in spamd (or, uh, relayd, or something
> new) and based on that it could choose whether to tarpit, greylist or
> transparent-forward the connection to the real mail server. This
> would also give a way to use dnswl.org's whitelist to avoid greylisting
> for those hosts where it just doesn't work well (gmail, office365 etc).

Interesting, I didn't even know that rsync blacklists existed. That was
the cause for confusion about Spamhaus's price earlier.

Would it make sense to enable a blacklist or two by default in spamd?
They seem to be an effectively necessary part of a sane mail server
configuration these days.

Re: spamd - blacklists

[Stuart Henderson]

On 2016/03/15 12:55, Craig Skinner wrote:
> Hi Stuart,
> 
> On 2016-03-14 Mon 16:27 PM |, Stuart Henderson wrote:
> > 
> > There aren't many who provide their whole dataset to anyone other
> > than paying customers - e.g. Spamhaus' rsync feeds are for
> > organisations with >5000 users and cost US$1700+/year.
> > 
> 
> I've found these free rsync feeds useful:
> 
> The Passive Spam Block List (collates IPs sending to spam traps):
> http://psbl.org/howto/
> CBL (SpamHaus) writes: "The PSBL is a solid and reliable DNSBL.
> Amazingly effective for such a modest effort. Generally recommended"
> http://www.abuseat.org/faq.html
> 
> UCE Protect (IPs sending to spam traps, and more aggresive options):
> http://www.uceprotect.net/en/index.php?m=6=10
> 
> The Composite Blocking List (CBL - a big part of SpamHaus DNSRBLs) can
> be rsync'd after rego (free, execpt for spam filter service operators):
> http://www.abuseat.org/faq.html
> 
> There are a few more paid rsync lists here:
> http://en.wikipedia.org/wiki/Comparison_of_DNS_blacklists

Ah that is a useful page. Maybe we could list it, e.g.

Index: spamd.conf
===
RCS file: /cvs/src/etc/mail/spamd.conf,v
retrieving revision 1.5
diff -u -p -r1.5 spamd.conf
--- spamd.conf  14 Mar 2016 21:36:52 -  1.5
+++ spamd.conf  15 Mar 2016 13:27:04 -
@@ -13,8 +13,10 @@
 # Lists specified with the :white: capability apply to the previous
 # list with a :black: capability.
 #
-# As of November 2004, a place to search for blacklists is
-# http://spamlinks.net/filter-bl.htm
+# As of March 2016, a place to search for blacklists is
+# http://en.wikipedia.org/wiki/Comparison_of_DNS_blacklists
+# - most of these are DNS-based only and cannot be used with spamd(8),
+# but some of the lists also provide access to text files via rsync.
 
 all:\
:uatraps:nixspam:

> Generally, everything has changed from file feeds to DNS.

Yep, because for the more actively maintained ones 1) new entries show
up more quickly than any sane rsync interval, this is quite important
for good blocking these days 2) DNS is less resource intensive and more
easily distributed than rsync, and 3) importantly for the rbl providers,
it gives additional input to them about new mail sources (if an rbl
suddenly starts seeing queries from all over the world for a previously
unseen address, it's probably worth investigation - I am sure this is
why some of the commercial antispam operators provide free DNS-based
lookups for smaller orgs).

A more flexible approach would be to skip the PF table integration
completely and do DNS lookups in spamd (or, uh, relayd, or something
new) and based on that it could choose whether to tarpit, greylist or
transparent-forward the connection to the real mail server. This
would also give a way to use dnswl.org's whitelist to avoid greylisting
for those hosts where it just doesn't work well (gmail, office365 etc).

Re: spamd - blacklists

[Craig Skinner]

Hi Stuart,

On 2016-03-14 Mon 16:27 PM |, Stuart Henderson wrote:
> 
> There aren't many who provide their whole dataset to anyone other
> than paying customers - e.g. Spamhaus' rsync feeds are for
> organisations with >5000 users and cost US$1700+/year.
> 

I've found these free rsync feeds useful:

The Passive Spam Block List (collates IPs sending to spam traps):
http://psbl.org/howto/
CBL (SpamHaus) writes: "The PSBL is a solid and reliable DNSBL.
Amazingly effective for such a modest effort. Generally recommended"
http://www.abuseat.org/faq.html

UCE Protect (IPs sending to spam traps, and more aggresive options):
http://www.uceprotect.net/en/index.php?m=6=10

The Composite Blocking List (CBL - a big part of SpamHaus DNSRBLs) can
be rsync'd after rego (free, execpt for spam filter service operators):
http://www.abuseat.org/faq.html

There are a few more paid rsync lists here:
http://en.wikipedia.org/wiki/Comparison_of_DNS_blacklists

Generally, everything has changed from file feeds to DNS.

Re: spamd - blacklists

[Bob Beck]

So I'd say remove it until we decide on an alternative :)

On Mon, Mar 14, 2016 at 10:27 AM, Stuart Henderson  wrote:
> On 2016/03/14 10:20, Michael McConville wrote:
>> Craig Skinner wrote:
>> > Hi Hans,
>> >
>> > On 2016-03-14 Mon 11:49 AM |, hans wrote:
>> > > On Mar 13 18:56:00, mm...@mykolab.com wrote:
>> > > > hans wrote:
>> > > > > The link to "the place to search for blacklists" is dead.
>> > > >
>> > > > Might be better to replace it than to remove it.
>> > >
>> > > Sure. Any suggestions?
>> > >
>> >
>> > Some DNSRBLs are available as files or rsync feeds.
>> >
>> > It takes a bit of digging about, so start with effective ones:
>> >
>> > http://www.intra2net.com/en/support/antispam/
>> > http://www.spamcannibal.org/dnsbl_compare.shtml
>> > http://en.wikipedia.org/wiki/Comparison_of_DNS_blacklists
>> > http://multirbl.valli.org/list/
>> >
>> > Offenders can be checked in all at once on:
>> > http://multirbl.valli.org/dnsbl-lookup/
>>
>> I think Spamhaus is the canonical one these days.
>>
>
> There aren't many who provide their whole dataset to anyone other
> than paying customers - e.g. Spamhaus' rsync feeds are for
> organisations with >5000 users and cost US$1700+/year.
>

Re: spamd - blacklists

[Stuart Henderson]

On 2016/03/14 10:20, Michael McConville wrote:
> Craig Skinner wrote:
> > Hi Hans,
> > 
> > On 2016-03-14 Mon 11:49 AM |, hans wrote:
> > > On Mar 13 18:56:00, mm...@mykolab.com wrote:
> > > > hans wrote:
> > > > > The link to "the place to search for blacklists" is dead.
> > > > 
> > > > Might be better to replace it than to remove it.
> > > 
> > > Sure. Any suggestions?
> > > 
> > 
> > Some DNSRBLs are available as files or rsync feeds.
> > 
> > It takes a bit of digging about, so start with effective ones:
> > 
> > http://www.intra2net.com/en/support/antispam/
> > http://www.spamcannibal.org/dnsbl_compare.shtml
> > http://en.wikipedia.org/wiki/Comparison_of_DNS_blacklists
> > http://multirbl.valli.org/list/
> > 
> > Offenders can be checked in all at once on:
> > http://multirbl.valli.org/dnsbl-lookup/
> 
> I think Spamhaus is the canonical one these days.
> 

There aren't many who provide their whole dataset to anyone other
than paying customers - e.g. Spamhaus' rsync feeds are for
organisations with >5000 users and cost US$1700+/year.

Re: spamd - blacklists

[Michael McConville]

Craig Skinner wrote:
> Hi Hans,
> 
> On 2016-03-14 Mon 11:49 AM |, hans wrote:
> > On Mar 13 18:56:00, mm...@mykolab.com wrote:
> > > hans wrote:
> > > > The link to "the place to search for blacklists" is dead.
> > > 
> > > Might be better to replace it than to remove it.
> > 
> > Sure. Any suggestions?
> > 
> 
> Some DNSRBLs are available as files or rsync feeds.
> 
> It takes a bit of digging about, so start with effective ones:
> 
> http://www.intra2net.com/en/support/antispam/
> http://www.spamcannibal.org/dnsbl_compare.shtml
> http://en.wikipedia.org/wiki/Comparison_of_DNS_blacklists
> http://multirbl.valli.org/list/
> 
> Offenders can be checked in all at once on:
> http://multirbl.valli.org/dnsbl-lookup/

I think Spamhaus is the canonical one these days.

Re: spamd - blacklists

[Craig Skinner]

Hi Hans,

On 2016-03-14 Mon 11:49 AM |, hans wrote:
> On Mar 13 18:56:00, mm...@mykolab.com wrote:
> > hans wrote:
> > > The link to "the place to search for blacklists" is dead.
> > 
> > Might be better to replace it than to remove it.
> 
> Sure. Any suggestions?
> 

Some DNSRBLs are available as files or rsync feeds.

It takes a bit of digging about, so start with effective ones:

http://www.intra2net.com/en/support/antispam/
http://www.spamcannibal.org/dnsbl_compare.shtml
http://en.wikipedia.org/wiki/Comparison_of_DNS_blacklists
http://multirbl.valli.org/list/

Offenders can be checked in all at once on:
http://multirbl.valli.org/dnsbl-lookup/

Re: spamd - blacklists

[hans]

On Mar 13 18:56:00, mm...@mykolab.com wrote:
> hans wrote:
> > The link to "the place to search for blacklists" is dead.
> 
> Might be better to replace it than to remove it.

Sure. Any suggestions?

> > Index: spamd.conf
> > ===
> > RCS file: /cvs/src/etc/mail/spamd.conf,v
> > retrieving revision 1.4
> > diff -u -p -r1.4 spamd.conf
> > --- spamd.conf  14 May 2012 16:58:46 -  1.4
> > +++ spamd.conf  13 Mar 2016 14:15:54 -
> > @@ -12,9 +12,6 @@
> >  # "all" must be here, and defines the order in which lists are applied.
> >  # Lists specified with the :white: capability apply to the previous
> >  # list with a :black: capability.
> > -#
> > -# As of November 2004, a place to search for blacklists is
> > -# http://spamlinks.net/filter-bl.htm
> >  
> >  all:\
> > :uatraps:nixspam:
> > 

Re: spamd - blacklists

[Michael McConville]

hans wrote:
> The link to "the place to search for blacklists" is dead.

Might be better to replace it than to remove it.

> Index: spamd.conf
> ===
> RCS file: /cvs/src/etc/mail/spamd.conf,v
> retrieving revision 1.4
> diff -u -p -r1.4 spamd.conf
> --- spamd.conf14 May 2012 16:58:46 -  1.4
> +++ spamd.conf13 Mar 2016 14:15:54 -
> @@ -12,9 +12,6 @@
>  # "all" must be here, and defines the order in which lists are applied.
>  # Lists specified with the :white: capability apply to the previous
>  # list with a :black: capability.
> -#
> -# As of November 2004, a place to search for blacklists is
> -# http://spamlinks.net/filter-bl.htm
>  
>  all:\
>   :uatraps:nixspam:
> 

spamd - blacklists

[hans]

The link to "the place to search for blacklists" is dead.

Jan

Index: spamd.conf
===
RCS file: /cvs/src/etc/mail/spamd.conf,v
retrieving revision 1.4
diff -u -p -r1.4 spamd.conf
--- spamd.conf  14 May 2012 16:58:46 -  1.4
+++ spamd.conf  13 Mar 2016 14:15:54 -
@@ -12,9 +12,6 @@
 # "all" must be here, and defines the order in which lists are applied.
 # Lists specified with the :white: capability apply to the previous
 # list with a :black: capability.
-#
-# As of November 2004, a place to search for blacklists is
-# http://spamlinks.net/filter-bl.htm
 
 all:\
:uatraps:nixspam: