Re: [mailop] Should mail servers publish IPv6 MX records? Could this harm your spam filtering?

[Bill Cole]

On 10 Jun 2018, at 17:26, Al Iverson via mailop wrote:


  example.net   IN   MX   0   .


Nice. It's been listed in a few best practice documents as well as 
RFC 7505 for more than a few years. Good to see it getting some 
traction.


I cry a little inside every time I see a null MX, think of the lost
opportunity to be handling it as a spamtrap domain instead.


It's not a lost opportunity, it's aging. A null MX is like a charred 
white oak barrel for a mail-dead domain.


___
mailop mailing list
mailop@mailop.org
https://chilli.nosignal.org/cgi-bin/mailman/listinfo/mailop

Re: [mailop] Should mail servers publish IPv6 MX records? Could this harm your spam filtering?

[Stefano Bagnara]

On Fri, 8 Jun 2018 at 18:14, Stefano Bagnara  wrote:
> On Fri, 8 Jun 2018 at 17:53, Michael Peddemors  wrote:
> > [...]
> > And while using that as feedback might seem the logical conclusion, in
> > the real world we still see more feedback reports from legitimate email
> > the customer should have wanted, vs emails tagged as spam that are spam.
>
> Well, this is very surprising to me. Anyone else record similar scenario?

Michael, I just noticed that I read your sentence in a wrong way. I read "more
no-spam reports than spam reports" while I should have read "most of
the spam reports received are submitted by mistake".
So my previous reply was written after this misunderstanding.

Now, if most spam reports are submitted by mistake by people that
doesn't really want to complain/stop received emails from that sender
then this make it even more important to be able to collect false
positives (or are you telling that user-originated-feedback is not
trustable enough that you can't use it for "false positives
reports"?).

On Fri, 8 Jun 2018 at 17:53, Michael Peddemors  wrote:
> IMHO, and the way most of our platforms are designed to work.. Empower
> the users when you can.. but block the worst of the worst..
>
> * Block at SMTP via RBL's that have very low false positive rates

How can you tell an RBL have "very low false positive rates" if most
of the users of that RBL use it at SMTP time (or if the "not-spam"
reports don't flow to the RBL operator)? This is a dog biting its
tail...

I read "Very low false positive rates" as "Very low
collected-false-positive rates" where we miss a
"collected-false-positive against false-positives rate".

In my small system I don't have a "false positive report", so If I
silently drop 5% of the traffic randomly at the smtp level I hardly
get any false positive report... this doesn't make the "randomly
dropping 5% of the traffic" a good/trustable block. Most people don't
know someone is trying to write them until they see the email and you
get a false positive only when the sender phone-call the recipient to
ask why he didn't reply (their next attempt only have 5% of chance to
get blocked again, so 0,25% to block 2 sends, and they won't care to
call me to complain).

Even if you have a good "rate" you could have a bad statistical
distribution (see my B2C vs B2B example at the end of this reply).

I don't want to delegitimize those RBL, but the "false positive loop"
is one of the key aspects of an automated system and its importance
increases as the "spam report loop" quality decrease. IMHO today there
are major holes in the false positive loop and this didn't improve at
all through the years.

How can the quality of the spam report loop be low?
1) you get it from the final user and, just like you told us, most of
them don't use this feature correctly.
2) you automate it with no "volume comparison", so you only evaluate
reports and not "everything else"
3) you works with hashes/fingerprints instead of full messages (it's
harder to manually review listing/delisting from something you can't
really "read").

Why is this "in topic"?
1) IPv6 enable more "scattering" and IP based RBL are less effective
for low-volume senders (the lower the volume, the lower the
statistical significance, the higher the error)
2) If you move to a fingerprint (content) based blacklist and you do
that working mainly with hashes (like CloudMark Authority) you don't
know anymore what you are really going to block, until you start
blocking it.

I'm sure CloudMark (CA: CloudMark Authority) have major "false
positive loops" but I'm aware of multimillion-inboxes-providers using
it only to block emails and to send "spam reports" (no false positive
report loops).

To name names, in Italy "Libero.it" (ItaliaOnLine) the largest italian
inbox provider (10-15 million inboxes) uses CloudMark and AFAIK send
them spam reports, non-spam reports and (I'm less confident about
this, but I think they do) "volume data" about fingerprints and they
never reject at SMTP time because og this.

On the other side "Aruba" the largest B2B italian inbox provider (7
millions inboxes) uses CloudMark and AFAIK send them spam reports, DO
NOT send "non-spam reports" and on recipient option they may reject
emails (not at SMTP time, but via bounce... but the recipient doesn't
get the email).

So, if you do B2C or mixed B2B/B2C to Italy Cloudmark works mostly
well (they easily block fingerprints, but then they get false positive
reports and unblock), but if you only do B2B traffic you'll often hit
some Cloudmark block with many false positives that are not collected
by Cloudmark that can't never decide to unlist on its own.

PS: I talk about CloudMark because I happen to have good knowledge of
that filter and I think it is one of the "smartest" filters out there
WRT to distributed content filtering and I think it is "100% IPv6
ready" as its design does not depend on IPs (they have "Sender
intelligence" for that), but still show issues. And 

Re: [mailop] Should mail servers publish IPv6 MX records? Could this harm your spam filtering?

[Grant Taylor via mailop]

On 06/10/2018 03:26 PM, Al Iverson via mailop wrote:
I cry a little inside every time I see a null MX, think of the lost 
opportunity to be handling it as a spamtrap domain instead.


~chuckle~

Fair enough.

I also view it as an opportunity to accidentally leak email that ends up 
being feed into a spam trap.




--
Grant. . . .
unix || die

___
mailop mailing list
mailop@mailop.org
https://chilli.nosignal.org/cgi-bin/mailman/listinfo/mailop

Re: [mailop] Should mail servers publish IPv6 MX records? Could this harm your spam filtering?

[Al Iverson via mailop]

> >   example.net   IN   MX   0   .
>
> Nice. It's been listed in a few best practice documents as well as RFC 7505 
> for more than a few years. Good to see it getting some traction.

I cry a little inside every time I see a null MX, think of the lost
opportunity to be handling it as a spamtrap domain instead.

-- 
al iverson // 312-725-0130 // miami
http://www.aliverson.com
http://www.spamresource.com

___
mailop mailing list
mailop@mailop.org
https://chilli.nosignal.org/cgi-bin/mailman/listinfo/mailop

Re: [mailop] Should mail servers publish IPv6 MX records? Could this harm your spam filtering?

[Steve Atkins]

> On Jun 10, 2018, at 10:18 AM, Grant Taylor via mailop  
> wrote:
> 
> On 06/09/2018 09:32 AM, Andrew C Aitchison wrote:
>> If a domain has no MX record, do all servers deliver to an  record, as 
>> required by (at least) RFC3974, or do some email systems ignore domains with 
>> no MX and no A record ?
> 
> My personal stance has been "Don't be surprised if the lack of an MX fails 
> when you want it to work." and "Don't be surprised if you do receive email to 
> servers without an MX."
> 
> Isn't ambiguity fun.
> 
> I've seen people put MX records in for every single host in their network 
> referring back to a central email infrastructure that knew how to properly 
> handle email for said systems.
> 
> I've also started seeing a few Null MX records.
> 
>   example.net   IN   MX   0   .

Nice. It's been listed in a few best practice documents as well as RFC 7505 for 
more than a few years. Good to see it getting some traction.

Cheers,
  Steve
___
mailop mailing list
mailop@mailop.org
https://chilli.nosignal.org/cgi-bin/mailman/listinfo/mailop

Re: [mailop] Should mail servers publish IPv6 MX records? Could this harm your spam filtering?

[Grant Taylor via mailop]

On 06/09/2018 09:32 AM, Andrew C Aitchison wrote:
If a domain has no MX record, do all servers deliver to an  record, 
as required by (at least) RFC3974, or do some email systems ignore 
domains with no MX and no A record ?


My personal stance has been "Don't be surprised if the lack of an MX 
fails when you want it to work." and "Don't be surprised if you do 
receive email to servers without an MX."


Isn't ambiguity fun.

I've seen people put MX records in for every single host in their 
network referring back to a central email infrastructure that knew how 
to properly handle email for said systems.


I've also started seeing a few Null MX records.

   example.net   IN   MX   0   .



--
Grant. . . .
unix || die

___
mailop mailing list
mailop@mailop.org
https://chilli.nosignal.org/cgi-bin/mailman/listinfo/mailop

Re: [mailop] Should mail servers publish IPv6 MX records? Could this harm your spam filtering?

[John Levine]

In article  you write:
>>> If a domain has no MX record, do all servers deliver to an  record,
>>> as required by (at least) RFC3974,
>
>You'd expect, No MX record, no mail delivery.  MX is related to
>hostname, not the transport stack.

Only if you'd never read RFC 5321, particularly section 5.2.  Every v6
MTA of which I am aware will fall back to both  and A if there is
no MX.  I had a long argument about this in the IETF making all the
points people here else did and I lost, on the basis that it's too
late to change SMTP now.


___
mailop mailing list
mailop@mailop.org
https://chilli.nosignal.org/cgi-bin/mailman/listinfo/mailop

Re: [mailop] Should mail servers publish IPv6 MX records? Could this harm your spam filtering?

[Mal via mailop]

On 10/06/2018 3:16 AM, Stefano Bagnara wrote:
> On Sat, 9 Jun 2018 at 17:36, Andrew C Aitchison  
> wrote:
>> I'm curious.
>> If a domain has no MX record, do all servers deliver to an  record,
>> as required by (at least) RFC3974,

You'd expect, No MX record, no mail delivery.  MX is related to
hostname, not the transport stack.

If an MX record exists, a sending MTAs will deliver to the resolved
hostname with the first MX priority.  That may be a host that only
resolves a  record (ie, has no 'A' record for that hostname).  You
can run an IPv6 only MTA as your first priority receiver.  For sender
MTAs who only have IPv4 connectivity, you would need to operate a dual
stack receiving MTA, on your second priority.

Works a treat.

> For sure any server without Ipv6 connectivity won't go to  and
> Ipv6 connectivity is not mandatory.

Its time operators took IPv6 as mandatory - its not 2005 anymore.  IPv6
is well down the road.

IPv6 / DNSSEC / DANE TLS SMTP (TLSA) .. these should all be considered
mandatory to any decent operator in 2018.

Mal



___
mailop mailing list
mailop@mailop.org
https://chilli.nosignal.org/cgi-bin/mailman/listinfo/mailop

Re: [mailop] Should mail servers publish IPv6 MX records? Could this harm your spam filtering?

[Matthias Leisi]

> If the industry had moved to a reputation model, it would be easier to 
> discuss "how bad is it" and whether it's bad enough to block at IP time, or 
> whether you mix it into your spam score.

Isn’t this what postscreen_dnsbl_sites is doing, for example?

> Will SMTP be the last hold-out on IPv4?

At dnswl.org , IPv6 by volume was 0.06% over the past three 
days, and 1.34% of the netranges we list are IPv6. (Yes, that includes a lot of 
small-ish site which use our data and which may have lower adoption of IPv6 
than larger recipients.)

— Matthias

___
mailop mailing list
mailop@mailop.org
https://chilli.nosignal.org/cgi-bin/mailman/listinfo/mailop

Re: [mailop] Should mail servers publish IPv6 MX records? Could this harm your spam filtering?

[Matthias Leisi]

> Isn't the simplest way to handle this is to treat IPv6 at the /64 or smaller 
> level? 

There is no broad consensus yet on where IPv6 reputation should be attached to. 
Cheap hosting providers handing out individual /128s to customers…

Discovery protocols to find the „right“ prefix length to query in a particular 
situation have not resulted in tangible results. 

— Matthias


___
mailop mailing list
mailop@mailop.org
https://chilli.nosignal.org/cgi-bin/mailman/listinfo/mailop

Re: [mailop] Should mail servers publish IPv6 MX records? Could this harm your spam filtering?

[Stefano Bagnara]

On Fri, 8 Jun 2018 at 19:31, Michael Peddemors  wrote:
> On 18-06-08 09:14 AM, Stefano Bagnara wrote:
> > [...]
> > In fact I still have to understand how spam reports and false positive
> > reports are collected in the whole plesk world (I guess you know what
> > I'm talking about): [...]
>
> Well, like any system, I think that you are probably looking at this too
> simply, it could be that the system administrator has chosen to reject
> instead of flag.. It depends on the system implementation.  This isn't
> related to any specific RBL, but how it is used.
>
> Some use it to 'block', some use it to tag, and it doesn't matter which
> RBL it is, SpamHaus, SpamRats, UCE-Protect, Invalument, Barracuda.. (or
> a subset of their offerings) or a more specific purpose list..

I agree, but when a lot of users for an RBL use it to reject at the
smtp level then you loose your false-positive feedbacks.
If there a big sample of users that use the list but are not able to
report false positives, then you can't say anymore that the list have
"low false positive".

In order to try to go back in topic, I think this "false positive
loop" that is often missing for smaller users that simply stack up a
number of DNSBLs and reject for every match, is an issue and an issue
that need more attention and priority than the ipv6 stuff.
IPv6 will increase the fragmentation and more fragmentation will mean
more servers with low volumes and more servers with low volumes means
much more chance of false positive listings... and if you don't have
an almost 100% coverage for false positive collections this is going
to fall apart.

Unfortunately I don't have a recipe or a proposal about how we could
make it possible for recipients (they are the first source of the
truth wrt spam) to report false positives for email they didn't
receive at all because of the RBL used at SMTP rejection.
Also there are a lot of final users using emails systems with
mark-as-spam and mark-as-no-spam buttons that in the end do not
generate spam reports or false positive reports to the systems
resposible for that classification.

> [...]
> But it does depend on the technology being used by that operator.  Some
> will only be able to do it at one level or another, but it still creates
> less problems than if they didn't have it in place.

I don't fully agree.
1) In the plesk world (as I took that as an example) there are
defaults and  the defaults are not an "operator" stuff (AFAIK the
DNSBL enabled by default don't get any false positive reports from
plesk users)
2) When we talk about RBL we are not only talking about the technology
itself, but also in how they are (and can be realistically) operated..
The whole IPv6 RBL discussion is about that, IMHO... the technology
itself could work... but in order to make it work you would have
higher costs and a lot more resources to be invested in the way you
"operate" it, in order to get similar efficiency. So we can't prescind
from how RBLs are "usually" operated when we try to understand their
status and their potentials.

Stefano

___
mailop mailing list
mailop@mailop.org
https://chilli.nosignal.org/cgi-bin/mailman/listinfo/mailop

Re: [mailop] Should mail servers publish IPv6 MX records? Could this harm your spam filtering?

[Michael Peddemors]

On 18-06-08 09:14 AM, Stefano Bagnara wrote:

On Fri, 8 Jun 2018 at 17:53, Michael Peddemors  wrote:

[...]
And while using that as feedback might seem the logical conclusion, in
the real world we still see more feedback reports from legitimate email
the customer should have wanted, vs emails tagged as spam that are spam.


Well, this is very surprising to me. Anyone else record similar scenario?


For the record, we are only collecting optics on Microsoft Feedbacks at 
this time, so it might be other systems have less customers marking 
messages that are not spam, but legitimate messages from friends.. 
confirmed double opt-in mailing lists etc..



I know much more users that use the "mark as spam" on their inboxes
than users that dig in their junk folder to mark something as "non
spam".
I'd love to get some stats about "mark as spam" vs "mark as non spam"
from Google or Microsoft, just to get some real data about this!

And I know a lot of systems where there's no spam folder in "inbound"
but there's a "mark as spam" so that you can only make reports for
spam but you can't really make reports for non spam.

In fact I still have to understand how spam reports and false positive
reports are collected in the whole plesk world (I guess you know what
I'm talking about): I've had multiple recipients using plesk that were
willing to report false positives for the MIPSPACE-POOR list (or other
lists used there by default) but I've not been able to explain them
how to report the false positive and had to explain them how to remove
the MIPSPACE-POOR lookup. I really tried hard to find a better way
than explaining them how to disable that list, but failed.


Well, like any system, I think that you are probably looking at this too 
simply, it could be that the system administrator has chosen to reject 
instead of flag.. It depends on the system implementation.  This isn't 
related to any specific RBL, but how it is used.


Some use it to 'block', some use it to tag, and it doesn't matter which 
RBL it is, SpamHaus, SpamRats, UCE-Protect, Invalument, Barracuda.. (or 
a subset of their offerings) or a more specific purpose list..


(eg, there are system admin's that simply block China IP space for 
instance, or IP(s) from specific hosting providers)


It will always be up to the administrator, and in the cases of some more 
robust platforms, the domain owner, or the end user to decide for 
themselves which reputation systems or IP lists that they want to 
'block' vs 'tag'.


It is hard to get optics into the individual's operational objects, eg 
one email admin has more strict belief's than another, because it works 
'best' for 'his' operation and/or customers.


Often, the operator may get fed up, and put a list into blocking mode, 
because he gets nobody complaining.  Or a person who has a reputation 
list that generated only few false positives, they will let the sender 
deal with the RBL operator, or if they get enough false positives, they 
will move it to tagging only, so the end user can say 'This is not 
Junk', and allow future email from that sender.  And if they still get 
too many, they may just use it as an indicator with other factors.


But it does depend on the technology being used by that operator.  Some 
will only be able to do it at one level or another, but it still creates 
less problems than if they didn't have it in place.




--
"Catch the Magic of Linux..."

Michael Peddemors, President/CEO LinuxMagic Inc.
Visit us at http://www.linuxmagic.com @linuxmagic
A Wizard IT Company - For More Info http://www.wizard.ca
"LinuxMagic" a Registered TradeMark of Wizard Tower TechnoServices Ltd.

604-682-0300 Beautiful British Columbia, Canada

This email and any electronic data contained are confidential and intended
solely for the use of the individual or entity to which they are addressed.
Please note that any views or opinions presented in this email are solely
those of the author and are not intended to represent those of the company.

___
mailop mailing list
mailop@mailop.org
https://chilli.nosignal.org/cgi-bin/mailman/listinfo/mailop

Re: [mailop] Should mail servers publish IPv6 MX records? Could this harm your spam filtering?

[Rob McEwen]

On 6/8/2018 11:50 AM, Michael Peddemors wrote:
It is sometimes important to point out that the email marketing is a 
multi-billion dollar business.. The spam protection and RBL operators 
get very little money if any in comparison..


I was reading an ebook on marketing this past week - mistakenly thinking 
that it would have good suggestions for non-spamming social media 
advertising - but it was really all about exploiting email lists - to 
its credit, it was all whitehat permissions-based confirmed-option 
stuff. So it wasn't a "how to spam" book. The real life examples of 
financial rewards from this business that it provided, when done well, 
were so shockingly lucrative and EASY - that it made me wonder, "what am 
I doing trying to run a blacklist... when I could be doing THIS!"


...and everything else you said below is excellent!

They are always going to have more money to spend on finding ways to 
get their email delivered, that the poor front line systems admin's 
and RBL operators trying to deal with customer complaints..


So, there is the aspect that the RBL's and system admin's are going to 
need cost effective ways to address this, and that is sometimes more 
important than 'perfect' ways to address the issues.


IMHO, and the way most of our platforms are designed to work.. Empower 
the users when you can.. but block the worst of the worst..


* Block at SMTP via RBL's that have very low false positive rates
* Try to use RBL's that allow those few false positives to be quickly 
removed.
* Tag as spam via RBL's where the system admin SHOULD be addressing 
the egress problem, but where the source could be a shared 
environment, eg a mix of good and bad, where the bad complaints 
outnumber the good..
* Increase the 'score' via RBL's where it is too big to block, but the 
problem users aren't being addressed by the system operators.


* Allow Users to make the final decision, eg, this is not spam/this is 
spam.


Anyone who says they can provide 100% accurate spam protection, is 
lying to your face.  In the real world, there will always be an email 
that one person wants, and the other person doesn't want, so you can't 
argue, you have to let the customer decide.


And while using that as feedback might seem the logical conclusion, in 
the real world we still see more feedback reports from legitimate 
email the customer should have wanted, vs emails tagged as spam that 
are spam.


Probably, because most systems doing ingress spam protection are 
pretty good accurate nowadays..


But more onus has to be put on the senders networks..

Not to call anyone out, but as far as too big to block goes, those 
senders have a lot more money to invest in egress filtering, and have 
better optics into the behavior, than the poor guys on the receiving 
end, however as history shows, the senders often don't worry about it 
as much, even though they have the budgets, UNTIL it impacts 
legitimate business..


And there are hosting companies out there that simply have a blind eye 
towards the activity on their networks..


So SOMETIMES RBL operators and/or system admins have only one cost 
effective way to address the issue, and that is to put the IP(s) 
and/or networks on the 'rejection' list.. until conditions approved..


Still way too much IPv4 space is wasted by known spammer havens.. And 
when it comes to IPv6, well years of reputation history can be undone 
in a single moment, when you allow a whole brand new internet of 
'fresh' IP(s) to be used.. eg IPv6


Have a happy Spam Free Weekend all..






--
Rob McEwen
https://www.invaluement.com
 



___
mailop mailing list
mailop@mailop.org
https://chilli.nosignal.org/cgi-bin/mailman/listinfo/mailop

Re: [mailop] Should mail servers publish IPv6 MX records? Could this harm your spam filtering?

[Rob McEwen]

On 6/8/2018 12:02 PM, David Hofstee wrote:
Earlier you stated that larger setups depended on having blacklists at 
the gate to keep processing manageable but which results in less 
weighed filtering. [see #5]


yes, but I wasn't referring to ALL blacklists - just a handful of the 
most effective and reliable ones that have extremely low FPs - and then 
the others would ideally be using for scoring


But these setups (often) still lack feedback mechanisms that the 
Yahoo's, Hotmail's and Gmail's of this world have... Feedback from 
users. It also lacks methods that need large quantities of "historical 
data". Both add information back into the content filter to improve it.


fwif, while, not perfect, invaluement has a number of such feedback 
mechanisms in place that constantly help improve the quality and 
effectiveness of our data. (I really don't want to provide a more 
specific/detailed answer.)


Hey Rob, I think that you have a very good blacklist. My point is that 
I think "an excellent blacklist is not good enough". Because you need 
a better (and faster) integration between the content filter (using 
e.g. reputation data from you), the traffic limiting mechanisms, the 
feedback mechanisms (e.g. users marking as spam, or the other way 
around) and "email history of your domain".


It is precisely BECAUSE invaluement DOES have things... that helps 
invaluement to be "a very good blacklist" (though there is always room 
for improvement, and what you envision might be even more comprehensive 
than what we do/have!). But some of your suggestions may require 
paradigm changes in how spam filters and blacklists interact. There are 
probably some good ideas there, but coordinating these with distributed 
hardware and software spam filtering is no trivial task. (it is bad 
enough that so many spam filters STILL don't have a way to add custom 
3rd party URI blacklists - they just hardwire in SURBL, and you can't 
add URIBL or Spamhaus' DBL or ivmURI - and this technology has been 
around for 13 or so years, so things like this are not easy!)


> (6) nobody in this thread ever claimed that blacklists-alone are 
sufficient for having good spam filtering
Rob, you as a small set of people, are capable to and have enough 
access to improve on the current situation. I also think that the 
Google's, Microsofts and Yahoo's are a systemic issue in the email 
world. This would include Proofpoint and SpamExperts as well. It is 
becoming harder and harder to have your own server and not be troubled 
by spam unless you use this small set of services. "They" are 
unwilling to share their methods with the rest of us (which I can 
understand but which results into "the rest" not having that knowledge).


I think that we, "the rest", should develop better ways to filter 
spam. That was my goal of this discussion. To make you conscious of 
the fact that we need more than IP/domain blacklists. To be able to 
level up with the proprietary solutions. Instead of being stuck with 
the idea that we should stick to what we have (because we have it).


Fair points. but, (1) invaluement is priced reasonable fwiw, so 
hopefully that helps! -AND- (2) if you only knew the blood-sweat-tears, 
over many many years, required to accumulate this knowledge and 
technology! Plus, there are many risks involved too! (even occasional 
death threats, and a constant litigation liability) Again, fwiw... not 
to complain, and I think that invaluement is going to pay off amazingly 
well in the not too-distant future, and eventually make this whole 
effort extremely financially rewarding - but a few important pieces to 
the puzzle need to come together first. I'm working on that. In the 
meantime, I don't think anyone should be very envious of my situation 
(morbid fascination would probably more appropriate!) But these may be 
the reasons why there are not as many high quality DNSBLs.


--
Rob McEwen
https://www.invaluement.com



___
mailop mailing list
mailop@mailop.org
https://chilli.nosignal.org/cgi-bin/mailman/listinfo/mailop

Re: [mailop] Should mail servers publish IPv6 MX records? Could this harm your spam filtering?

[Stefano Bagnara]

On Fri, 8 Jun 2018 at 17:53, Michael Peddemors  wrote:
> [...]
> And while using that as feedback might seem the logical conclusion, in
> the real world we still see more feedback reports from legitimate email
> the customer should have wanted, vs emails tagged as spam that are spam.

Well, this is very surprising to me. Anyone else record similar scenario?

I know much more users that use the "mark as spam" on their inboxes
than users that dig in their junk folder to mark something as "non
spam".
I'd love to get some stats about "mark as spam" vs "mark as non spam"
from Google or Microsoft, just to get some real data about this!

And I know a lot of systems where there's no spam folder in "inbound"
but there's a "mark as spam" so that you can only make reports for
spam but you can't really make reports for non spam.

In fact I still have to understand how spam reports and false positive
reports are collected in the whole plesk world (I guess you know what
I'm talking about): I've had multiple recipients using plesk that were
willing to report false positives for the MIPSPACE-POOR list (or other
lists used there by default) but I've not been able to explain them
how to report the false positive and had to explain them how to remove
the MIPSPACE-POOR lookup. I really tried hard to find a better way
than explaining them how to disable that list, but failed.

Stefano

___
mailop mailing list
mailop@mailop.org
https://chilli.nosignal.org/cgi-bin/mailman/listinfo/mailop

Re: [mailop] Should mail servers publish IPv6 MX records? Could this harm your spam filtering?

[David Hofstee]

> (1) First, I "eat my own dogfood",  ...
Yes, that was clear.

> (2) A large percentage of invaluement subscribers use SpamAssassin
So this should work somewhat. If you have the capacity to let everything be
processed by the SA content filter. Earlier you stated that larger setups
depended on having blacklists at the gate to keep processing manageable but
which results in less weighed filtering. [see #5]

But these setups (often) still lack feedback mechanisms that the Yahoo's,
Hotmail's and Gmail's of this world have... Feedback from users. It also
lacks methods that need large quantities of "historical data". Both
add information back into the content filter to improve it.

> (3) I'm certain that some portion of invaluement subscribers have BETTER
filtering ...- they all have excellent filtering in the areas of their spam
filtering that invaluement attempts to improve! :)
Hey Rob, I think that you have a very good blacklist. My point is that I
think "an excellent blacklist is not good enough". Because you need a
better (and faster) integration between the content filter (using e.g.
reputation data from you), the traffic limiting mechanisms, the feedback
mechanisms (e.g. users marking as spam, or the other way around) and "email
history of your domain".

> (4) You seem to be very confused about what I mean when I talk about how
there has to be some justified level of "collateral damage" these days, due
to the very high frequency of hijacked accounts
I have seen it all... Up to the level of terrorism. I agree with you on
that.

> (5) Also, a large percentage of Invaluement subscribers choose to block
at the perimeter
See #2.

> (6) nobody in this thread ever claimed that blacklists-alone are
sufficient for having good spam filtering
Rob, you as a small set of people, are capable to and have enough access to
improve on the current situation. I also think that the Google's,
Microsofts and Yahoo's are a systemic issue in the email world. This would
include Proofpoint and SpamExperts as well. It is becoming harder and
harder to have your own server and not be troubled by spam unless you use
this small set of services. "They" are unwilling to share their
methods with the rest of us (which I can understand but which results into
"the rest" not having that knowledge).

I think that we, "the rest", should develop better ways to filter spam.
That was my goal of this discussion. To make you conscious of the fact that
we need more than IP/domain blacklists. To be able to level up with the
proprietary solutions. Instead of being stuck with the idea that we should
stick to what we have (because we have it).

Anyway, take it as it is. I hope you have a great weekend.

Yours,


David




On 8 June 2018 at 16:27, Rob McEwen  wrote:

> On 6/8/2018 5:49 AM, David Hofstee wrote:
>
>> > ... score of the sending-IP, which is similar to what you've described,
>> correct?
>> Correct.
>>
>> So you have these mechanisms in place. But your customers, who get access
>> to the invaluement RBL, do not.  Am I correct? If I am, it still results in
>> the conclusion that blacklists are not sufficient to have a resulting good
>> spam filter. You would be ok, the list would not have false positives,
>> but your customers would not be sufficiently covered once bad guys get
>> smarter.
>>
>
>
> David,
>
> You've made so many false assumptions to come to these conclusions... and
> taken things I've said out of context to get there... I had a hard time
> knowing where to begin!
>
> (1) First, I "eat my own dogfood", even for my own mailbox! In our own
> spam filtering system, we score ALL invaluement blacklists "above
> threshold". However, in VERY RARE situations, a message will get delivered
> in our mail system where (a) it had one hit on one invaluement list, (b)
> NOTHING else spammy triggered, (c) and some rules kicked in that lowered
> the spam score just barely below threshold -BUT GUESS WHAT?- the vast
> majority of the time that happens, it ends up being a FALSE NEGATIVE - then
> I'm jealous of my own customers whose systems didn't deliver those spams to
> their users' inboxes!
>
> (2) A large percentage of invaluement subscribers use SpamAssassin, and
> likewise use a multi-tiered scoring system where they score blacklists
> higher if that blacklist (a) had fewer FPs, -AND- (b) the FPs it generates
> are more likely to result in extremely rare and/or extremely justified FPs.
> And they score OTHER DNSBLs lower for those which have a higher frequency
> of hits on desired mail. And some have similar scoring options with other
> spam filtering systems in addition to SpamAssassin.
>
> (3) I'm certain that some portion of invaluement subscribers have BETTER
> filtering than we have for our boutique mail hosting system, but that
> circumstance is somewhat rare ONLY because we can afford to put a lot more
> resources per mailbox into this, due to our DNSBL effectively subsidizing
> our small mail hosting system. (But this helps us to 

Re: [mailop] Should mail servers publish IPv6 MX records? Could this harm your spam filtering?

[Michael Peddemors]

On 18-06-08 08:21 AM, Stefano Bagnara wrote:

If you really think that rejecting email from senders that want to
optimize their costs is a good strategy
Well, IPv6 is simply a way to make email sending cheaper. So not
supporting Ipv6 is an effective way to dump cheap sending.

I guess anyone with a good corpus can easily check that "inexpensive
ESP" are not more spammy than "fortune 500 ESP".

Someone proposed to simply add some cost to every SMTP transaction as
a way to stop the spam, some blacklist offer paid unlisting services,
too... but spammers sometimes have more money to send email than the
average user IMHO.


While this thread has gone far off the original topic..

It is sometimes important to point out that the email marketing is a 
multi-billion dollar business..


The spam protection and RBL operators get very little money if any in 
comparison..


They are always going to have more money to spend on finding ways to get 
their email delivered, that the poor front line systems admin's and RBL 
operators trying to deal with customer complaints..


So, there is the aspect that the RBL's and system admin's are going to 
need cost effective ways to address this, and that is sometimes more 
important than 'perfect' ways to address the issues.


IMHO, and the way most of our platforms are designed to work.. Empower 
the users when you can.. but block the worst of the worst..


* Block at SMTP via RBL's that have very low false positive rates
* Try to use RBL's that allow those few false positives to be quickly 
removed.
* Tag as spam via RBL's where the system admin SHOULD be addressing the 
egress problem, but where the source could be a shared environment, eg a 
mix of good and bad, where the bad complaints outnumber the good..
* Increase the 'score' via RBL's where it is too big to block, but the 
problem users aren't being addressed by the system operators.


* Allow Users to make the final decision, eg, this is not spam/this is spam.

Anyone who says they can provide 100% accurate spam protection, is lying 
to your face.  In the real world, there will always be an email that one 
person wants, and the other person doesn't want, so you can't argue, you 
have to let the customer decide.


And while using that as feedback might seem the logical conclusion, in 
the real world we still see more feedback reports from legitimate email 
the customer should have wanted, vs emails tagged as spam that are spam.


Probably, because most systems doing ingress spam protection are pretty 
good accurate nowadays..


But more onus has to be put on the senders networks..

Not to call anyone out, but as far as too big to block goes, those 
senders have a lot more money to invest in egress filtering, and have 
better optics into the behavior, than the poor guys on the receiving 
end, however as history shows, the senders often don't worry about it as 
much, even though they have the budgets, UNTIL it impacts legitimate 
business..


And there are hosting companies out there that simply have a blind eye 
towards the activity on their networks..


So SOMETIMES RBL operators and/or system admins have only one cost 
effective way to address the issue, and that is to put the IP(s) and/or 
networks on the 'rejection' list.. until conditions approved..


Still way too much IPv4 space is wasted by known spammer havens.. And 
when it comes to IPv6, well years of reputation history can be undone in 
a single moment, when you allow a whole brand new internet of 'fresh' 
IP(s) to be used.. eg IPv6


Have a happy Spam Free Weekend all..




--
"Catch the Magic of Linux..."

Michael Peddemors, President/CEO LinuxMagic Inc.
Visit us at http://www.linuxmagic.com @linuxmagic
A Wizard IT Company - For More Info http://www.wizard.ca
"LinuxMagic" a Registered TradeMark of Wizard Tower TechnoServices Ltd.

604-682-0300 Beautiful British Columbia, Canada

This email and any electronic data contained are confidential and intended
solely for the use of the individual or entity to which they are addressed.
Please note that any views or opinions presented in this email are solely
those of the author and are not intended to represent those of the company.

___
mailop mailing list
mailop@mailop.org
https://chilli.nosignal.org/cgi-bin/mailman/listinfo/mailop

Re: [mailop] Should mail servers publish IPv6 MX records? Could this harm your spam filtering?

[Jim Popovitch via mailop]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA512

On Fri, 2018-06-08 at 17:21 +0200, Stefano Bagnara wrote:
> On Fri, 8 Jun 2018 at 16:47, Jim Popovitch via mailop  org> wrote:
> > On Fri, 2018-06-08 at 10:27 -0400, Rob McEwen wrote:
> > > there has to be some justified level of "collateral damage" these
> > > days, due to the very high frequency of hijacked accounts,
> > > hijacked
> > > websites, and spamming ESP customers (from ESP that are overall
> > > good).
> > 
> > Rather than dumping a piece of technology (ipv6), dump the ESPs
> > that
> > enable cheap sending. (Win! Win!).  If those ESP customers had to
> > build
> > out their own infrastructure then they would take better care of
> > it...
> > regardless of ipv4, ipv6, ipv8, etc.
> 
> If you really think that rejecting email from senders that want to
> optimize their costs is a good strategy
> Well, IPv6 is simply a way to make email sending cheaper. So not
> supporting Ipv6 is an effective way to dump cheap sending.
> 
> I guess anyone with a good corpus can easily check that "inexpensive
> ESP" are not more spammy than "fortune 500 ESP".
> 
> Someone proposed to simply add some cost to every SMTP transaction as
> a way to stop the spam, some blacklist offer paid unlisting services,
> too... but spammers sometimes have more money to send email than the
> average user IMHO.

Fair points!  My comment was mostly tongue-in-cheek, but there is
something to be said about how relatively easy it is to send bulk spam.

- -Jim P.
-BEGIN PGP SIGNATURE-

iQIzBAEBCgAdFiEEPxwe8uYBnqxkbORSJxVetMRaJwUFAlsaomoACgkQJxVetMRa
JwVGKA/+JNNIAJq05rmovo8xZxPQZqUsdUJ5Y+ZqaMIynlGNg6tLy+qfKFF1wtFa
dF01Rhyo0L3f7muWU4UDG7+o2nCp0n6elwvrwZtMnC6KnkEaNM8EhxOOq0b6L5ad
Nj/0m0jkvz2R5eoIpfcN17u3TG3cafn4iWMzXJlJXs/gvpwggIN8NPHXS7mry7WW
LC0m1Zr2wMc6396TEky5LCTFPsqdTSENhh9krJsN4xYCJmcggUd7vokLjqe7tPuA
KmoftehvJ1Tyfeav7R8IY7GMhE3lJMLnlo4sdprg++U9PphSYeVtdeb+OHpdmHu1
JkS0Dl5ttpTvWqmVILtZOx7l2IwdrKtcErW0r435sFTJysqrbrxRkzEOMUrg/L8P
ycvQMMgo6CK4NzZ3NatJnRe1frvLpsrWtvdyV6XxsMNC1vGq/ITWdYe8TknwDF8O
vMMYNL5z4/CMu/YgV28QVF5ZyAP3aXNb8Z6co8+FGINtyU8O4XM+1WWo6JWQeIKS
zZ1ddv99PNhYJFhgWoI7GTPoa76pXsL7mWV1Qopd6vvCQkdU0CzoxXpZe/lbhIO7
ztR+AXw+k82lc4dGyTfuyn33hwgshI7LkndxLyU2c49xHitHkIfWqcPQUy/q0qTZ
gzvAEzNypr59YI1Hv9Vr8gtRA+23fEigc/kJMzMKRypZNVJ2eo0=
=QYPF
-END PGP SIGNATURE-


___
mailop mailing list
mailop@mailop.org
https://chilli.nosignal.org/cgi-bin/mailman/listinfo/mailop

Re: [mailop] Should mail servers publish IPv6 MX records? Could this harm your spam filtering?

[Stefano Bagnara]

On Fri, 8 Jun 2018 at 16:47, Jim Popovitch via mailop  wrote:
> On Fri, 2018-06-08 at 10:27 -0400, Rob McEwen wrote:
> > there has to be some justified level of "collateral damage" these
> > days, due to the very high frequency of hijacked accounts, hijacked
> > websites, and spamming ESP customers (from ESP that are overall
> > good).
>
> Rather than dumping a piece of technology (ipv6), dump the ESPs that
> enable cheap sending. (Win! Win!).  If those ESP customers had to build
> out their own infrastructure then they would take better care of it...
> regardless of ipv4, ipv6, ipv8, etc.

If you really think that rejecting email from senders that want to
optimize their costs is a good strategy
Well, IPv6 is simply a way to make email sending cheaper. So not
supporting Ipv6 is an effective way to dump cheap sending.

I guess anyone with a good corpus can easily check that "inexpensive
ESP" are not more spammy than "fortune 500 ESP".

Someone proposed to simply add some cost to every SMTP transaction as
a way to stop the spam, some blacklist offer paid unlisting services,
too... but spammers sometimes have more money to send email than the
average user IMHO.

Stefano

___
mailop mailing list
mailop@mailop.org
https://chilli.nosignal.org/cgi-bin/mailman/listinfo/mailop

Re: [mailop] Should mail servers publish IPv6 MX records? Could this harm your spam filtering?

[Jim Popovitch via mailop]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA512

On Fri, 2018-06-08 at 10:27 -0400, Rob McEwen wrote:
> there has to be some justified level of "collateral damage" these
> days, due to the very high frequency of hijacked accounts, hijacked
> websites, and spamming ESP customers (from ESP that are overall
> good). 

Rather than dumping a piece of technology (ipv6), dump the ESPs that
enable cheap sending. (Win! Win!).  If those ESP customers had to build
out their own infrastructure then they would take better care of it...
regardless of ipv4, ipv6, ipv8, etc.

- -Jim P.
-BEGIN PGP SIGNATURE-

iQIzBAEBCgAdFiEEPxwe8uYBnqxkbORSJxVetMRaJwUFAlsaloYACgkQJxVetMRa
JwUxXxAAu0CqqhxNX+IbYfyJBZWZwfEwLrZ7Ku4DPBTLWab4YAxas9uewt1sMgO1
6TpmV4cVM3C1z/Co+duFEsYfadDcOY6WEuYQAaxC5sVQXLZqba3lrjVMG5Bstm5O
qUXD9EGyieE7un96yP/WhBpTdIxQwRpq5piXknu7FVQqarqDfLwNm2osprpAD1IG
GZnV0V52HqImaW6QFDtBonX9OhAOPzwS7m6Z3DMRzkrYFQKYA594WC1q3Q15g4ZB
tYRPlTX1ReHuESGOaHrgFCAKowgMAxPgxDCT2FsYltqdh+3gSf+0YNpV5TcBTaBZ
SX4uJoetZWFdWc+9Kf7DeFzVUl+ACqOCtXDFECwHXuE56yLFyk8dn1lMBtdtqk5S
07vk7X4B0De2wklbn8dBX2dpXQBkYAOE19VnVsf4Ad2A7bXn7BE2PMdnTEVf6r8l
mhefDGOvgYYP07QLMn/lQOUhstnp64CLwWLPCzlWlvnIXDrKE8XkQJl8ac4EfSqd
DmYH7W3kfP094d4O0Bv8hvAkLY44PRAeHOjuntWC/v1g4PQTKejWzwfHHyccX1dK
iY0/6vEyh2ALuwqLfL+ei+iiAU+TD7ZJTu/b+UBLDwefMHtImjSs2jKEenQhFmIt
mRyrwN+IL2jjlqv4CYN0MpnCvPqqsT0l9WX5xMyDm9ZKOU3pbWE=
=mvOW
-END PGP SIGNATURE-


___
mailop mailing list
mailop@mailop.org
https://chilli.nosignal.org/cgi-bin/mailman/listinfo/mailop

Re: [mailop] Should mail servers publish IPv6 MX records? Could this harm your spam filtering?

[Rob McEwen]

On 6/8/2018 5:49 AM, David Hofstee wrote:
> ... score of the sending-IP, which is similar to what you've 
described, correct?

Correct.

So you have these mechanisms in place. But your customers, who get 
access to the invaluement RBL, do not.  Am I correct? If I am, 
it still results in the conclusion that blacklists are not sufficient 
to have a resulting good spam filter. You would be ok, the list would 
not have false positives, but your customers would not be sufficiently 
covered once bad guys get smarter.



David,

You've made so many false assumptions to come to these conclusions... 
and taken things I've said out of context to get there... I had a hard 
time knowing where to begin!


(1) First, I "eat my own dogfood", even for my own mailbox! In our own 
spam filtering system, we score ALL invaluement blacklists "above 
threshold". However, in VERY RARE situations, a message will get 
delivered in our mail system where (a) it had one hit on one invaluement 
list, (b) NOTHING else spammy triggered, (c) and some rules kicked in 
that lowered the spam score just barely below threshold -BUT GUESS 
WHAT?- the vast majority of the time that happens, it ends up being a 
FALSE NEGATIVE - then I'm jealous of my own customers whose systems 
didn't deliver those spams to their users' inboxes!


(2) A large percentage of invaluement subscribers use SpamAssassin, and 
likewise use a multi-tiered scoring system where they score blacklists 
higher if that blacklist (a) had fewer FPs, -AND- (b) the FPs it 
generates are more likely to result in extremely rare and/or extremely 
justified FPs. And they score OTHER DNSBLs lower for those which have a 
higher frequency of hits on desired mail. And some have similar scoring 
options with other spam filtering systems in addition to SpamAssassin.


(3) I'm certain that some portion of invaluement subscribers have BETTER 
filtering than we have for our boutique mail hosting system, but that 
circumstance is somewhat rare ONLY because we can afford to put a lot 
more resources per mailbox into this, due to our DNSBL effectively 
subsidizing our small mail hosting system. (But this helps us to make 
the quality of invaluement data even better, which only benefits ALL of 
our subscribers - the opposite would be true if we neglected our own 
filtering system!) But since our subscribers all use a VARIETY of 
technologies and approaches to spam, and since invaluement data is only 
tacking a subset of all spam, and isn't trying to be a comprehensive 
solution for blocking all spam - every client has a unique situation. 
And I'm certain that at least SOME of them have even BETTER spam 
filtering than what we have for our mail hosting customers. But they all 
share one thing in common - they all have excellent filtering in the 
areas of their spam filtering that invaluement attempts to improve! :)


(4) You seem to be very confused about what I mean when I talk about how 
there has to be some justified level of "collateral damage" these days, 
due to the very high frequency of hijacked accounts, hijacked websites, 
and spamming ESP customers (from ESP that are overall good). Keep in 
mind that, ALL of these added together at one time - can be STILL be an 
astronomically tiny percentage-wise when it comes to how much the 
collateral damage impacts the average end user, yet can still be 
tremendously harmful to the company with the security problem, since the 
problem CONCENTRATES there. To give an extreme example, suppose a small 
business with 25 employees, who averages 500 outbound legit emails a day 
- had a security lapse and their server starts attempting to send out 
200,000+ egregious spams per day - and now their 500 outbound legit 
emails are getting blocked in many places. The chances that an ISP with 
10,000 mailboxes - or even 1M mailboxes - is going to be impacted by the 
collateral damage - and have to deal with user complaints about false 
positives - is extremely rare - yet this small business is going to have 
a very very bad day that day! In a situation like this, their sending IP 
is likely to get blacklisted on Invaluement and/or Spamhaus - but will 
likely also get delisted fairly soon after they submit a delist request 
and/or fix the problem (but that could take longer if they are doing 
stupid stuff - like having a poorly formed PTR record that looks dynamic 
and doesn't properly convey identity and reputation... but I digress) 
And then other higher-FP blacklists will do a lot of similar listings, 
except they'll also include situations where the 
spam-to-collateral-damage ratios are NOT so clear cut. These other lists 
are better for scoring - AND MANY (OR MOST?) INVALUEMENT SUBSCRIBERS 
HAVE THE TECHNOLOGY TO DO THAT SCORING. (did you not know that?)


(5) Also, a large percentage of Invaluement subscribers choose to block 
at the perimeter (at connection, without accepting the message) based on 
Zen (from Spamhaus), ivmSIP, and ivmSIP/24 - and are 

Re: [mailop] Should mail servers publish IPv6 MX records? Could this harm your spam filtering?

[Stefano Bagnara]

On Fri, 8 Jun 2018 at 13:57, David Hofstee  wrote:
> Hi Stefano,
>
> The only problem I see with Cloudmark is that they are not just a reputation 
> provider, but a spamfilter provider with access to all the data. Which has 
> been acquired by Proofpoint.

Well it is a mix of reputation and filter.. the filtering is "ON/OFF"
and not reputation/score based. If one fingerprint is in the blocklist
the email is spam, otherwise is ham.
The "reputation" is involved in the way they take into consideration
the reporters or they decide to block or unblock a give fingerprint.
So it is not so different from an IP RBL.

What do you mean with "spamfilter provider with access to all the data."?
They don't see the whole email flow.. they mostly get fingerprint
lists for email being marked as spam or "non-spam" and an installable
peace of software (that you can add to your mailserver) do the
fingerprint extraction and is able to check if any fingerprint is
currently blocked, and then you can do whatever you want with this
information.

An email will result in 5, 10, 30 fingerprints depending on the number
of "intesting" things Cloudmark identify.

> I'm asking myself the question if the fingerprints they collect are GDPR 
> proof (although Jaren may comment on that).

We'll probably know this in a few years , as GDPR will start getting
used for real...
In the mean time they mostly deal with cryptographic irreversible
hashes of the email "fingerprints". GDPR call this
"pseudo-anonymization" because you if know the original personal data
you can find the fingerprint for it.

For example whenever an email contains a link or plaintext sequence
that sounds like an url and the domain is "gravatar.com" they will ad
a "AxTCswu-:8" fingerprint for that domain. This "AxTCswu-:8"
is the one that is blocked if they want to start blocking every email
referencing gravatar and the "block packages" their users download via
DNS micro-updates will simply contain this fingerprint as the
filtering can be done "offline" once you have the latest updates.
There are specific domains where the hash will be computed for every
subdomain and there are specific domains where another hash will be
computed depending on the path, e.g:
gallery.mailchimp.com, googleusercontent.com, www.youtube.com,
www.dropbox.com, goo.gl, tinypic.com and many other will generate a
"something:20" fingerprint that will be specific to a part of the
path. For url shortner is the full url, for others like
gallery.mailchimp it will include the "user specific" part of the url
so that they can identify a single mailchimp sender with a specific
fingerprint.
This "special" domain list is part of the "data", not of the
code/algorythm... When you receive the micro-updates with the
blacklisted fingerprints you may also receive updates about the
patterns that will generate the :20 fingerprints.

This is just a basic case of the most common patterns from CloudMark
Authority, but there is a lot more and their system is really
interesting: it is the most advanced of its kind. I think it is much
better than using a bunch of IP/URI BL at the SMTP level, but it is
far from perfect. E.g: given they don't have "volume" informations
they very easily block "new" fingerprints because of a couple of "mark
as spam" actions even if there were 2 "mark as spam" for 1 million
email delivered... then, you'll start being junked and they easily
will get "non-spam" reports and remove the fingerprint from the
block... I see a lot of new senders blocked at least once in their
first send and then unblocked after a couple of "junked" emails. Of
course this only works if you send blocked email to spam and you get
the "not-spam" reports.

IMHO this is GDPR compliant, not so different from IP RBL: they deal
with clear-text IP addresses and IP addresses are not even
pseudo-anonymized.
That said GPDR in the end doesn't make much differences between
personal data and pseudo-anonymized personal data... they both require
the same "legal" treatment, but if you use pseudo-anonymization you
can say you care more about the privacy leaks issues.

Stefano

>
> Yours,
>
>
> David
>
> On 8 June 2018 at 12:35, Stefano Bagnara  wrote:
>>
>> On Fri, 8 Jun 2018 at 11:53, David Hofstee  
>> wrote:
>> > [...]
>> > I also think that there is space for a reputation provider which can:
>> > - Identify more than just IP addresses and domains from an email.
>>
>> This is what CloudMark Authority does about this, but you enable a new
>> set of issues that have been just fixed in the IP/domain world thanks
>> to DMARC (I wrote an answer to the SDLU sister-thread).
>> IIRC Vipul's Razor used the same fingerprinting concepts and ended up
>> using a DNSBL of "fingerprints". Vipul founded Cloudmark and I don't
>> know what is the current status of the Razor project.
>>
>> > - Is able to process feedback from domain owners and recipients in an 
>> > automated, quick, effective and anonymous enough way (with the GDPR et 
>> > al). 

Re: [mailop] Should mail servers publish IPv6 MX records? Could this harm your spam filtering?

[David Hofstee]

Hi Stefano,

The only problem I see with Cloudmark is that they are not just a
reputation provider, but a spamfilter provider with access to all the data.
Which has been acquired by Proofpoint.

I'm asking myself the question if the fingerprints they collect are GDPR
proof (although Jaren may comment on that).

Yours,


David

On 8 June 2018 at 12:35, Stefano Bagnara  wrote:

> On Fri, 8 Jun 2018 at 11:53, David Hofstee 
> wrote:
> > [...]
> > I also think that there is space for a reputation provider which can:
> > - Identify more than just IP addresses and domains from an email.
>
> This is what CloudMark Authority does about this, but you enable a new
> set of issues that have been just fixed in the IP/domain world thanks
> to DMARC (I wrote an answer to the SDLU sister-thread).
> IIRC Vipul's Razor used the same fingerprinting concepts and ended up
> using a DNSBL of "fingerprints". Vipul founded Cloudmark and I don't
> know what is the current status of the Razor project.
>
> > - Is able to process feedback from domain owners and recipients in an
> automated, quick, effective and anonymous enough way (with the GDPR et al).
> Feedback is key.
>
> I strongly agree with "Feedback is key" both for "spam reports" and
> "non-spam reports" (and considering that "non-spam" only flows if you
> didn't block at the SMTP level).
> Unfortunately once you adopt SMTP reject based on a blacklist then you
> accept to stop getting false positives about that traffic, so you
> really stop monitoring the effectiveness of that block.
>
> The issue with this is that you have to start building a reputation
> not only for IP/domains/other email content fingerprints (sender
> stuff), but also need to build a reputation for feedback providers
> (recipient stuff) and maybe you also need a reputation management for
> people asking delisting (consultant, ESP...)
>
> A lot of data to mix.. so you start building some machine learning to
> deal with that automatically and then you end up with SmartScreen and
> no one, included your creator, will know why some message have been
> blocked or not ;-)   (no offense to SmartScreen, I know how hard is to
> deal with this stuff, but I receive Office365 invoices in spam, in my
> Office365 inbox.).
>
> An FBL system "ala Google PT" (so only aggregated data not enabling
> list washing) but open to multiple receivers could help adding more
> accountability to ESP to do their own filtering part (mainly with B2B
> emails, as with B2C they already have microsoft/yahoo/google data).
>
> Stefano
>
> ___
> mailop mailing list
> mailop@mailop.org
> https://chilli.nosignal.org/cgi-bin/mailman/listinfo/mailop
>



-- 
--
My opinion is mine.
___
mailop mailing list
mailop@mailop.org
https://chilli.nosignal.org/cgi-bin/mailman/listinfo/mailop

Re: [mailop] Should mail servers publish IPv6 MX records? Could this harm your spam filtering?

[Stefano Bagnara]

On Fri, 8 Jun 2018 at 11:53, David Hofstee  wrote:
> [...]
> I also think that there is space for a reputation provider which can:
> - Identify more than just IP addresses and domains from an email.

This is what CloudMark Authority does about this, but you enable a new
set of issues that have been just fixed in the IP/domain world thanks
to DMARC (I wrote an answer to the SDLU sister-thread).
IIRC Vipul's Razor used the same fingerprinting concepts and ended up
using a DNSBL of "fingerprints". Vipul founded Cloudmark and I don't
know what is the current status of the Razor project.

> - Is able to process feedback from domain owners and recipients in an 
> automated, quick, effective and anonymous enough way (with the GDPR et al). 
> Feedback is key.

I strongly agree with "Feedback is key" both for "spam reports" and
"non-spam reports" (and considering that "non-spam" only flows if you
didn't block at the SMTP level).
Unfortunately once you adopt SMTP reject based on a blacklist then you
accept to stop getting false positives about that traffic, so you
really stop monitoring the effectiveness of that block.

The issue with this is that you have to start building a reputation
not only for IP/domains/other email content fingerprints (sender
stuff), but also need to build a reputation for feedback providers
(recipient stuff) and maybe you also need a reputation management for
people asking delisting (consultant, ESP...)

A lot of data to mix.. so you start building some machine learning to
deal with that automatically and then you end up with SmartScreen and
no one, included your creator, will know why some message have been
blocked or not ;-)   (no offense to SmartScreen, I know how hard is to
deal with this stuff, but I receive Office365 invoices in spam, in my
Office365 inbox.).

An FBL system "ala Google PT" (so only aggregated data not enabling
list washing) but open to multiple receivers could help adding more
accountability to ESP to do their own filtering part (mainly with B2B
emails, as with B2C they already have microsoft/yahoo/google data).

Stefano

___
mailop mailing list
mailop@mailop.org
https://chilli.nosignal.org/cgi-bin/mailman/listinfo/mailop

Re: [mailop] Should mail servers publish IPv6 MX records? Could this harm your spam filtering?

[David Hofstee]

Hi Rob,

> ... score of the sending-IP, which is similar to what you've described,
correct?
Correct.

So you have these mechanisms in place. But your customers, who get access
to the invaluement RBL, do not.  Am I correct? If I am, it still results in
the conclusion that blacklists are not sufficient to have a resulting good
spam filter. You would be ok, the list would not have false positives,
but your customers would not be sufficiently covered once bad guys get
smarter.

I also think that there is space for a reputation provider which can:
- Identify more than just IP addresses and domains from an email.
- Is able to process feedback from domain owners and recipients in an
automated, quick, effective and anonymous enough way (with the GDPR et al).
Feedback is key.

Yours,


David


On 7 June 2018 at 17:29, Rob McEwen  wrote:

> On 6/7/2018 9:45 AM, David Hofstee wrote:
>
>> Isn't it time conclude that "separate IP blacklists" combined with
>> "separate content filters" are not sufficient any more? Because you need
>> one to interact with the other? You need the content filter to steer the IP
>> blacklist (and other traffic limiting methods like throttling and
>> greylisting).
>>
>> In this sense, many IP blacklists have always been indicators of
>> reputation instead of being used to block traffic "without questions".
>> Adding to a spam score.
>>
>> I think that these more complicated spam filters need a lot of data to
>> work (both the email and how people react to it). That is not easy to
>> obtain for smaller domains. I guess there is a technical challenge in
>> that...
>>
>
>
> David,
>
> If I had tried to cover all such details in that article (and other
> similar things that you could also have mentioned, too) I would have had to
> write a book, not an article. In fact, my own filtering does such things -
> but I can afford the extra processing, where I accept every entire spam
> message and then combine all such processing as you described - and even
> having them dynamically interact in the ways you've described, and in other
> ways, too.
>
> For example, one of the 3rd party command-line anti-spam content filters
> that I use in my spam filtering is good at blocking elusive spams, but has
> just a few too many false positives. Therefore, I dynamically alter its
> spam scoring in my system based on the sending IP's reputation, increasing
> that score if the IP isn't in my whitelist, and then further altering its
> score based on my systems' overall reputation score of the sending-IP,
> which is similar to what you've described, correct?
>
> But I can afford to put much time and money into my filter per user - even
> if that time and cost isn't justified by the overall spam filtering
> revenue. I can afford this precicely because my anti-spam business
> essentially subsidizes my small mail hosting and spam filtering business,
> which mostly exists as a way to keep my finger on the pulse of what my
> typical DNSBL subscriber experiences. HOWEVER - many businesses don't have
> this flexibility and/or they are stuck with a "canned" anti-spam solution
> from a software or hardware vendor that doesn't provide such flexibility.
> (and not all email admins are coders/programmers! nor should they have to
> be!) And, as I mentioned, others have such massively high volumes of
> inbound mail that they DEPEND on significantly reducing the volume of spam
> (with IPv4 blacklists) before it hits any kind of content filtering.
>
> And these are the more common situations - those with situations like your
> situation or mine (for example, most of my spam filter is self-programmed!)
> ...are more rare.
>
> BTW - for anyone just joining this thread, here is the article being
> discussed:
> https://www.linkedin.com/pulse/should-mail-servers-publish-
> ipv6-mx-records-rob-mcewen/
>
>
> --
> Rob McEwen
> https://www.invaluement.com
>
>
> ___
> mailop mailing list
> mailop@mailop.org
> https://chilli.nosignal.org/cgi-bin/mailman/listinfo/mailop
>



-- 
--
My opinion is mine.
___
mailop mailing list
mailop@mailop.org
https://chilli.nosignal.org/cgi-bin/mailman/listinfo/mailop

Re: [mailop] Should mail servers publish IPv6 MX records? Could this harm your spam filtering?

[Rob McEwen]

On 6/7/2018 6:55 PM, Brandon Long wrote:
Note I'm not saying that IP time blocking isn't useful. My issue is, 
are most RBL's good for IP time blocking? An RBL is a statement that 
everything from that IP is bad, but the truth of that statement varies 
greatly based on the RBL in question. But, in the end, what everyone 
seems to have is that hammer, and so the little built-in software 
support is for RBLs and using them either at connection time or 
letting the mail through. If the industry had moved to a reputation 
model, it would be easier to discuss "how bad is it" and whether it's 
bad enough to block at IP time, or whether you mix it into your spam 
score.




Brandon,

Much of what you're recommending... has essentially been done - or dealt 
with - once you "peel back the layers"...


First I should mention that, because of the following three factors, the 
zero-false-positive blacklist ideal... is no longer possible:


(A) legit websites getting hacked by spammers (who install spam content 
pages) is epidemic


(B) legit (generally non-spamming) mail servers getting an account 
hijacked by spammers is epidemic


(C) normally good ESPs or hosters getting a "bad apple" spammer customer 
is still a constant problem (though some ESPs do a MUCH better job than 
others of vetting their new customers!)


So this list above is made up of generally good senders who normally 
would never get blacklisted - but due to a security lapse - they then 
sometimes they get blacklisted, and for good cause! Not ever 
blacklisting those would be too large of a loophole for spammers, and 
the hijacked sites and accounts wouldn't get fixed! They would just 
build up - and many admins wouldn't have a clue about their security 
problem.


And then there are occasional dark-gray-hat ESPs and hosters who start 
out with BOTH legit customers and spamming customers -AND- who do little 
or no vetting - they don't pursue spammers - they just let the chips 
fall where they may without vetting their customers very well  - and 
these days - those are now ALWAYS going to downward spiral into 
blacklist hell. There is no way to avoid that, since spammers are 
attracted to that like flies on a picnic lunch - and the collateral 
damage is well deserved, but won't last long anyways as legit customers 
will flee - and that is a GOOD thing!


So, at the end of the day, you're left with 4 categories of blacklists:

(1) blacklists which do a GREAT JOB of keeping false positives to a 
minimum by carefully weighing the potential collateral damage of such a 
listing into the listing decision - and do a great job of delisting 
those who have fixed their problems  - and have good 
metrics/algorithms/procedures for delistings, but do NOT allow "too 
easy-off", but which do self-correct for situations that got fixed. (and 
they don't leave already-fixed-situations blacklisted for many days or 
weeks after the problem is fixed). These blacklists will still have some 
occasional collateral damage, but it is a tiny fraction of a percent of 
the whole  - and therefore these are safe to use for outright blocking. 
(for sending IP lists - examples in this category would definitely 
include Spamhaus and invaluement... MAYBE one or two others?)


(2) same, but they do a GOOD job of this, not a great job (so these 
would be better for high scoring, but you might get too many FPs if you 
score above threshold)


(3) same, but a mediocre job - so for these, I might add recommend 
adding 1 or 2 points in SpamAssassin - but these lists are STILL 
extremely valuable because they can often put the spam from a normally 
legit source that was hijacked "over the top" - yet without causing 
massive FPs from that sender's legit email. (whereas blacklists in the 
first category which aim for extremely low FPs - had to pass on such 
listings)


(4) all of the above - except the list is too "easy off" - helping 
snowshoe and other more deliberate spammers to get themselves delisted 
too easily (sometimes just in time for their next spam campaign!) These 
can often be scored above threshold - but don't count on these for 
blocking the more shrewd spammers!


ALL of these play a role in spam filtering. All of them! To varying 
degrees - they are ALL useful. And they ALL help reduce the load on 
content filters (though the 1st category does the most "heavy lifting" 
for  minimizing resource usage)


So to answer your question, a whole echo-system/culture has evolved 
around this... which effectively deals with the challenges you mentioned!


NOW - WHAT TO DO ABOUT HOSTERS/ESPs WHO DON'T CARE (or don't care much)?

With invaluement - we've had a situations where an ESP or hoster got so 
aggressively bad (often, talking the good talk publicly, but not 
matching that talk with actions) - that we've had to yank their IPs or 
domains out of our whitelist - and there is one particularly painful 
ongoing situation right now that is causing a little more collateral 
damage than 

Re: [mailop] Should mail servers publish IPv6 MX records? Could this harm your spam filtering?

[Brandon Long via mailop]

Note I'm not saying that IP time blocking isn't useful.

My issue is, are most RBL's good for IP time blocking?

An RBL is a statement that everything from that IP is bad, but the truth of
that statement varies greatly based on the RBL in question.

But, in the end, what everyone seems to have is that hammer, and so the
little built-in software support is for RBLs and using them either at
connection time or letting the mail through.

If the industry had moved to a reputation model, it would be easier to
discuss "how bad is it" and whether it's bad enough to block at IP time, or
whether you mix it into your spam score.

As for the large mailbox providers being too big to block, I think that's
really more of a question of how many false positives are you willing to
hit with your block, and the fact that you're arguing for something that's
a blacklist.  There are definitely large spammers out there which are pure
blocks, and definitely being blocked helps push back on folks to clean up
their act, and the RBLs are good for that part of the eco-system.  But they
will have a false positive aspect to them, and how much (by percentage and
absolute volume) an RBL is willing to put up with varies greatly as well.
What a large mailbox provider gives is the largest example of that fact,
that the ban hammer has affects the innocent as well.

And that's even before you get into the realm of grey mail, the mail that's
spam depending on the eye of the beholder, did this opt-in list overstep
it's bounds, is the content a little too much, did this sales women reach
out to one too many potential new clients (manually!).

Will SMTP be the last hold-out on IPv4?

Brandon

On Thu, Jun 7, 2018 at 8:41 AM Rob McEwen  wrote:

> On 6/7/2018 9:45 AM, David Hofstee wrote:
> > Isn't it time conclude that "separate IP blacklists" combined with
> > "separate content filters" are not sufficient any more? Because you
> > need one to interact with the other? You need the content filter to
> > steer the IP blacklist (and other traffic limiting methods like
> > throttling and greylisting).
> >
> > In this sense, many IP blacklists have always been indicators of
> > reputation instead of being used to block traffic "without questions".
> > Adding to a spam score.
> >
> > I think that these more complicated spam filters need a lot of data to
> > work (both the email and how people react to it). That is not easy to
> > obtain for smaller domains. I guess there is a technical challenge in
> > that...
>
>
> David,
>
> If I had tried to cover all such details in that article (and other
> similar things that you could also have mentioned, too) I would have had
> to write a book, not an article. In fact, my own filtering does such
> things - but I can afford the extra processing, where I accept every
> entire spam message and then combine all such processing as you
> described - and even having them dynamically interact in the ways you've
> described, and in other ways, too.
>
> For example, one of the 3rd party command-line anti-spam content filters
> that I use in my spam filtering is good at blocking elusive spams, but
> has just a few too many false positives. Therefore, I dynamically alter
> its spam scoring in my system based on the sending IP's reputation,
> increasing that score if the IP isn't in my whitelist, and then further
> altering its score based on my systems' overall reputation score of the
> sending-IP, which is similar to what you've described, correct?
>
> But I can afford to put much time and money into my filter per user -
> even if that time and cost isn't justified by the overall spam filtering
> revenue. I can afford this precicely because my anti-spam business
> essentially subsidizes my small mail hosting and spam filtering
> business, which mostly exists as a way to keep my finger on the pulse of
> what my typical DNSBL subscriber experiences. HOWEVER - many businesses
> don't have this flexibility and/or they are stuck with a "canned"
> anti-spam solution from a software or hardware vendor that doesn't
> provide such flexibility. (and not all email admins are
> coders/programmers! nor should they have to be!) And, as I mentioned,
> others have such massively high volumes of inbound mail that they DEPEND
> on significantly reducing the volume of spam (with IPv4 blacklists)
> before it hits any kind of content filtering.
>
> And these are the more common situations - those with situations like
> your situation or mine (for example, most of my spam filter is
> self-programmed!) ...are more rare.
>
> BTW - for anyone just joining this thread, here is the article being
> discussed:
>
> https://www.linkedin.com/pulse/should-mail-servers-publish-ipv6-mx-records-rob-mcewen/
>
> --
> Rob McEwen
> https://www.invaluement.com
>
>
> ___
> mailop mailing list
> mailop@mailop.org
> https://chilli.nosignal.org/cgi-bin/mailman/listinfo/mailop
>
___

Re: [mailop] Should mail servers publish IPv6 MX records? Could this harm your spam filtering?

[Rob McEwen]

On 6/7/2018 9:45 AM, David Hofstee wrote:
Isn't it time conclude that "separate IP blacklists" combined with 
"separate content filters" are not sufficient any more? Because you 
need one to interact with the other? You need the content filter to 
steer the IP blacklist (and other traffic limiting methods like 
throttling and greylisting).


In this sense, many IP blacklists have always been indicators of 
reputation instead of being used to block traffic "without questions". 
Adding to a spam score.


I think that these more complicated spam filters need a lot of data to 
work (both the email and how people react to it). That is not easy to 
obtain for smaller domains. I guess there is a technical challenge in 
that...



David,

If I had tried to cover all such details in that article (and other 
similar things that you could also have mentioned, too) I would have had 
to write a book, not an article. In fact, my own filtering does such 
things - but I can afford the extra processing, where I accept every 
entire spam message and then combine all such processing as you 
described - and even having them dynamically interact in the ways you've 
described, and in other ways, too.


For example, one of the 3rd party command-line anti-spam content filters 
that I use in my spam filtering is good at blocking elusive spams, but 
has just a few too many false positives. Therefore, I dynamically alter 
its spam scoring in my system based on the sending IP's reputation, 
increasing that score if the IP isn't in my whitelist, and then further 
altering its score based on my systems' overall reputation score of the 
sending-IP, which is similar to what you've described, correct?


But I can afford to put much time and money into my filter per user - 
even if that time and cost isn't justified by the overall spam filtering 
revenue. I can afford this precicely because my anti-spam business 
essentially subsidizes my small mail hosting and spam filtering 
business, which mostly exists as a way to keep my finger on the pulse of 
what my typical DNSBL subscriber experiences. HOWEVER - many businesses 
don't have this flexibility and/or they are stuck with a "canned" 
anti-spam solution from a software or hardware vendor that doesn't 
provide such flexibility. (and not all email admins are 
coders/programmers! nor should they have to be!) And, as I mentioned, 
others have such massively high volumes of inbound mail that they DEPEND 
on significantly reducing the volume of spam (with IPv4 blacklists) 
before it hits any kind of content filtering.


And these are the more common situations - those with situations like 
your situation or mine (for example, most of my spam filter is 
self-programmed!) ...are more rare.


BTW - for anyone just joining this thread, here is the article being 
discussed:

https://www.linkedin.com/pulse/should-mail-servers-publish-ipv6-mx-records-rob-mcewen/

--
Rob McEwen
https://www.invaluement.com


___
mailop mailing list
mailop@mailop.org
https://chilli.nosignal.org/cgi-bin/mailman/listinfo/mailop

Re: [mailop] Should mail servers publish IPv6 MX records? Could this harm your spam filtering?

[David Hofstee]

Hi Rob,

Isn't it time conclude that "separate IP blacklists" combined with
"separate content filters" are not sufficient any more? Because you need
one to interact with the other? You need the content filter to steer the IP
blacklist (and other traffic limiting methods like throttling and
greylisting).

In this sense, many IP blacklists have always been indicators of reputation
instead of being used to block traffic "without questions". Adding to a
spam score.

I think that these more complicated spam filters need a lot of data to work
(both the email and how people react to it). That is not easy to obtain for
smaller domains. I guess there is a technical challenge in that...

Yours,


David


On 7 June 2018 at 04:10, Rob McEwen  wrote:

> On 6/6/2018 8:11 PM, Brandon Long wrote:
>
>> Isn't the simplest way to handle this is to treat IPv6 at the /64 or
>> smaller level?
>>
>
> Brandon,
>
> Everything I've stated in that article, and in my comments on this thread
> - came with the foreknowledge that many of those who have discussed
> implementing IPv6 DNSBLs - have concluded that it would be wise to make the
> smallest block at the /64 level. I've already considered this. Your
> suggestion doesn't change a thing I've stated. And it doesn't change the
> "lack of scarcity" issues that will help IPv6 senders easily acquire new
> IPv6 space, where their paid hosters are not as concerned as much about
> soiled IP space, and start falling in love with spammers' money all over
> again. It also doesn't change the magical listwashing opportunities for
> sending spams on a one-IP (or /64) per recipient - then listwashing the
> recipients when that sending IPv6 (or IPv6 /64) gets listed. It doesn't
> change opportunities for them to do one-off sending where the resulting
> DNSBL entries greatly bloat the size of the DNSBL with entries that are
> absolutely worthless. There is just so much you glossed over or
> overlooked...
>
> I'm also not clear that content level scanning is really so much more
>> expensive that it can't be invested in.  "Here's a nickel kid, buy yourself
>> another VM" or something.
>>
>
> I've seen spam filters which were serving several thousand mailboxes and
> running on abundantly fast hardware - brought to their knees due to
> sending-IP filtering temporarily breaking due to a glitch, where everything
> then had to be fully content filtered. The problem here is that you're
> greatly underestimating the VAST difference in resources used when a
> majority of the spam is blocked by IPv4 RBLs at connection - vs - the
> resources needed to accept the entire message and then doing content
> filtering. This can really add up - to a point where it isn't "buying
> another cheap VM"... instead... that becomes... "quadruple your hosting
> budget", or worse. And this can also apply to large ISPs, too. I recall
> chatting with an admin from a really large ISP (~30M mailboxes) some years
> ago - and he told me - "we get 10s of thousand of messages per second, and
> we couldn't even begin to handle that volume without rejecting a large
> percentage of those at the connection based on the sending IP" - their
> expenses would instantly grow by many millions of dollars if they suddenly
> had to depend on much more content filtering.
>
> Also - large e-mail hosters who have teams of in-house software engineers
> (such as Microsoft, Google, Proofpoint, etc) are at a distinct advantage
> here because they have more options to do customized programming that
> enables them to implement strategies that will work better for IPv6 (such
> as filtering based on the DKIM domain, etc). In contrast, many smaller and
> medium sized businesses that used "canned" software or hardware-appliance
> solutions - are limited by the options presented in their software or
> hardware. And, unlike Google and Microsoft, they don't have the option to
> procure more hosting at essentially wholesale prices! So just because these
> issues are no big deal to YOU (Google) - doesn't mean that they don't
> present hardships to these other senders who might not have the flexibility
> and options you handle (potentially) too-fast paradigm shifts like this.
>
> So I'm just trying to help these other types to have the information they
> need to make "informed decisions" - which can be factored into their
> particular situation. Certainly, they shouldn't be pressured into
> publishing IPv6 records (and such pressure is already happening!),
> especially without knowing the rest of these facts - and especially not by
> others who have institutional advantages for handling IPv6 - who then might
> not empathize with their situation.
>
> Ironically, (and sadly) we ALREADY have had a situation for many years
> where the increasing complexity of managing a mail server had motivated
> many to abandon it and move their email to the cloud. This is not all bad -
> but I think the very large amount of that which occurred - was bad for the
> industry. We're now 

Re: [mailop] Should mail servers publish IPv6 MX records? Could this harm your spam filtering?

[John Levine]

In article  
you write:
>Isn't the simplest way to handle this is to treat IPv6 at the /64 or
>smaller level?

That's what Spamhaus does.  They made rbldnsd serve v6 CIDRs like it serves v4.

Apropos of Steve's comment about blowing caches, I did some
simulations a while ago of various ways to publish v6 DNSBLs.  I was
surprised to find that even v4 BL queries aren't cached much.  There's
a handful of heavily used cache entries for hosts that send lots of
mail, and a long tail of IPs that are hit once or twice and then never
again.

Also, anyone who uses BLs at scale has local mirrors, and querying a
local mirror is as fast as querying a local cache.

R's,
John

___
mailop mailing list
mailop@mailop.org
https://chilli.nosignal.org/cgi-bin/mailman/listinfo/mailop

Re: [mailop] Should mail servers publish IPv6 MX records? Could this harm your spam filtering?

[Steve Atkins]

> On Jun 6, 2018, at 5:11 PM, Brandon Long via mailop  wrote:
> 
> 
> 
> Isn't the simplest way to handle this is to treat IPv6 at the /64 or smaller 
> level?  More likely, because most people use IPv4, the RBL's just don't have 
> the data sources they need to populate the data, not because of some inherent 
> size problem with the data.
> 

IPv6 blacklists served over DNS using "regular" DNS infrastructure risk blowing 
out the caches of recursive resolvers (theoretically) if the lookup is done by 
/128 - there are potentially many, many queries you might have to make without 
getting cache hits. It's better with /64, but that's potentially not as 
selective as you might want while still meaning more cache hits (in theory) as 
/48s are handed out like candy in a way /16s aren't.

I'm not sure I believe that it's an actual problem today, or one that's likely 
in the future, but there is a potential issue there.

Distributing IPv6 reputation data via something other than DNS eliminates the 
issue. It can still be provided to the MXes via DNS, just directly from a local 
authoritative server rather than via a caching resolver.

That'd be better in many respects. (The history of BGP not being trivial to 
feed into mailservers and the coincidence that m4-ed sendmail.cf can be 
persuaded to do DNS lookups are the only reason we're where we are.)


> I'm also not clear that content level scanning is really so much more 
> expensive that it can't be invested in.  "Here's a nickel kid, buy yourself 
> another VM" or something.  More likely, there's a trade-off in trusting RBLs 
> completely vs how much mail you receive, and as you scale up, the more 
> numerous the false positives from RBLs become (not as a fraction but as an 
> absolute number)  and the more effort you need to put into doing more 
> complicated evaluations even as your traffic is higher.

I think content scanning is critical. A significant fraction of the spam I see 
- and a large fraction of the spam that's not trivially blocked - is coming 
from shared infrastructure (whether that be ESPs, Large Webmail Providers or 
Large Hosted Business Apps). Content can block that. Source IP based blocking 
can't, really, as the sources are shared between legitimate users and spammers. 
And, to wander back to the topic, the majority of spam I see on IPv6 comes from 
those sorts of provider.

Cheers,
  Steve
___
mailop mailing list
mailop@mailop.org
https://chilli.nosignal.org/cgi-bin/mailman/listinfo/mailop

Re: [mailop] Should mail servers publish IPv6 MX records? Could this harm your spam filtering?

[Brandon Long via mailop]

On Wed, Jun 6, 2018 at 4:48 PM Rob McEwen  wrote:

> On 6/6/2018 6:32 PM, Steve Atkins wrote:
>
> To answer the question in the title: "Probably, yes. Only if your spam 
> filtering is bad." As Rob mentions in his article in the case he's discussing 
> the spam would have been blocked if they'd had better spam filtering in place.
>
>
> Steve,
>
> (1) in that particular example, the technology used (uri blacklists
> checking domains in the body of the message) only applies to a subset of
> all incoming IPv6 spams, and the portion it doesn't apply to is very
> significant.
>
> (2) Also, had my subscriber (discussed in that article) not published IPv6
> MX records, those incoming IPv6 spams would have been sent via IPv4, and
> MUCH of that spam would have been easily blocked by low-FP IPv4 blacklists
> like Spamhaus' IPv4 blacklists and invaluement's IPv4 blacklists. In
> contrast, IPv6 filtering cannot possibly scale as well as IPv4 since the
> resulting increase in content filtering would be order of magnitudes more
> resource-expensive (CPU and RAM) than blocking so many of those connections
> via low-FP IPv4 blacklists at the perimeter. (especially when running the
> IPv4 lists locally in rbldnsd)
>
> Therefore, because of (1), IPv6 spam filtering can't possibly be *as*
> good (all else being equal) as spam filtering on a system that only accepts
> IPv4 connections. Because of (2) the filtering can't be as fast/efficient
> and doesn't scale nearly as well as IPv4 filtering - and the quality of
> one's content spam filtering doesn't change that. In fact, filtering IPv6
> spam will cause an INCREASE in necessary investments in more types of
> content filtering (to compensate for these not being blockable anymore via
> IPv4 blacklists), and that only further exacerbates the resource problems,
> which then only makes IPv6 filtering even LESS scalable.
>
> Steve - you have some valid points, but I think your "probably, yes"
> answer to the question in the title of my article fails to factor in these
> things. Due to these things I mentioned above, even someone with "good
> filtering" (who publishes IPv6 MX records) is still going to need EVEN MORE
> resource-expensive content filtering, which will require MORE hardware to
> run this increased content filtering, and such increases in content
> filtering does not scale very well (or at least don't scale
> inexpensively!). Also, their content filtering is STILL going to have some
> false positives in situations where the spam would have been easily blocked
> by IP4v blacklists had those IPv6 MX records not been published, and the
> content filtering missed. Also, the example of that spam being blockable
> via ivmURI was somewhat anecdotal. While ivmURI can greatly help to block
> IPv6-sent spams that otherwise would have been blocked by IPv4 IP
> blacklists, ivmURI doesn't solve the entire problem, nor can ANY content
> filtering be nearly as efficient as when an IPv4 DNSBL blocks the spam at
> the perimeter! In fact, many medium and large systems heavily depend on
> their content filters getting a reduced spam volume due to IPv4 blocking a
> high percentage of such spams BEFORE the body of the message is accepted
> (before DATA).
>
Isn't the simplest way to handle this is to treat IPv6 at the /64 or
smaller level?  More likely, because most people use IPv4, the RBL's just
don't have the data sources they need to populate the data, not because of
some inherent size problem with the data.

I'm also not clear that content level scanning is really so much more
expensive that it can't be invested in.  "Here's a nickel kid, buy yourself
another VM" or something.  More likely, there's a trade-off in trusting
RBLs completely vs how much mail you receive, and as you scale up, the more
numerous the false positives from RBLs become (not as a fraction but as an
absolute number)  and the more effort you need to put into doing more
complicated evaluations even as your traffic is higher.

Brandon
___
mailop mailing list
mailop@mailop.org
https://chilli.nosignal.org/cgi-bin/mailman/listinfo/mailop

Re: [mailop] Should mail servers publish IPv6 MX records? Could this harm your spam filtering?

[Rob McEwen]

On 6/6/2018 6:32 PM, Steve Atkins wrote:

To answer the question in the title: "Probably, yes. Only if your spam filtering is 
bad." As Rob mentions in his article in the case he's discussing the spam would have 
been blocked if they'd had better spam filtering in place.


Steve,

(1) in that particular example, the technology used (uri blacklists 
checking domains in the body of the message) only applies to a subset of 
all incoming IPv6 spams, and the portion it doesn't apply to is very 
significant.


(2) Also, had my subscriber (discussed in that article) not published 
IPv6 MX records, those incoming IPv6 spams would have been sent via 
IPv4, and MUCH of that spam would have been easily blocked by low-FP 
IPv4 blacklists like Spamhaus' IPv4 blacklists and invaluement's IPv4 
blacklists. In contrast, IPv6 filtering cannot possibly scale as well as 
IPv4 since the resulting increase in content filtering would be order of 
magnitudes more resource-expensive (CPU and RAM) than blocking so many 
of those connections via low-FP IPv4 blacklists at the perimeter. 
(especially when running the IPv4 lists locally in rbldnsd)


Therefore, because of (1), IPv6 spam filtering can't possibly be /as/ 
good (all else being equal) as spam filtering on a system that only 
accepts IPv4 connections. Because of (2) the filtering can't be as 
fast/efficient and doesn't scale nearly as well as IPv4 filtering - and 
the quality of one's content spam filtering doesn't change that. In 
fact, filtering IPv6 spam will cause an INCREASE in necessary 
investments in more types of content filtering (to compensate for these 
not being blockable anymore via IPv4 blacklists), and that only further 
exacerbates the resource problems, which then only makes IPv6 filtering 
even LESS scalable.


Steve - you have some valid points, but I think your "probably, yes" 
answer to the question in the title of my article fails to factor in 
these things. Due to these things I mentioned above, even someone with 
"good filtering" (who publishes IPv6 MX records) is still going to need 
EVEN MORE resource-expensive content filtering, which will require MORE 
hardware to run this increased content filtering, and such increases in 
content filtering does not scale very well (or at least don't scale 
inexpensively!). Also, their content filtering is STILL going to have 
some false positives in situations where the spam would have been easily 
blocked by IP4v blacklists had those IPv6 MX records not been published, 
and the content filtering missed. Also, the example of that spam being 
blockable via ivmURI was somewhat anecdotal. While ivmURI can greatly 
help to block IPv6-sent spams that otherwise would have been blocked by 
IPv4 IP blacklists, ivmURI doesn't solve the entire problem, nor can ANY 
content filtering be nearly as efficient as when an IPv4 DNSBL blocks 
the spam at the perimeter! In fact, many medium and large systems 
heavily depend on their content filters getting a reduced spam volume 
due to IPv4 blocking a high percentage of such spams BEFORE the body of 
the message is accepted (before DATA).


--
Rob McEwen
https://www.invaluement.com


___
mailop mailing list
mailop@mailop.org
https://chilli.nosignal.org/cgi-bin/mailman/listinfo/mailop

Re: [mailop] Should mail servers publish IPv6 MX records? Could this harm your spam filtering?

[Steve Atkins]

> On Jun 6, 2018, at 1:41 PM, SM  wrote:
> 
> Hi Rob,
> At 01:08 PM 06-06-2018, Rob McEwen wrote:
>> Here is an article I posted on Linkedin about spam filtering IPv6-sent email.
>> 
>> "Should mail servers publish IPv6 MX records? Could this harm your spam 
>> filtering?"
> 
> In other words, DNSBLs have a scalability problem.

DNSBLs do. Reputation providers keyed on peer IPv6 address don't.

To answer the question in the title: "Probably, yes. Only if your spam 
filtering is bad.".

As Rob mentions in his article in the case he's discussing the spam would have 
been blocked if they'd had better spam filtering in place.

Cheers,
  Steve


___
mailop mailing list
mailop@mailop.org
https://chilli.nosignal.org/cgi-bin/mailman/listinfo/mailop

Re: [mailop] Should mail servers publish IPv6 MX records? Could this harm your spam filtering?

[SM]

Hi Rob,
At 01:08 PM 06-06-2018, Rob McEwen wrote:

Here is an article I posted on Linkedin about spam filtering IPv6-sent email.

"Should mail servers publish IPv6 MX records? Could this harm your 
spam filtering?"


In other words, DNSBLs have a scalability problem.

Regards,
-sm 



___
mailop mailing list
mailop@mailop.org
https://chilli.nosignal.org/cgi-bin/mailman/listinfo/mailop

[mailop] Should mail servers publish IPv6 MX records? Could this harm your spam filtering?

[Rob McEwen]

Here is an article I posted on Linkedin about spam filtering IPv6-sent 
email.


"Should mail servers publish IPv6 MX records? Could this harm your spam 
filtering?"


https://www.linkedin.com/pulse/should-mail-servers-publish-ipv6-mx-records-rob-mcewen/

--
Rob McEwen
https://www.invaluement.com


___
mailop mailing list
mailop@mailop.org
https://chilli.nosignal.org/cgi-bin/mailman/listinfo/mailop