Re: Consolidation of Email Platforms Bad for Email?

[Owen DeLong]

> On Sep 8, 2020, at 4:38 AM, Eliot Lear via NANOG  wrote:
> 
> I'm sure Dave Crocker has thoughts about this, but it has come up elsewhere.  
> There are both positives and negatives about having such a consolidation.  
> The positive is that it a small club can establish ground rules for how they 
> will handle various forms of attacks, including BGP hijacking, DKIM, SPF, and 
> other forms of validation to identify fraudulent mail, etc.  Also, if you 
> have a whole lot of postfixes and sendmails running around, that's a whole 
> lot of code to patch when things go wrong.  A small number of MSPs can devote 
> a lot of time and paid eyes on code.  They can also very quickly spot new 
> attack trends.

All true…

> 
> On the other hand, that means that it becomes difficult to become a new 
> entrant, because one doesn't easily get one's mail accepted.  Lots of 
> grey/blacklisting (forgive the use of the term).  Also, when one of those 
> systems fails, it takes down a vast number of customers.  Furthermore, it 
> represents a massive concentration of private information that can be 
> monetized.

You’ve also left out:

Economic incentives to make questionable use of mail content and user data.
Economic incentives to make life difficult for new entrants.
Economic incentives to avoid transparency or convenience in addressing user 
concerns about erroneously rejected email.
Reduction in consumer choice (if there are a handful of providers and they all 
provide essentially the same (crappy) level of service, then what can the 
consumer do about it?
#include 

Owen

> 
> Eliot
> 
> On 08.09.20 00:27, Mike Hammett via NANOG wrote:
>> I originally asked on mailops, but here is a much wider net and I suspect 
>> there's a lot of overlap in interest.
>> 
>> 
>> I had read an article one time, somewhere about the ongoing consolidation of 
>> e-mail into a handful of providers was bad for the Internet as a whole. It 
>> was some time ago and thus, the details have escaped me, so I was looking to 
>> refresh my recollection.
>> 
>> Have any of you read a similar article before? If so, can you link me to it?
>> 
>> 
>> 
>> -
>> Mike Hammett
>> Intelligent Computing Solutions
>> http://www.ics-il.com 
>> 
>> Midwest-IX
>> http://www.midwest-ix.com 
>> 
> 

Re: Consolidation of Email Platforms Bad for Email?

[Don Gould via NANOG]

I find this question interesting (obviously because I'm responding to 
the list) and have done for decades.


Providing a reasonable email solution has become more and more complex 
while public perception is that email should be, and is, free.


I see lots of sides to this debate, some have already been covered by 
many of you already.


* Stuff has to be secure

* When stuff becomes insecure it starts to cause headaches for others.

* Keeping stuff secure gets harder and harder

* Customers want more and more features

* Customers should pay for some features/service

* Some IT folk are standing up systems to help others reduce costs - 
again causing headaches for others


* Some IT folk have set up expensive systems, funded by data mining and 
not customers.


* Some IT folk simply object to data mining - some folk act on that 
objection.


* There's a lot of 'activism' in the email space and has been for a very 
long time.


* Some of the 'big providers' take some of the heat out of the activism, 
which only winds up some IT folk even more.


* Knowledge and skills with people who can, and will, set up small 
systems is thinning as demand is growing.


* Some want to grow and drive others to rise up their skills.

* Some of those "drivers", I think [1], 'attack' learners, not unlike 
throwing the Apollo crew in a rocket simulator, hoping they will rise up 
their skills.


* With limited revenue, and constant 'driver training', some eventually 
abandon the game.


* Some view that driving training is important if you want to have skin 
in the game, but quickly forget their time is funded and they're not 
funding idealism.


* Some see their lunch being taken by a rise of good 'free' software.  
Some react by [1] driving more updates, features and improvements 
'help', which just overwhelms small operators.


* Some had no choice but to stand up small systems but 'now free 
offerings' have empowered them to abandon the space.


* Some have no thought around the issues, others simply don't care - 
some days there are just bigger fish.


Personally, I identify with some of these issues, and perhaps there's 
more, but it's the 'fish' question that right now connects with me the 
most...


https://scontent.fhlz1-1.fna.fbcdn.net/v/t1.0-9/118984848_10158758280448988_8560408895957059983_n.jpg?_nc_cat=105&_nc_sid=8bfeb9&_nc_ohc=VvSoKwD8SqkAX8hIeXE&_nc_ht=scontent.fhlz1-1.fna=69fc9c56a2e95fabe5cb637ba294ab35=5F7F5EB4

In a country of 5 million people, this graphic says we have ~18,000 
people waiting for social housing.  The idealist in me has turned it's 
attention, and while I still operate my own mail systems (mainly because 
I like to able to back it up and add capacity more quickly and I have 
trust issues with big providers changing the rules mid-stream), I to am 
leaning closer and closer to calling time...


...anyway, thanks for your eye balls, I'm off to put some paint on a 
building ready to launch a community housing trust to address that 
graphic.




[1] - Tin Foil Hat time.

D


On 2020-09-09 05:25, Barry Shein via NANOG wrote:

This is being portrayed a little too "either/or", that if you get spam
etc from $BIGEMAIL you, service provider, block them.

What goes on is multi-layer spam blocking using various tools rather
than host/server blocking except as a last resort.

So we'll block/toss/etc a lot of the malmail from $BIGEMAIL w/o
generally blocking their servers.

If we get a huge attack we have thresholds at which point we might
block them for two hours (whatever) hoping it stops on its own or
$BIGMAIL stops it.

But those are pretty high thresholds and obviously can cause problems
for our customers in delayed email but so can our mail servers being
pounded on. Those $BIGMAIL delivery servers have a lot more computrons
than we do.

Aside: What's astounding to me is how little any of this has changed,
other than consolidation perhaps -- remember when AOL's servers
pounding you with spam could bring you to your knees? I do -- in over
20 years.


--
Don Gould
5 Cargill Place
Richmond
Christchurch, New Zealand
Mobile/Telegram: + 64 21 114 0699
www.bowenvale.co.nz

Re: Consolidation of Email Platforms Bad for Email?

[Barry Shein via NANOG]

This is being portrayed a little too "either/or", that if you get spam
etc from $BIGEMAIL you, service provider, block them.

What goes on is multi-layer spam blocking using various tools rather
than host/server blocking except as a last resort.

So we'll block/toss/etc a lot of the malmail from $BIGEMAIL w/o
generally blocking their servers.

If we get a huge attack we have thresholds at which point we might
block them for two hours (whatever) hoping it stops on its own or
$BIGMAIL stops it.

But those are pretty high thresholds and obviously can cause problems
for our customers in delayed email but so can our mail servers being
pounded on. Those $BIGMAIL delivery servers have a lot more computrons
than we do.

Aside: What's astounding to me is how little any of this has changed,
other than consolidation perhaps -- remember when AOL's servers
pounding you with spam could bring you to your knees? I do -- in over
20 years.

-- 
-Barry Shein

Software Tool & Die| b...@theworld.com | http://www.TheWorld.com
Purveyors to the Trade | Voice: +1 617-STD-WRLD   | 800-THE-WRLD
The World: Since 1989  | A Public Information Utility | *oo*

Re: Consolidation of Email Platforms Bad for Email?

[Eliot Lear via NANOG]

On 08.09.20 16:59, Matt Harris wrote:

The positive is that it a small club can establish ground rules for
how they will handle various forms of attacks, including BGP
hijacking, DKIM, SPF, and other forms of validation to identify
fraudulent mail, etc.  [...] They can also very quickly spot new
attack trends.

> In theory, but the current state of what's coming out of sendgrid
> implies otherwise. 

It's not theory but history.  They have spotted those sorts of trends
quickly in the past (see below).  They may not tell you they have
spotted the trends.

> Once you get into that small club, it's just as hard to get kicked
> out, and unfortunately that means that if abuse, UCE, etc is coming
> from those hosts, they've got an even higher chance of hitting your
> inbox.

This depends on the nature of the incident, but if their evil bit gets
set and if their size is Size XL, then it is indeed hard to give them
the boot.

> So while in theory it might work the way you're thinking, in practice
> it hasn't because once you are in that club, a lot of the financial
> motivation to prevent abuse of your service - that is, inbox
> deliverability for your client base - goes away.

I disagree, but we aren't going to debate incentive models here. 
Suffice it to say that the big guys spending money on this, as they do,
belies your point.  A good example was one such very large provider
tracking hijacked BGP announcements and then releasing that information
to shut down a huge swathe of sources all at once.

However...

> That deliverability isn't likely to change for the negative on any
> scale that you care about once you're "in". But to be "in" you have to
> be at a huge scale. The small players are the ones who get hurt, and
> spam still gets through just fine only now via different means.

Yes.  That was why I said that there is good and bad.  Were we to take
this to extremes, we see why FB can curate their messages and keep spam
to a bear minimum, as they really do control the horizontal and the
vertical (two sided market).

>
> Also oligopolies in general are bad for everyone except the owners
> thereof and should be discouraged on principle. 

Not that I disagree (this comes to you by way of my dinky little VM),
but that's not the topic at hand.

Eliot




OpenPGP_0x87B66B46D9D27A33_and_old_rev.asc
Description: application/pgp-keys


OpenPGP_signature
Description: OpenPGP digital signature

Re: Consolidation of Email Platforms Bad for Email?

[Rob McEwen via NANOG]

On 9/8/2020 10:59 AM, Matt Harris via NANOG wrote:
Once you get into that small club, it's just as hard to get kicked 
out, and unfortunately that means that if abuse, UCE, etc is coming 
from those hosts, they've got an even higher chance of hitting your 
inbox. So while in theory it might work the way you're thinking, in 
practice it hasn't because once you are in that club, a lot of the 
financial motivation to prevent abuse of your service - that is, inbox 
deliverability for your client base - goes away.


+1

Likewise, we're at a point now where if a criminal phish or virus comes 
from the largest few email hosters, and you provide them emails with 
full headers - the accounts do NOT get shut down. They literally don't 
think this is their problem. And likewise, data storage sites 
(GoogleDrive, OneDrive, etc) from the largest providers often will host 
malware for weeks or months without being shut down - or the malware at 
least persists for many days after being reported. The same is often 
true for their redirectors.


Wwhat is frustrating is that the long-standing industry standard of 
"you're responsible both for what you both send and host - even if the 
malware wasn't intended" - seems to be lost.


Likewise, back in the spring months of 2018, google's "goo[.]gl" 
shortner went crazy for a few months, and was being MASSIVELY abused by 
spammers, and was being used as an "end run around" URI DNSBLs (SURBL, 
URIBL, ivmURI, DBL). I collected 15K examples of abused shortners that 
were "live", and sent those to Google. At the time I sent those, only 
about 500 of that 15K had been shut down. What was infuriating was that 
80% of these 15K shortners were pointing to only 12 spammer's domains. 
These should have been trivial to prevent!


The OTHER infuriating thing was that my INITIAL response from my 
contacts at Google was - (I paraphrase) "other spam filters should just 
follow the redirect, and block these spams based on the URI it redirects 
to" - WOW! I sent them a very stern email about that. (and for 
comparison, abused Bitly shortners were mostly getting shut down within 
2 hours - so "everyone does it" was NOT a decent excuse!)


Like I said - the long-standing industry standard of "you're response 
both for what you both send and host - even if the malware wasn't 
intended" - seems to be lost on some of these large providers.


Thankfully, this had a happy ending. After some "tough love" - Google 
replied back and said (I paraphrase), "we were planning on shutting that 
down - or at least shutting down the ability to add new ones - and due 
to your feedback - we're going to push that up a few months" - and so 
soon afterwards, they finally did terminate those 15K shortners - and 
stopped allowing new ones. So this is to Google's credit - but the 
problem had persisted for months - and it seemed like a lot of 
cultural/industry standards in the Internet Security industry seemed 
lost on them.


Sadly, while this situation had a good ending - similar problems with 
the largest providers persist. At the same time, they sure can be 
draconian in how they block smaller providers who had a rare and 
short-lived security incident. The hypocrisy is incredible. For example, 
Microsoft will sometimes *permanently* block a small email hoster for a 
short one or two hour compromised email account situation that caused 
spam to be sent from that small hosters - but that was quickly fixed - 
even if that hoster sends MUCH legit email. It almost FEELS like 
extortion - since many of the IT people running those small-ish servers 
sometimes get frustrating - and move their email to the cloud - and then 
guess who OFTEN gets their email hosting business?


-- Rob McEwen, invaluement

Re: Consolidation of Email Platforms Bad for Email?

[Caesar Kabalan via NANOG]

In many ways I see this similarly to the consolidation of browsers, but less 
consolidated. I think about the advantages and disadvantages of the prominence 
of Chrome (65%), Safari (20%), Firefox/Samsung/Edge/Opera/etc (15%). With 
Chrome we’ve seen Google move the browser and related standards forward through 
sheer marketshare. CSS/HTML/JS standards live and die by Chrome support and 
that’s both good and bad. They have made great and opinionated strides when it 
comes to SSL/TLS. For example, Google effectively killed Symantec’s certificate 
business because it was mismanaged. They also effectively got rid of EV certs 
and pushed secure-by-default web server design where HTTPS appeared normal, but 
warnings all over the place for non-encrypted connections. On the other hand, 
Google is fairly disliked in the privacy community and those communities prefer 
independent Firefox.

For email, I can see similar issues, mostly around security. If Microsoft were 
to decide security mechanism X is not worth the effort they can effectively 
decide to not implement it. What will internet users do, block all Microsoft 
email services? Conversely they could come up with their own security 
mechanisms and effectively force the rest of the world to adopt it. I do think 
centralization of email providers provides little potential for negative impact 
aside from operational issues. For example, outages probably have a wider 
impact due to number of users, but I can’t realistically see a scenario where 
Microsoft/Google does something “bad” with their email platform that affects 
the rest of the ecosystem.

Caesar Kabalan

From: NANOG 
Date: Tuesday, September 8, 2020 at 4:47 AM
To: Mike Hammett , NANOG 
Subject: Re: Consolidation of Email Platforms Bad for Email?

I'm sure Dave Crocker has thoughts about this, but it has come up elsewhere.  
There are both positives and negatives about having such a consolidation.  The 
positive is that it a small club can establish ground rules for how they will 
handle various forms of attacks, including BGP hijacking, DKIM, SPF, and other 
forms of validation to identify fraudulent mail, etc.  Also, if you have a 
whole lot of postfixes and sendmails running around, that's a whole lot of code 
to patch when things go wrong.  A small number of MSPs can devote a lot of time 
and paid eyes on code.  They can also very quickly spot new attack trends.



On the other hand, that means that it becomes difficult to become a new 
entrant, because one doesn't easily get one's mail accepted.  Lots of 
grey/blacklisting (forgive the use of the term).  Also, when one of those 
systems fails, it takes down a vast number of customers.  Furthermore, it 
represents a massive concentration of private information that can be monetized.



Eliot


On 08.09.20 00:27, Mike Hammett via NANOG wrote:
I originally asked on mailops, but here is a much wider net and I suspect 
there's a lot of overlap in interest.


I had read an article one time, somewhere about the ongoing consolidation of 
e-mail into a handful of providers was bad for the Internet as a whole. It was 
some time ago and thus, the details have escaped me, so I was looking to 
refresh my recollection.

Have any of you read a similar article before? If so, can you link me to it?



-
Mike Hammett
Intelligent Computing Solutions
http://www.ics-il.com

Midwest-IX
http://www.midwest-ix.com

[https://www.gore.com/sites/g/files/ypyipe116/files/2017-03/Gore_logo_0.png]

This email may contain trade secrets or privileged, undisclosed or otherwise 
confidential information. If you have received this email in error, you are 
hereby notified that any review, copying or distribution of it is strictly 
prohibited. Please inform us immediately and destroy the original transmittal. 
Thank you for your cooperation.

Re: Consolidation of Email Platforms Bad for Email?

[Matt Harris via NANOG]

Matt Harris|Infrastructure Lead Engineer
816-256-5446|Direct
Looking for something?
Helpdesk Portal|Email Support|Billing Portal
We build and deliver end-to-end IT solutions.
On Tue, Sep 8, 2020 at 6:43 AM Eliot Lear via NANOG  wrote:

> The positive is that it a small club can establish ground rules for how
> they will handle various forms of attacks, including BGP hijacking, DKIM,
> SPF, and other forms of validation to identify fraudulent mail, etc.  [...]
> They can also very quickly spot new attack trends.
>
In theory, but the current state of what's coming out of sendgrid implies
otherwise. Once you get into that small club, it's just as hard to get
kicked out, and unfortunately that means that if abuse, UCE, etc is coming
from those hosts, they've got an even higher chance of hitting your inbox.
So while in theory it might work the way you're thinking, in practice it
hasn't because once you are in that club, a lot of the financial motivation
to prevent abuse of your service - that is, inbox deliverability for your
client base - goes away. That deliverability isn't likely to change for the
negative on any scale that you care about once you're "in". But to be "in"
you have to be at a huge scale. The small players are the ones who get
hurt, and spam still gets through just fine only now via different means.

Also oligopolies in general are bad for everyone except the owners thereof
and should be discouraged on principle.

Re: Consolidation of Email Platforms Bad for Email?

[Eliot Lear via NANOG]

I'm sure Dave Crocker has thoughts about this, but it has come up
elsewhere.  There are both positives and negatives about having such a
consolidation.  The positive is that it a small club can establish
ground rules for how they will handle various forms of attacks,
including BGP hijacking, DKIM, SPF, and other forms of validation to
identify fraudulent mail, etc.  Also, if you have a whole lot of
postfixes and sendmails running around, that's a whole lot of code to
patch when things go wrong.  A small number of MSPs can devote a lot of
time and paid eyes on code.  They can also very quickly spot new attack
trends.


On the other hand, that means that it becomes difficult to become a new
entrant, because one doesn't easily get one's mail accepted.  Lots of
grey/blacklisting (forgive the use of the term).  Also, when one of
those systems fails, it takes down a vast number of customers. 
Furthermore, it represents a *massive* concentration of private
information that can be monetized.


Eliot


On 08.09.20 00:27, Mike Hammett via NANOG wrote:
> I originally asked on mailops, but here is a much wider net and I
> suspect there's a lot of overlap in interest.
>
>
> I had read an article one time, somewhere about the ongoing
> consolidation of e-mail into a handful of providers was bad for the
> Internet as a whole. It was some time ago and thus, the details have
> escaped me, so I was looking to refresh my recollection.
>
> Have any of you read a similar article before? If so, can you link me
> to it?
>
>
>
> -
> Mike Hammett
> Intelligent Computing Solutions
> http://www.ics-il.com
>
> Midwest-IX
> http://www.midwest-ix.com
>


OpenPGP_0x87B66B46D9D27A33_and_old_rev.asc
Description: application/pgp-keys


OpenPGP_signature
Description: OpenPGP digital signature

Re: Consolidation of Email Platforms Bad for Email?

[Suresh Ramasubramanian via NANOG]

I don’t know. Do I miss the days of every person and their dog running a mail 
server on a Linux server in a basement cupboard?

Huge crowds and high drama on nanae and spam-l type places

You never know whether your mail is going to get through or not because of 
weird and wonderful notions about spam filtering

No shortage of open relays and hacked Matt Wright formmail.pl

Whoever heard of backup?
(etc)




--srs

From: NANOG  on behalf of Mike 
Hammett via NANOG 
Sent: Tuesday, September 8, 2020 3:57:27 AM
To: NANOG 
Subject: Consolidation of Email Platforms Bad for Email?

I originally asked on mailops, but here is a much wider net and I suspect 
there's a lot of overlap in interest.


I had read an article one time, somewhere about the ongoing consolidation of 
e-mail into a handful of providers was bad for the Internet as a whole. It was 
some time ago and thus, the details have escaped me, so I was looking to 
refresh my recollection.

Have any of you read a similar article before? If so, can you link me to it?



-
Mike Hammett
Intelligent Computing Solutions
http://www.ics-il.com

Midwest-IX
http://www.midwest-ix.com

Consolidation of Email Platforms Bad for Email?

[Mike Hammett via NANOG]

I originally asked on mailops, but here is a much wider net and I suspect 
there's a lot of overlap in interest. 



I had read an article one time, somewhere about the ongoing consolidation of 
e-mail into a handful of providers was bad for the Internet as a whole. It was 
some time ago and thus, the details have escaped me, so I was looking to 
refresh my recollection. 


Have any of you read a similar article before? If so, can you link me to it? 



- 
Mike Hammett 
Intelligent Computing Solutions 
http://www.ics-il.com 

Midwest-IX 
http://www.midwest-ix.com