Sir, this is a Wendy's. On Wed, Jul 2, 2025 at 3:47 PM Rich Kulawiec via NANOG < [email protected]> wrote:
> On Sun, May 25, 2025 at 11:20:16AM +0200, Tom Ivar Helbekkmo via NANOG > wrote: > > First: SPF/DKIM/DMARC are not about spam, so that part is irrelevant. > > Perhaps you don't remember this, but when SPF was announced, its home > page read: > > "Spam as a technical problem is solved by SPF." > > Shortly thereafter, spammers became far and away the largest early > adopters of SPF. > > Of course, that claim was eventually disappeared down the memory hole. > > > Now onto the long, and I do mean REALLY LONG part of this. Sorry, > it's a complex issue and it takes time to explain all the pieces. > > Here's a one-sentence summary: > > All these alleged anti-email-forgery technologies have accomplished is > to make the email forgery problem MUCH worse than before they existed. > > > > Introduction > ------------ > > I've never considered email forgery to be a significant problem -- > not when compared to the other problems we face. > > But let's put my opinion aside for a moment, and let's presume that email > forgery really is a significant problem -- one so serious that it's worth > adding an enormous amount of fragile complexity to an ecosystem already > under serious stress from spam and other attacks/abuse. Let's assume > that it's worth breaking email forwarding (working fine for decades) > and mailing lists (working fine for decades, and clearly the best mass > collaboration/communication mechanism we have) and adding enormous cost, > effort, and complexity to every email system. > > There's a problem with that: email forgery can't be solved. > > Even if if these byzantine hacks were perfectly designed (they're not) > and globally deployed (not happening) and correctly deployed everywhere > (nope) and always worked perfectly (also no) and weren't attacked (they > are) and so on, even if everything worked *exactly* as intended...the > overall impact of all this effort and expense and brittle complexity is > to make the email forgery problem MUCH worse. > > I've explained this before (briefly) but let me add a lot more detail > this time around. > > > Part A: contributory factors > ---------------------------- > > A1. We've had a problem with fake/junk domains for decades. It was bad > enough when there were only 7 gTLDs, but now there are now ~1000 new gTLDs > that nobody needed or wanted except (a) registrars, who of course want to > make the cash registers ring and (b) abusers/attackers, who buy domains in > bulk, burn through them, and then buy more. This symbiotic relationship > works beautifully for both parties involved but not for anyone else. > And of course registrars are performing no due diligence whatsoever > (why should they?) so it's trivial for abusers/attackers to register > example.xyz or example.lol or example-com.xyz or example-site.red or > whatever, where "example" is a name explicitly designed to suggest that > they're some other Internet operation -- e.g., Amazon, Paypal, Google, > Joe's Donuts, etc. > > I've been studying large samples of domain data for multiple decades, > and there are enormous numbers of these fake/junk domains. Many are > abandoned once they've served their purpose, but new ones are being > registered constantly. And every time a new TLD is put into production, > there's a land rush as abusers/attackers swarm it in an attempt to be > the first to register the optimal fake domains. > > I see no signs that this problem will be solved, because the people > responsible for creating it don't want it to be solved. If these > registrars performed any kind of due diligence, it would (a) cost them > money and (b) reduce their sales. There's no reason in the world for > them to do that. > > Bottom line: it's cheap, simple, and easy to register an arbitrary number > of domains whose names correspond to any operation that abusers/attackers > wish to target. > > You want a real-world example? Okay. Here's an example: > > Infrastructure Laundering: Blending in with the Cloud > > https://krebsonsecurity.com/2025/01/infrastructure-laundering-blending-in-with-the-cloud/ > > U.S. Sanctions Cloud Provider "Funnull" as Top Source of "Pig > Butchering" Scams > > https://krebsonsecurity.com/2025/05/u-s-sanctions-cloud-provider-funnull-as-top-source-of-pig-butchering-scams/ > > Complete List of Domains Attributed to Funnull > > https://www.ic3.gov/CSA/2025/Complete_List_of_Domains_Attributed_to_Funnull.zip > > There are 332,696 lines in that list. Well over a quarter million > fake/junk > domains used by just one scamming operation. And as large as this is, > it's the > tip of the tip of the tip of the iceberg. > > Let me also ask a very pointed question: if someone shows up at your > hosting > or cloud operation, and they want to deploy, let's say, 52,782 domains: do > you > think that maybe, just *maybe*, you ought to ask a few questions about > what the heck they're up to? > > If you're Microsoft or Amazon (see Krebs articles above): nope. Which > brings > me to: > > A2. There are all kinds of hosting/cloud operations out there which will > happily take example.xyz's money because they perform the same level of > due diligence as registrars in (1): none. These operations know full well > that example.xyz is not Example Inc, and they know full well what they're > doing, but because example.xyz pays the bills and they're repeat/bulk > customers, the hosts/clouds just take the money and look the other way. > > This isn't a small problem. It's massive. And it includes a lot of > hosting/cloud operations that should know better. And probably *do* > know better, but: due diligence costs money and takes effort, it's > cheaper and easier to just blow it off and only take action belatedly -- > long after the damage is done -- if there's any bad press. > > Bottom line: even the most obvious scam/fraud operations have no problem > finding hosting. > > A3. The combination of (A1) and (A2) mean that you can go out today > and register 1000, 10000, 100000 fake domains impersonating the Example, > Inc's of the world and you can find hosting for them, and you can do it > all at a bulk rate discount. Your only real difficulty will be competing > with the other people doing the same thing. And unless you do something > that generates a sufficient amount of sufficiently bad press, it's very > unlikely that a registrar or host will take any action to shut down > anything you're during, and even if they do that, they'll only shut down > the small part directly involved -- everything else you have in place will > be just fine. And then, on top of all this, they'll slow-walk whatever > they do in order to keep the income coming as long as they possibly can. > > A4. User interfaces in email clients -- whether standalone pieces > of software, or webmail interfaces that run in a web browser -- are > increasingly being modified to obfuscate sender addresses. In other > words, rather than displaying: > > From: Joe Smith <[email protected]> > > they display: > > From: Joe Smith > > and some kind of user action is required to actually see the address. > > The aggregate effect of this, across all UIs across all clients across > all users, is to train users to ignore actual email addresses in favor > of the text strings that purport to be someone. > > (Did YOU check the actual email address in this message to see if it's > really from me?) > > A5. One consequence of (A4) is that users are increasingly unable > and/or unwilling to understand email addresses. They can't distinguish > @example.com from @example.xyz or @example-com.lol or @exammple.com > or @exammple.somethingsomething.fun. > > (Yes, yes, there are *some* users that can do this. A few clueful > ones. As in: "A person is smart. People are..." C'mon, you know > the rest of this.) > > This loss of comprehension/function resulting from (A4) and (A5) works > beautifully for the operators of domains in (A1) and (A2): they don't > even have to try hard. Even sloppy half-ass typosquatted domains (and > subdomains) work just fine. > > [ Let me interject that while I was drafting this I read about the > plans by Mozilla to completely screw up the address bar in Firefox > -- because of course leaving the simplest, most fundamental, and > most important thing in the browser alone was never a choice for > these incompetent clowns. If I understand what they're going to > do correctly, and I really hope I don't, they're going to make > life MUCH easier for forgers/scammers/et.al. who are deploying > fake domains. ] > > A6. User interfaces in email clients are being modified to provide > signaling to users that a given message has been "authenticated" or > "verified" or "signed" or "certified" or some similar designation. > This is of course the result of a successful check of whichever > anti-forgery mechanism(s) the receiving email server chose to utilize. > > And just as users being trained to accept (A4), they're being trained to > accept this. They believe what's in front of them on the screen. > > But of course this means *nothing*: a message from [email protected] will > be dutifully marked (provided the keepers of example.xyz set things up > correctly and everything's working) as authentic even though example.xyz > is a fake domain that has nothing to do with the real Example Inc. and > Joe is a fake person. > > In other words, obviously fake messages from obviously fake domains are > treated the same as authentic messages from authentic domains. > > ("But couldn't...". No. It's not a tractable problem, because it would > require enumerating the authentic domain(s) of every operation on the > Internet, i.e. determining that example.com is the "real Example Inc. -- > for millions of example.com's, example.edu's, example.net's, etc. And not > only would this be incredibly expensive, it would break badly as companies > and domains change ownership or go out of business, it would greatly > favor large operations over small ones, it would quickly be corrupted by > the people with the biggest wallets, it would... So don't even go there.) > > This is analogous to the wry saying about https:// -- yeah, it means > you have a (putatively) secure connection, but you could have a secure > connection to the worst people in the world. > > A7. Many Example Inc's of the world don't send plaintext email messages; > they mark them up with HTML and graphics and so on, which means that > they're teaching their users that any message which looks like it's from > them is really from them. > > They're training their users to fall for forgeries/phishes. > > And years (decades now) of this training have paid off: it's much, > MUCH easier to trick users now than it was 20 years ago. > > A8. Similarly, many Example Inc's of the world include URLs in their > messages. They've spent years (decades now) training users to utilize > those URLs, and of course users have dutifully learned to do this, > and once again: > > They're training their users to fall for forgeries/phishes. > > And years (decades now) of *this* training have also paid off: again, > it's much easier to trick users now than it was 20 years ago. > > A9. The combination of (A7) and (A8) means that there's no help coming > from the Example Inc's of the world. They'll continue to train > their users to be victims while exhorting them to not be victims. > The relevance to my point here? These email worst practices distract > users from the small pieces of critical information that *might* give > them a slim chance of spotting a forgery. They don't cause the problem, > but they certainly exacerbate it. > > A10. The aggregate impact of (A1)-(A9) is: users see a message that looks > as they expect it to -- it has the pretty graphics and typography of > Example Inc. It claims to be from Example Inc. It's marked as "verified" > in their email UI. It has a domain that looks kinda like Example Inc's > (if they even bother to check, which almost none of them will). > > What do you *think* they're going to do? I can tell you what they won't > do: they won't take the time to actually study the email address and see > that it's @example-com-completely-fake.xyz, or to study the embedded URLs, > and even if they do, a substantial fraction of them Will Not Get It. > > If you think I'm wrong about this, then I must ask: have you actually > spent any time with a significant number of users? > > > Part B: large email operations > ------------------------------ > > It's not always necessary to go through the trouble and expense of > registering domains, arranging for hosting, etc. > > It's easy to set up an arbitrary number of accounts that claim to be > arbitrary people/companies and spam/phish/whatever from them at will, > because there are some very large mail operations that make this easy. > > And because those same operations have implemented all this anti-forgery > tech, those messages will dutifully pass every check that can be done > on them. > > And before someone trots out the "...but they're so large and it's SOOOO > hard..." nonsense as an excuse for the malfeasance of these very large > email operations: no. If you can't run an enormous operation properly, > you shouldn't *build* an enormous operation. It's irresponsible. > It's unprofessional. It's unethical. > > Especially when it would be trivial for these operations to throw 1,000 > people and a billion dollars at the problem tomorrow. (We *know* they > have those resources, because they're already wasting them on bogus > "AI" development.) > > Let me note, in fairness, that there are also some other rather large > operations that -- as far as I can tell by measuring across hundreds of > domains over several decades -- do a very good job. The problem is that > they're the exception. > > > Recap of part A and part B: > --------------------------- > > The combination of (A) and (B) means that we have an environment where > it's cheap, easy, and simple to register an arbitrary number of domains > masquerading as Example Inc. It's also cheap, easy, and simple to find > hosting for them. And then -- thanks to very poor choices in user > interfaces combined with lack of user knowledge/effort, it's cheap, > easy, and simple to convince lots of people that completely fake messages > from completely fake domains are authentic. And there's even a pretty > good chance that forgeries which don't even use a fake domain will work: > there are certainly unlimited opportunities to try -- also cheap, easy, > and simple. > > The sad reality is that while it's true that thanks to this anti-forgery > tech, abusers will have a hard time successfully forging messages from > the real example.com and getting them delivered: they don't have to. > > > Part C: bots > ------------ > > And then it gets even worse, because of a problem that still exists > but doesn't garner the headlines that it did years ago. > > The problem is bots. Remember this? > > Vint Cerf: one quarter of all computers part of a botnet > http://arstechnica.com/news.ars/post/20070125-8707.html > > That was 18 years ago. And in the ensuing time, *nothing* has happened > to seriously address that problem. (Yes, yes, I'm aware of all the > headline-grabbing when someone somewhere dismantles a botnet, but > those are momentary disruptions at best.) > > On the other hand, lots of things have happened to make the bot problem > worse, including (1) more computers (2) more users (3) vastly more > sophisticated bot-creating malware (4) bot-creating malware targeting > non-Windows systems (5) major improvements in botnet C&C (6) the IOT > -- remember, the "S" in "IOT" stands for security. And (7) thanks to > Microsoft's "Recall", things will get worse yet again. > > Oh, and let's (8) toss in AI, because the people frantically developing > those systems want to spend lots of time and money hyping them but can't > be bothered to put in any guardrails or do any real testing or even > concern themselves for a moment about how this tech will be misused by > some of the worst people on the planet. I'll bet even money that there's > already a botnet being run by a commercial AI. It's an obvious move, > don't you think? > > Note that: > > C1. Anyone in control of a bot can utilize any email credentials stored > on or transiting that bot at will. If Joe has an address @example.org > and one at @aol.com, then the new owner of Joe's system can send outbound > messages via those at will. > > C2. They can also rummage through all email stored on that system or > in the accounts from (1), which means that -- if they want -- they can > easily make some first-order guesses at who's likely to believe that > a message which claims to be from Joe AND that their email system's UI > says is from Joe, really is from Joe. > > C3. This is already bad if Joe is just some guy and Joe's system is just > some random laptop somewhere. But what if it's not? What if Joe works > at Example Inc. and this is his desktop system and it's loaded with > customer correspondence and it's connected to his work email account, > also loaded with customer correspondence? *Now* the new owner of Joe's > system is in an ideal situation to phish or scam all of those people, > because such a message really will come from Joe's email address via > Joe's work email server, it'll pass all the anti-forgery checks anyone > cares to run, and it'll dutifully be marked as "authentic" in most/all > those email UIs used by its recipients. > > So even if the content is a little sketchy -- maybe enough to raise > an eyebrow or two among recipients -- they'll look at their screen, > see that the message is marked as real, see that the fancy graphics > and typography look real, AND THEY'LL BELIEVE IT. Of course they will, > they've been trained to, see A7 and A8 and above > > C4. And what about the tiny fraction of people who will catch on that > think that's something amiss? What, EXACTLY, do you propose that they > do about it? Example Inc has probably done its very best to turn itself > into an impenetrable corporation where it's nearly impossible to reach > anyone in a position to understand a problem like this and take prompt > action to fix it. Do they have working postmaster@, security@, abuse@ > addresses -- something that every novice sysadmin on this planet should > know about? Probably not. Do they have a published security contact? > Probably not. Do they have any mechanism whereby someone who calls in can > reach anyone useful? Probably not. So even if one of those rare, clueful, > diligent people spots this and tries to report it, they're probably > going to give up in frustration long before they achieve anything. > > C5. And even if -- by some miracle of science -- someone in (C4) manages > to reach a contact inside Example Inc who understands the problem...the > most likely outcome is that Example Inc will ignore it and pretend it > doesn't exist. The next most likely outcome is that they'll quietly fix > it but say nothing. Why should they inform or warn anyone? That's bad > for business. And since there are no significant penalties for anyone > involved, their best bet is to disappear the problem and claim it > never happened. (If caught? Deny everything. If REALLY caught? > Blame a miscommunication and throw a third-level manager under the bus.) > > Sadly, (C4) and (C5) are not hypothetical. They're why I don't even > bother any more. Why should I invest my time and unpaid labor when > these operations can't even be bothered to do the simplest, easiest, > most obvious things like supporting RFC 2142 addresses and reading > what shows up there, acting on it promptly, and publicly acknowledging > what's happened? > > How many email accounts are affected by bot proliferation? A first-order > estimate may be found by multiplying (a) an estimate of the worldwide > population of bots by (b) an estimate of the average number of email > accounts used/present on those bots. I think the floor for (a) is 750M > and 1.5B is much more likely. (Go back to Cerf's estimate from 18 years > ago and extrapolate.) As to (b) I think it's probably between 1 and 2 -- > let's go with 1.5, probably low but that may also account for duplicate > email accounts across bots. > > This gives us a quite conservative back-of-the-envelope guesstimate > somewhere north of 1B. > > One billion compromised email accounts. > > And you want to tell me it's possible to do something truly meaningful > about email forgery? Okay. Fine. *Fix this problem first*, then > we'll talk. > > I'll wait. Let me know when you're done. > > Note that not only do you have to (1) remediate all those accounts, > you'll also need to (2) remediate all the bot'd systems and then > (3) keep them that way -- which means discovering the root cause(s) of > their > compromise and fixing it. (3) will be very difficult if the root cause is > something like "the user clicks on every shiny thing they see". But if > you're going to claim it's possible to do something truly meaningful > about email forgery, then you must do all those steps first. > > > Part D: email account compromises > --------------------------------- > > And (C) isn't even all the compromised email accounts, because we also > have to think about all the cases where only an email account (not an > entire system) has been compromised. I don't know how many there are > (nobody does), but I strongly suspect it's also an enormous (hundreds of > millions or more) number. One way to approach an estimate of this is to > observe the marketplaces where they're being sold, and in those places, > there are lots people selling them in bulk: how many hundred thousand > would you like? > > (Do recall that a few years ago, EVERY Yahoo email account was > compromised. That was a neat 3 billion accounts right there. > > Every single Yahoo account was hacked - 3 billion in all > > https://money.cnn.com/2017/10/03/technology/business/yahoo-breach-3-billion-accounts/index.html > > And as anyone who's paying attention to the daily parade of security > breaches and dataloss incidents is painfully aware: this isn't an > isolated case. It's the norm.) > > This problem is exacerbated by the complete lack of support at some very > large email operations: even when someone who's the real owner of one > of these compromised email accounts detects it and make a good-faith > effort to fix the problem...yeah, good luck with that. Their chances > of recovering their own email account are pretty much zero thanks to > awful procedures and non-existent customer service. Thus there's a > high probabilty that any compromised email account is going to stay > compromised until it no longer exists. > > Again, if you want to tell me that something serious is going to happen > WRT email forgery: *you have to fix this too*. Let me know when that's > done as well. > > > Part E: email server compromises > -------------------------------- > > And *then*, on top of (C) and (D), we have to consider the cases where > email servers themselves have been compromised, and there's a steady > parade of those. Hardly a day goes without another announcement of > a massive security breach that involves most/all of some operation's > infrastructure, > > Those announcements are only the tip of the iceberg, since pretty > much everyone does their best to conceal such things until either the > press reports on them and/or they're legally required to report them... > and even *then* they'll sometimes persist in denial and minimization. > > I'm looking right at you, Oracle. > > Anyway, compromise of an email server means that it's not only possible to > undetectably forge email from every account on it, it's also possible to > create new ones and do the same with those. And of course the new owners > of these servers have access to everything stored on or transiting them, > which provides all kinds of resources that are useful in figuring out > who to target, how best to target them, etc. > > And yet once again, if you want to tell me that something serious is > going to happen WRT email forgery: *you have to fix this too*. First. > > Yeah, I'll wait for this too. Sure I will. > > > Summary of (C) (D) (E): > ------------------------ > > So whether (C) email accounts have been compromised via bot'd systems > or (D) just the email accounts themselves have been taken over or > (E) entire email servers have been comprommised, outbound messages from > all of those will pass any anti-forgery check anybody feels like running. > This problem is already well past "enormous" and will continue to > get worse, as various ill-conceived bandaids (e.g. "let's get rid of > passwords and use passkeys", a breathtakingly stupid and cruel idea) > are haphazardly slapped on the problem because nobody wants to spend > the massive resources it would take to solve it properly -- one system, > one account, one server at a time. Nobody wants to do the hard boring > work that might actually succeed in the real world *without* creating > more problems than it claims to solve. > > Until then, we're living in a world where "one billion compromised email > accounts" is very likely a serious underestimate. And there will be more > tomorrow -- doubly so because attackers know everything I just said. > They recognize that the ill-advised rollout of anti-forgery tech that > doesn't actually solve the forgery problem has created a great opportunity > for them...and they're all over it. > > > Conclusions? Well, maybe. ;) > ----------------------------- > > As I've said before, if you want to solve email forgery for a real-world > value of "solve" at the scale of the Internet, then you have to solve > the underlying security problems first. > > ALL OF THEM. > > And nobody's interested in doing that, because "doing nothing" and > "pretending those problems don't exist" is not only far easier, it's > far more profitable. Plus it's much more fun to invent technology > and promote it and deploy it and declare "Mission Accomplished" and > take credit for solving the problem than it is to do the hard work > of actually solving the problem. > > So what's really transpired here, as a consequence of this headlong rush > to "solve" email forgery, is that all these elaborate mechanisms have > been deployed on top of an ecosystem that's broken all the way down. > > They're gold leaf on a rotting garbage pile. > > They don't work because they can't work. They've make-believe -- and > unfortunately *users are being trained to believe in them*, which is > going to lead to very bad real-world outcomes. > > Note well: none of this commentary has anything to do with the actual > mechanics of SPF or DMARC or DKIM or anything else, which is why > I haven't discussed their specifics. The specifics *do not matter*. > If someone comes up with another one of these tomorrow and writes an > RFC and releases code and tells us all about it: that's not going to > work either -- for the same reasons. > > Yes, in some ideal best-case world, maybe, MAYBE, this elaborate > anti-forgery tech might have a fighting chance. (Although: if we were > living in that ideal best-case world, we probably wouldn't need it. > Do recall that we did just fine for many years without any of this, > modulo the occasional prankster or idiot.) > > But this is not the ideal best-case world. This is a world with massive > problems that nobody wants to solve -- and as long as this writeup is, > I've only listed some of them. (Yeah, it's worse. It's always worse.) > > ---rsk > > _______________________________________________ > NANOG mailing list > > https://lists.nanog.org/archives/list/[email protected]/message/J4QLVGCKA6NG4WQY5WGANML2DYLODWUY/ > _______________________________________________ NANOG mailing list https://lists.nanog.org/archives/list/[email protected]/message/E4QDYX4Z3DSTXWM2UYFXBEJSUYIBAJE2/
