Re: [mailop] greylisting, SendGrid is deleting your mail
It appears that Al Iverson via mailop said: >What if we just got to the heart of the matter and admitted that >greylisting is useless 2023? Because it's still quite useful if you do it sensibly. Here's what my logs say for the past two years, by number of IPs greylist but never retried: 13573 greylist and did retry: 9299 whitelisted or previously retried: 6519 More than half of the addresses I greylist never retry, and spot checks of the logs show that's overwhelmingly bots. I suspect that Sendgrid's problem is that they gush so much spam that mail systems have cranked the spam filters way up, so they run into delivery problems that better behaved mail systems don't have. I have a bunch of special rules that whitelist a modest set of known good return addresses and report most of the rest as spam to them, for what little good that does. R's, John ___ mailop mailing list mailop@mailop.org https://list.mailop.org/listinfo/mailop
Re: [mailop] SendGrid is deleting your mail
What if we just got to the heart of the matter and admitted that greylisting is useless 2023? It was meant to annoy botnet spammers etc. who, once upon a time, did not have big iron and big disks to be able to queue and retry. That reality basically no longer exists. I tested greylisting myself this year in my own mail-in-a-box setup and found it was just a pain in the ass that delayed password reset emails while I spent too much time trying to whitelist stuff I cared about receiving in a timely fashion. It's a 1990s style configuration option that makes little sense in 2023 and I don't really see any data proving that it actually does anything that inhibits modern spam. The botnets can retry, too. Cheers, Al Iverson -- Al Iverson / Deliverability blogging at www.spamresource.com Subscribe to the weekly newsletter at wombatmail.com/sr.cgi DNS Tools at xnnd.com / (312) 725-0130 / Chicago (Central Time) ___ mailop mailing list mailop@mailop.org https://list.mailop.org/listinfo/mailop
Re: [mailop] SendGrid is deleting your mail
I thought I was a wild one with testing in production with SpamAssassin rules set to 0 score. That way I could determine the impact without making the impact at all, or without adding resource overhead to other parties just because I was playing around. I think it's a good point here that we all wish everyone would do the same thing, but you can't just get upset at people reacting to others who aren't doing things the way we wish they were, and not also get upset at the people who they're reacting to. In a perfect world we'd all do the same things all of the time, in reality there's nuance and not making adjustments for it is to be willfully ignorant of it. We all do what we have to do at the end of the day. I'm no fan of SpamGrid, I'm actually genuinely surprised to see that anyone works on the project at all anymore. I thought Twilio was just going to milk it until it's revenue dropped below overhead and then repurpose the IPs / sell off the brand. Since, you know, it seems like so little there has changed from an external perspective since this: Sendgrid Under Siege from Hacked Accounts - Krebs on Security [1] But credit where it's due, someone digging into response codes and reacting to the strange ways that everyone else handles their own mail servers, I respect the work. On 2023-06-23 16:28, Michael Orlitzky via mailop wrote: I don't necessarily disagree with this, but here's another perspective. Blacklists are hard to test in production because they rely on a third party. When testing a new config, it's common (at least in postfix land) to switch your reject code from 5xx to 4xx while you see how things play out. Basically, you're giving yourself time to look at the logs and decide if you're losing mail that you don't want to lose. When you're satisfied, you switch the reject code back to 5xx. There are several config parameters designed for this. Links: -- [1] https://krebsonsecurity.com/2020/08/sendgrid-under-siege-from-hacked-accounts/___ mailop mailing list mailop@mailop.org https://list.mailop.org/listinfo/mailop
Re: [mailop] SendGrid is deleting your mail
Default behavior is always to retry 4xx if there isn't a rule in place that overrides it. In this case, it was, and still is (i'm working on it) hitting another rule that was put in place to *not* retry a very specific 4xx that would never result in a successful delivery. The full response in question is this: 450 4.3.2 Please retry immediately. If your message was rejected by a blacklist, see http://michael.orlitzky.com/articles/so_youre_blacklisted.xhtml for more information. The bit of text in the string causing the trouble is the word blacklist. Unfortunately, we see a number of 4xx responses that contain the word blacklist and retries never succeed. For example, "450 4.3.2 your message was rejected because you're on a blacklist" is *not *retried because it's useless and *not* retrying is the appropriate thing to do for our customers. The response we are talking about here matches the response code, enhanced code, and contains the word blacklist. It's a rule collision that we can and will solve for. What really confuses me about this discussion is the almost religious fervor around the idea that under no circumstances should a 4xx not be retried and under no circumstances should a 5xx ever be retried. Anyone who has spent more than an hour reviewing SMTP responses for a large-scale mail system knows this is not true. But here we are. Smart, well intended people saying what is known to be false. It's Pravda-esque. It's the Emperor's New Clothes. I'm happy to be the one to say the emperor is naked and take my licks here. On Fri, Jun 23, 2023 at 12:15 PM Andy Smith via mailop wrote: > Hi, > > On Fri, Jun 23, 2023 at 08:03:40PM +0200, Carsten Schiefner via mailop > wrote: > > how about elaborating a bit further on the whats and whys of your setup? > > Maybe some of us could learn something from that, or maybe SendGrid > would consider that to be giving an advantage to competitors. Really > what I am interested in is the justification for not even retrying > once when receiving the "450 4.3.2 Please retry immediately" > response described by the OP. > > If I understand correctly, the OP had experienced missing mail from > SendGrid previously, had asked why there had not been retries on > a 4xx, and was told that SendGrid uses a complex rule set to decide > whether to actually retry for any given 4xx or 5xx. The OP then > tested that by coming up with "450 4.3.2 Please retry immediately", > which also did not receive any retries at all. > > So this implies that if SendGrid sees a "450 4.3.2" response that it > does not otherwise have a special rule for (or is it any 4xx that > there is no rule for?) then discarding the mail without any retries > at all is what happens by default. > > The idea of not retrying at all on certain specific 4xx responses > doesn't sit that well with me, but I can kinda sorta see why a large > sender might want to do that if they were really sure. But here > seems to be the implication that it's actually quite easy to trigger > that behaviour, unless by some stroke of bad luck "450 4.3.2 Please > retry immediately" happened to match an existing rule. > > It would be useful to know if it is the case that if one uses a 4xx > response that SendGrid hasn't seen before, it's going to result in > no actual retries. > > Thanks, > Andy > > -- > https://bitfolk.com/ -- No-nonsense VPS hosting > ___ > mailop mailing list > mailop@mailop.org > https://list.mailop.org/listinfo/mailop > ___ mailop mailing list mailop@mailop.org https://list.mailop.org/listinfo/mailop
Re: [mailop] SendGrid is deleting your mail
[NOTE: There's absolutely never any need to CC me individually when posting to this or any other list I'm on. Or not on.] On 2023-06-23 at 12:46:58 UTC-0400 (Fri, 23 Jun 2023 09:46:58 -0700) Luke via mailop is rumored to have said: That's right, Bill. It's so simple. All the ESPs and MTAs out there have tools built in to create and customize response handling rules for absolutely no reason. In 30 years of administering mail systems, including a wide variety of MTAs and a couple of MLMs, I've never found it useful to significantly modify the basic error codes or qualitatively vary reactions to them. Not sure how I'd even do that in Postfix, I'm pretty sure it would require hacking the code in ways that it is not designed for. Yes, it's easy to just never retry any 4xx, no, you can't adjust reaction to a 4xx code based on the text part or retry after a 5xx. We actually just do it for fun. We know that every single 4xx should be retried and every single 5xx should not be retried. But we thought it would be neat to have a table with hundreds of custom rules just to make it sound more complicated than it is. Sounds wild, but people do crazy stuff based on unsupportable hunches unencumbered by real data, particularly when it involves the mixing of computer and human behaviors. It would hardly be novel for an ESP to engage in self-deception about what they need to be the essentially arcane and complex lore of how to send bulk email. I've seen a few robust attempts to comprehensively violate the rules on not interpreting the text part of SMTP replies which, under careful analysis, were on-balance net negatives. The very meager discernible benefits could never justify the concrete cost of building and maintaining a system of dozens or hundreds of custom rules. Maybe you do it differently or better, but I'm a skeptic. So, you're right, it's just because we are big and we don't think the rules apply to us. Super productive take you have there. If I ever thought that I was being 'productive' for Twilio I would bill for it. I can tell you've really given this some thought. More than you can imagine... Nothing in the past 25 years has persuaded me that the "ESP" industry is anything but parasitic and harmful, as it is built on fundamental falsehoods. Your sneering is neither original, surprising, nor persuasive. It is clear that you believe you are exempt from the basic rules of SMTP interoperability and I've seen that before, often. My assertion of *why* you believe that is of course just a guess, but in my experience it has always boiled down to scale and a refusal to accept some fundamental truths of email. If you have evidence that it is actually worthwhile for you, you could just say that. My experience includes evidence that it is not. Maybe that has changed in 2023, but I doubt it. -- Bill Cole b...@scconsult.com or billc...@apache.org (AKA @grumpybozo and many *@billmail.scconsult.com addresses) Not Currently Available For Hire ___ mailop mailing list mailop@mailop.org https://list.mailop.org/listinfo/mailop
Re: [mailop] SendGrid is deleting your mail
On Fri, 23 Jun 2023, Carsten Schiefner via mailop wrote: Hi, Luke (& all) - how about elaborating a bit further on the whats and whys of your setup? Because at first sight it is indeed a bit hard to understand why SendGrid may not be in a position to follow the RFCs and the thereof derived and sort of well-established practices of handling server responses. This might actually take a bit of heat out of this thread. And after all, noboy here is really already at the top of their learning curve as I'd assume - and so additional knowledge is quite likely welcome. Luke did give us some examples of what he is up against in February, on the thread "Gmail blocking of good customer" https://list.mailop.org/private/mailop/2023-February/thread.html The first message in this long thread (50+ messages) is https://list.mailop.org/private/mailop/2023-February/024397.html and Luke made about six contributions. Perhaps the most pertinent one was https://list.mailop.org/private/mailop/2023-February/024461.html where Luke said: 4.1.1 Recipient Address Rejected: this one is tough because with some MXs retries result in delivery, in others, it's a dead end. Dynamic rule handling per receiving MX would be awesome, but it would require machine learning to accomplish at scale. which Michael Orlitzky suggested https://list.mailop.org/private/mailop/2023-February/024463.html might be because of "postfix's unverified_recipient_reject_code". I would think that considering the MX when overriding the response code would be a worthwhile addition to try, whether the machine learning is AI-based or hard coded. -- Andrew C. Aitchison Kendal, UK and...@aitchison.me.uk ___ mailop mailing list mailop@mailop.org https://list.mailop.org/listinfo/mailop
Re: [mailop] SendGrid is deleting your mail
Hi, On Fri, Jun 23, 2023 at 08:03:40PM +0200, Carsten Schiefner via mailop wrote: > how about elaborating a bit further on the whats and whys of your setup? Maybe some of us could learn something from that, or maybe SendGrid would consider that to be giving an advantage to competitors. Really what I am interested in is the justification for not even retrying once when receiving the "450 4.3.2 Please retry immediately" response described by the OP. If I understand correctly, the OP had experienced missing mail from SendGrid previously, had asked why there had not been retries on a 4xx, and was told that SendGrid uses a complex rule set to decide whether to actually retry for any given 4xx or 5xx. The OP then tested that by coming up with "450 4.3.2 Please retry immediately", which also did not receive any retries at all. So this implies that if SendGrid sees a "450 4.3.2" response that it does not otherwise have a special rule for (or is it any 4xx that there is no rule for?) then discarding the mail without any retries at all is what happens by default. The idea of not retrying at all on certain specific 4xx responses doesn't sit that well with me, but I can kinda sorta see why a large sender might want to do that if they were really sure. But here seems to be the implication that it's actually quite easy to trigger that behaviour, unless by some stroke of bad luck "450 4.3.2 Please retry immediately" happened to match an existing rule. It would be useful to know if it is the case that if one uses a 4xx response that SendGrid hasn't seen before, it's going to result in no actual retries. Thanks, Andy -- https://bitfolk.com/ -- No-nonsense VPS hosting ___ mailop mailing list mailop@mailop.org https://list.mailop.org/listinfo/mailop
Re: [mailop] SendGrid is deleting your mail
Hi, Luke (& all) - how about elaborating a bit further on the whats and whys of your setup? Because at first sight it is indeed a bit hard to understand why SendGrid may not be in a position to follow the RFCs and the thereof derived and sort of well-established practices of handling server responses. This might actually take a bit of heat out of this thread. And after all, noboy here is really already at the top of their learning curve as I'd assume - and so additional knowledge is quite likely welcome. Best, -C. -- Von meiner Hängematte aus gesendet. -Original Message- From: Luke via mailop To: Bill Cole Cc: Luke via mailop Sent: Fr., 23 Juni 2023 18:53 Subject: Re: [mailop] SendGrid is deleting your mail That's right, Bill. It's so simple. All the ESPs and MTAs out there have tools built in to create and customize response handling rules for absolutely no reason. We actually just do it for fun. We know that every single 4xx should be retried and every single 5xx should not be retried. But we thought it would be neat to have a table with hundreds of custom rules just to make it sound more complicated than it is. So, you're right, it's just because we are big and we don't think the rules apply to us. Super productive take you have there. I can tell you've really given this some thought. On Fri, Jun 23, 2023, 7:37 AM Bill Cole via mailop wrote: > On 2023-06-22 at 19:14:15 UTC-0400 (Thu, 22 Jun 2023 16:14:15 -0700) > Luke via mailop > is rumored to have said: > > > > Unfortunately we cant make a rule that retries all 4xx and doesn't > > retry > > all 5xx. Despite very smart, well intended people writing RFCs and > > other > > specs that make this feel super clean cut in an acedemic or lab > > environment, it's just not how it works in the wild. > > And yet somehow it is how the overwhelming majority of mail systems have > dealt with SMTP reply codes, for decades, successfully. > > SendGrid's deliverability problems are precisely because you and your > coworkers think that you are so big that the rules can't apply to you. > That you are somehow different and special. > > You are wrong. You're nothing special. You're just big. Get over > yourselves. > > > > -- > Bill Cole > b...@scconsult.com or billc...@apache.org > (AKA @grumpybozo and many *@billmail.scconsult.com addresses) > Not Currently Available For Hire > ___ > mailop mailing list > mailop@mailop.org > https://list.mailop.org/listinfo/mailop > ___ mailop mailing list mailop@mailop.org https://list.mailop.org/listinfo/mailop
Re: [mailop] SendGrid is deleting your mail
That's right, Bill. It's so simple. All the ESPs and MTAs out there have tools built in to create and customize response handling rules for absolutely no reason. We actually just do it for fun. We know that every single 4xx should be retried and every single 5xx should not be retried. But we thought it would be neat to have a table with hundreds of custom rules just to make it sound more complicated than it is. So, you're right, it's just because we are big and we don't think the rules apply to us. Super productive take you have there. I can tell you've really given this some thought. On Fri, Jun 23, 2023, 7:37 AM Bill Cole via mailop wrote: > On 2023-06-22 at 19:14:15 UTC-0400 (Thu, 22 Jun 2023 16:14:15 -0700) > Luke via mailop > is rumored to have said: > > > > Unfortunately we cant make a rule that retries all 4xx and doesn't > > retry > > all 5xx. Despite very smart, well intended people writing RFCs and > > other > > specs that make this feel super clean cut in an acedemic or lab > > environment, it's just not how it works in the wild. > > And yet somehow it is how the overwhelming majority of mail systems have > dealt with SMTP reply codes, for decades, successfully. > > SendGrid's deliverability problems are precisely because you and your > coworkers think that you are so big that the rules can't apply to you. > That you are somehow different and special. > > You are wrong. You're nothing special. You're just big. Get over > yourselves. > > > > -- > Bill Cole > b...@scconsult.com or billc...@apache.org > (AKA @grumpybozo and many *@billmail.scconsult.com addresses) > Not Currently Available For Hire > ___ > mailop mailing list > mailop@mailop.org > https://list.mailop.org/listinfo/mailop > ___ mailop mailing list mailop@mailop.org https://list.mailop.org/listinfo/mailop
Re: [mailop] SendGrid is deleting your mail
That's a very silly interpretation of what was said. On Thu, Jun 22, 2023, 11:01 PM Slavko via mailop wrote: > Dňa 22. júna 2023 23:14:15 UTC používateľ Luke via mailop < > mailop@mailop.org> napísal: > > >Unfortunately we cant make a rule that retries all 4xx and doesn't retry > >all 5xx. > > In other words, you are not able to ensure delivery, if target rejects with > 4xx code, and you are not able to stop resend, if target rejects with 5xx > code. > > And this all just because you have big amount of messages and smart > rules? > > Something is rotten in the state of Denmark... > > regards > > > -- > Slavko > https://www.slavino.sk/ > ___ > mailop mailing list > mailop@mailop.org > https://list.mailop.org/listinfo/mailop > ___ mailop mailing list mailop@mailop.org https://list.mailop.org/listinfo/mailop
Re: [mailop] SendGrid is deleting your mail
On 2023-06-22 at 19:14:15 UTC-0400 (Thu, 22 Jun 2023 16:14:15 -0700) Luke via mailop is rumored to have said: Unfortunately we cant make a rule that retries all 4xx and doesn't retry all 5xx. Despite very smart, well intended people writing RFCs and other specs that make this feel super clean cut in an acedemic or lab environment, it's just not how it works in the wild. And yet somehow it is how the overwhelming majority of mail systems have dealt with SMTP reply codes, for decades, successfully. SendGrid's deliverability problems are precisely because you and your coworkers think that you are so big that the rules can't apply to you. That you are somehow different and special. You are wrong. You're nothing special. You're just big. Get over yourselves. -- Bill Cole b...@scconsult.com or billc...@apache.org (AKA @grumpybozo and many *@billmail.scconsult.com addresses) Not Currently Available For Hire ___ mailop mailing list mailop@mailop.org https://list.mailop.org/listinfo/mailop
Re: [mailop] SendGrid is deleting your mail
On Thu, 2023-06-22 at 16:14 -0700, Luke wrote: > It's specifically targeting the response code and string you provided: 450 > 4.3.2 Please retry immediately Ok, I've spent the morning patching out a few more of our MTA's 4xx responses with that exact text. Some of them contained important diagnostic information, so I'm sabotaging my coworkers (and everyone else who sends us mail) to make this work. Supposing it does, it will have fixed the problem for one person, with one MTA, with one set of custom patches. You're still deleting everyone else's mail. ___ mailop mailing list mailop@mailop.org https://list.mailop.org/listinfo/mailop
Re: [mailop] SendGrid is deleting your mail
Dnia 22.06.2023 o godz. 16:14:15 Luke via mailop pisze: > > Unfortunately we cant make a rule that retries all 4xx and doesn't retry > all 5xx. Why? 4xx means the target says "I want you to retry". 5xx means the target says "I don't want you to retry". Regardless if this is deliberate, or by mistake on target's side, why can't you just be a good netizen and follow it? -- Regards, Jaroslaw Rafa r...@rafa.eu.org -- "In a million years, when kids go to school, they're gonna know: once there was a Hushpuppy, and she lived with her daddy in the Bathtub." ___ mailop mailing list mailop@mailop.org https://list.mailop.org/listinfo/mailop
Re: [mailop] SendGrid is deleting your mail
Dňa 22. júna 2023 23:14:15 UTC používateľ Luke via mailop napísal: >Unfortunately we cant make a rule that retries all 4xx and doesn't retry >all 5xx. In other words, you are not able to ensure delivery, if target rejects with 4xx code, and you are not able to stop resend, if target rejects with 5xx code. And this all just because you have big amount of messages and smart rules? Something is rotten in the state of Denmark... regards -- Slavko https://www.slavino.sk/ ___ mailop mailing list mailop@mailop.org https://list.mailop.org/listinfo/mailop
