[mailop] SendGrid is deleting your mail (again)

2023-07-11 Thread Michael Orlitzky via mailop
Early yesterday morning, our upstream circuit was down from about 2:30am until 5:00am EDT. During that time, our MX was not answering; so, no 4xx or 5xx response involved. It's been over 24h, and I'm (personally) still missing five GitHub notifications from that time: 4:03am EDT,

Re: [mailop] SendGrid is deleting your mail

2023-06-25 Thread Slavko via mailop
Dňa 25. júna 2023 15:39:44 UTC používateľ Andy Smith via mailop napísal: >I think that you are perhaps only considering this from the >perspective of a sender. When it comes to choosing text for SMTP >responses there are many different types of person involved and many >of them will not be

Re: [mailop] SendGrid is deleting your mail

2023-06-25 Thread Dmytro Homoniuk via mailop
Yes, me being only on the senders' side for now is driving some of the bias. I'll work on that. I agree with the rest of your mail, except for "actionable" being equal to "how to deliver this message in future" - I would consider *"we'll never accept mail from you"* to be also very actionable with

Re: [mailop] SendGrid is deleting your mail

2023-06-25 Thread Andy Smith via mailop
Hi Dmytro, On Sun, Jun 25, 2023 at 02:28:33PM +0100, Dmytro Homoniuk via mailop wrote: > 450 4.3.2 Local problem - couldn't query foobar blacklist > > I do think this very hypothetical example is a bit of an outlier. It's > providing non-actionable information to the sending system: it should

Re: [mailop] SendGrid is deleting your mail

2023-06-25 Thread Michael Rathbun via mailop
On Sun, 25 Jun 2023 14:28:33 +0100, Dmytro Homoniuk via mailop wrote: >*In a very non-confrontational way* I want to express my opinion and to >note that's pretty much how senders operate right now: too often the smtp >code and enhanced code the recipient system returns have nothing to do with

Re: [mailop] SendGrid is deleting your mail

2023-06-25 Thread Michael Orlitzky via mailop
On Sun, 2023-06-25 at 14:28 +0100, Dmytro Homoniuk via mailop wrote: > > 450 4.3.2 Please retry immediately. If your message was rejected by a > blacklist, see for more information. > > Now this just adds needless ambiguity: was it rejected because of the > blacklist or not? Am I on a

Re: [mailop] SendGrid is deleting your mail

2023-06-25 Thread Dmytro Homoniuk via mailop
On 6/24/23 14:03, Andy Smith via mailop wrote: If this sort of thing is common amongst large senders, does that mean that we should all be combing our 4xx responses for "triggering" words like "blacklist" and "blocklist"? And how are we to keep up to date with the heuristics of multiple senders?

Re: [mailop] SendGrid is deleting your mail

2023-06-24 Thread Matt Palmer via mailop
On Fri, Jun 23, 2023 at 11:13:56PM -0500, Al Iverson via mailop wrote: > What if we just got to the heart of the matter and admitted that > greylisting is useless 2023? That feels like a bit of a strawman, insofar as greylisting is *far* from the only reason why a 4xx could be emitted. - Matt

Re: [mailop] SendGrid is deleting your mail

2023-06-24 Thread Dave Crocker via mailop
On 6/23/2023 9:13 PM, Al Iverson via mailop wrote: What if we just got to the heart of the matter and admitted that greylisting is useless 2023? This prompts wondering whether it's time for the email anti-abuse community to have a discussion about mechanisms that used to be useful but are

Re: [mailop] SendGrid is deleting your mail

2023-06-24 Thread John Levine via mailop
It appears that Bill Cole via mailop said: >On 2023-06-24 at 00:13:56 UTC-0400 (Fri, 23 Jun 2023 23:13:56 -0500) >Al Iverson via mailop >is rumored to have said: > >> What if we just got to the heart of the matter and admitted that >> greylisting is useless 2023? > >+1 > >It was always a cute

Re: [mailop] SendGrid is deleting your mail

2023-06-24 Thread Louis Laureys via mailop
Fully agree! You make a good argument. I didn't mean to say that this issue should be ignored entirely. I would argue that their matchers could be more specific for such a drastic action as "drop the message and don't retry". I just don't agree with the whole sentiment that sendgrid has

Re: [mailop] SendGrid is deleting your mail

2023-06-24 Thread Louis Laureys via mailop
Why take the time just to victim blame? That wasn't my intention. This has turned into a discussion, and I wanted to add my opinion. I'm not really defending them, I just think you're getting very angry about something that doesn't seem all that odd to me personally. The reason I wanted to

Re: [mailop] SendGrid is deleting your mail

2023-06-24 Thread Bill Cole via mailop
On 2023-06-24 at 00:13:56 UTC-0400 (Fri, 23 Jun 2023 23:13:56 -0500) Al Iverson via mailop is rumored to have said: What if we just got to the heart of the matter and admitted that greylisting is useless 2023? +1 It was always a cute trick that just happened to mostly work but unlike

Re: [mailop] SendGrid is deleting your mail

2023-06-24 Thread Andy Smith via mailop
Hello, On Sat, Jun 24, 2023 at 10:18:26AM +, Louis Laureys via mailop wrote: > I was with you until it was revealed you mention a blacklist in > your response. Sendgrid assumes that the words in the response > actually have something to do with the reason it's being > temporarily rejected

Re: [mailop] SendGrid is deleting your mail

2023-06-24 Thread Michael Orlitzky via mailop
On Sat, 2023-06-24 at 10:18 +, Louis Laureys via mailop wrote: > Michael, I was with you until it was revealed you mention a blacklist in your > response. Sendgrid assumes that the words in the response actually have > something to do with the reason it's being temporarily rejected, which is a

Re: [mailop] SendGrid is deleting your mail

2023-06-24 Thread Michael Orlitzky via mailop
On Sat, 2023-06-24 at 08:42 +, Andy Smith via mailop wrote: > > In the specific case at hand it seems that the OP is greylisting > SendGrid for being on some sort of blocklist, and the specific > mention of "blacklist" in the 4xx response is hitting a SendGrid > heuristic that says "don't

Re: [mailop] SendGrid is deleting your mail

2023-06-24 Thread Slavko via mailop
Dňa 24. júna 2023 8:42:59 UTC používateľ Andy Smith via mailop napísal: >I wouldn't go as far as to characterise my belief that 4xx should >always be retried to be as firm as fanaticism or religion. Like I >say, I can see why in some specific cases large senders might not >want to. The

Re: [mailop] SendGrid is deleting your mail

2023-06-24 Thread Louis Laureys via mailop
Michael, I was with you until it was revealed you mention a blacklist in your response. Sendgrid assumes that the words in the response actually have something to do with the reason it's being temporarily rejected, which is a fair assumption to make even if it doesn't follow spec. Then using

Re: [mailop] SendGrid is deleting your mail

2023-06-23 Thread Al Iverson via mailop
What if we just got to the heart of the matter and admitted that greylisting is useless 2023? It was meant to annoy botnet spammers etc. who, once upon a time, did not have big iron and big disks to be able to queue and retry. That reality basically no longer exists. I tested greylisting myself

Re: [mailop] SendGrid is deleting your mail

2023-06-23 Thread Jarland Donnell via mailop
I thought I was a wild one with testing in production with SpamAssassin rules set to 0 score. That way I could determine the impact without making the impact at all, or without adding resource overhead to other parties just because I was playing around. I think it's a good point here that

Re: [mailop] SendGrid is deleting your mail

2023-06-23 Thread Luke via mailop
Default behavior is always to retry 4xx if there isn't a rule in place that overrides it. In this case, it was, and still is (i'm working on it) hitting another rule that was put in place to *not* retry a very specific 4xx that would never result in a successful delivery. The full response in

Re: [mailop] SendGrid is deleting your mail

2023-06-23 Thread Bill Cole via mailop
[NOTE: There's absolutely never any need to CC me individually when posting to this or any other list I'm on. Or not on.] On 2023-06-23 at 12:46:58 UTC-0400 (Fri, 23 Jun 2023 09:46:58 -0700) Luke via mailop is rumored to have said: That's right, Bill. It's so simple. All the ESPs and MTAs out

Re: [mailop] SendGrid is deleting your mail

2023-06-23 Thread Andrew C Aitchison via mailop
On Fri, 23 Jun 2023, Carsten Schiefner via mailop wrote: Hi, Luke (& all) - how about elaborating a bit further on the whats and whys of your setup? Because at first sight it is indeed a bit hard to understand why SendGrid may not be in a position to follow the RFCs and the thereof derived

Re: [mailop] SendGrid is deleting your mail

2023-06-23 Thread Andy Smith via mailop
Hi, On Fri, Jun 23, 2023 at 08:03:40PM +0200, Carsten Schiefner via mailop wrote: > how about elaborating a bit further on the whats and whys of your setup? Maybe some of us could learn something from that, or maybe SendGrid would consider that to be giving an advantage to competitors. Really

Re: [mailop] SendGrid is deleting your mail

2023-06-23 Thread Carsten Schiefner via mailop
sage- From: Luke via mailop To: Bill Cole Cc: Luke via mailop Sent: Fr., 23 Juni 2023 18:53 Subject: Re: [mailop] SendGrid is deleting your mail That's right, Bill. It's so simple. All the ESPs and MTAs out there have tools built in to create and customize response handling rules for absolu

Re: [mailop] SendGrid is deleting your mail

2023-06-23 Thread Luke via mailop
That's right, Bill. It's so simple. All the ESPs and MTAs out there have tools built in to create and customize response handling rules for absolutely no reason. We actually just do it for fun. We know that every single 4xx should be retried and every single 5xx should not be retried. But we

Re: [mailop] SendGrid is deleting your mail

2023-06-23 Thread Luke via mailop
That's a very silly interpretation of what was said. On Thu, Jun 22, 2023, 11:01 PM Slavko via mailop wrote: > Dňa 22. júna 2023 23:14:15 UTC používateľ Luke via mailop < > mailop@mailop.org> napísal: > > >Unfortunately we cant make a rule that retries all 4xx and doesn't retry > >all 5xx. > >

Re: [mailop] SendGrid is deleting your mail

2023-06-23 Thread Bill Cole via mailop
On 2023-06-22 at 19:14:15 UTC-0400 (Thu, 22 Jun 2023 16:14:15 -0700) Luke via mailop is rumored to have said: Unfortunately we cant make a rule that retries all 4xx and doesn't retry all 5xx. Despite very smart, well intended people writing RFCs and other specs that make this feel super

Re: [mailop] SendGrid is deleting your mail

2023-06-23 Thread Michael Orlitzky via mailop
On Thu, 2023-06-22 at 16:14 -0700, Luke wrote: > It's specifically targeting the response code and string you provided: 450 > 4.3.2 Please retry immediately Ok, I've spent the morning patching out a few more of our MTA's 4xx responses with that exact text. Some of them contained important

Re: [mailop] SendGrid is deleting your mail

2023-06-23 Thread Jaroslaw Rafa via mailop
Dnia 22.06.2023 o godz. 16:14:15 Luke via mailop pisze: > > Unfortunately we cant make a rule that retries all 4xx and doesn't retry > all 5xx. Why? 4xx means the target says "I want you to retry". 5xx means the target says "I don't want you to retry". Regardless if this is deliberate, or by

Re: [mailop] SendGrid is deleting your mail

2023-06-23 Thread Slavko via mailop
Dňa 22. júna 2023 23:14:15 UTC používateľ Luke via mailop napísal: >Unfortunately we cant make a rule that retries all 4xx and doesn't retry >all 5xx. In other words, you are not able to ensure delivery, if target rejects with 4xx code, and you are not able to stop resend, if target rejects

Re: [mailop] SendGrid is deleting your mail

2023-06-22 Thread Luke via mailop
It's specifically targeting the response code and string you provided: 450 4.3.2 Please retry immediately Unfortunately we cant make a rule that retries all 4xx and doesn't retry all 5xx. Despite very smart, well intended people writing RFCs and other specs that make this feel super clean cut in

Re: [mailop] SendGrid is deleting your mail

2023-06-22 Thread Michael Orlitzky via mailop
On Thu, 2023-06-22 at 10:52 -0700, Luke via mailop wrote: > I got a rule in place that should cover this. I'll monitor and make sure > the desired behavior is occurring. > Thanks, but which "this"? Should all 4xx replies now get a retry, or just the ones containing a certain phrase? (And which

Re: [mailop] SendGrid is deleting your mail

2023-06-22 Thread Luke via mailop
I got a rule in place that should cover this. I'll monitor and make sure the desired behavior is occurring. On Wed, Jun 21, 2023, 7:27 PM Michael Orlitzky wrote: > On 2023-06-21 18:39:59, Luke wrote: > > Ahh. Thats funny. Apologies. You'll have to refresh my memory. > > > > Happy to help if I

Re: [mailop] SendGrid is deleting your mail

2023-06-22 Thread Hans-Martin Mosner via mailop
Am 22.06.23 um 06:52 schrieb Matt Harris via mailop: On Wed, Jun 21, 2023 at 6:11 PM Sebastian Nielsen via mailop wrote: >>The RFC forbids doing that, and I argued against it The RFC and reality is two different things. If a client don't want to retry, I think they are free to

Re: [mailop] SendGrid is deleting your mail

2023-06-21 Thread Matt Harris via mailop
On Wed, Jun 21, 2023 at 6:11 PM Sebastian Nielsen via mailop < mailop@mailop.org> wrote: > >>The RFC forbids doing that, and I argued against it > > The RFC and reality is two different things. If a client don't want to > retry, I think they are free to choose to not retry. > This is a terrible

Re: [mailop] SendGrid is deleting your mail

2023-06-21 Thread Michael Orlitzky via mailop
On 2023-06-22 04:21:59, Sebastian Nielsen via mailop wrote: > >> We update kernels, reload AV signatures, have databases go down, > >> accidentally crash postfix during OS upgrades, typo config files, etc. > > Couldn't you make so if the inner servers are in trouble or go down > or similar,

Re: [mailop] SendGrid is deleting your mail

2023-06-21 Thread Michael Orlitzky via mailop
On 2023-06-21 18:39:59, Luke wrote: > Ahh. Thats funny. Apologies. You'll have to refresh my memory. > > Happy to help if I can. Give me the full response, code and string, and > tell me how you'd like us to handle it and if it makes sense, I can make > the necessary changes. > In this case,

Re: [mailop] SendGrid is deleting your mail

2023-06-21 Thread Sebastian Nielsen via mailop
>> We update kernels, reload AV signatures, have databases go down, >> accidentally crash postfix during OS upgrades, typo config files, etc. Couldn't you make so if the inner servers are in trouble or go down or similar, then the perimeter server will buffer the email for you without a

Re: [mailop] SendGrid is deleting your mail

2023-06-21 Thread Luke via mailop
Ahh. Thats funny. Apologies. You'll have to refresh my memory. Happy to help if I can. Give me the full response, code and string, and tell me how you'd like us to handle it and if it makes sense, I can make the necessary changes. On Wed, Jun 21, 2023, 6:31 PM Michael Orlitzky wrote: > On

Re: [mailop] SendGrid is deleting your mail

2023-06-21 Thread Michael Orlitzky via mailop
On 2023-06-21 18:19:41, Luke via mailop wrote: > > I work at sendgrid and manage response handling. If someone were to reach > out with an obvious problem, I'd be willing and able to adjust our response > handling appropriately. > You're the guy I mentioned the problem to last time.

Re: [mailop] SendGrid is deleting your mail

2023-06-21 Thread Michael Orlitzky via mailop
On 2023-06-22 02:05:40, Sebastian Nielsen via mailop wrote: > >> They were going to get a 4xx anyway. I changed the message to *help* > >> SendGrid. > > Yes but if you can change the message for SendGrid only, you can > accept the mail and let it through... Apparently you were able to > send

Re: [mailop] SendGrid is deleting your mail

2023-06-21 Thread Luke via mailop
An alternative approach would be to admit that response handling at massive sclae is very difficult to get 100% right. Give the sender the benefit of the doubt that they are trying to do the right thing and attempt to reach someone who works there to see if they can help. You could try mailop,

Re: [mailop] SendGrid is deleting your mail

2023-06-21 Thread Sebastian Nielsen via mailop
>> They were going to get a 4xx anyway. I changed the message to *help* >> SendGrid. Yes but if you can change the message for SendGrid only, you can accept the mail and let it through >> Where do I find out what the IP/domain is? Is it in my mail logs, Apparently you were able to send custom

Re: [mailop] SendGrid is deleting your mail

2023-06-21 Thread Michael Orlitzky via mailop
On 2023-06-22 00:32:37, Sebastian Nielsen via mailop wrote: > Why even send retry requests to SendGrid? > Just accept the email, whats the problem? > > If your antivirus or mail scanning solution requires some time, > buffer the email at your server instead. I do understand it creates > the

Re: [mailop] SendGrid is deleting your mail

2023-06-21 Thread Sebastian Nielsen via mailop
>>The RFC forbids doing that, and I argued against it The RFC and reality is two different things. If a client don't want to retry, I think they are free to choose to not retry. Why even send retry requests to SendGrid? Just accept the email, whats the problem? If your antivirus or mail

[mailop] SendGrid is deleting your mail

2023-06-21 Thread Michael Orlitzky via mailop
The last time SendGrid's failure to retry 4xx came up here, their response was that, when determining whether or not to retry, they analyze not only the 4xx code but also the associated text. If (based on those data) a retry is statistically unlikely, they won't do it. The RFC forbids doing that,