Re: [mailop] A proposal for automated management of mail sending limits

[David Hofstee]

Hi Ken,

The part after the snippet, is where I explain why I said that.

Only in recent years has PowerMTA 4.5 added some more granular options,
maybe after me complaining about it. It used to be per (set of) domains
only (and the added routing). I looked at the 4.5 features and I still
presume this falls short. I don't know if you have seen how Message
Systems' "Adaptive Delivery" works? It only works for about 75 ISPs and
only for fixed domains (barely adequate in my opinion). Not sure what Green
Arrow can do.

The problem I see is that it requires extensive tuning. Small shops don't
do that because they lack expertise, data and time. If we want email to
remain accessible, we should remove this obstacle.

Yours,


David

On 14 November 2017 at 16:24, Ken O'Driscoll  wrote:

> On Tue, 2017-11-14 at 10:05 +0100, David Hofstee wrote:
> > I agree that it is a problem. I do think this could be done at connection
> > time only. Of of the tricky parts is that all mail servers I know have
> > trouble with throttling.
> [...snip...]
>
> Traditional MTAs often respond by queuing and re-trying. That works for
> small delays, such as "Server Busy - Try Later" but doesn't really scale
> for the throttling experienced during bulk sending. You can use multiple
> queues but that only results in email often being delayed unnecessarily.
>
> However, Specialist MTA designed for bulk sending, such as GreenArrow or
> PowerMTA can be configured to control mail flow on a very granular level on
> a per provider, IP, MX etc. basis. They also support dynamically changing
> mail flow in response to throttling.
>
> So legitimate bulk emails sent though a specialist MTA aren't typically
> unduly hindered by pre-acceptance throttling etc.
>
> But I do think that this issue isn't as much an engineering problem as a
> communications one.
>
> Ken.
>
>
> --
> Ken O'Driscoll / We Monitor Email
> t: +353 1 254 9400 | w: www.wemonitoremail.com
>
> Need to understand deliverability? Now there's a book:
> www.wemonitoremail.com/book
>
>
> ___
> mailop mailing list
> mailop@mailop.org
> https://chilli.nosignal.org/cgi-bin/mailman/listinfo/mailop
>



-- 
--
My opinion is mine.
___
mailop mailing list
mailop@mailop.org
https://chilli.nosignal.org/cgi-bin/mailman/listinfo/mailop

Re: [mailop] A proposal for automated management of mail sending limits

[Federico Santandrea]

Thanks to everyone for taking the time to read and give feedback.


On 13/11/2017 18:58, Steve Atkins wrote:
If the ISP were to publish the constraints that were applicable to 
SMTP from strangers they'd often be low enough that many senders 
wouldn't be able to comply with them while still delivering the mail 
they need to deliver, so I can't picture a choice of values to 
publish that would be of value to (or could be followed by) most 
senders.


That's what I meant, limits for "strangers"; because of course every
peer who establishes any kind of reputation would be subject to
limitations unique to that peer and that mail flow, and know better. It
was meant to show a starting point for newly or rarely seen providers.


On 14/11/2017 16:24, Ken O'Driscoll wrote:

However, Specialist MTA designed for bulk sending, such as GreenArrow
or PowerMTA can be configured to control mail flow on a very granular
level on a per provider, IP, MX etc. basis. They also support
dynamically changing mail flow in response to throttling.


I am aware of how specialist MTA can assess sending limits on their
own; what I was seeking was a less intrusive alternative to the de facto
standard practice of slap-on-the-wrist-until-you-know, i.e. using
throttling as signalling.

After reading everyone's replies, I realize this is not as useful as I
thought at first. In fact anyone using appropriate software would only
have any use for this for the first handful of messages to somewhere
new, which when you send volumes is a negligible part of the whole. And
then you'd need to bump against the limits anyhow if you need to send
more.

I agree this is probably not worth pursuing.

--
Federico

___
mailop mailing list
mailop@mailop.org
https://chilli.nosignal.org/cgi-bin/mailman/listinfo/mailop

Re: [mailop] A proposal for automated management of mail sending limits

[Ken O'Driscoll]

On Tue, 2017-11-14 at 10:05 +0100, David Hofstee wrote:
> I agree that it is a problem. I do think this could be done at connection
> time only. Of of the tricky parts is that all mail servers I know have
> trouble with throttling.
[...snip...]

Traditional MTAs often respond by queuing and re-trying. That works for
small delays, such as "Server Busy - Try Later" but doesn't really scale
for the throttling experienced during bulk sending. You can use multiple
queues but that only results in email often being delayed unnecessarily.

However, Specialist MTA designed for bulk sending, such as GreenArrow or
PowerMTA can be configured to control mail flow on a very granular level on
a per provider, IP, MX etc. basis. They also support dynamically changing
mail flow in response to throttling.

So legitimate bulk emails sent though a specialist MTA aren't typically
unduly hindered by pre-acceptance throttling etc. 

But I do think that this issue isn't as much an engineering problem as a 
communications one.

Ken. 


-- 
Ken O'Driscoll / We Monitor Email
t: +353 1 254 9400 | w: www.wemonitoremail.com

Need to understand deliverability? Now there's a book:
www.wemonitoremail.com/book


___
mailop mailing list
mailop@mailop.org
https://chilli.nosignal.org/cgi-bin/mailman/listinfo/mailop

Re: [mailop] A proposal for automated management of mail sending limits

[Maarten Oelering]

I agree with Ken that the need for guidelines is well intended. Many legitimate 
senders want to avoid pushing too hard and wasting resources on both ends. But 
I think exchanging specific limits between receiver and sender via a new 
mechanism is infeasible, and may not even be needed.

Why not start with bringing some clarity to SMTP replies and sender guidelines? 
Enhanced status codes or clear messages can tell the sender to (temporarily) 
stop sending from an IP or domain. Professional MTAs can already handle this 
using adaptive delivery settings.

I know that some are seeking the exact message rate that they are allowed to 
send, but reputation and throttling is not exact science. I don't see what's 
wrong with sending a burst of messages, stopping at the first reputation error, 
and resuming after some predefined delay. 

Maarten Oelering
Postmastery

> On 14 Nov 2017, at 10:05, David Hofstee  wrote:
> 
> Hi Ken,
> 
> I agree that it is a problem. I do think this could be done at connection 
> time only. Of of the tricky parts is that all mail servers I know have 
> trouble with throttling. They can throttle on a (set of) recipient domain(s), 
> but not on a cluster of MXs from e.g. Microsoft. E.g. they put hotmail, 
> outlook, msn in one queue but forget to put email from businesses that use 
> Office365 in the same queue. In order to fix that more easily, the mail 
> server should disclose an identifyer where the volume should be counted 
> under. In that way a server could know when a new connection/domain is part 
> of that throttling (and shut that connection down if necessary).
> 
> e.g.
> 250-THROTTLING-IDENTIFIER hotmail-scoijwesoifjwhps [QUIT here if you 
> realize it would violate the throttling quota]
> 250-THROTTLING-CONNECTIONS 10
> 250-THROTTLING-EMAILS-CONNECTION 20
> 250-THROTTLING-RATE-MINUTE 20
> 250-THROTTLING-RATE-HOUR 200
> 250-THROTTLING-RATE-DAY 500
> 250-THROTTLING-DATASIZE-CONNECTION 100MB 
> 
> or
> 250-THROTTLING-IDENTIFIER hotmail-scoijwesoifjwhps
> 250-THROTTLING-GRAYLISTING-MINUTE 120
> 
> It would then be up to the MTA to group all these domains under that 
> identifier and abide to the throttling requirements. One thing is that these 
> throttling parameters must be adjustable in the same connection (so they can 
> react to emails you send).
> 
> Yours,
> 
> 
> David
> 
> 
> 
> On 13 November 2017 at 20:41, Ken O'Driscoll  > wrote:
> On Mon, 2017-11-13 at 09:58 -0800, Steve Atkins wrote:
> > (If this proposal were coming out of a group of major ISPs or a few of
> > the larger inbound mail appliance or service providers as "this is
> > something we want to do" I'd consider it differently than it coming from
> > the high volume email deployer side of things. There's a long history of
> > bulk mail senders going "just tell us exactly what we need to do so
> > you'll deliver our email, and we'll do it!" and it's not something that
> > ever really leads anywhere.)
> 
> I understand this sentiment but I believe these days the only reason that
> most of them want to know "what the rules are" is because they want to
> comply but it's often not evident how. 
> 
> The quality (and existence) of postmaster portals and knowledge bases
> varies greatly. The informative value of SMTP error messages varies
> greatly. The ability to communicate with a postmaster varies greatly. 
> 
> You can't really blame them (particularly the smaller ones) for sometimes
> being confused about what's needed to get their transactional or opt-in
> email into inboxes.
> 
> Ken.
> 
> -- 
> Ken O'Driscoll / We Monitor Email
> t: +353 1 254 9400  | w: www.wemonitoremail.com 
> 
> 
> Need to understand deliverability? Now there's a book:
> www.wemonitoremail.com/book 
> 
> 
> ___
> mailop mailing list
> mailop@mailop.org 
> https://chilli.nosignal.org/cgi-bin/mailman/listinfo/mailop 
> 
> 
> 
> 
> -- 
> --
> My opinion is mine.
> ___
> mailop mailing list
> mailop@mailop.org
> https://chilli.nosignal.org/cgi-bin/mailman/listinfo/mailop

___
mailop mailing list
mailop@mailop.org
https://chilli.nosignal.org/cgi-bin/mailman/listinfo/mailop

Re: [mailop] A proposal for automated management of mail sending limits

[David Hofstee]

Hi Ken,

I agree that it is a problem. I do think this could be done at connection
time only. Of of the tricky parts is that all mail servers I know have
trouble with throttling. They can throttle on a (set of) recipient
domain(s), but not on a cluster of MXs from e.g. Microsoft. E.g. they put
hotmail, outlook, msn in one queue but forget to put email from businesses
that use Office365 in the same queue. In order to fix that more easily, the
mail server should disclose an identifyer where the volume should be
counted under. In that way a server could know when a new connection/domain
is part of that throttling (and shut that connection down if necessary).

e.g.
250-THROTTLING-IDENTIFIER hotmail-scoijwesoifjwhps [QUIT here if you
realize it would violate the throttling quota]
250-THROTTLING-CONNECTIONS 10
250-THROTTLING-EMAILS-CONNECTION 20
250-THROTTLING-RATE-MINUTE 20
250-THROTTLING-RATE-HOUR 200
250-THROTTLING-RATE-DAY 500
250-THROTTLING-DATASIZE-CONNECTION 100MB

or
250-THROTTLING-IDENTIFIER hotmail-scoijwesoifjwhps
250-THROTTLING-GRAYLISTING-MINUTE 120

It would then be up to the MTA to group all these domains under that
identifier and abide to the throttling requirements. One thing is that
these throttling parameters must be adjustable in the same connection (so
they can react to emails you send).

Yours,


David



On 13 November 2017 at 20:41, Ken O'Driscoll  wrote:

> On Mon, 2017-11-13 at 09:58 -0800, Steve Atkins wrote:
> > (If this proposal were coming out of a group of major ISPs or a few of
> > the larger inbound mail appliance or service providers as "this is
> > something we want to do" I'd consider it differently than it coming from
> > the high volume email deployer side of things. There's a long history of
> > bulk mail senders going "just tell us exactly what we need to do so
> > you'll deliver our email, and we'll do it!" and it's not something that
> > ever really leads anywhere.)
>
> I understand this sentiment but I believe these days the only reason that
> most of them want to know "what the rules are" is because they want to
> comply but it's often not evident how.
>
> The quality (and existence) of postmaster portals and knowledge bases
> varies greatly. The informative value of SMTP error messages varies
> greatly. The ability to communicate with a postmaster varies greatly.
>
> You can't really blame them (particularly the smaller ones) for sometimes
> being confused about what's needed to get their transactional or opt-in
> email into inboxes.
>
> Ken.
>
> --
> Ken O'Driscoll / We Monitor Email
> t: +353 1 254 9400 | w: www.wemonitoremail.com
>
> Need to understand deliverability? Now there's a book:
> www.wemonitoremail.com/book
>
>
> ___
> mailop mailing list
> mailop@mailop.org
> https://chilli.nosignal.org/cgi-bin/mailman/listinfo/mailop
>



-- 
--
My opinion is mine.
___
mailop mailing list
mailop@mailop.org
https://chilli.nosignal.org/cgi-bin/mailman/listinfo/mailop

Re: [mailop] A proposal for automated management of mail sending limits

[Ken O'Driscoll]

On Mon, 2017-11-13 at 09:58 -0800, Steve Atkins wrote:
> (If this proposal were coming out of a group of major ISPs or a few of
> the larger inbound mail appliance or service providers as "this is
> something we want to do" I'd consider it differently than it coming from
> the high volume email deployer side of things. There's a long history of
> bulk mail senders going "just tell us exactly what we need to do so
> you'll deliver our email, and we'll do it!" and it's not something that
> ever really leads anywhere.)

I understand this sentiment but I believe these days the only reason that
most of them want to know "what the rules are" is because they want to
comply but it's often not evident how. 

The quality (and existence) of postmaster portals and knowledge bases
varies greatly. The informative value of SMTP error messages varies
greatly. The ability to communicate with a postmaster varies greatly. 

You can't really blame them (particularly the smaller ones) for sometimes
being confused about what's needed to get their transactional or opt-in
email into inboxes.

Ken.

-- 
Ken O'Driscoll / We Monitor Email
t: +353 1 254 9400 | w: www.wemonitoremail.com

Need to understand deliverability? Now there's a book:
www.wemonitoremail.com/book


___
mailop mailing list
mailop@mailop.org
https://chilli.nosignal.org/cgi-bin/mailman/listinfo/mailop

Re: [mailop] A proposal for automated management of mail sending limits

[ml+mailop]

Just a note:
draft-santos-smtpgrey suggests to tell the sender when to try again,
e.g., 451 Greylisted, please try again in # seconds

Maybe something like that would be the right approach for this case
too?  You could announce the limits for the client in the greeting,
e.g.,
250-CONCURRENT-CONNECTIONS 10
250-RATE 5/minute
or maybe in replies at some later stage.

___
mailop mailing list
mailop@mailop.org
https://chilli.nosignal.org/cgi-bin/mailman/listinfo/mailop

Re: [mailop] A proposal for automated management of mail sending limits

[Ken O'Driscoll]

On Mon, 2017-11-13 at 18:05 +0100, Federico Santandrea wrote:
> Would this fill anyone else's needs and benefit the community?
> Any feedback is appreciated.

Most pre-acceptance filtering is dynamic and based on other sender
reputational indicators meaning throttling thresholds can vary dramatically
based on the characteristics of a sender at a particular point in time.

Additionally, MTAs which are designed for high volume sending typically
detect throttling and ease off that particular mailbox provider meaning
warm-up plans don't really need to super-granular based on individual MXes.

Throttling at pre-acceptance stops a lot of bad stuff, including viruses.
The worst it might do to legitimate email is delay it slightly.

I'm not sure that there is a problem here which needs solving.

Ken.

-- 
Ken O'Driscoll / We Monitor Email
t: +353 1 254 9400 | w: www.wemonitoremail.com

Need to understand deliverability? Now there's a book:
www.wemonitoremail.com/book


___
mailop mailing list
mailop@mailop.org
https://chilli.nosignal.org/cgi-bin/mailman/listinfo/mailop

Re: [mailop] A proposal for automated management of mail sending limits

[Steve Atkins]

> On Nov 13, 2017, at 9:05 AM, Federico Santandrea 
>  wrote:
> 
> Hello,
> 
> I am working on a rough draft for a protocol meant to facilitate exchange of 
> deliverability information among ESPs and mailbox providers.
> 
> This arose from the observation that providers who choose to publish a 
> description of what they consider to be an acceptable mail flow, do this each 
> in their own way (FAQs, postmaster pages, helpdesks) and one has to keep 
> track of everything everywhere.
> 
> I thought I could ask here for relevant opinions before getting too deep into 
> it, as I don't want to elaborate on what could be a known or unknown bad 
> idea; this seemed a suitable initial discussion venue to me.
> 
> The draft text can be found at:
> https://www.ietf.org/id/draft-santandrea-mail-limits-00.txt
> 
> Would this fill anyone else's needs and benefit the community?
> Any feedback is appreciated.

The difficult issue here is people+policies+politics, not a communication 
protocol.

ISPs usually don't have particularly consistently defined delivery constraints. 
Constraints are likely to vary (wildly! by many orders of magnitude!) depending 
on the history of the smtp peer, both very recent and over multiple weeks.

If the ISP were to publish the constraints that were applicable to SMTP from 
strangers they'd often be low enough that many senders wouldn't be able to 
comply with them while still delivering the mail they need to deliver, so I 
can't picture a choice of values to publish that would be of value to (or could 
be followed by) most senders.

Those dynamic policies vary from provider to provider, and aren't implemented 
consistently enough that you can necessarily sum them up with a few common 
variables.

And I would bet that at many ISPs you couldn't find someone who could tell you 
what their constraints are, as they're tweaked in an ad-hoc way using knobs 
that don't directly map on to the variables you're looking for. And even if you 
could on Monday, that doesn't mean it'd be the same by Wednesday.

Those providers that do publish constraints often aren't publishing the real 
constraints - they're publishing a baseline of "if you're doing more than this, 
and you're getting rejections or deferrals, don't bother contacting us". That's 
fine for customer support pages, but not something you'd want to feed into 
sending automation. Publishing those in machine-readable form risks people 
putting too much trust in them.

(If this proposal were coming out of a group of major ISPs or a few of the 
larger inbound mail appliance or service providers as "this is something we 
want to do" I'd consider it differently than it coming from the high volume 
email deployer side of things. There's a long history of bulk mail senders 
going "just tell us exactly what we need to do so you'll deliver our email, and 
we'll do it!" and it's not something that ever really leads anywhere.)

Cheers,
  Steve


___
mailop mailing list
mailop@mailop.org
https://chilli.nosignal.org/cgi-bin/mailman/listinfo/mailop