Re: SMTP session caching

[Marco Pizzoli]

Hello Wietse,
I was reading this topic with interest because this issue affects me very
much as well.

I had a slowness issue when sending to @hotmail.* .  I tried all the
Microsoft services in order to check how I was seen from them and I got
confirmation that I was OK. I received this information not only through MS
SNDS[1] but also agreeing a sending window with one of their engineers who
observed my actual sending and investigated evidence of troubles.

During a massive sending I was able to complete all destinations in 1hour
BUT Microsoft's. (and Apple's)
By looking at my logs I was capable of noticing they were slowly accepting
my TCP requests, or, said better, slowing down everything related to the
connection establishment.

To complete my sending to @hotmail.* I was spending more than 12 hours...

I changed my client configuration, for that dedicated transport, disabling
STARTTLS support and so making effectively use of the connection cache.
y using the connection cache I managed to get that sending complete in
approx 1hour.

I then applied the same configuration for Apple domains and a couple of
local national ISPs.

I could afford to proceed like that because:
1. these emails are just marketing
2. those mailbox providers are not providing evidence to the end user that
the email was sent in clear text instead of TLS-secured. Were they to do
the same as Gmail does, the red lock to email received in plain text, I
will need to revert my configuration and eventually get my sending slow
again...

Thank you in advance for everything you can develop in order to overcome
this limitation.
Marco

P.s. As a side note: the second limitation I face in using Postfix for bulk
email sending is the missing "MX-rollup" feature I already reported twice
in this list:
http://postfix.1071664.n5.nabble.com/Feature-request-MX-rollup-td90800.html


[1] https://postmaster.live.com/snds/



Virus-free.
www.avg.com

<#m_6573517841975336091_DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>

On Mon, Mar 19, 2018 at 4:28 PM, Wietse Venema  wrote:

> Emmanuel Fust?:
> > Is there any document that describe how interprocess notification is
> > done in postfix ? More precisely the scheduler -> delivery agent
> > notification ?
>
> There is no public documentation for *internal* Postfix interfaces,
> so that I can change that without breaking non-Postfix software
> that depends on Postfix internals.
>
> I agree with Viktor that updating the TLS proxy is the more feasible
> approach (caching the TLS sessions outside delivery agents). I also
> don't believe in changing the scheduler-to-delivery agent protocol.
> The Postfix approach is not to make an existing thing more complex,
> but by combining less complex things together.
>
> Please provide quantitative evidence that connection reuse is
> necessary to get mail into the 'big providers' (i.e. they punish
> connection rate and message rate differently).
>
> Wietse
>

Re: SMTP session caching

[Emmanuel Fusté]

Le 19/03/2018 à 17:32, Msd a écrit :

Le 19/03/2018 à 16:28, Wietse Venema a écrit :

Please provide quantitative evidence that connection reuse is
necessary to get mail into the 'big providers' (i.e. they punish
connection rate and message rate differently).

Hi Wietse,

The biggest email service provider in France : Orange/Wanadoo.

We needed to disable TLS to reuse connections :
postconf -P "orange/unix/smtp_tls_security_level=none"

Here are their rules, got from ab...@orange.fr and confirmed by our logs :
- Maximum 1000 connections per hour
- Maximum 100 emails by connection
- Maximum 3 simultaneous connections

Yes and as they count connection at the IP level and have two domains 
behind the same MX, you should reduce the 
transport_destination_concurrency_limit to 1 to not exceed the 3 
simultaneous connections limit. (you could theoretically reach 6 with 
local transport trick because the single MX for these domains has two IP 
address...)


---
Emmanuel.

Re: SMTP session caching

[Emmanuel Fusté]

Le 19/03/2018 à 16:42, Emmanuel Fusté a écrit :

Le 19/03/2018 à 16:28, Wietse Venema a écrit :

Emmanuel Fust?:

Is there any document that describe how interprocess notification is
done in postfix ? More precisely the scheduler -> delivery agent
notification ?

There is no public documentation for *internal* Postfix interfaces,
so that I can change that without breaking non-Postfix software
that depends on Postfix internals.

I agree with Viktor that updating the TLS proxy is the more feasible
approach (caching the TLS sessions outside delivery agents). I also
don't believe in changing the scheduler-to-delivery agent protocol.
The Postfix approach is not to make an existing thing more complex,
but by combining less complex things together.

Please provide quantitative evidence that connection reuse is
necessary to get mail into the 'big providers' (i.e. they punish
connection rate and message rate differently).

Ok I will try to collect data with/without con reuse and get back here.
While reviewing logs, I see that for thousand email being delivered to 
the two offending domains, each time the same smtp process will pick the 
next mail for the same domain.
So in this particular case, for the most part of the traffic, simply 
delaying the closing of the connection (of ~2s max)  while taking the 
next email would be sufficient to opportunistically reuse the connection 
most the time without any inter-process  notification modification or 
delivery agent TLS refactoring.


I will collect data for TLS vs clear with conn reuse for next week.

---
Emmanuel.

Re: SMTP session caching

[Msd]

Le 19/03/2018 à 16:28, Wietse Venema a écrit :

Please provide quantitative evidence that connection reuse is
necessary to get mail into the 'big providers' (i.e. they punish
connection rate and message rate differently).


Hi Wietse,

The biggest email service provider in France : Orange/Wanadoo.

We needed to disable TLS to reuse connections :
postconf -P "orange/unix/smtp_tls_security_level=none"

Here are their rules, got from ab...@orange.fr and confirmed by our logs :
- Maximum 1000 connections per hour
- Maximum 100 emails by connection
- Maximum 3 simultaneous connections

A website about that :
http://www.delivernow.eu/en/european-isp/orange-protects-its-messaging-infrastructure-with-cloudmark-antispam-solution-3

Sample of SMTP reply in case of problem :


smtp-in.orange.fr[193.252.22.65] refused to talk to me: 421 mwinf5c01 ME
Service refuse. Veuillez essayer plus tard. Service refused, please try
later. OFR005_105 [105]


host smtp.wanadoo.fr[80.12.242.15] refused to talk to me: 421 
mwinf5c26 ME Trop de connexions, veuillez verifier votre 
configuration. Too many connections, slow down. OFR004_104 [104]



Guillaume

Re: SMTP session caching

[Emmanuel Fusté]

Le 19/03/2018 à 16:28, Wietse Venema a écrit :

Emmanuel Fust?:

Is there any document that describe how interprocess notification is
done in postfix ? More precisely the scheduler -> delivery agent
notification ?

There is no public documentation for *internal* Postfix interfaces,
so that I can change that without breaking non-Postfix software
that depends on Postfix internals.

I agree with Viktor that updating the TLS proxy is the more feasible
approach (caching the TLS sessions outside delivery agents). I also
don't believe in changing the scheduler-to-delivery agent protocol.
The Postfix approach is not to make an existing thing more complex,
but by combining less complex things together.

Please provide quantitative evidence that connection reuse is
necessary to get mail into the 'big providers' (i.e. they punish
connection rate and message rate differently).

Ok I will try to collect data with/without con reuse and get back here.

---
Emmanuel.

Re: SMTP session caching

[Wietse Venema]

Emmanuel Fust?:
> Is there any document that describe how interprocess notification is 
> done in postfix ? More precisely the scheduler -> delivery agent 
> notification ?

There is no public documentation for *internal* Postfix interfaces,
so that I can change that without breaking non-Postfix software
that depends on Postfix internals.

I agree with Viktor that updating the TLS proxy is the more feasible
approach (caching the TLS sessions outside delivery agents). I also
don't believe in changing the scheduler-to-delivery agent protocol.
The Postfix approach is not to make an existing thing more complex,
but by combining less complex things together.

Please provide quantitative evidence that connection reuse is
necessary to get mail into the 'big providers' (i.e. they punish
connection rate and message rate differently).

Wietse

Re: Recording of DANE talk at ICANN61

[Dominic Raferd]

On 19 March 2018 at 09:12, Mike Guelfi  wrote:

> The m3u contains a link to:
> http://audio.icann.org/meetings/sju61/sju61-OPEN-2018-03-14-T1732-208bc-
> zYhNI147Nrs4gtkXUVItrT4uukdYi3nR-en-02.mp3
>
> Which does work...
>
> On 19 Mar. 2018 19:50, Dominic Raferd  wrote:
>
> On 17 March 2018 at 19:42, Viktor Dukhovni 
> wrote:
>
>
> My segment starts at 16 minutes into the recording and ends 51 minutes
> into the recording:
>
>   http://audio.icann.org/meetings/sju61/sju61-OPEN-2018-03-14-
> T1732-208bc-zYhNI147Nrs4gtkXUVItrT4uukdYi3nR-en.m3u
>
>
​ Thanks, yes the mp3 link works. On my Windows 10 machines the original
m3u file is downloaded by Chrome and then opens in Groove Music which
throws an error.

Re: SMTP session caching

[Emmanuel Fusté]

Le 18/03/2018 à 19:23, Viktor Dukhovni a écrit :

On Mar 18, 2018, at 1:55 PM, Matus UHLAR - fantomas  wrote:


* Caching open TLS connections in the smtp(8) delivery agent for
reuse by scheduling repeated deliveries to the same delivery
agent runs into complex scheduling difficulties.  The presence
of open connections don't, and should not IMHO alter scheduler
priority.  Messages should still leave in approximately FIFO
order (subject to (n)qmgr's periodic preemption of messages that
split into many envelopes).

I believe routing multiple messages for the same destination through an
array of currently open connections (held by different processes) should
not be slower than opening new connection for each message.


This would be a rather non-trivial effort.  And downstream MTAs
at major providers seem to frown on connection re-use these days.

is there an evidence about this?


While traffic to smaller destinations is typically less
performance-critical.  So while having TLS connection caching
would be nice, it is not clear that it is a high priority given
the high cost and possibly only moderate benefit.

that is imho still the question. But I'm more curious about questions above

Sorry, no cycles to dissect this point by point.  My view is as stated in
the previous message:

   * I believe that support for this requires proxy processes to handle TLS
 on behalf of (multiple) SMTP delivery agents.  The connection cache
 would then hold a connection to the proxy, rather than a connection to
 the remote destination.

   * The feature is attractive, but not urgently compelling.  It is complex
 to implement, so I cannot predict when work in that direction might begin.

Ok, this is for the connection reuse in the traditional postfix 
sense.Some "opportunistic" reuse.


But would it be difficult to implement something for the case when the 
connection reuse could be planned by the scheduler before submission to 
the agents ?


The most problematic case (and the case for which we would gain a lot) 
is when we cross by a factor the transport_destination_concurrency_limit 
of mail waiting to be delivered to a destination in the mailq.


In this case, instead of pushing/notifying smtp one mail at time, could 
the notifying scheme be extended to be able to push a list of mail to be 
processed by one smtp agent ?
With purely static parameters (crossing factor and fixed queue length) 
it could be a huge step forward for the most common case, even with low 
queue length and could be later enhanced if wanted (dynamic or adaptive 
queue length for example).


In my case, I already have transport_destination_concurrency_limit=1 for 
the problematic destinations and many parallels servers with their own 
public IP. At the scale of my country, the problematic destination is 
like google or yahoo for the world. Without connection reuse for 
lowering my tcp connexion rate, my servers are always throttled 
(temporary blacklisted).
Now my only option would be to disable TLS for these domains, but my 
client will put me on fire or more in that case |-)).


Is there any document that describe how interprocess notification is 
done in postfix ? More precisely the scheduler -> delivery agent 
notification ?


---
Emmanuel.

Re: Recording of DANE talk at ICANN61

[Mike Guelfi]

The m3u contains a link to: http://audio.icann.org/meetings/sju61/sju61-OPEN-2018-03-14-T1732-208bc-zYhNI147Nrs4gtkXUVItrT4uukdYi3nR-en-02.mp3Which does work...On 19 Mar. 2018 19:50, Dominic Raferd  wrote:On 17 March 2018 at 19:42, Viktor Dukhovni  wrote:
[ Also posted to dane-us...@sys4.de, please pardon the duplication if
  you're reading both lists.  I'm planning to also post to exim-users
  and d...@ietf.org ]

I gave a talk about DANE for SMTP at the ICANN61 conference last week.
Audio and slides are available, but not a synchronized recording so if
you want to follow along you'll need to figure out the slide transitions
from the context of the audio.  I was promised 45 minutes, had too much
material even for that, but only got 35 minutes, and yet managed to get
to most of the key points.  I hope and think that the pace was not too
fast to get the points across.

My segment starts at 16 minutes into the recording and ends 51 minutes
into the recording:

  http://audio.icann.org/meetings/sju61/sju61-OPEN-2018-03-14-T1732-208bc-zYhNI147Nrs4gtkXUVItrT4uukdYi3nR-en.m3u

The slides are at:

  https://static.ptbl.co/static/attachments/169319/1520904692.pdf

The slides have additional material in the Appendix section.  Please
take the key rotation advice in the slides seriously and apply it to
improve your own practices.  May your TLSA records never fail to
validate, even briefly...

--
        Viktor.​There may be a problem with the audio link - I can't get it to play.​

Re: Recording of DANE talk at ICANN61

[Dominic Raferd]

On 17 March 2018 at 19:42, Viktor Dukhovni 
wrote:

>
> [ Also posted to dane-us...@sys4.de, please pardon the duplication if
>   you're reading both lists.  I'm planning to also post to exim-users
>   and d...@ietf.org ]
>
> I gave a talk about DANE for SMTP at the ICANN61 conference last week.
> Audio and slides are available, but not a synchronized recording so if
> you want to follow along you'll need to figure out the slide transitions
> from the context of the audio.  I was promised 45 minutes, had too much
> material even for that, but only got 35 minutes, and yet managed to get
> to most of the key points.  I hope and think that the pace was not too
> fast to get the points across.
>
> My segment starts at 16 minutes into the recording and ends 51 minutes
> into the recording:
>
>   http://audio.icann.org/meetings/sju61/sju61-OPEN-2018-03-14-T1732-208bc-
> zYhNI147Nrs4gtkXUVItrT4uukdYi3nR-en.m3u
>
> The slides are at:
>
>   https://static.ptbl.co/static/attachments/169319/1520904692.pdf
>
> The slides have additional material in the Appendix section.  Please
> take the key rotation advice in the slides seriously and apply it to
> improve your own practices.  May your TLSA records never fail to
> validate, even briefly...
>
> --
> Viktor.


​There may be a problem with the audio link - I can't get it to play.​