Re: Any update on lazy load SNI?

[Pierre Allétru]

Thank you for the information Joel, very helpful! We've started doing the
exact same thing actually, with good ol' ssl_certificate_by_lua, until we
realized this wouldn't work with STARTTLS/STLS.

We'd like that to work though and we can't seem to find a solution if
Dovecot can't smoothly handle SNI at scale.

--
Pierre Allétru
06 70 55 08 35
pierre.alle...@gmail.com

Le jeu. 3 nov. 2022, 14:32, Joel A. Chornik  a
écrit :

> What we do is have openresty(nginx) sit as a reverse proxy on top of
> dovecot, and use lua to dynamically load certificates using sni.
>
> We have a large userbase (100k+) and works without issues, except that it
> does not work with STARTTLS, only IMAP+TLS. Has not been an issue, as we
> setup users using autodiscover/autoconfig or as a fallback it is the
> default config in most user agents.
>
> Hope it helps
> Joel Chornik
>
> >
> > On 3 Nov 2022, at 10:24, Pierre Allétru 
> wrote:
> >
> > Pierre Allétru
>

Re: Any update on lazy load SNI?

[Joel A. Chornik]

What we do is have openresty(nginx) sit as a reverse proxy on top of dovecot, 
and use lua to dynamically load certificates using sni.

We have a large userbase (100k+) and works without issues, except that it does 
not work with STARTTLS, only IMAP+TLS. Has not been an issue, as we setup users 
using autodiscover/autoconfig or as a fallback it is the default config in most 
user agents.

Hope it helps
Joel Chornik

> 
> On 3 Nov 2022, at 10:24, Pierre Allétru  wrote:
> 
> Pierre Allétru

Any update on lazy load SNI?

[Pierre Allétru]

Hi folks,

We need to use SNI with Dovecot at a relatively large scale and I was
wondering if there's any update on the ability to:

1 - Lazy load SNI certificates when they are needed instead of loading them
all at once during startup/reload, thus taking a lot of memory and being
very slow.
2 - Not having to reload Dovecot every time a new domain cert is added in
conf.
3 - Or at least have Dovecot keep processing clients while the slow reload
happens.

This has been asked already in a 2016 thread:
https://www.dovecot.org/pipermail/dovecot/2016-November/106075.html.

Regarding point 3 there's a reply from Aki saying it was in the internal
tasklist (
https://www.dovecot.org/pipermail/dovecot/2016-November/106089.html).

Anybody knows if some progress has been made on these subjects? I can't
anything in the docs or any recent information anywhere else.

Thanks,

--
Pierre Allétru
06 70 55 08 35
pierre.alle...@gmail.com

Re: Multiple names in local_name for UCC certificates (was lazy-load SNI?)

[Aki Tuomi]

This has been merged, and hopefully will make it still to 2.2.27 release.

Aki Tuomi
Dovecot oy

On 01.12.2016 09:45, Aki Tuomi wrote:
> Thank you, we'll start looking at this.
>
> Aki
>
> On 01.12.2016 09:44, J. Nick Koston wrote:
>> Hi Aki & Felipe,
>>
>> Attached is an implementation of supporting multiple domains in local_name.
>>
>> Example
>>
>> local_name "mail.domain.tld domain.tld mx.domain.tld" { ... }
>>
>> This can significantly reduce memory usage when using
>> a UCC certificate with multiple names by only loading
>> the certificate and key once.
>>
>>
>>
>> And the pull request…..
>>
>> https://github.com/dovecot/core/pull/24 
>> 
>>
>> Thanks
>> -Nick
>>
>>
>>> On Nov 17, 2016, at 5:27 AM, J. Nick Koston  wrote:
>>>
>>> Aki,
>>>
>>> Multiple local_names would be ideal to accommodate certificates that have 
>>> multiple names.  The way I’m reading the code it looks like its having to 
>>> pay for the memory for every name on the certificate because a unique CTX 
>>> is being created for each name even if they are all on a single certificate.
>>>
>>> This would be a big memory win for anyone using a certificate with multiple 
>>> names on it.
>>>
>>> Thanks
>>> -Nick
>>> cPanel Inc
>>>
>>>
 Dear Aki et al.,

How straightforward would it be to implement the following or similar 
 syntax:

 local_name foo.tld www.foo.tld mail.foo.tld bar.tld {
   ...
 }

 ??

Thank you!

 -FG

 --

 Subject: Digest Footer

 ___
 dovecot mailing list
 dovecot@dovecot.org
 http://dovecot.org/cgi-bin/mailman/listinfo/dovecot

 --

 End of dovecot Digest, Vol 163, Issue 34
 

Re: Multiple names in local_name for UCC certificates (was lazy-load SNI?)

[Aki Tuomi]

Thank you, we'll start looking at this.

Aki

On 01.12.2016 09:44, J. Nick Koston wrote:
> Hi Aki & Felipe,
>
> Attached is an implementation of supporting multiple domains in local_name.
>
> Example
>
> local_name "mail.domain.tld domain.tld mx.domain.tld" { ... }
>
> This can significantly reduce memory usage when using
> a UCC certificate with multiple names by only loading
> the certificate and key once.
>
>
>
> And the pull request…..
>
> https://github.com/dovecot/core/pull/24 
> 
>
> Thanks
> -Nick
>
>
>> On Nov 17, 2016, at 5:27 AM, J. Nick Koston  wrote:
>>
>> Aki,
>>
>> Multiple local_names would be ideal to accommodate certificates that have 
>> multiple names.  The way I’m reading the code it looks like its having to 
>> pay for the memory for every name on the certificate because a unique CTX is 
>> being created for each name even if they are all on a single certificate.
>>
>> This would be a big memory win for anyone using a certificate with multiple 
>> names on it.
>>
>> Thanks
>> -Nick
>> cPanel Inc
>>
>>
>>> Dear Aki et al.,
>>>
>>> How straightforward would it be to implement the following or similar 
>>> syntax:
>>>
>>> local_name foo.tld www.foo.tld mail.foo.tld bar.tld {
>>>   ...
>>> }
>>>
>>> ??
>>>
>>> Thank you!
>>>
>>> -FG
>>>
>>> --
>>>
>>> Subject: Digest Footer
>>>
>>> ___
>>> dovecot mailing list
>>> dovecot@dovecot.org
>>> http://dovecot.org/cgi-bin/mailman/listinfo/dovecot
>>>
>>> --
>>>
>>> End of dovecot Digest, Vol 163, Issue 34
>>> 
>

Multiple names in local_name for UCC certificates (was lazy-load SNI?)

[J. Nick Koston]

0001-config-Match-multiple-names-in-local_name.patch
Description: Binary data


smime.p7s
Description: S/MIME cryptographic signature

Re: lazy-load SNI?

[Felipe Gasper]

> On Nov 11, 2016, at 9:06 AM, Aki Tuomi  wrote:
> 
> If you are interested in testing, please find patch attached that allows you 
> to specify
> 
> local_name *.foo.bar {
> }
> 
> or
> 
> local_name *.*.foo.bar {
> }
> 

Dear Aki et al.,

How straightforward would it be to implement the following or similar 
syntax:

local_name foo.tld www.foo.tld mail.foo.tld bar.tld {
...
}

??

Thank you!

-FG

Re: lazy-load SNI?

[Arkadiusz Miśkiewicz]

On Friday 11 of November 2016, KSB wrote:
> >>> Great! Seems to be working fine for my usage and makes my configs 50%
> >>> smaller (which is gigantic improvement). Will do more testing though.
> >>> 
> >>> Thanks!
> 
> A little bit offtopic, but what is the point of using imap/pop SNI?

> All
> clients want to connect to their own domain or what?

Yes.

-- 
Arkadiusz Miśkiewicz, arekm / ( maven.pl | pld-linux.org )

Re: lazy-load SNI?

[Felipe Gasper]

> On Nov 11, 2016, at 1:29 PM, KSB  wrote:
> 
 
 Great! Seems to be working fine for my usage and makes my configs 50%
 smaller (which is gigantic improvement). Will do more testing though.
 
 Thanks!
 
> 
> A little bit offtopic, but what is the point of using imap/pop SNI? All 
> clients want to connect to their own domain or what?
> 

Yes. It allows more comprehensive branding via the domain name.

-FG

Re: lazy-load SNI?

[KSB]

Great! Seems to be working fine for my usage and makes my configs 50%
smaller (which is gigantic improvement). Will do more testing though.

Thanks!




A little bit offtopic, but what is the point of using imap/pop SNI? All 
clients want to connect to their own domain or what?


--
Kaspars

Re: lazy-load SNI?

[Arkadiusz Miśkiewicz]

On Friday 11 of November 2016, Aki Tuomi wrote:
> On 11.11.2016 19:17, Arkadiusz Miśkiewicz wrote:
> > On Friday 11 of November 2016, Aki Tuomi wrote:
> >> If you are interested in testing, please find patch attached that allows
> >> you to specify
> >> 
> >> local_name *.foo.bar {
> >> }
> >> 
> >> or
> >> 
> >> local_name *.*.foo.bar {
> >> }
> >> 
> >> so basically you can now use certificate name matching rules for
> >> local_name. It made most sense.
> > 
> > Great! Seems to be working fine for my usage and makes my configs 50%
> > smaller (which is gigantic improvement). Will do more testing though.
> > 
> > Thanks!
> > 
> > 
> > 
> > What about dovecot stopping processing new clients when reload is in
> > progress problem -  is it possible to make it behave better? To minimize
> > (or avoid) "downtime".
> > 
> > How to reproduce - just create config file with 20 000 - 50 000 entries
> > 
> > local_name hostXexample.com {
> > 
> >ssl_cert =  >ssl_key =  > 
> > }
> > 
> > where cert.pem contains some full chain (CA cert + intermediate + cert +
> > key).
> > 
> > Start dovecot and then doveadm reload should take long time. Enough for
> > noticing that dovecot stops processing clients.
> > 
> >> Aki Tuomi
> >> Dovecot oy
> 
> That is something that will happen later. Can't give any date, but it's
> in our internal tasklist.

Ok, thanks.

Just making sure that this (stopping processing clients) and lazy-loading of 
thousands of SSL certs itself are treated by dovecot team as two separate 
issues (and tons of SSL certs simply helps to notice first issue).

And was hoping that stopping processing clients issue is easy/easier to solve 
(but looks like that's not the case).

> Aki


-- 
Arkadiusz Miśkiewicz, arekm / ( maven.pl | pld-linux.org )

Re: lazy-load SNI?

[Aki Tuomi]

On 11.11.2016 19:17, Arkadiusz Miśkiewicz wrote:

On Friday 11 of November 2016, Aki Tuomi wrote:


If you are interested in testing, please find patch attached that allows
you to specify

local_name *.foo.bar {
}

or

local_name *.*.foo.bar {
}

so basically you can now use certificate name matching rules for
local_name. It made most sense.

Great! Seems to be working fine for my usage and makes my configs 50% smaller
(which is gigantic improvement). Will do more testing though.

Thanks!



What about dovecot stopping processing new clients when reload is in progress
problem -  is it possible to make it behave better? To minimize (or avoid)
"downtime".

How to reproduce - just create config file with 20 000 - 50 000 entries

local_name hostXexample.com {
   ssl_cert = 
Aki Tuomi
Dovecot oy


That is something that will happen later. Can't give any date, but it's 
in our internal tasklist.


Aki

Re: lazy-load SNI?

[Arkadiusz Miśkiewicz]

On Friday 11 of November 2016, Aki Tuomi wrote:

> If you are interested in testing, please find patch attached that allows
> you to specify
> 
> local_name *.foo.bar {
> }
> 
> or
> 
> local_name *.*.foo.bar {
> }
> 
> so basically you can now use certificate name matching rules for
> local_name. It made most sense.

Great! Seems to be working fine for my usage and makes my configs 50% smaller 
(which is gigantic improvement). Will do more testing though.

Thanks!



What about dovecot stopping processing new clients when reload is in progress 
problem -  is it possible to make it behave better? To minimize (or avoid) 
"downtime".

How to reproduce - just create config file with 20 000 - 50 000 entries

local_name hostXexample.com {
  ssl_cert =  Aki Tuomi
> Dovecot oy

-- 
Arkadiusz Miśkiewicz, arekm / ( maven.pl | pld-linux.org )

Re: lazy-load SNI?

[Aki Tuomi]

On 11.11.2016 12:22, Arkadiusz Miśkiewicz wrote:

On Friday 11 of November 2016, Felipe Gasper wrote:

Hello,

We’re rolling out large SNI deployments for our mail servers. Each 
domain
gets an entry like this in the config:

local_name mail.foo.com {
 ssl_cert = 
Lack of glob/regexp support here is also a problem (for me). I could have 50%
smaller config if local_name supported regexp matching, so it would be
possible to do:

local_name ^(pop3|imap)\.foo\.com {
...
}

or even with glob like *.foo.com matching.


There are a couple problems we’re finding with this approach:

1) Dovecot wants to load everything at once, which has some machines taking
up many GiB of memory just for Dovecot. Is there any way to defer loading
of an SSL cert until a client actually requests it?

No - thread here http://www.dovecot.org/list/dovecot/2016-October/105855.html

Memory is one thing.

The other is that dovecot stops accepting clients when huge config reload
happens (I guess it's a design problem since it makes no sense to do that in
any case. Clients should be processed without gap using old config until new
config is loaded and ready to go).

And third problem is that there is hardcoded 10s limit for reloading which in
case thousands of certificates is way too short limit. Anyway if you hit that
limit it's already lost case due to earlier problem.


2) Any time we add or remove a domain, Dovecot’s SNI config matrix needs to
be rebuilt. Is there a way to handle SNI requests dynamically via some
sort of configuration plugin, so we wouldn’t need to rebuild the config on
domain add/remove? I looked through the docs but couldn’t see a way to do
this.

That's unavoidable for now :-(

Here we started analyzing maillog and put into dovecot config only these ssl
certs for domains that are actually used with TLS. It's very ugly and short-
sighted approach but hopefuly proper solution will be implemented by dovecot
team before all people start to use TLS.
  

Thank you in advance!

-Felipe Gasper
Mississauga, ON




If you are interested in testing, please find patch attached that allows 
you to specify


local_name *.foo.bar {
}

or

local_name *.*.foo.bar {
}

so basically you can now use certificate name matching rules for 
local_name. It made most sense.


This should apply cleanly to 2.2.26.0.

---
Aki Tuomi
Dovecot oy
>From 1016a956f6fb8a14e5d24372dc8ed3615b2ecc10 Mon Sep 17 00:00:00 2001
From: Aki Tuomi 
Date: Fri, 11 Nov 2016 13:13:29 +0200
Subject: [PATCH 1/3] lib-dns: Add DNS specific matching algorithms

RFC4343 and RFCRFC4592 compare and match algorithms
---
 src/lib-dns/Makefile.am |  6 ++--
 src/lib-dns/dns-util.c  | 85 +
 src/lib-dns/dns-util.h  | 36 +
 3 files changed, 125 insertions(+), 2 deletions(-)
 create mode 100644 src/lib-dns/dns-util.c
 create mode 100644 src/lib-dns/dns-util.h

diff --git a/src/lib-dns/Makefile.am b/src/lib-dns/Makefile.am
index 751860a..4a0e8bc 100644
--- a/src/lib-dns/Makefile.am
+++ b/src/lib-dns/Makefile.am
@@ -4,10 +4,12 @@ AM_CPPFLAGS = \
 	-I$(top_srcdir)/src/lib
 
 libdns_la_SOURCES = \
-	dns-lookup.c
+	dns-lookup.c \
+	dns-util.c
 
 headers = \
-	dns-lookup.h
+	dns-lookup.h \
+	dns-util.h
 
 pkginc_libdir=$(pkgincludedir)
 pkginc_lib_HEADERS = $(headers)
diff --git a/src/lib-dns/dns-util.c b/src/lib-dns/dns-util.c
new file mode 100644
index 000..3ed7244
--- /dev/null
+++ b/src/lib-dns/dns-util.c
@@ -0,0 +1,85 @@
+/* Copyright (c) 2010-2016 Dovecot authors, see the included COPYING file */
+#include "lib.h"
+#include "dns-util.h"
+
+/**
+  return first position from b->a of c or a if not found
+ */
+static inline
+const char *strchr_ba(const char *a, const char *b, char c)
+{
+	for(;b>a && *b != c; b--);
+	return b;
+}
+
+int dns_ncompare(const char *a, const char *b, size_t n)
+{
+	if (a == NULL && b == NULL) return 0;
+	if (a == NULL && b != NULL) return 1;
+	if (a != NULL && b == NULL) return -1;
+
+	for(size_t i = 0; i < n &&
+			  *a != '\0' &&
+			  *b != '\0' &&
+			  dns_tolower(*a) == dns_tolower(*b);
+	i++, a++, b++);
+
+	return dns_tolower(*a) - dns_tolower(*b);
+}
+
+int dns_compare(const char *a, const char *b)
+{
+	return dns_ncompare(a, b, (size_t)-1);
+}
+
+int dns_compare_labels(const char *a, const char *b)
+{
+	if (a == NULL && b == NULL) return 0;
+	if (a == NULL && b != NULL) return 1;
+	if (a != NULL && b == NULL) return -1;
+
+	const char *ptr_a = a + strlen(a);
+	const char *ptr_b = b + strlen(b);
+	const char *label_a = ptr_a, *label_b = ptr_b;
+	int comp = 0;
+
+	while(comp == 0 && ptr_a > a && ptr_b > b) {
+		/* look for start of label, including dot */
+		label_a = strchr_ba(a, ptr_a, '.');
+		label_b = strchr_ba(b, ptr_b, '.');
+		if (ptr_a - label_a != ptr_b - label_b)
+			/* compare labels up to minimum length
+			   but include \0 to make sure that we
+			   don't consider alpha and alphabet
+			   equal */
+			return 

Re: lazy-load SNI?

[Felipe Gasper]

> On Nov 11, 2016, at 5:36 AM, Aki Tuomi  wrote:
> 
> Hi!
> 
> We are going to do some changes at some point how the certs are loaded and 
> handled to alleviate this. The idea is not yet ripe, so I won't go into too 
> much detail, but idea is to move the cert storage from protocol login 
> processes to elsewhere.
> 

In the other thread 
(http://www.dovecot.org/list/dovecot/2016-October/105855.html) there is mention 
of Exim. Exim actually allows a pretty flexible SNI-time cert load. Dovecot’s 
config doesn’t have the run-time variables that would allow this, but maybe 
there could be some sort of pluggable mechanism to show Dovecot where the cert 
for a given FQDN is? 

It’d be great if a little bit of logic could “teach” Dovecot for each system, 
like:

sub certificate_path ($fqdn) {
return "$WHERE_MY_CERTS_ARE/$fqdn.pem";
}

-FG

Re: lazy-load SNI?

[Aki Tuomi]

> On November 11, 2016 at 12:22 PM Arkadiusz Miśkiewicz  wrote:
> 
> 
> On Friday 11 of November 2016, Felipe Gasper wrote:
> > Hello,
> > 
> > We’re rolling out large SNI deployments for our mail servers. Each 
> > domain
> > gets an entry like this in the config:
> > 
> > local_name mail.foo.com {
> > ssl_cert =  > ssl_key =  > }
> 
> Lack of glob/regexp support here is also a problem (for me). I could have 50% 
> smaller config if local_name supported regexp matching, so it would be 
> possible to do:
> 
> local_name ^(pop3|imap)\.foo\.com {
> ...
> }
> 
> or even with glob like *.foo.com matching.
> 
> > 
> > There are a couple problems we’re finding with this approach:
> > 
> > 1) Dovecot wants to load everything at once, which has some machines taking
> > up many GiB of memory just for Dovecot. Is there any way to defer loading
> > of an SSL cert until a client actually requests it?
> 
> No - thread here http://www.dovecot.org/list/dovecot/2016-October/105855.html
> 
> Memory is one thing.
> 
> The other is that dovecot stops accepting clients when huge config reload 
> happens (I guess it's a design problem since it makes no sense to do that in 
> any case. Clients should be processed without gap using old config until new 
> config is loaded and ready to go).
> 
> And third problem is that there is hardcoded 10s limit for reloading which in 
> case thousands of certificates is way too short limit. Anyway if you hit that 
> limit it's already lost case due to earlier problem.
> 
> > 
> > 2) Any time we add or remove a domain, Dovecot’s SNI config matrix needs to
> > be rebuilt. Is there a way to handle SNI requests dynamically via some
> > sort of configuration plugin, so we wouldn’t need to rebuild the config on
> > domain add/remove? I looked through the docs but couldn’t see a way to do
> > this.
> 
> That's unavoidable for now :-(
> 
> Here we started analyzing maillog and put into dovecot config only these ssl 
> certs for domains that are actually used with TLS. It's very ugly and short-
> sighted approach but hopefuly proper solution will be implemented by dovecot 
> team before all people start to use TLS.
>  
> > Thank you in advance!
> > 
> > -Felipe Gasper
> > Mississauga, ON
> 
> 
> -- 
> Arkadiusz Miśkiewicz, arekm / ( maven.pl | pld-linux.org )

Hi!

We are going to do some changes at some point how the certs are loaded and 
handled to alleviate this. The idea is not yet ripe, so I won't go into too 
much detail, but idea is to move the cert storage from protocol login processes 
to elsewhere.

The local_name matching can probably be fixed faster, it could use the same 
rules as matching cert names generally do.

Aki

Re: lazy-load SNI?

[Arkadiusz Miśkiewicz]

On Friday 11 of November 2016, Felipe Gasper wrote:
> Hello,
> 
>   We’re rolling out large SNI deployments for our mail servers. Each 
> domain
> gets an entry like this in the config:
> 
> local_name mail.foo.com {
> ssl_cert =  ssl_key =  }

Lack of glob/regexp support here is also a problem (for me). I could have 50% 
smaller config if local_name supported regexp matching, so it would be 
possible to do:

local_name ^(pop3|imap)\.foo\.com {
...
}

or even with glob like *.foo.com matching.

> 
>   There are a couple problems we’re finding with this approach:
> 
> 1) Dovecot wants to load everything at once, which has some machines taking
> up many GiB of memory just for Dovecot. Is there any way to defer loading
> of an SSL cert until a client actually requests it?

No - thread here http://www.dovecot.org/list/dovecot/2016-October/105855.html

Memory is one thing.

The other is that dovecot stops accepting clients when huge config reload 
happens (I guess it's a design problem since it makes no sense to do that in 
any case. Clients should be processed without gap using old config until new 
config is loaded and ready to go).

And third problem is that there is hardcoded 10s limit for reloading which in 
case thousands of certificates is way too short limit. Anyway if you hit that 
limit it's already lost case due to earlier problem.

> 
> 2) Any time we add or remove a domain, Dovecot’s SNI config matrix needs to
> be rebuilt. Is there a way to handle SNI requests dynamically via some
> sort of configuration plugin, so we wouldn’t need to rebuild the config on
> domain add/remove? I looked through the docs but couldn’t see a way to do
> this.

That's unavoidable for now :-(

Here we started analyzing maillog and put into dovecot config only these ssl 
certs for domains that are actually used with TLS. It's very ugly and short-
sighted approach but hopefuly proper solution will be implemented by dovecot 
team before all people start to use TLS.
 
>   Thank you in advance!
> 
> -Felipe Gasper
> Mississauga, ON


-- 
Arkadiusz Miśkiewicz, arekm / ( maven.pl | pld-linux.org )

Re: lazy-load SNI?

[Aki Tuomi]

On 11.11.2016 01:02, Felipe Gasper wrote:

Hello,

We’re rolling out large SNI deployments for our mail servers. Each 
domain gets an entry like this in the config:

local_name mail.foo.com {
 ssl_cert = 

Unfortunately it's not possible now, it has been asked before though. We 
have this feature request in our list but cannot give any date when it 
would be available.


Aki Tuomi

Dovecot oy

lazy-load SNI?

[Felipe Gasper]

Hello,

We’re rolling out large SNI deployments for our mail servers. Each 
domain gets an entry like this in the config:

local_name mail.foo.com {
ssl_cert =