Re: Replication going away?

[Michael Slusarz via dovecot]

Hello all,

I want to provide a brief overview regarding various questions surrounding 
features that are being removed from Dovecot CE going forward.

We are currently working on providing updated/improved website info and 
documentation that will better explain exactly what is being maintained in CE.  
However, the desire to have unified messaging clashes with the Engineering 
Team's desire to continue to push code to the open source repository when it is 
ready...

So I want to educate on just a few points here, with the promise that further 
information will be provided in the future.

A reminder that Dovecot is commercial software, and has been since Timo made 
this decision 13 years ago.  Dovecot is not maintained by a community of 
volunteers.  We continue to be lucky that Timo remains Dovecot's Chief 
Architect today, but there are 20 dedicated Dovecot employees, plus additional 
Open-Xchange support staff, that are working on the software everyday.  This is 
carrier-grade software, which requires significant resources to maintain.

Dovecot CE is the open source version of this commercial product (currently, 
Dovecot Pro).  Dovecot CE is not a separate project - it is maintained as part 
of the day-to-day maintenance of Pro.

Every single person that works for Dovecot/OX is extremely proud and dedicated 
to releasing as much software as we can to open source.  CE is able to take 
advantage of this situation to provide features that would not be allowed in a 
purely voluntary project (for example, there are 5 full time QA people working 
on what is eventually released as Dovecot CE).

However, there remains a delicate balance of what we can openly release and 
what we need to be able to commercially provide in order to keep the lights on 
(which allows us to continue to provide open releases...).  This is a difficult 
juggling act, and is one that is always prone to recalibration in any open 
software product, not just Dovecot.

Dovecot CE has always been 100% open source, and will continue to be so.  
Nothing is changing in the future.  Dovecot CE has been, and will always 
continue to be, fully compliant with open source principles (see 
https://opensource.org/osd/).

For a variety of software, maintenance, and (yes) business reasons, there comes 
a time when decisions need to be made to move beyond existing software.  This 
is completely normal in software development, and there is no "open source" 
duty to continue to maintain software that is no longer useful (or, is broken 
or is unmaintained or is not longer best practices or is no longer commercially 
viable or is duplicative of other features that exist or )  That decision 
is what is being done for a selection of longstanding Dovecot features.  It is 
time to move on from them.  There are valid reasons to do so.

If you disagree: the software is open source.  You can continue to use the 
existing software, adapt it to your needs, move to a different solution, or 
whatever else.  

To focus development efforts, and to provide extreme clarity for users going 
forward, Dovecot CE for the first time has adopted a defined Vision Statement: 
"To provide the world's premier open source, standards compliant, 
full-featured, single node email backend server."  This vision formulation was 
made to ensure that CE users continue to receive world class, stable, tested, 
modern, secure email software going forward.  Maintaining features that have 
existed since the mid-2000s (replication, Directors), at the expense of moving 
the software forward to adapt to new paradigms (cloud, containers, 
storage-layer replication, statelessness) is not the proper choice.

These Dovecot CE feature decisions are mine.  If you are unhappy with them, I 
ask that you direct your vitriol directly (and privately) to me.  The Dovecot 
Team does fantastic work and has provided software, under open source 
principles, that runs millions of email servers around the world.  They 
continue to provide invaluable feedback internally in determining the proper 
balance between open and commercial considerations.  They deserve to be thanked 
by the community, not vilified.

michael
___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: Replication going away?

[gene heskett]

On 7/17/23 12:21, Stephan Bosch wrote:


Op 17-7-2023 om 17:54 schreef Steven Varco:

However, I understand some had a better experience with it. I am curious
if someone will fork dovecot and restore the beloved feature.
With all the recent actions going on, clearly targeting in getting 
more paying „pro“ customers (and nothing else!) the dovecot project 
will walk on thin ice and there is a risk (or chance :) ) that it will 
get forked, all the good dovecot developers walking over to the fork 
and dovecot let down to a product no one uses anymore.
It happend before, i.ex. nagios/icinga, CentOS/Alma-/Rocky Linux and 
other big, commonly used software projects.



Sadly, Dovecot is a project with only a few prolific developers, all of 
which work for the company as far as I know. So, the scenario you 
describe is pretty unlikely. Maybe new community developers will stand 
up now, but that is something I think we can only welcome. And forks 
don't need to be adversarial like that.


Regards,

This is a point where we should all remember TANSTAAFL. Its a law you 
cannot break no matter how many useless MBA's you hire. Developers like 
to eat regular and they can't do that on nothing but our good will. 
Dovecot has been good to us, so much so its become the default mail 
server all over this pale blue dot.  Other than the monthly fees we pay 
our ISP's has any one of us donated a stock tank full of ice & beer for 
a summer picnic? Or $500 for a costly bug fixed in 2 hours?  TANSTAAFL 
folks, There really Ain't No Such Thing As A Free Lunch.


Stephan.

___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org


Cheers, Gene Heskett.
--
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author, 1940)
If we desire respect for the law, we must first make the law respectable.
 - Louis D. Brandeis
Genes Web page 

___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: Replication going away?

[Stephan Bosch]

Op 17-7-2023 om 17:54 schreef Steven Varco:

However, I understand some had a better experience with it. I am curious
if someone will fork dovecot and restore the beloved feature.

With all the recent actions going on, clearly targeting in getting more paying 
„pro“ customers (and nothing else!) the dovecot project will walk on thin ice 
and there is a risk (or chance :) ) that it will get forked, all the good 
dovecot developers walking over to the fork and dovecot let down to a product 
no one uses anymore.
It happend before, i.ex. nagios/icinga, CentOS/Alma-/Rocky Linux and other big, 
commonly used software projects.



Sadly, Dovecot is a project with only a few prolific developers, all of 
which work for the company as far as I know. So, the scenario you 
describe is pretty unlikely. Maybe new community developers will stand 
up now, but that is something I think we can only welcome. And forks 
don't need to be adversarial like that.


Regards,


Stephan.

___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: Replication going away?

[Steven Varco]

> However, I understand some had a better experience with it. I am curious
> if someone will fork dovecot and restore the beloved feature.

With all the recent actions going on, clearly targeting in getting more paying 
„pro“ customers (and nothing else!) the dovecot project will walk on thin ice 
and there is a risk (or chance :) ) that it will get forked, all the good 
dovecot developers walking over to the fork and dovecot let down to a product 
no one uses anymore.
It happend before, i.ex. nagios/icinga, CentOS/Alma-/Rocky Linux and other big, 
commonly used software projects.

I for my part will also try to stay as long as possible on dovecot 2.x with 
director and replicator, hoping for a fork which includes those features again 
in a distant future.

Steven

-- 
https://steven.varco.ch/ 
https://www.tech-island.com/ 

___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: Many IDLE imap processes but very few connections moved to imap-hibernate

[D D]

Well I'm not sure what happened but the issue seems to have resolved itself 
somehow:

$ ps aux | grep "dovecot/imap" | wc -l
2168
$ ps aux | grep "dovecot/imap" | grep IDLE | wc -l
56
$ ps aux | grep "imap-hibernate"
syslog863411  0.4  0.0  16660 15296 ?S09:07   1:40 
dovecot/imap-hibernate [2298 connections]

Since my initial message we played with the following settings which seem 
unrelated to hibernation:

mail_debug
haproxy_trusted_networks
mmap_disable
mail_fsync
mail_nfs_index
mail_nfs_storage
mail_max_userip_connections

I'll try to provide more info if the issue arises again.
___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: Replication going away?

[Emmanuel Dreyfus]

On Sun, Jul 09, 2023 at 10:36:35PM +0300, Vladimir Mishonov via dovecot wrote:
> Looking at the commit details, it appears that it completely removes the
> replication feature.

I am not very upset with that news, since all I have been able to do
with replicator was loosing mail. 

However, I understand some had a better experience with it. I am curious
if someone will fork dovecot and restore the beloved feature.

-- 
Emmanuel Dreyfus
m...@netbsd.org
___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Tons of imap-login processes despite client_limit very high

[D D]

Hi Dovecot community.

We're seeing a ton of imap-login processes running even when using high 
performance mode 
(https://doc.dovecot.org/admin_manual/login_processes/#high-performance-mode). 
According to the docs:

"process_min_avail should be set to be at least the number of CPU cores in the 
system, so that all of them will be used. Otherwise new processes are created 
only once an existing one’s connection count reaches client_limit"

We have process_min_avail=4, client_limit=0 and default_client_limit=20. So 
we'd expect seeing only 4 imap-login processes serving a ton of connections 
each. Yet, we see thousands of imap-login processes (more than half of all the 
imap processes):

$ ps aux | grep imap-login | wc -l
1278
$ ps aux | grep imap | wc -l
2154

We use verbose_proctitle=yes and a there seem to be 2 types of these processes, 
about half for a single IP and about half we suspect for multiple IPs:

$ ps aux  | grep imap-log
_apt 1941081  0.0  0.0  10420  8700 ?S14:57   0:00 
dovecot/imap-login [84.115.232.178 TLS proxy]
_apt 1941589  0.0  0.0  10532  8648 ?S14:57   0:00 
dovecot/imap-login [119.202.86.160 TLS proxy]
_apt 1941789  0.0  0.0  10188  8620 ?S14:57   0:00 
dovecot/imap-login [0 pre-login + 2 TLS proxies]
_apt 1942144  0.0  0.0  10716  8748 ?S14:57   0:00 
dovecot/imap-login [0 pre-login + 3 TLS proxies]
_apt 1942428  0.0  0.0  10800  8712 ?S14:57   0:00 
dovecot/imap-login [5.41.100.37 TLS proxy]
...
$ ps aux  | grep imap-log | grep pre-login | wc -l
624
$ ps aux  | grep imap-log | grep -v pre-login | wc -l
654

Is having so many imap-login processes normal with our config? Did we 
misunderstand the docs or is there something wrong here?


default_client_limit = 1048576
default_process_limit = 20

service imap-login {
  # client_limit = 0 # default is 0
  # process_limit = 0 # default is 0
  service_count = 100
  process_min_avail = 4 
  vsz_limit = 2G

  inet_listener imap {
  }
  inet_listener imaps {
haproxy = yes
port = 994
  }
}
___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: Replication going away?

[Nikolaos Milas via dovecot]

On 17/7/2023 1:24 μ.μ., David Zambonini wrote:
With respect, I'm not sure why these scripts are considered a suitable 
replacement, because they're not, and it's obvious no real attempt was 
made to make them so.

...


+1000

We recognize and respect all Dovecot Team's efforts and goodwill for 
many years now. You guys rock.


But please do not mess thousands of mail setups around the globe where 
Replication is already in use.


Please continue the support of these really valuable - for a really huge 
number of mail admins - features.


I strongly believe that the vast majority of the community will be very 
disappointed by this feature removal and it will be a pity for such a 
rich and collaborative community.


My 2c.

All the best,
Nick

___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: Replication going away?

[Michael Grimm via dovecot]

Emmanuel Fusté  wrote:

> Le dim. 16 juil. 2023, 18:55, Aki Tuomi via dovecot  a 
> écrit :
>
>> Yes, director and replicator are removed, and won't be available for pro 
>> users either.

Why in hell would one remove replicator? It's working for years now. Yes, I 
recall issues in the beginning, and others and me helped Timo in 
debugging/testing. After that it runs without any flaws.

So why removing it?

>> Regards to replication, doveadm sync is not being removed. So you can still 
>> run 
>> doveadm sync on your system to have a primary / backup setup

AND: What do you believe an alternative should be, for a failover scenario of 
two IMAP servers?
doveadm sync is not! That's why replicator has been implemented!

> That's completely crazy ! 

+1

Regards,
Michael

___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: Replication going away?

[David Zambonini]

On 17/07/2023 11:41, Aki Tuomi wrote:


You do note that most of the issues you just described were actually
issues with director itself. The director did not do any automatic healing, 
load balancing etc, which is why we have a Pro offering that provides actual 
clustering component (called Palomar architecture).


I'd not raised those points. Load balancers handle first level load balancing in 
both scenarios, in any case. poolmon has always adequately managed healing.


To reiterate, we know director mappings are a live mapping set, not a permanent 
record; there's literally a timeout. Unless I'm in a hurry to re-equalise, I've 
seldom had to manually intervene. It sorts itself out in a few days.


What I've never had to do is constantly mess around in a database and kick users 
manually all the time as a _requirement_ just to keep things running. We both 
know it's not an adequate replacement for director.



Those Lua scripts are OK replacement for a do it yourself environment.


Putting aside the subtext of the retiring of enterprise features outside of Pro, 
with Pro the issue of the cost involved in being forced to shift to an entirely 
different architecture now exists.


--
David Zambonini



___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: Problem connecting with desktop client

[Wolfgang Paul Rauchholz]

What I did and fixed the problem was to reset the listen value the default:
"listen = *, ::"
I am running my home server under dynamic DNS. Therefor I cannot set the IP

Thanks for pointing me in the right direction.

Un saludo,

Wolfgang Rauchholz
+34 627 994 977
https://www.linkedin.com/in/wolfgangrauchholz/




On Mon, Jul 17, 2023 at 1:17 PM Aki Tuomi 
wrote:

> Try
>
> listen = 0.0.0.0
>
> or
>
> listen = 79.152.236.25, 127.0.0.1
>
> instead.
>
> Aki
>
> > On 17/07/2023 14:07 EEST Wolfgang Paul Rauchholz 
> wrote:
> >
> >
> > Hello Aki,
> >
> > Thanks for picking up the topic.
> >
> > [root@home wp.rauchholz]# doveconf listen
> > listen = ipv4
> >
> > root@home wp.rauchholz]# ss -lnpt | grep dovecot
> > LISTEN 0 100 79.152.236.25:143 (http://79.152.236.25:143/) 0.0.0.0:*
> users:(("dovecot",pid=803194,fd=35))
> > LISTEN 0 100 79.152.236.25:993 (http://79.152.236.25:993/) 0.0.0.0:*
> users:(("dovecot",pid=803194,fd=36))
> >
> > Wolfgang
> >
> >
> >
> >
> > Wolfgang Rauchholz
> > +34 627 994 977
> > https://www.linkedin.com/in/wolfgangrauchholz/
> >
> >
> >
> >
> > On Mon, Jul 17, 2023 at 11:59 AM Aki Tuomi 
> wrote:
> > >
> > >  > On 17/07/2023 12:37 EEST Wolfgang Paul Rauchholz <
> wp.rauchh...@gmail.com> wrote:
> > >  >
> > >  >
> > >  > I run my home server under Rocky Linux 9. The server is modem /
> router and as such has two firewall interfaces; internal and external.
> > >  > Dovecot version isdovecot-2.3.16-8.el9.x86_64
> > >  > kernel is: 5.14.0-284.18.1.el9_2.x86_64
> > >  > My domain is wo-lar.com (http://wo-lar.com) (http://wo-lar.com)
> > >  > Postfix and Dovecot are up and running, and I can send and receive
> emails from CLI.
> > >  > But I cannot connect from desktop clients. I get the following
> error message: Server message: Can't connect to host "tcp://wo-lar.com:143
> (http://wo-lar.com:143) (http://wo-lar.com:143)"
> > >  >
> > >  >
> > >  > I tried to telnet from my desktop and server. Results are the same:
> > >  >
> > >  > * I always get a connection refused: telnet wo-lar.com (
> http://wo-lar.com) (http://wo-lar.com) 143 telnet / telnet  IP> 143. On server only: telnet 127.0.0.1 143
> > >  > * telnet wo-lar 143 (without .com!) establishes aconnection
> > >  > [root@home wp.rauchholz]# telnet wo-lar 143
> > >  > Trying 79.152.236.25...
> > >  > Connected to wo-lar.
> > >  > Escape character is '^]'.
> > >  > OK [CAPABILITY IMAP4rev1 SASL-IR LOGIN-REFERRALS ID ENABLE IDLE
> LITERAL+ STARTTLS AUTH=PLAIN AUTH=LOGIN] Dovecot ready.
> > >  >
> > >  > I went through all kinds for conf files and search for wo-lar
> string, but can't find it anywhere
> > >  > Where is the mistake hiding?
> > >  > Thanks for helping.
> > >  >
> > >  > Wolfgang
> > >
> > >  What is your `listen` setting in dovecot.conf?
> > >
> > >  you can check with `doveconf listen`
> > >
> > >  Aki
> > >
> > ___
> > dovecot mailing list -- dovecot@dovecot.org
> > To unsubscribe send an email to dovecot-le...@dovecot.org
>
___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: Problem connecting with desktop client

[Aki Tuomi via dovecot]

Try 

listen = 0.0.0.0

or

listen = 79.152.236.25, 127.0.0.1

instead.

Aki

> On 17/07/2023 14:07 EEST Wolfgang Paul Rauchholz  
> wrote:
> 
> 
> Hello Aki,
> 
> Thanks for picking up the topic.
> 
> [root@home wp.rauchholz]# doveconf listen
> listen = ipv4
> 
> root@home wp.rauchholz]# ss -lnpt | grep dovecot
> LISTEN 0 100 79.152.236.25:143 (http://79.152.236.25:143/) 0.0.0.0:* 
> users:(("dovecot",pid=803194,fd=35)) 
> LISTEN 0 100 79.152.236.25:993 (http://79.152.236.25:993/) 0.0.0.0:* 
> users:(("dovecot",pid=803194,fd=36))
> 
> Wolfgang
> 
> 
> 
> 
> Wolfgang Rauchholz
> +34 627 994 977
> https://www.linkedin.com/in/wolfgangrauchholz/
> 
> 
> 
> 
> On Mon, Jul 17, 2023 at 11:59 AM Aki Tuomi  wrote:
> > 
> >  > On 17/07/2023 12:37 EEST Wolfgang Paul Rauchholz 
> >  wrote:
> >  > 
> >  > 
> >  > I run my home server under Rocky Linux 9. The server is modem / router 
> > and as such has two firewall interfaces; internal and external.
> >  > Dovecot version isdovecot-2.3.16-8.el9.x86_64
> >  > kernel is: 5.14.0-284.18.1.el9_2.x86_64
> >  > My domain is wo-lar.com (http://wo-lar.com) (http://wo-lar.com)
> >  > Postfix and Dovecot are up and running, and I can send and receive 
> > emails from CLI.
> >  > But I cannot connect from desktop clients. I get the following error 
> > message: Server message: Can't connect to host "tcp://wo-lar.com:143 
> > (http://wo-lar.com:143) (http://wo-lar.com:143)"
> >  > 
> >  > 
> >  > I tried to telnet from my desktop and server. Results are the same:
> >  > 
> >  > * I always get a connection refused: telnet wo-lar.com 
> > (http://wo-lar.com) (http://wo-lar.com) 143 telnet / telnet  
> > 143. On server only: telnet 127.0.0.1 143
> >  > * telnet wo-lar 143 (without .com!) establishes aconnection
> >  > [root@home wp.rauchholz]# telnet wo-lar 143
> >  > Trying 79.152.236.25...
> >  > Connected to wo-lar.
> >  > Escape character is '^]'.
> >  > OK [CAPABILITY IMAP4rev1 SASL-IR LOGIN-REFERRALS ID ENABLE IDLE LITERAL+ 
> > STARTTLS AUTH=PLAIN AUTH=LOGIN] Dovecot ready.
> >  > 
> >  > I went through all kinds for conf files and search for wo-lar string, 
> > but can't find it anywhere
> >  > Where is the mistake hiding?
> >  > Thanks for helping.
> >  > 
> >  > Wolfgang
> >  
> >  What is your `listen` setting in dovecot.conf?
> >  
> >  you can check with `doveconf listen`
> >  
> >  Aki
> > 
> ___
> dovecot mailing list -- dovecot@dovecot.org
> To unsubscribe send an email to dovecot-le...@dovecot.org
___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: Problem connecting with desktop client

[Wolfgang Paul Rauchholz]

Hello Aki,

Thanks for picking up the topic.

[root@home wp.rauchholz]# doveconf listen
listen = ipv4

root@home wp.rauchholz]# ss -lnpt | grep dovecot
LISTEN 0  10079.152.236.25:1430.0.0.0:*
 users:(("dovecot",pid=803194,fd=35))

LISTEN 0  10079.152.236.25:9930.0.0.0:*
 users:(("dovecot",pid=803194,fd=36))

Wolfgang



Wolfgang Rauchholz
+34 627 994 977
https://www.linkedin.com/in/wolfgangrauchholz/



On Mon, Jul 17, 2023 at 11:59 AM Aki Tuomi 
wrote:

>
> > On 17/07/2023 12:37 EEST Wolfgang Paul Rauchholz 
> wrote:
> >
> >
> > I run my home server under Rocky Linux 9. The server is modem / router
> and as such has two firewall interfaces; internal and external.
> > Dovecot version isdovecot-2.3.16-8.el9.x86_64
> > kernel is: 5.14.0-284.18.1.el9_2.x86_64
> > My domain is wo-lar.com (http://wo-lar.com)
> > Postfix and Dovecot are up and running, and I can send and receive
> emails from CLI.
> > But I cannot connect from desktop clients. I get the following error
> message: Server message: Can't connect to host "tcp://wo-lar.com:143 (
> http://wo-lar.com:143)"
> >
> >
> > I tried to telnet from my desktop and server. Results are the same:
> >
> >   * I always get a connection refused: telnet wo-lar.com (
> http://wo-lar.com) 143 telnet / telnet  143. On server
> only: telnet 127.0.0.1 143
> >   * telnet wo-lar 143 (without .com!) establishes aconnection
> > [root@home wp.rauchholz]# telnet wo-lar 143
> > Trying 79.152.236.25...
> > Connected to wo-lar.
> > Escape character is '^]'.
> > OK [CAPABILITY IMAP4rev1 SASL-IR LOGIN-REFERRALS ID ENABLE IDLE
> LITERAL+ STARTTLS AUTH=PLAIN AUTH=LOGIN] Dovecot ready.
> >
> > I went through all kinds for conf files and search for wo-lar string,
> but can't find it anywhere
> > Where is the mistake hiding?
> > Thanks for helping.
> >
> > Wolfgang
>
> What is your `listen` setting in dovecot.conf?
>
> you can check with `doveconf listen`
>
> Aki
>
___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: Replication going away?

[Aki Tuomi via dovecot]

> On 17/07/2023 13:24 EEST David Zambonini  wrote:
> 
>  
> On 16/07/2023 17:54, Aki Tuomi via dovecot wrote:
> > Hi!
> > 
> > Yes, director and replicator are removed, and won't be available for pro 
> > users either.
> > 
> > For NFS setups (or similar shared setups), we have documented a way to use 
> > Lua to run a director-like setup, see
> > 
> > https://doc.dovecot.org/3.0/configuration_manual/howto/director_with_lua/
> 
> With respect, I'm not sure why these scripts are considered a suitable 
> replacement, because they're not, and it's obvious no real attempt was made 
> to 
> make them so.
> 
> Putting aside things relatively trivial to add, like weighted instead of 
> random 
> mappings, the meat of the issue is:
> 
> When a backend is removed (fails, or gracefully taken out), the script remaps 
> connecting users that were mapped to it to a different backend. This sounds 
> obvious enough.
> 
> However, when that backend comes back up, users aren't mapped back onto it - 
> the 
> users now have mappings elsewhere. Maybe you got lucky and some users that 
> were 
> mapped onto it just didn't connect the whole time it was down, maybe you got 
> a 
> few new users, but for the majority of your active user base, you now have a 
> large imbalance in your user mappings. The backend has stopped existing for 
> them.
> 
> So you have to rebalance your user mappings manually.
> 
> Rebalancing requires going through all your user mappings. This quickly 
> becomes 
> prohibitively expensive as number of users increases. However, it also leads 
> us 
> on to the next issue.
> 
> Adding a new backend, or returning a failed backend, requires remapping your 
> users, but you can't remap one that's currently connected to a backend 
> without 
> risking a new connection to the wrong backend.
> 
> Either you don't balance/add backends in a way that covers any connected user 
> going forward (which isn't really useful) or you have to start kicking your 
> users (on all of your balancers individually, you don't have the control 
> director gave you any more!) and remapping to rebalance the overall 
> configuration.
> 
> We know that a user will immediately attempt to reconnect when it's kicked, 
> this 
> is the nature of imap clients. You now have a race condition, the only 
> solution 
> to which is to lock that user out while you update the database. I'm not even 
> sure how you'd cover this.
> 
> So it's all gone from automatic to painfully manual with large overheads and 
> race conditions.
> 
> At the very least, to retain the very basic level of suitability would 
> require:
> 
> 1. mappings in the database being given a TTL after which they're no longer 
> considered valid. This is trivial, however:
> 2. importantly, mappings exceeding TTL must *still* be considered valid if 
> the 
> user has existing connections on any balancer. Good luck with that!
> 
> And all this doesn't even cover the other deficiencies. Some easy enough to 
> add, 
> some (involving addressing all balancers at once) far less so.
> 
> Unfortunately I can't speak for our future plans; but I'd personally remain 
> on 
> 2.x director for the proxies for as long as possible, then potentially look 
> at 
> the feasibility of porting the director code to 3.x.
> 
> Again, I really don't understand the thinking behind saying the lua scripts 
> are 
> anything like a suitable replacement. Our team were considering using Pro 
> when 
> the decision to drop director was announced, we backed off quickly as a 
> result.
> 
> -- 
> David Zambonini
> 
> 

You do note that most of the issues you just described were actually
issues with director itself. The director did not do any automatic healing, 
load balancing etc, which is why we have a Pro offering that provides actual 
clustering component (called Palomar architecture).

Those Lua scripts are OK replacement for a do it yourself environment.

Aki
___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: Replication going away?

[David Zambonini]

On 16/07/2023 17:54, Aki Tuomi via dovecot wrote:

Hi!

Yes, director and replicator are removed, and won't be available for pro users 
either.

For NFS setups (or similar shared setups), we have documented a way to use Lua 
to run a director-like setup, see

https://doc.dovecot.org/3.0/configuration_manual/howto/director_with_lua/


With respect, I'm not sure why these scripts are considered a suitable 
replacement, because they're not, and it's obvious no real attempt was made to 
make them so.


Putting aside things relatively trivial to add, like weighted instead of random 
mappings, the meat of the issue is:


When a backend is removed (fails, or gracefully taken out), the script remaps 
connecting users that were mapped to it to a different backend. This sounds 
obvious enough.


However, when that backend comes back up, users aren't mapped back onto it - the 
users now have mappings elsewhere. Maybe you got lucky and some users that were 
mapped onto it just didn't connect the whole time it was down, maybe you got a 
few new users, but for the majority of your active user base, you now have a 
large imbalance in your user mappings. The backend has stopped existing for them.


So you have to rebalance your user mappings manually.

Rebalancing requires going through all your user mappings. This quickly becomes 
prohibitively expensive as number of users increases. However, it also leads us 
on to the next issue.


Adding a new backend, or returning a failed backend, requires remapping your 
users, but you can't remap one that's currently connected to a backend without 
risking a new connection to the wrong backend.


Either you don't balance/add backends in a way that covers any connected user 
going forward (which isn't really useful) or you have to start kicking your 
users (on all of your balancers individually, you don't have the control 
director gave you any more!) and remapping to rebalance the overall configuration.


We know that a user will immediately attempt to reconnect when it's kicked, this 
is the nature of imap clients. You now have a race condition, the only solution 
to which is to lock that user out while you update the database. I'm not even 
sure how you'd cover this.


So it's all gone from automatic to painfully manual with large overheads and 
race conditions.


At the very least, to retain the very basic level of suitability would require:

1. mappings in the database being given a TTL after which they're no longer 
considered valid. This is trivial, however:
2. importantly, mappings exceeding TTL must *still* be considered valid if the 
user has existing connections on any balancer. Good luck with that!


And all this doesn't even cover the other deficiencies. Some easy enough to add, 
some (involving addressing all balancers at once) far less so.


Unfortunately I can't speak for our future plans; but I'd personally remain on 
2.x director for the proxies for as long as possible, then potentially look at 
the feasibility of porting the director code to 3.x.


Again, I really don't understand the thinking behind saying the lua scripts are 
anything like a suitable replacement. Our team were considering using Pro when 
the decision to drop director was announced, we backed off quickly as a result.


--
David Zambonini








___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: Problem connecting with desktop client

[Aki Tuomi via dovecot]

> On 17/07/2023 12:37 EEST Wolfgang Paul Rauchholz  
> wrote:
> 
> 
> I run my home server under Rocky Linux 9. The server is modem / router and as 
> such has two firewall interfaces; internal and external.
> Dovecot version isdovecot-2.3.16-8.el9.x86_64
> kernel is: 5.14.0-284.18.1.el9_2.x86_64
> My domain is wo-lar.com (http://wo-lar.com)
> Postfix and Dovecot are up and running, and I can send and receive emails 
> from CLI.
> But I cannot connect from desktop clients. I get the following error message: 
> Server message: Can't connect to host "tcp://wo-lar.com:143 
> (http://wo-lar.com:143)"
> 
> 
> I tried to telnet from my desktop and server. Results are the same:
> 
>   * I always get a connection refused: telnet wo-lar.com (http://wo-lar.com) 
> 143 telnet / telnet  143. On server only: telnet 127.0.0.1 143
>   * telnet wo-lar 143 (without .com!) establishes aconnection
> [root@home wp.rauchholz]# telnet wo-lar 143
> Trying 79.152.236.25...
> Connected to wo-lar.
> Escape character is '^]'.
> OK [CAPABILITY IMAP4rev1 SASL-IR LOGIN-REFERRALS ID ENABLE IDLE LITERAL+ 
> STARTTLS AUTH=PLAIN AUTH=LOGIN] Dovecot ready.
> 
> I went through all kinds for conf files and search for wo-lar string, but 
> can't find it anywhere
> Where is the mistake hiding?
> Thanks for helping.
> 
> Wolfgang

What is your `listen` setting in dovecot.conf?

you can check with `doveconf listen`

Aki
___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Problem connecting with desktop client

[Wolfgang Paul Rauchholz]

I run my home server under Rocky Linux 9. The server is modem / router and
as such has two firewall interfaces; internal and external.

Dovecot version is dovecot-2.3.16-8.el9.x86_64
kernel is: 5.14.0-284.18.1.el9_2.x86_64

My domain is wo-lar.com

Postfix and Dovecot are up and running, and I can send and receive emails
from CLI.

But I cannot connect from desktop clients. I get the following error
message: Server message: Can't connect to host "tcp://wo-lar.com:143"


I tried to telnet from my desktop and server. Results are the same:


   - I always get a connection refused: telnet wo-lar.com 143 telnet /
   telnet  143. On server only: telnet 127.0.0.1 143
   - telnet wo-lar 143 (without .com!) establishes a connection
   [root@home wp.rauchholz]# telnet wo-lar 143
   Trying 79.152.236.25...
   Connected to wo-lar.
   Escape character is '^]'.
   OK [CAPABILITY IMAP4rev1 SASL-IR LOGIN-REFERRALS ID ENABLE IDLE LITERAL+
   STARTTLS AUTH=PLAIN AUTH=LOGIN] Dovecot ready.

I went through all kinds for conf files and search for wo-lar string, but
can't find it anywhere

Where is the mistake hiding?

Thanks for helping.


Wolfgang
___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org