Re: Replication going away?

[Peter Wienemann]

Hi,

On 2023-11-28 16:20:56 +0100, gerben.wie...@rna.nl wrote:

From what version on is replication gone? I am running 2.3.20 and it still 
there.


it will be gone from version 2.4/3.0 onward (see [0]).

Best regards,

Peter

[0] https://doc.dovecot.org/3.0/installation_guide/upgrading/from-2.3-to-3.0
___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: Replication going away?

[gerben . wierda]

>From what version on is replication gone? I am running 2.3.20 and it still 
>there.
___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: Replication going away?

[Noel Butler via dovecot]

On 26/07/2023 22:43, Marc wrote:


A dns query for imap.web.de address records (IN A) returns two ip
addresses.

And I'm betting each IP is a hardware load balancer with crap load of
servers behind each :)


I am converting a bit to containers and there are so many applications 
that are not able to properly resolve and handle errors. Once they have 
an ip they stop doing anything. That it is nicely setup on the server 
side means nothing. If you do this for outgoing email, lots of email 
clients fail switching to the 2nd ip.


Interesting, if server end is using L4 DSR they shouldnt tell the 
difference, but I can't comment on containers or VM's, as we do not play 
in the virtual world, these things need as much raw power as possible.


--
Regards,
Noel Butler

This Email, including attachments, may contain legally privileged 
information, therefore at all times remains confidential and subject to 
copyright protected under international law. You may not disseminate 
this message without the authors express written authority to do so.   
If you are not the intended recipient, please notify the sender then 
delete all copies of this message including attachments immediately. 
Confidentiality, copyright, and legal privilege are not waived or lost 
by reason of the mistaken delivery of this message.___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: Replication going away?

[Doug Hardie]

> On Jul 26, 2023, at 05:01, Paul Kudla  wrote:
> 
> 
> I know this might have already been answered
> 
> Can some one give a link to the paid site that does what dovecot project does 
> now 
> 
> more then happy to keep the lights on !
> 
> pls advise link ?
> 

I believe the URL is https://www.open-xchange.com 


-- Doug

___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

RE: Replication going away?

[Marc]

> 
> 
>   A dns query for imap.web.de address records (IN A) returns two ip
> addresses.
> 
> 
> And I'm betting each IP is a hardware load balancer with crap load of
> servers behind each :)
> 

I am converting a bit to containers and there are so many applications that are 
not able to properly resolve and handle errors. Once they have an ip they stop 
doing anything. That it is nicely setup on the server side means nothing. If 
you do this for outgoing email, lots of email clients fail switching to the 2nd 
ip.



___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: Replication going away?

[Paul Kudla]

I know this might have already been answered

Can some one give a link to the paid site that does what dovecot project 
does now 


more then happy to keep the lights on !

pls advise link ?




Happy Wednesday !!!
Thanks - paul

Paul Kudla


Scom.ca Internet Services 
004-1009 Byron Street South
Whitby, Ontario - Canada
L1N 4S3

Toronto 416.642.7266
Main 1.866.411.7266
Fax 1.888.892.7266
Email p...@scom.ca

On 7/26/2023 5:12 AM, Noel Butler via dovecot wrote:

On 20/07/2023 05:55, Gerald Galster wrote:




A dns query for imap.web.de address records (IN A) returns two ip 
addresses.
And I'm betting each IP is a hardware load balancer with crap load of 
servers behind each :)

--

Regards,
Noel Butler

This Email, including attachments, may contain legally privileged 
information, therefore at all times remains confidential and subject to 
copyright protected under international law. You may not disseminate 
this message without the authors express written authority to do so.   
If you are not the intended recipient, please notify the sender then 
delete all copies of this message including attachments immediately. 
Confidentiality, copyright, and legal privilege are not waived or lost 
by reason of the mistaken delivery of this message.




--
This message has been scanned for viruses and
dangerous content by *MailScanner* , and is
believed to be clean.

___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: Replication going away?

[Noel Butler via dovecot]

On 20/07/2023 05:55, Gerald Galster wrote:

A dns query for imap.web.de address records (IN A) returns two ip 
addresses.


And I'm betting each IP is a hardware load balancer with crap load of 
servers behind each :)


--
Regards,
Noel Butler

This Email, including attachments, may contain legally privileged 
information, therefore at all times remains confidential and subject to 
copyright protected under international law. You may not disseminate 
this message without the authors express written authority to do so.   
If you are not the intended recipient, please notify the sender then 
delete all copies of this message including attachments immediately. 
Confidentiality, copyright, and legal privilege are not waived or lost 
by reason of the mistaken delivery of this message.___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: Replication going away?

[Noel Butler via dovecot]

(I'm very late to this party so my comments may have been said at some 
point)


On 20/07/2023 03:53, Michael Peddemors wrote:

Real world is a bit different.. DNS Caching.. While DNS Round Robin is 
good enough to distribute loads, it isnt' a very good method for 
failover, even with a very short TTL.  Many home


No, history showed DNS round robin proved abysmal, it led to real load 
balancing hardware and software being born.


These changes don't affect us, we've never used director, hardware load 
balancers FTW, and no replicator, nightly snapshots and multiple levels 
of raid on a NAS backend, but I do see smaller installs where it may be 
preferable to buying a $200k NAS :)


However, for those with shoe string budgets, for load balancing, this 
can be overcome by a software version, there are some for no cost if you 
have a spare machine, you might even pick up a real cheap old hardware 
balancer on likes of ebay.


but it more of a last line failover, and during the time it takes for 
DNS to retry, and find another active node, an AWFUL lot of disgruntled 
customers will be calling ;)


Ahhh reminds me on the very early 90's :)

--
Regards,
Noel Butler

This Email, including attachments, may contain legally privileged 
information, therefore at all times remains confidential and subject to 
copyright protected under international law. You may not disseminate 
this message without the authors express written authority to do so.   
If you are not the intended recipient, please notify the sender then 
delete all copies of this message including attachments immediately. 
Confidentiality, copyright, and legal privilege are not waived or lost 
by reason of the mistaken delivery of this message.___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

RE: Replication going away?

[Marc]

> We are slowly but surely FORCED to use external/cloud services as Open
> (and even closed) source tools are no longer use-able on own/private
> infrastructures or need nuclear plan like budget and implementation
> time/ressources (Yes it is a bit exaggerated ;-) )
> 

There are still people out there that do not want to force everyone in the US 
cloud, they work on development strategies to support and facilitate micro 
organisations. Fact will always be that micro organisations have much better 
client support/relationship and people are just sick of the monopolistic cloud 
providers. Which at that point do not have even a financial advantage any more.
Software designs like eg dovecot will be phased out in the near future and 
replaced by modern applications, this will happen sooner than you think. I 
sincerely hope I can have a significant contribution in such transition.



___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: Replication going away?

[Emmanuel Fusté]

Le 20/07/2023 à 00:57, Michael Slusarz via dovecot a écrit :

On 07/19/2023 2:54 PM MDT Michael Grimm via dovecot  wrote:

Michael Slusarz via dovecot  wrote:

On 07/18/2023 9:00 AM MDT Gerald Galster  wrote:
While I understand it takes effort to maintain the replication plugin, this is 
especially problematic for small active/active high-availability deployments.

To clarify: replication absolutely does not provide "active/active".  
Replication was meant to copy data to a standby server, but you can't have concurrent 
mailbox access.  This is why directors existed.

That simply isn't true, and I am baffled that you don't know that replication 
works with a two server active/active setup for years now! Two separate 
instances (active/active) on two different continents are a completely reliable 
failover scenario for years now.

Very irritating to read such a statement.

You may be irritated, but my statement is accurate.

Eventually consistent replication is *NOT* active/active.  active/active has a 
very specific meaning (and is not the same as master/master).

Quotas and shared mailboxes are two troublesome concepts with replicator.  
Inconsistent mailbox views are a call center driver.  Neither of these would be 
an issue in a true active/active setup.  Forcing a user to a single node at any 
given time will prevent some (but not all) issues.

Replicator's scaling issue can't really be worked around, and was a main driver 
why Dovecot Pro was developed (example: one Pro customer migrating from 
CE/replicators saw a 90% decrease in server count).

Your positive individual experience does not change the inherent 
characteristics, and limitations, of the design.  If your setup works for you, 
in your particular circumstances, great!  But it doesn't work for everyone.  
There is a reason Dovecot development moved on from replicator based 
architecture 10+ years ago.


Your focusing on a very specific scenario, and throw the baby out with 
the bathwater.
Multiple shared client access on the same mailbox is a very specific 
scenario avoidable 99.99% of time as "true" active/active is never a 
real need.
No offense too but it seems that you are biased by your 
business/customers needs that real world main usage.

It is a little bit sad.
We are slowly but surely FORCED to use external/cloud services as Open 
(and even closed) source tools are no longer use-able on own/private 
infrastructures or need nuclear plan like budget and implementation 
time/ressources (Yes it is a bit exaggerated ;-) )


Regards,
Emmanuel.

___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: Replication going away?

[Emmanuel Fusté]

Le 19/07/2023 à 23:03, Michael Peddemors a écrit :


In theory, that is how it is SUPPOSED to work, in practice (and we 
have lots of history where customers ran into this problem when one 
went down), I believe that it was Outlook that didn't try an 
alternative IP address for a 20 min internal cache for instance, 
before a requery of the DNS was done, at which time it again would 
choose which IP to connect to.  As well, SOME modems would get the two 
results, and return only one to the client.  And lots of libraries we 
see, do the DNS query, get two IP results, but then only use the first 
one returned, etc..


The windows cache is supposed (and is confirmed on my side ) to work the 
same as other DNS cache: It will cache all the A records.

Outlook being a good IMAP client is another story :-)

Not arguing how it is supposed to work, just forewarning those to be 
ready when it doesn't work like the manual says.. (Everyone hates 
phone calls about email being down).


If you want to be certain, only a true load balancer will fit the bill.

Oh, and another PS.. IF you are going to do round robin, suggest you 
make two (2) MX records, and put two IPs in both, and then equal 
weight the two MX's.

That is exactly what should not be done.
Never put more than one IPv4 or IPv6 behind a FQDN pointed by a MX.
It will kill the proper HA algorithm build in the SMTP/MX protocol. You 
will introduce some unnecessary delivery delays/retry backoff in case of 
one server failure.
Put as many MX records has you have SMTP gateways. Or group some 
gateways behind some LB VIP if you have/need a high count of gateways.


Keeps a more even load, given those that only prefer the first MX 
returned, and those that prefer the last (spammers)



There is no ordering, round robin apply here too.

MX are for MTA to MTA communications. Talking about MUA/clients, they 
don't care/use MX.


Emmanuel.
___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: Replication going away?

[Eray Aslan]

On Mon, Jul 17, 2023 at 10:30:32PM -0600, Michael Slusarz via dovecot wrote:
> For a variety of software, maintenance, and (yes) business reasons, there 
> comes a time when decisions need to be made to move beyond existing software. 
>  This is completely normal in software development, and there is no "open 
> source" duty to continue to maintain software that is no longer useful (or, 
> is broken or is unmaintained or is not longer best practices or is no longer 
> commercially viable or is duplicative of other features that exist or )  
> That decision is what is being done for a selection of longstanding Dovecot 
> features.  It is time to move on from them.  There are valid reasons to do so.

You are breaking the social contract with your (unpaying) customers.
Expect push back and loss of goodwill. And hope that loss of goodwill
does not amount to much in the long run.

> These Dovecot CE feature decisions are mine.  If you are unhappy with them, I 
> ask that you direct your vitriol directly (and privately) to me.  The Dovecot 
> Team does fantastic work and has provided software, under open source 
> principles, that runs millions of email servers around the world.  They 
> continue to provide invaluable feedback internally in determining the proper 
> balance between open and commercial considerations.  They deserve to be 
> thanked by the community, not vilified.

The people I interacted with working on Dovecot are passinate about the
work they do. And those kind of people do not usually say "I dont care.
I just work here". Whether voiced or acted upon or not, expect loss of
goodwill in your developers as well.

-- 
Eray
___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: Replication going away?

[Michael Slusarz via dovecot]

> On 07/19/2023 2:54 PM MDT Michael Grimm via dovecot  
> wrote:
> 
> Michael Slusarz via dovecot  wrote:
> >> On 07/18/2023 9:00 AM MDT Gerald Galster  wrote:
> 
> >> While I understand it takes effort to maintain the replication plugin, 
> >> this is especially problematic for small active/active high-availability 
> >> deployments.
> > 
> > To clarify: replication absolutely does not provide "active/active".  
> > Replication was meant to copy data to a standby server, but you can't have 
> > concurrent mailbox access.  This is why directors existed.
> 
> That simply isn't true, and I am baffled that you don't know that replication 
> works with a two server active/active setup for years now! Two separate 
> instances (active/active) on two different continents are a completely 
> reliable failover scenario for years now.
> 
> Very irritating to read such a statement.

You may be irritated, but my statement is accurate.

Eventually consistent replication is *NOT* active/active.  active/active has a 
very specific meaning (and is not the same as master/master).

Quotas and shared mailboxes are two troublesome concepts with replicator.  
Inconsistent mailbox views are a call center driver.  Neither of these would be 
an issue in a true active/active setup.  Forcing a user to a single node at any 
given time will prevent some (but not all) issues.

Replicator's scaling issue can't really be worked around, and was a main driver 
why Dovecot Pro was developed (example: one Pro customer migrating from 
CE/replicators saw a 90% decrease in server count).

Your positive individual experience does not change the inherent 
characteristics, and limitations, of the design.  If your setup works for you, 
in your particular circumstances, great!  But it doesn't work for everyone.  
There is a reason Dovecot development moved on from replicator based 
architecture 10+ years ago.

michael
___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: Replication going away?

[Michael Grimm via dovecot]

Marc  wrote:

>> That simply isn't true, and I am baffled that you don't know that
>> replication works with a two server active/active setup for years now!
>> Two separate instances (active/active) on two different continents are a
>> completely reliable failover scenario for years now.
> 
> Maybe it works like this in your environment? Maybe if the load increases you 
> run into trouble? The director is making sure you never utilize an 
> active/active situation from the perspective of user access. The user is only 
> accessing one server. It is quite a different story when the same user starts 
> writing to both servers at the same time.

If I do rapidly inject tens of thousands of mails locally on both servers 
SIMULTANEOUSLY for the very same user I never ever loose one of it. Tested 
numerous times before rolling it out. In the very beginning of Timo's 
publishing replication it had had flaws, but other users and myself tested it 
while Timo enhances his code (and IIRC once even rewrote it from scratch). For 
years now it runs as expected and documented.

As mentioned in this thread this ist true for small setups.

Regards,
Michael

___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

RE: Replication going away?

[Marc]

> 
> That simply isn't true, and I am baffled that you don't know that
> replication works with a two server active/active setup for years now!
> Two separate instances (active/active) on two different continents are a
> completely reliable failover scenario for years now.
> 

Maybe it works like this in your environment? Maybe if the load increases you 
run into trouble? The director is making sure you never utilize an 
active/active situation from the perspective of user access. The user is only 
accessing one server. It is quite a different story when the same user starts 
writing to both servers at the same time. 




___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: Replication going away?

[Michael Peddemors]

On 2023-07-19 12:55, Gerald Galster wrote:

Le 19/07/2023 à 19:53, Michael Peddemors a écrit :

Real world is a bit different.. DNS Caching.. While DNS Round Robin is good 
enough to distribute loads, it isnt' a very good method for failover, even with 
a very short TTL.  Many home routers, still insist on caching results for a 
long time, no matter what the TTL says, and of course Windows internal caching 
etc..

Should not confuse the issue.. call it a 'poor man's load balancer' if you 
will, but it more of a last line failover, and during the time it takes for DNS 
to retry, and find another active node, an AWFUL lot of disgruntled customers 
will be calling ;)

Also so interesting to see some resolvers that don't think of using the second 
record, if the first one is down..


You're mixing things : DNS and Mail client behavior. It is a non sense.
A resolver will serve records, It does not use them and do not care of what is 
behind the record.
A good client use the lists (of A or ) records to connect to the server and 
will iterate on the list if the server behind the record is down.
And DNS caching do it job nothing less, nothing more and is out of the picture.


Emmanuel is right. Here's an example to clarify:

$ dig imap.web.de

;; ANSWER SECTION:
imap.web.de.226 IN  A   212.227.17.178
imap.web.de.226 IN  A   212.227.17.162

A dns query for imap.web.de address records (IN A) returns two ip addresses.
A local resolver receives those two ip addresses and usually passes them on
to clients while it may rotate the order, so that some clients will see
212.227.17.178, 212.227.17.162 and others will see 212.227.17.162, 
212.227.17.178.
It is possible to get the same order for subsequent requests but on a *global* 
scale
that roughly equals 50/50 loadbalancing.

Mail clients then connect to e.g. 212.227.17.178 and try 212.227.17.162 on 
connection
failure without any further dns involvement. Dns caching (ttl) is irrelevant in 
that case.


In theory, that is how it is SUPPOSED to work, in practice (and we have 
lots of history where customers ran into this problem when one went 
down), I believe that it was Outlook that didn't try an alternative IP 
address for a 20 min internal cache for instance, before a requery of 
the DNS was done, at which time it again would choose which IP to 
connect to.  As well, SOME modems would get the two results, and return 
only one to the client.  And lots of libraries we see, do the DNS query, 
get two IP results, but then only use the first one returned, etc..


Not arguing how it is supposed to work, just forewarning those to be 
ready when it doesn't work like the manual says.. (Everyone hates phone 
calls about email being down).


If you want to be certain, only a true load balancer will fit the bill.

Oh, and another PS.. IF you are going to do round robin, suggest you 
make two (2) MX records, and put two IPs in both, and then equal weight 
the two MX's.


Keeps a more even load, given those that only prefer the first MX 
returned, and those that prefer the last (spammers)



--
"Catch the Magic of Linux..."

Michael Peddemors, President/CEO LinuxMagic Inc.
Visit us at http://www.linuxmagic.com @linuxmagic
A Wizard IT Company - For More Info http://www.wizard.ca
"LinuxMagic" a Registered TradeMark of Wizard Tower TechnoServices Ltd.

604-682-0300 Beautiful British Columbia, Canada

This email and any electronic data contained are confidential and intended
solely for the use of the individual or entity to which they are addressed.
Please note that any views or opinions presented in this email are solely
those of the author and are not intended to represent those of the company.

___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: Replication going away?

[Michael Grimm via dovecot]

Michael Slusarz via dovecot  wrote:
>> On 07/18/2023 9:00 AM MDT Gerald Galster  wrote:

>> While I understand it takes effort to maintain the replication plugin, this 
>> is especially problematic for small active/active high-availability 
>> deployments.
> 
> To clarify: replication absolutely does not provide "active/active".  
> Replication was meant to copy data to a standby server, but you can't have 
> concurrent mailbox access.  This is why directors existed.

That simply isn't true, and I am baffled that you don't know that replication 
works with a two server active/active setup for years now! Two separate 
instances (active/active) on two different continents are a completely reliable 
failover scenario for years now.

Very irritating to read such a statement.

Regards,
Michael
___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: Replication going away?

[Gerald Galster]

>> While I understand it takes effort to maintain the replication plugin, this 
>> is especially problematic for small active/active high-availability 
>> deployments.
> 
> To clarify: replication absolutely does not provide "active/active".  
> Replication was meant to copy data to a standby server, but you can't have 
> concurrent mailbox access.  This is why directors existed.

I have to disagree, active/active with two distinct servers is a special case 
and two-way replication has been working reliably in production for years.
Here's why: those two imap servers are separate instances with local storage, 
local quota and local pruning. There is no shared medium access like nfs that 
could lead to data corruption and hence no directors needed. Moreover the dsync 
manpage states "doveadm-sync - Dovecot's two-way mailbox synchronization 
utility". It's more like IMAP-clients copying mails from one server to another 
where the most current state wins.


>> I guess there are lots of servers that use replication for just 50 or 100 
>> mailboxes. Cloudstorage (like S3) would be overkill for these.
>> 
>> Do you provide dovecot pro subscriptions for such small deployments?
> 
> A 50-100 mailbox user server will run Dovecot CE just fine.  Pro would be 
> overkill.
> 
> All current Dovecot development assumes that storage is decoupled from the 
> system.  Shared (as in network available) storage is what you need if you 
> want high availability, whether in Pro or CE.


Thanks for clarification. So even if mdbox is still available and replication 
(backup) would work with dsync on the command line, there is no signaling layer 
to auto-trigger replication because storage is decoupled and this functionality 
is not needed anymore.

Best regards,
Gerald
___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: Replication going away?

[Gerald Galster]

> Le 19/07/2023 à 19:53, Michael Peddemors a écrit :
>> Real world is a bit different.. DNS Caching.. While DNS Round Robin is good 
>> enough to distribute loads, it isnt' a very good method for failover, even 
>> with a very short TTL.  Many home routers, still insist on caching results 
>> for a long time, no matter what the TTL says, and of course Windows internal 
>> caching etc..
>> 
>> Should not confuse the issue.. call it a 'poor man's load balancer' if you 
>> will, but it more of a last line failover, and during the time it takes for 
>> DNS to retry, and find another active node, an AWFUL lot of disgruntled 
>> customers will be calling ;)
>> 
>> Also so interesting to see some resolvers that don't think of using the 
>> second record, if the first one is down..
>> 
> You're mixing things : DNS and Mail client behavior. It is a non sense.
> A resolver will serve records, It does not use them and do not care of what 
> is behind the record.
> A good client use the lists (of A or ) records to connect to the server 
> and will iterate on the list if the server behind the record is down.
> And DNS caching do it job nothing less, nothing more and is out of the 
> picture.

Emmanuel is right. Here's an example to clarify:

$ dig imap.web.de

;; ANSWER SECTION:
imap.web.de.226 IN  A   212.227.17.178
imap.web.de.226 IN  A   212.227.17.162

A dns query for imap.web.de address records (IN A) returns two ip addresses.
A local resolver receives those two ip addresses and usually passes them on
to clients while it may rotate the order, so that some clients will see
212.227.17.178, 212.227.17.162 and others will see 212.227.17.162, 
212.227.17.178.
It is possible to get the same order for subsequent requests but on a *global* 
scale
that roughly equals 50/50 loadbalancing.

Mail clients then connect to e.g. 212.227.17.178 and try 212.227.17.162 on 
connection
failure without any further dns involvement. Dns caching (ttl) is irrelevant in 
that case.

Best regards,
Gerald

___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: Replication going away?

[David Morsberger]

In order to prepare for the upcoming release:

Is there an estimated release date when replication will be removed?

What is the best way (reconfigure, turn off auto replication, etc.) to migrate 
to an environment to test and run using ‘doveadm sync -d’ in a scheduled job 
(e.g., cron)? 



Sent from my iPad

> On Jul 18, 2023, at 9:42 AM, Aki Tuomi via dovecot  
> wrote:
> 
> 
>> On 18/07/2023 15:19 EEST i...@joergschulz.de wrote:
>> 
>> 
>> Just to understand that correctly: I could setup a (cron) based process for 
>> doveadm sync, but no longer a setup like 
>> plugin { 
>>  mail_replica = tcp:$IMAP_REPLICA_SERVER:$IMAP_REPLICA_PORT 
>> } 
>> where the cron would lead to some delay and would have to check for 
>> concurrent jobs?
> 
> You can also have that too.
> 
> doveadm sync -d 
> 
> makes it use mail_replica setting.
> 
> Aki
> ___
> dovecot mailing list -- dovecot@dovecot.org
> To unsubscribe send an email to dovecot-le...@dovecot.org
___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: [EXT] RE: Replication going away?

[Michael Slusarz via dovecot]

> On 07/19/2023 12:51 PM MDT Marc  wrote:
>
> > A 50-100 mailbox user server will run Dovecot CE just fine.  Pro would
> > be overkill.
> 
> What is overkill? I always thought it had a bit more features and support.

For Pro 2.3, you need (at minimum) 7 Dovecot nodes + HA authentication + HA 
storage + (minimum) 3 Cassandra nodes if using object storage.  This is per 
site; most of our customers require data center redundancy as well, so multiply 
as needed.  And this is only email retrieval; this doesn't even begin to touch 
upon email transfer.

Email high availability isn't cheap.  (I would argue that if you truly need 
this sort of carrier-grade HA for 50 users, it makes much more sense to use 
email as-a-service than trying to do it yourself these days.  Unless you have 
very specific reasons and a ton of cash.)

michael
___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: Replication going away?

[Simon B]

On Wed, 19 Jul 2023 at 20:40, Michael Slusarz via dovecot
 wrote:
>
> > On 07/18/2023 9:00 AM MDT Gerald Galster  wrote:
> >
> > While I understand it takes effort to maintain the replication plugin, this 
> > is especially problematic for small active/active high-availability 
> > deployments.
>
> To clarify: replication absolutely does not provide "active/active".  
> Replication was meant to copy data to a standby server, but you can't have 
> concurrent mailbox access.  This is why directors existed.
>
>
> > I guess there are lots of servers that use replication for just 50 or 100 
> > mailboxes. Cloudstorage (like S3) would be overkill for these.
> >
> > Do you provide dovecot pro subscriptions for such small deployments?
>
> A 50-100 mailbox user server will run Dovecot CE just fine.  Pro would be 
> overkill.
>
> All current Dovecot development assumes that storage is decoupled from the 
> system.  Shared (as in network available) storage is what you need if you 
> want high availability, whether in Pro or CE.

That seems like an extraordinary callous and short-sighted decision,
but nevertheless does give some technical weight to your earlier
statement.

Regards

Simon
___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: Replication going away?

[Simon B]

On Tue, 18 Jul 2023 at 06:47, Michael Slusarz via dovecot
 wrote:
>
> Hello all,
>
> I want to provide a brief overview regarding various questions surrounding 
> features that are being removed from Dovecot CE going forward.
>
> We are currently working on providing updated/improved website info and 
> documentation that will better explain exactly what is being maintained in 
> CE.  However, the desire to have unified messaging clashes with the 
> Engineering Team's desire to continue to push code to the open source 
> repository when it is ready...
>
> So I want to educate on just a few points here, with the promise that further 
> information will be provided in the future.
>
> A reminder that Dovecot is commercial software, and has been since Timo made 
> this decision 13 years ago.  Dovecot is not maintained by a community of 
> volunteers.  We continue to be lucky that Timo remains Dovecot's Chief 
> Architect today, but there are 20 dedicated Dovecot employees, plus 
> additional Open-Xchange support staff, that are working on the software 
> everyday.  This is carrier-grade software, which requires significant 
> resources to maintain.
>
> Dovecot CE is the open source version of this commercial product (currently, 
> Dovecot Pro).  Dovecot CE is not a separate project - it is maintained as 
> part of the day-to-day maintenance of Pro.
>
> Every single person that works for Dovecot/OX is extremely proud and 
> dedicated to releasing as much software as we can to open source.  CE is able 
> to take advantage of this situation to provide features that would not be 
> allowed in a purely voluntary project (for example, there are 5 full time QA 
> people working on what is eventually released as Dovecot CE).
>
> However, there remains a delicate balance of what we can openly release and 
> what we need to be able to commercially provide in order to keep the lights 
> on (which allows us to continue to provide open releases...).  This is a 
> difficult juggling act, and is one that is always prone to recalibration in 
> any open software product, not just Dovecot.
>
> Dovecot CE has always been 100% open source, and will continue to be so.  
> Nothing is changing in the future.  Dovecot CE has been, and will always 
> continue to be, fully compliant with open source principles (see 
> https://opensource.org/osd/).
>
> For a variety of software, maintenance, and (yes) business reasons, there 
> comes a time when decisions need to be made to move beyond existing software. 
>  This is completely normal in software development, and there is no "open 
> source" duty to continue to maintain software that is no longer useful (or, 
> is broken or is unmaintained or is not longer best practices or is no longer 
> commercially viable or is duplicative of other features that exist or )  
> That decision is what is being done for a selection of longstanding Dovecot 
> features.  It is time to move on from them.  There are valid reasons to do so.
>
> If you disagree: the software is open source.  You can continue to use the 
> existing software, adapt it to your needs, move to a different solution, or 
> whatever else.
>
> To focus development efforts, and to provide extreme clarity for users going 
> forward, Dovecot CE for the first time has adopted a defined Vision 
> Statement: "To provide the world's premier open source, standards compliant, 
> full-featured, single node email backend server."  This vision formulation 
> was made to ensure that CE users continue to receive world class, stable, 
> tested, modern, secure email software going forward.  Maintaining features 
> that have existed since the mid-2000s (replication, Directors), at the 
> expense of moving the software forward to adapt to new paradigms (cloud, 
> containers, storage-layer replication, statelessness) is not the proper 
> choice.

I cannot imagine you will devote any time to justifying this
statement, so I won't ask, but for a list of mainly technical
participants, I imagine people cleverer than me have issues with the
lack of substance.

> These Dovecot CE feature decisions are mine.  If you are unhappy with them, I 
> ask that you direct your vitriol directly (and privately) to me.  The Dovecot 
> Team does fantastic work and has provided software, under open source 
> principles, that runs millions of email servers around the world.  They 
> continue to provide invaluable feedback internally in determining the proper 
> balance between open and commercial considerations.  They deserve to be 
> thanked by the community, not vilified.

Just to be clear, no one - involved with the project, or the
commercial operations - should be vilified for any decision taken at a
senior level.

But from my detached perspective - I do not use the feature - the real
failure here is the lack of communication from the senior level.   Aki
has had to defend this decision for months without your support and
communication.

Similar to the complete absence 

RE: Replication going away?

[Marc]

> 
> A 50-100 mailbox user server will run Dovecot CE just fine.  Pro would
> be overkill.
> 

What is overkill? I always thought it had a bit more features and support.
___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: Replication going away?

[Michael Slusarz via dovecot]

> On 07/18/2023 9:00 AM MDT Gerald Galster  wrote:
>
> While I understand it takes effort to maintain the replication plugin, this 
> is especially problematic for small active/active high-availability 
> deployments.

To clarify: replication absolutely does not provide "active/active".  
Replication was meant to copy data to a standby server, but you can't have 
concurrent mailbox access.  This is why directors existed.


> I guess there are lots of servers that use replication for just 50 or 100 
> mailboxes. Cloudstorage (like S3) would be overkill for these.
> 
> Do you provide dovecot pro subscriptions for such small deployments?

A 50-100 mailbox user server will run Dovecot CE just fine.  Pro would be 
overkill.

All current Dovecot development assumes that storage is decoupled from the 
system.  Shared (as in network available) storage is what you need if you want 
high availability, whether in Pro or CE.

michael
___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: Replication going away?

[Emmanuel Fusté]

Le 19/07/2023 à 19:53, Michael Peddemors a écrit :
Real world is a bit different.. DNS Caching.. While DNS Round Robin is 
good enough to distribute loads, it isnt' a very good method for 
failover, even with a very short TTL.  Many home routers, still insist 
on caching results for a long time, no matter what the TTL says, and 
of course Windows internal caching etc..


Should not confuse the issue.. call it a 'poor man's load balancer' if 
you will, but it more of a last line failover, and during the time it 
takes for DNS to retry, and find another active node, an AWFUL lot of 
disgruntled customers will be calling ;)


Also so interesting to see some resolvers that don't think of using 
the second record, if the first one is down..



You're mixing things : DNS and Mail client behavior. It is a non sense.
A resolver will serve records, It does not use them and do not care of 
what is behind the record.
A good client use the lists (of A or ) records to connect to the 
server and will iterate on the list if the server behind the record is down.
And DNS caching do it job nothing less, nothing more and is out of the 
picture.


Emmanuel.


___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: Replication going away?

[Michael Peddemors]

Real world is a bit different.. DNS Caching.. While DNS Round Robin is 
good enough to distribute loads, it isnt' a very good method for 
failover, even with a very short TTL.  Many home routers, still insist 
on caching results for a long time, no matter what the TTL says, and of 
course Windows internal caching etc..


Should not confuse the issue.. call it a 'poor man's load balancer' if 
you will, but it more of a last line failover, and during the time it 
takes for DNS to retry, and find another active node, an AWFUL lot of 
disgruntled customers will be calling ;)


Also so interesting to see some resolvers that don't think of using the 
second record, if the first one is down..


On 2023-07-18 17:09, Gerald Galster wrote:

While I understand it takes effort to maintain the replication plugin, this is 
especially problematic for small active/active high-availability deployments.
I guess there are lots of servers that use replication for just 50 or 100 
mailboxes. Cloudstorage (like S3) would be overkill for these.


Even without active/active, it's super useful for the simple
active/backup configuration which I use on my personal mail server


This depends heavily on individual usage. Coming from an active/active
deployment it's a major step backwards though: usually two servers
are running independently in geographically dispersed datacenters.
High-availabilty is achieved by a simple DNS entry that returns two
ip addresses, one from each datacenter. Under normal circumstances
that gives you 50/50 loadbalancing without loadbalancers, without
additional components that can fail. In case one datacenter goes down,
and that happens to every datacenter at some time, the other datacenter
takes over - automatically, without any configuration changes.
Additionally mail user agents (Outlook, Thunderbird, ...) don't need
special configuration. If one ip address is unrechable they connect
to the other one obtained via DNS and users can quite seemlessly send
and receive email again. After the outage ceased and the other
datacenter is back online again, there is nothing to do.
No configuration changes, no error prone manual synchronization or
promoting passive to active - it just works and heals itself.
Being used to a carefree setup like that you don't want to go back.

Of course there are other possibilities like nfs, glusterfs, gfs2,
zfs snapshots, ceph, minio or dsync backup but they all have their own
drawbacks. For small mailservers that want high availability dsync
replication is quite the perfect solution.



setup (one colo box, one home server) and a small company mail
server; as such I'm pretty sad to see it go. Still, it is up
to OX where they want to put their resources.


Well, it seems the dsync replication function is still there,
just the replication plugin that notifies what to replicate
is deprectated. Of course it's OX's decision, I'm just hoping
they were not aware how useful replication is in the before
mentioned scenario.

Moreover I'm quite sure this kind of small-scale replication
does not have any impact on customers upgrading to the new
cloud architecture. Big customers will go for cloud because
it scales way better and does not have replication induced
performance penalties and small customers probably can't
afford to upgrade because it's too pricey.



I guess losing repl probably doesn't affect larger ISP type setups
so much; it seems a bit more common to use shared storage (e.g.
maildirs on an nfs appliance or similar) in those cases if they're
actually running their own storage.


Do you provide dovecot pro subscriptions for such small deployments?


Unless I misunderstood the message (and I don't think I did), repl
was removed in pro too. (I don't expect that pro is available on my
usual choice of OS anyway..).


As I understood it dsync is still working. Replication configured via
ssh is calling dsync under the hood, so if local storage and index/log
formats don't change for single deployments, it seems to be more of
a political decision. I know maintenance is not for free, that's why
I suggested to think about a dovecot small/medium business edition
with a more affordable price tag.

Best regards,
Gerald
___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org



--
"Catch the Magic of Linux..."

Michael Peddemors, President/CEO LinuxMagic Inc.
Visit us at http://www.linuxmagic.com @linuxmagic
A Wizard IT Company - For More Info http://www.wizard.ca
"LinuxMagic" a Registered TradeMark of Wizard Tower TechnoServices Ltd.

604-682-0300 Beautiful British Columbia, Canada

This email and any electronic data contained are confidential and intended
solely for the use of the individual or entity to which they are addressed.
Please note that any views 

Re: Replication going away?

[Gerald Galster]

>> While I understand it takes effort to maintain the replication plugin, this 
>> is especially problematic for small active/active high-availability 
>> deployments.
>> I guess there are lots of servers that use replication for just 50 or 100 
>> mailboxes. Cloudstorage (like S3) would be overkill for these.
> 
> Even without active/active, it's super useful for the simple
> active/backup configuration which I use on my personal mail server

This depends heavily on individual usage. Coming from an active/active
deployment it's a major step backwards though: usually two servers
are running independently in geographically dispersed datacenters.
High-availabilty is achieved by a simple DNS entry that returns two
ip addresses, one from each datacenter. Under normal circumstances
that gives you 50/50 loadbalancing without loadbalancers, without
additional components that can fail. In case one datacenter goes down,
and that happens to every datacenter at some time, the other datacenter
takes over - automatically, without any configuration changes.
Additionally mail user agents (Outlook, Thunderbird, ...) don't need
special configuration. If one ip address is unrechable they connect
to the other one obtained via DNS and users can quite seemlessly send
and receive email again. After the outage ceased and the other
datacenter is back online again, there is nothing to do.
No configuration changes, no error prone manual synchronization or
promoting passive to active - it just works and heals itself.
Being used to a carefree setup like that you don't want to go back.

Of course there are other possibilities like nfs, glusterfs, gfs2,
zfs snapshots, ceph, minio or dsync backup but they all have their own
drawbacks. For small mailservers that want high availability dsync
replication is quite the perfect solution.


> setup (one colo box, one home server) and a small company mail
> server; as such I'm pretty sad to see it go. Still, it is up
> to OX where they want to put their resources.

Well, it seems the dsync replication function is still there,
just the replication plugin that notifies what to replicate
is deprectated. Of course it's OX's decision, I'm just hoping
they were not aware how useful replication is in the before
mentioned scenario.

Moreover I'm quite sure this kind of small-scale replication
does not have any impact on customers upgrading to the new
cloud architecture. Big customers will go for cloud because
it scales way better and does not have replication induced
performance penalties and small customers probably can't
afford to upgrade because it's too pricey.


> I guess losing repl probably doesn't affect larger ISP type setups
> so much; it seems a bit more common to use shared storage (e.g.
> maildirs on an nfs appliance or similar) in those cases if they're
> actually running their own storage.
> 
>> Do you provide dovecot pro subscriptions for such small deployments?
> 
> Unless I misunderstood the message (and I don't think I did), repl
> was removed in pro too. (I don't expect that pro is available on my
> usual choice of OS anyway..).

As I understood it dsync is still working. Replication configured via
ssh is calling dsync under the hood, so if local storage and index/log
formats don't change for single deployments, it seems to be more of
a political decision. I know maintenance is not for free, that's why
I suggested to think about a dovecot small/medium business edition
with a more affordable price tag.

Best regards,
Gerald
___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: Replication going away?

[Stuart Henderson]

On 2023-07-18, Gerald Galster  wrote:
> While I understand it takes effort to maintain the replication plugin, this 
> is especially problematic for small active/active high-availability 
> deployments.
> I guess there are lots of servers that use replication for just 50 or 100 
> mailboxes. Cloudstorage (like S3) would be overkill for these.

Even without active/active, it's super useful for the simple
active/backup configuration which I use on my personal mail server
setup (one colo box, one home server) and a small company mail
server; as such I'm pretty sad to see it go. Still, it is up
to OX where they want to put their resources.

I guess losing repl probably doesn't affect larger ISP type setups
so much; it seems a bit more common to use shared storage (e.g.
maildirs on an nfs appliance or similar) in those cases if they're
actually running their own storage.

> Do you provide dovecot pro subscriptions for such small deployments?

Unless I misunderstood the message (and I don't think I did), repl
was removed in pro too. (I don't expect that pro is available on my
usual choice of OS anyway..).


___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: Replication going away?

[Steven Varco]

> While I understand it takes effort to maintain the replication plugin, this 
> is especially problematic for small active/active high-availability 
> deployments.
> I guess there are lots of servers that use replication for just 50 or 100 
> mailboxes. Cloudstorage (like S3) would be overkill for these.
> 
> Do you provide dovecot pro subscriptions for such small deployments?

I’m running such a setup for 5-10 mailboxes.. :D


-- 
https://steven.varco.ch/ 

___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: Replication going away?

[Gerald Galster]

>> Just to understand that correctly: I could setup a (cron) based process for 
>> doveadm sync, but no longer a setup like 
>> plugin { 
>>  mail_replica = tcp:$IMAP_REPLICA_SERVER:$IMAP_REPLICA_PORT 
>> } 
>> where the cron would lead to some delay and would have to check for 
>> concurrent jobs?
> 
> You can also have that too.
> 
> doveadm sync -d 
> 
> makes it use mail_replica setting.

@Aki:

Is it possible to monitor actions that would have triggered replication in 
dovecot < 2.4, e.g. parsing logs or a lua script to imitate the previous 
behaviour?


@Michael Slusarz:

While I understand it takes effort to maintain the replication plugin, this is 
especially problematic for small active/active high-availability deployments.
I guess there are lots of servers that use replication for just 50 or 100 
mailboxes. Cloudstorage (like S3) would be overkill for these.

Do you provide dovecot pro subscriptions for such small deployments?

The basic replication functionality with dsync seems to be available in future 
versions. Would you consider releasing a "dovecot smb" version for small/medium 
businesses that maintains the replication plugin (without director) for a 
yearly subscription fee? Otherwise those affected would have to look for 
alternatives. The recent release of rust-written Stalwart mailserver comes to 
mind, that bundles a lot of functionality (smtp, pop, imap, jmap, s3, sieve, 
dkim/arc/spf/dmarc, dane, mta-sas, ...): https://stalw.art/blog/

Best regards,
Gerald
___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: Replication going away?

[Aki Tuomi via dovecot]

> On 18/07/2023 15:19 EEST i...@joergschulz.de wrote:
> 
>  
> Just to understand that correctly: I could setup a (cron) based process for 
> doveadm sync, but no longer a setup like 
> plugin { 
>   mail_replica = tcp:$IMAP_REPLICA_SERVER:$IMAP_REPLICA_PORT 
> } 
> where the cron would lead to some delay and would have to check for 
> concurrent jobs?

You can also have that too.

doveadm sync -d 

makes it use mail_replica setting.

Aki
___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: Replication going away?

[info]

Just to understand that correctly: I could setup a (cron) based process for 
doveadm sync, but no longer a setup like 
plugin { 
  mail_replica = tcp:$IMAP_REPLICA_SERVER:$IMAP_REPLICA_PORT 
} 
where the cron would lead to some delay and would have to check for concurrent 
jobs?
___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: Replication going away?

[Michael Slusarz via dovecot]

Hello all,

I want to provide a brief overview regarding various questions surrounding 
features that are being removed from Dovecot CE going forward.

We are currently working on providing updated/improved website info and 
documentation that will better explain exactly what is being maintained in CE.  
However, the desire to have unified messaging clashes with the Engineering 
Team's desire to continue to push code to the open source repository when it is 
ready...

So I want to educate on just a few points here, with the promise that further 
information will be provided in the future.

A reminder that Dovecot is commercial software, and has been since Timo made 
this decision 13 years ago.  Dovecot is not maintained by a community of 
volunteers.  We continue to be lucky that Timo remains Dovecot's Chief 
Architect today, but there are 20 dedicated Dovecot employees, plus additional 
Open-Xchange support staff, that are working on the software everyday.  This is 
carrier-grade software, which requires significant resources to maintain.

Dovecot CE is the open source version of this commercial product (currently, 
Dovecot Pro).  Dovecot CE is not a separate project - it is maintained as part 
of the day-to-day maintenance of Pro.

Every single person that works for Dovecot/OX is extremely proud and dedicated 
to releasing as much software as we can to open source.  CE is able to take 
advantage of this situation to provide features that would not be allowed in a 
purely voluntary project (for example, there are 5 full time QA people working 
on what is eventually released as Dovecot CE).

However, there remains a delicate balance of what we can openly release and 
what we need to be able to commercially provide in order to keep the lights on 
(which allows us to continue to provide open releases...).  This is a difficult 
juggling act, and is one that is always prone to recalibration in any open 
software product, not just Dovecot.

Dovecot CE has always been 100% open source, and will continue to be so.  
Nothing is changing in the future.  Dovecot CE has been, and will always 
continue to be, fully compliant with open source principles (see 
https://opensource.org/osd/).

For a variety of software, maintenance, and (yes) business reasons, there comes 
a time when decisions need to be made to move beyond existing software.  This 
is completely normal in software development, and there is no "open source" 
duty to continue to maintain software that is no longer useful (or, is broken 
or is unmaintained or is not longer best practices or is no longer commercially 
viable or is duplicative of other features that exist or )  That decision 
is what is being done for a selection of longstanding Dovecot features.  It is 
time to move on from them.  There are valid reasons to do so.

If you disagree: the software is open source.  You can continue to use the 
existing software, adapt it to your needs, move to a different solution, or 
whatever else.  

To focus development efforts, and to provide extreme clarity for users going 
forward, Dovecot CE for the first time has adopted a defined Vision Statement: 
"To provide the world's premier open source, standards compliant, 
full-featured, single node email backend server."  This vision formulation was 
made to ensure that CE users continue to receive world class, stable, tested, 
modern, secure email software going forward.  Maintaining features that have 
existed since the mid-2000s (replication, Directors), at the expense of moving 
the software forward to adapt to new paradigms (cloud, containers, 
storage-layer replication, statelessness) is not the proper choice.

These Dovecot CE feature decisions are mine.  If you are unhappy with them, I 
ask that you direct your vitriol directly (and privately) to me.  The Dovecot 
Team does fantastic work and has provided software, under open source 
principles, that runs millions of email servers around the world.  They 
continue to provide invaluable feedback internally in determining the proper 
balance between open and commercial considerations.  They deserve to be thanked 
by the community, not vilified.

michael
___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: Replication going away?

[gene heskett]

On 7/17/23 12:21, Stephan Bosch wrote:


Op 17-7-2023 om 17:54 schreef Steven Varco:

However, I understand some had a better experience with it. I am curious
if someone will fork dovecot and restore the beloved feature.
With all the recent actions going on, clearly targeting in getting 
more paying „pro“ customers (and nothing else!) the dovecot project 
will walk on thin ice and there is a risk (or chance :) ) that it will 
get forked, all the good dovecot developers walking over to the fork 
and dovecot let down to a product no one uses anymore.
It happend before, i.ex. nagios/icinga, CentOS/Alma-/Rocky Linux and 
other big, commonly used software projects.



Sadly, Dovecot is a project with only a few prolific developers, all of 
which work for the company as far as I know. So, the scenario you 
describe is pretty unlikely. Maybe new community developers will stand 
up now, but that is something I think we can only welcome. And forks 
don't need to be adversarial like that.


Regards,

This is a point where we should all remember TANSTAAFL. Its a law you 
cannot break no matter how many useless MBA's you hire. Developers like 
to eat regular and they can't do that on nothing but our good will. 
Dovecot has been good to us, so much so its become the default mail 
server all over this pale blue dot.  Other than the monthly fees we pay 
our ISP's has any one of us donated a stock tank full of ice & beer for 
a summer picnic? Or $500 for a costly bug fixed in 2 hours?  TANSTAAFL 
folks, There really Ain't No Such Thing As A Free Lunch.


Stephan.

___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org


Cheers, Gene Heskett.
--
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author, 1940)
If we desire respect for the law, we must first make the law respectable.
 - Louis D. Brandeis
Genes Web page 

___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: Replication going away?

[Stephan Bosch]

Op 17-7-2023 om 17:54 schreef Steven Varco:

However, I understand some had a better experience with it. I am curious
if someone will fork dovecot and restore the beloved feature.

With all the recent actions going on, clearly targeting in getting more paying 
„pro“ customers (and nothing else!) the dovecot project will walk on thin ice 
and there is a risk (or chance :) ) that it will get forked, all the good 
dovecot developers walking over to the fork and dovecot let down to a product 
no one uses anymore.
It happend before, i.ex. nagios/icinga, CentOS/Alma-/Rocky Linux and other big, 
commonly used software projects.



Sadly, Dovecot is a project with only a few prolific developers, all of 
which work for the company as far as I know. So, the scenario you 
describe is pretty unlikely. Maybe new community developers will stand 
up now, but that is something I think we can only welcome. And forks 
don't need to be adversarial like that.


Regards,


Stephan.

___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: Replication going away?

[Steven Varco]

> However, I understand some had a better experience with it. I am curious
> if someone will fork dovecot and restore the beloved feature.

With all the recent actions going on, clearly targeting in getting more paying 
„pro“ customers (and nothing else!) the dovecot project will walk on thin ice 
and there is a risk (or chance :) ) that it will get forked, all the good 
dovecot developers walking over to the fork and dovecot let down to a product 
no one uses anymore.
It happend before, i.ex. nagios/icinga, CentOS/Alma-/Rocky Linux and other big, 
commonly used software projects.

I for my part will also try to stay as long as possible on dovecot 2.x with 
director and replicator, hoping for a fork which includes those features again 
in a distant future.

Steven

-- 
https://steven.varco.ch/ 
https://www.tech-island.com/ 

___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: Replication going away?

[Emmanuel Dreyfus]

On Sun, Jul 09, 2023 at 10:36:35PM +0300, Vladimir Mishonov via dovecot wrote:
> Looking at the commit details, it appears that it completely removes the
> replication feature.

I am not very upset with that news, since all I have been able to do
with replicator was loosing mail. 

However, I understand some had a better experience with it. I am curious
if someone will fork dovecot and restore the beloved feature.

-- 
Emmanuel Dreyfus
m...@netbsd.org
___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: Replication going away?

[Nikolaos Milas via dovecot]

On 17/7/2023 1:24 μ.μ., David Zambonini wrote:
With respect, I'm not sure why these scripts are considered a suitable 
replacement, because they're not, and it's obvious no real attempt was 
made to make them so.

...


+1000

We recognize and respect all Dovecot Team's efforts and goodwill for 
many years now. You guys rock.


But please do not mess thousands of mail setups around the globe where 
Replication is already in use.


Please continue the support of these really valuable - for a really huge 
number of mail admins - features.


I strongly believe that the vast majority of the community will be very 
disappointed by this feature removal and it will be a pity for such a 
rich and collaborative community.


My 2c.

All the best,
Nick

___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: Replication going away?

[Michael Grimm via dovecot]

Emmanuel Fusté  wrote:

> Le dim. 16 juil. 2023, 18:55, Aki Tuomi via dovecot  a 
> écrit :
>
>> Yes, director and replicator are removed, and won't be available for pro 
>> users either.

Why in hell would one remove replicator? It's working for years now. Yes, I 
recall issues in the beginning, and others and me helped Timo in 
debugging/testing. After that it runs without any flaws.

So why removing it?

>> Regards to replication, doveadm sync is not being removed. So you can still 
>> run 
>> doveadm sync on your system to have a primary / backup setup

AND: What do you believe an alternative should be, for a failover scenario of 
two IMAP servers?
doveadm sync is not! That's why replicator has been implemented!

> That's completely crazy ! 

+1

Regards,
Michael

___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: Replication going away?

[David Zambonini]

On 17/07/2023 11:41, Aki Tuomi wrote:


You do note that most of the issues you just described were actually
issues with director itself. The director did not do any automatic healing, 
load balancing etc, which is why we have a Pro offering that provides actual 
clustering component (called Palomar architecture).


I'd not raised those points. Load balancers handle first level load balancing in 
both scenarios, in any case. poolmon has always adequately managed healing.


To reiterate, we know director mappings are a live mapping set, not a permanent 
record; there's literally a timeout. Unless I'm in a hurry to re-equalise, I've 
seldom had to manually intervene. It sorts itself out in a few days.


What I've never had to do is constantly mess around in a database and kick users 
manually all the time as a _requirement_ just to keep things running. We both 
know it's not an adequate replacement for director.



Those Lua scripts are OK replacement for a do it yourself environment.


Putting aside the subtext of the retiring of enterprise features outside of Pro, 
with Pro the issue of the cost involved in being forced to shift to an entirely 
different architecture now exists.


--
David Zambonini



___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: Replication going away?

[Aki Tuomi via dovecot]

> On 17/07/2023 13:24 EEST David Zambonini  wrote:
> 
>  
> On 16/07/2023 17:54, Aki Tuomi via dovecot wrote:
> > Hi!
> > 
> > Yes, director and replicator are removed, and won't be available for pro 
> > users either.
> > 
> > For NFS setups (or similar shared setups), we have documented a way to use 
> > Lua to run a director-like setup, see
> > 
> > https://doc.dovecot.org/3.0/configuration_manual/howto/director_with_lua/
> 
> With respect, I'm not sure why these scripts are considered a suitable 
> replacement, because they're not, and it's obvious no real attempt was made 
> to 
> make them so.
> 
> Putting aside things relatively trivial to add, like weighted instead of 
> random 
> mappings, the meat of the issue is:
> 
> When a backend is removed (fails, or gracefully taken out), the script remaps 
> connecting users that were mapped to it to a different backend. This sounds 
> obvious enough.
> 
> However, when that backend comes back up, users aren't mapped back onto it - 
> the 
> users now have mappings elsewhere. Maybe you got lucky and some users that 
> were 
> mapped onto it just didn't connect the whole time it was down, maybe you got 
> a 
> few new users, but for the majority of your active user base, you now have a 
> large imbalance in your user mappings. The backend has stopped existing for 
> them.
> 
> So you have to rebalance your user mappings manually.
> 
> Rebalancing requires going through all your user mappings. This quickly 
> becomes 
> prohibitively expensive as number of users increases. However, it also leads 
> us 
> on to the next issue.
> 
> Adding a new backend, or returning a failed backend, requires remapping your 
> users, but you can't remap one that's currently connected to a backend 
> without 
> risking a new connection to the wrong backend.
> 
> Either you don't balance/add backends in a way that covers any connected user 
> going forward (which isn't really useful) or you have to start kicking your 
> users (on all of your balancers individually, you don't have the control 
> director gave you any more!) and remapping to rebalance the overall 
> configuration.
> 
> We know that a user will immediately attempt to reconnect when it's kicked, 
> this 
> is the nature of imap clients. You now have a race condition, the only 
> solution 
> to which is to lock that user out while you update the database. I'm not even 
> sure how you'd cover this.
> 
> So it's all gone from automatic to painfully manual with large overheads and 
> race conditions.
> 
> At the very least, to retain the very basic level of suitability would 
> require:
> 
> 1. mappings in the database being given a TTL after which they're no longer 
> considered valid. This is trivial, however:
> 2. importantly, mappings exceeding TTL must *still* be considered valid if 
> the 
> user has existing connections on any balancer. Good luck with that!
> 
> And all this doesn't even cover the other deficiencies. Some easy enough to 
> add, 
> some (involving addressing all balancers at once) far less so.
> 
> Unfortunately I can't speak for our future plans; but I'd personally remain 
> on 
> 2.x director for the proxies for as long as possible, then potentially look 
> at 
> the feasibility of porting the director code to 3.x.
> 
> Again, I really don't understand the thinking behind saying the lua scripts 
> are 
> anything like a suitable replacement. Our team were considering using Pro 
> when 
> the decision to drop director was announced, we backed off quickly as a 
> result.
> 
> -- 
> David Zambonini
> 
> 

You do note that most of the issues you just described were actually
issues with director itself. The director did not do any automatic healing, 
load balancing etc, which is why we have a Pro offering that provides actual 
clustering component (called Palomar architecture).

Those Lua scripts are OK replacement for a do it yourself environment.

Aki
___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: Replication going away?

[David Zambonini]

On 16/07/2023 17:54, Aki Tuomi via dovecot wrote:

Hi!

Yes, director and replicator are removed, and won't be available for pro users 
either.

For NFS setups (or similar shared setups), we have documented a way to use Lua 
to run a director-like setup, see

https://doc.dovecot.org/3.0/configuration_manual/howto/director_with_lua/


With respect, I'm not sure why these scripts are considered a suitable 
replacement, because they're not, and it's obvious no real attempt was made to 
make them so.


Putting aside things relatively trivial to add, like weighted instead of random 
mappings, the meat of the issue is:


When a backend is removed (fails, or gracefully taken out), the script remaps 
connecting users that were mapped to it to a different backend. This sounds 
obvious enough.


However, when that backend comes back up, users aren't mapped back onto it - the 
users now have mappings elsewhere. Maybe you got lucky and some users that were 
mapped onto it just didn't connect the whole time it was down, maybe you got a 
few new users, but for the majority of your active user base, you now have a 
large imbalance in your user mappings. The backend has stopped existing for them.


So you have to rebalance your user mappings manually.

Rebalancing requires going through all your user mappings. This quickly becomes 
prohibitively expensive as number of users increases. However, it also leads us 
on to the next issue.


Adding a new backend, or returning a failed backend, requires remapping your 
users, but you can't remap one that's currently connected to a backend without 
risking a new connection to the wrong backend.


Either you don't balance/add backends in a way that covers any connected user 
going forward (which isn't really useful) or you have to start kicking your 
users (on all of your balancers individually, you don't have the control 
director gave you any more!) and remapping to rebalance the overall configuration.


We know that a user will immediately attempt to reconnect when it's kicked, this 
is the nature of imap clients. You now have a race condition, the only solution 
to which is to lock that user out while you update the database. I'm not even 
sure how you'd cover this.


So it's all gone from automatic to painfully manual with large overheads and 
race conditions.


At the very least, to retain the very basic level of suitability would require:

1. mappings in the database being given a TTL after which they're no longer 
considered valid. This is trivial, however:
2. importantly, mappings exceeding TTL must *still* be considered valid if the 
user has existing connections on any balancer. Good luck with that!


And all this doesn't even cover the other deficiencies. Some easy enough to add, 
some (involving addressing all balancers at once) far less so.


Unfortunately I can't speak for our future plans; but I'd personally remain on 
2.x director for the proxies for as long as possible, then potentially look at 
the feasibility of porting the director code to 3.x.


Again, I really don't understand the thinking behind saying the lua scripts are 
anything like a suitable replacement. Our team were considering using Pro when 
the decision to drop director was announced, we backed off quickly as a result.


--
David Zambonini








___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: Replication going away?

[William Edwards via dovecot]

> Op 16 jul. 2023 om 17:57 heeft Aki Tuomi  het 
> volgende geschreven:
> 
> Hi!
> 
> Yes, director and replicator are removed, and won't be available for pro 
> users either.
> 
> For NFS setups (or similar shared setups), we have documented a way to use 
> Lua to run a director-like setup, see 
> 
> https://doc.dovecot.org/3.0/configuration_manual/howto/director_with_lua/ 

It would probably be good to share this link in the aforementioned Director 
removal thread. 

> 
> Regards to replication, doveadm sync is not being removed. So you can still 
> run doveadm sync on your system to have a primary / backup setup.
> 
> Aki
> 
>> On 16/07/2023 18:34 EEST William Edwards via dovecot  
>> wrote:
>> 
>> 
>> Top posting because nothing specific to reply to, sorry. Not exactly sure, 
>> but there’s another thread about the removal of Director in favour of 
>> Dovecot Pro on 3.x. Perhaps this change is related.
>> 
>> William Edwards
>> 
 Op 16 jul. 2023 om 16:33 heeft Daniele  het 
 volgende geschreven:
>>> 
>>> Hello,
>>> 
>>> Just like Vladimir, I'm a bit concerned about this change, and I'd really 
>>> appreciate if someone could let us know if the replication feature (that 
>>> works so well!) will be replaced or removed; and, in case of removal, what 
>>> would be recommended replacement?
>>> Thanks in advance and best regards,
>>> Daniele
>>> 
 On 09-Jul-23 9:36 PM, Vladimir Mishonov via dovecot wrote:
 Hello everyone.
 
 Just saw this commit in the official Github repo:
 
 https://github.com/dovecot/core/commit/4c04e4c30fd4817a8b0e11d04d9681173f696f41#diff-5f643d8b0d1eea65d0f3c749d14d42b25a9d60f0f149bface862f5ff348412c8
  
 
 Looking at the commit details, it appears that it completely removes the 
 replication feature. I'm a bit perplexed by this change and am not sure 
 what might be the justification for it. Personally, I find replication to 
 be very useful, as it allows me to maintain a synchronized mirror of all 
 of my mailboxes on my home server, for use as backup in case the primary 
 server goes down for some reason.
 
 Perhaps there's some sort of replacement being planned for this feature? 
 Or maybe the relevant code is simply going to be refactored to a plugin or 
 external program, and there's nothing to worry about at all?
 
 In any case, I'd greatly appreciate if one of the developers could comment 
 on this change.
 
>>> 
>>> ___
>>> dovecot mailing list -- dovecot@dovecot.org
>>> To unsubscribe send an email to dovecot-le...@dovecot.org
>>> 
>> 
>> ___
>> dovecot mailing list -- dovecot@dovecot.org
>> To unsubscribe send an email to dovecot-le...@dovecot.org
> 

___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: Replication going away?

[Emmanuel Fusté]

Le dim. 16 juil. 2023, 18:55, Aki Tuomi via dovecot  a
écrit :

> Hi!
>
> Yes, director and replicator are removed, and won't be available for pro
> users either.
>
> For NFS setups (or similar shared setups), we have documented a way to use
> Lua to run a director-like setup, see
>
> https://doc.dovecot.org/3.0/configuration_manual/howto/director_with_lua/
>
> Regards to replication, doveadm sync is not being removed. So you can
> still run doveadm sync on your system to have a primary / backup setup
>

That's completely crazy ! What is the rationale ?

Emmanuel.

>
___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: Replication going away?

[Jim Popovitch via dovecot]

On Sun, 2023-07-16 at 19:54 +0300, Aki Tuomi via dovecot wrote:
> Hi!
> 
> Yes, director and replicator are removed

Are you able to provide details on why this is happening?

-Jim P.

___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: Replication going away?

[Aki Tuomi via dovecot]

Hi!

Yes, director and replicator are removed, and won't be available for pro users 
either.

For NFS setups (or similar shared setups), we have documented a way to use Lua 
to run a director-like setup, see 

https://doc.dovecot.org/3.0/configuration_manual/howto/director_with_lua/ 

Regards to replication, doveadm sync is not being removed. So you can still run 
doveadm sync on your system to have a primary / backup setup.

Aki

> On 16/07/2023 18:34 EEST William Edwards via dovecot  
> wrote:
> 
>  
> Top posting because nothing specific to reply to, sorry. Not exactly sure, 
> but there’s another thread about the removal of Director in favour of Dovecot 
> Pro on 3.x. Perhaps this change is related.
> 
> William Edwards
> 
> > Op 16 jul. 2023 om 16:33 heeft Daniele  het volgende 
> > geschreven:
> > 
> > Hello,
> > 
> > Just like Vladimir, I'm a bit concerned about this change, and I'd really 
> > appreciate if someone could let us know if the replication feature (that 
> > works so well!) will be replaced or removed; and, in case of removal, what 
> > would be recommended replacement?
> > Thanks in advance and best regards,
> > Daniele
> > 
> >> On 09-Jul-23 9:36 PM, Vladimir Mishonov via dovecot wrote:
> >> Hello everyone.
> >> 
> >> Just saw this commit in the official Github repo:
> >> 
> >> https://github.com/dovecot/core/commit/4c04e4c30fd4817a8b0e11d04d9681173f696f41#diff-5f643d8b0d1eea65d0f3c749d14d42b25a9d60f0f149bface862f5ff348412c8
> >>  
> >> 
> >> Looking at the commit details, it appears that it completely removes the 
> >> replication feature. I'm a bit perplexed by this change and am not sure 
> >> what might be the justification for it. Personally, I find replication to 
> >> be very useful, as it allows me to maintain a synchronized mirror of all 
> >> of my mailboxes on my home server, for use as backup in case the primary 
> >> server goes down for some reason.
> >> 
> >> Perhaps there's some sort of replacement being planned for this feature? 
> >> Or maybe the relevant code is simply going to be refactored to a plugin or 
> >> external program, and there's nothing to worry about at all?
> >> 
> >> In any case, I'd greatly appreciate if one of the developers could comment 
> >> on this change.
> >> 
> > 
> > ___
> > dovecot mailing list -- dovecot@dovecot.org
> > To unsubscribe send an email to dovecot-le...@dovecot.org
> > 
> 
> ___
> dovecot mailing list -- dovecot@dovecot.org
> To unsubscribe send an email to dovecot-le...@dovecot.org
___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: Replication going away?

[William Edwards via dovecot]

Top posting because nothing specific to reply to, sorry. Not exactly sure, but 
there’s another thread about the removal of Director in favour of Dovecot Pro 
on 3.x. Perhaps this change is related.

William Edwards

> Op 16 jul. 2023 om 16:33 heeft Daniele  het volgende 
> geschreven:
> 
> Hello,
> 
> Just like Vladimir, I'm a bit concerned about this change, and I'd really 
> appreciate if someone could let us know if the replication feature (that 
> works so well!) will be replaced or removed; and, in case of removal, what 
> would be recommended replacement?
> Thanks in advance and best regards,
> Daniele
> 
>> On 09-Jul-23 9:36 PM, Vladimir Mishonov via dovecot wrote:
>> Hello everyone.
>> 
>> Just saw this commit in the official Github repo:
>> 
>> https://github.com/dovecot/core/commit/4c04e4c30fd4817a8b0e11d04d9681173f696f41#diff-5f643d8b0d1eea65d0f3c749d14d42b25a9d60f0f149bface862f5ff348412c8
>>  
>> 
>> Looking at the commit details, it appears that it completely removes the 
>> replication feature. I'm a bit perplexed by this change and am not sure what 
>> might be the justification for it. Personally, I find replication to be very 
>> useful, as it allows me to maintain a synchronized mirror of all of my 
>> mailboxes on my home server, for use as backup in case the primary server 
>> goes down for some reason.
>> 
>> Perhaps there's some sort of replacement being planned for this feature? Or 
>> maybe the relevant code is simply going to be refactored to a plugin or 
>> external program, and there's nothing to worry about at all?
>> 
>> In any case, I'd greatly appreciate if one of the developers could comment 
>> on this change.
>> 
> 
> ___
> dovecot mailing list -- dovecot@dovecot.org
> To unsubscribe send an email to dovecot-le...@dovecot.org
> 

___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: Replication going away?

[Daniele]

Hello,

Just like Vladimir, I'm a bit concerned about this change, and I'd 
really appreciate if someone could let us know if the replication 
feature (that works so well!) will be replaced or removed; and, in case 
of removal, what would be recommended replacement?

Thanks in advance and best regards,
Daniele

On 09-Jul-23 9:36 PM, Vladimir Mishonov via dovecot wrote:

Hello everyone.

Just saw this commit in the official Github repo:

https://github.com/dovecot/core/commit/4c04e4c30fd4817a8b0e11d04d9681173f696f41#diff-5f643d8b0d1eea65d0f3c749d14d42b25a9d60f0f149bface862f5ff348412c8 



Looking at the commit details, it appears that it completely removes 
the replication feature. I'm a bit perplexed by this change and am not 
sure what might be the justification for it. Personally, I find 
replication to be very useful, as it allows me to maintain a 
synchronized mirror of all of my mailboxes on my home server, for use 
as backup in case the primary server goes down for some reason.


Perhaps there's some sort of replacement being planned for this 
feature? Or maybe the relevant code is simply going to be refactored 
to a plugin or external program, and there's nothing to worry about at 
all?


In any case, I'd greatly appreciate if one of the developers could 
comment on this change.




___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org