Re: ot: how to t/s TBird problems ?
Voytek Eymont writes: > I've enabled logging as per your suggestion: > > -rw--- 1 vmail vmail 127 Oct 16 21:38 20221016-213738.25640.1.in > -rw--- 1 vmail vmail 8546603 Oct 16 21:38 20221016-213738.25640.1.out > -rw--- 1 vmail vmail 96 Oct 16 21:58 20221016-215757.26075.1.in > -rw--- 1 vmail vmail 8343463 Oct 16 21:58 20221016-215757.26075.1.out > > # cat 20221016-213738.25640.1.in > 1665916659.491025 STAT > 1665916659.550829 LIST > 1665916676.430794 UIDL > 1665916693.761281 RETR 114437 > 1665916694.440965 QUIT > # cat 20221016-215757.26075.1.in > 1665917878.786953 STAT > 1665917878.863136 LIST > 1665917905.610805 UIDL > 1665917924.491198 QUIT > # > > what should I look in the .out file ? > > some of the file is like: > > 1665916661.234807 114436 70097 > 1665916661.234814 114437 154498 > 1665916661.234821 . > 1665916676.430870 +OK > 1665916676.981415 1 24b95283283a > 1665916676.981459 2 24ba5283283a > > > 1665916679.434297 114436 00033fcf5283283a > 1665916679.434327 114437 00033fd05283283a > 1665916679.434349 . > 1665916694.048139 +OK 154498 octets > 1665916694.048199 Return-Path: > I haven't seen anyone else replying, but there doesn't seem anything anomalous with the output. The session commands-repliesd is is more or less what I expect, although to make sense of this, you'll have to splice the input and output files together using timestamps to see the sequential flow of data. I forget what the symptoms you originally reported, but theoretically, you could simulate either client or server by feeding in the above data and see how the other end behaves. If dovecot is serving out the correct data, then TB is somehow misinterpreting it. > on an uneducated guess, the mailbox is just 'too large' ? > POP has difficulty handling so many files ? Typically, if some resource limit is hit, one side or the other will create a log or notification. Your INBOX is large, but not outrageous. You can test it directly by creating smaller subsets of the INBOX messages and see if the problem goes away. Joseph Tam
Re: The end of Dovecot Director?
On 2022-10-21 11:38, Heiko Schlittermann wrote: Apparently, Dovecot Director is going to be removed in the next major version of Dovecot and the commercial Dovecot cluster architecture will be its successor: We - the communitiy - are free to continue development of the director. So, who's going to fork dovecot (director)? Ciao - Frank
Re: dovecot mailing list (this mailing list), DKIM, SPF and DMARC
Trojitá, a fast Qt IMAP e-mail client http://www.trojita.flaska.net/ I also use http://opendkim.org/ http://www.trusteddomain.org/opendmarc/ as milters on Postfix Active development, I'm sure they could all use some help, or forks for alternatives, I don't know, I'm not involved in development per se, just a user, and I have to get off the property of any of these places with my code before anything happens. All that Finnish osalliyhdistys and by the time a Swede gets online all hell breaks loose./ On Friday, October 21, 2022 1:50:43 PM AKDT, hi@zakaria.website wrote: On 2022-10-11 14:05, Benny Pedersen wrote: hi@zakaria.website skrev den 2022-10-11 13:42: ... Indeed, it's because you set the following headers in dkim signing headers:- from : subject : date : to : message-id Although not sure why you've added some space, as per standards I think only colon separated list its the compliant format like the following:- from:subject:date:to:message-id Anyhow this is my final update, the previous headers set which I included wasnt perfect as cc header was causing a trouble, given it can fail at some point e.g. when replying more than one time to the same recipient through a mailing list, and mind me OX and iRedMail, I had to check your signing headers set, hopefully you are ok for me to present it here as the optimal one to avoid DKIM failures:- OX:- Date:From:To:In-Reply-To:References:Subject:From IRM:- x-mailer:message-id:in-reply-to:to:references:date:subject :mime-version:content-transfer-encoding:content-type:from iRedMail seems to be the best headers set given it includes X-Mailer header, which enhances signature validity, when client uses specific mail client app, although it can be faked yet one must know which client app the sender would use and if was able to have information to this length I guess signature validity would be an easy task to break it further. Also, I was advised by a friend to duplicate the signing headers in order to disallow spoofing signature further, while I couldnt see how nor populate a proof of concept, I removed it but if someone understand it, I would appreciate their elaboration, surely with thanks :) Good luck. Zakaria.
Re: dovecot mailing list (this mailing list), DKIM, SPF and DMARC
On 2022-10-11 14:05, Benny Pedersen wrote: hi@zakaria.website skrev den 2022-10-11 13:42: On 2022-09-13 13:10, Benny Pedersen wrote: hi@zakaria.website skrev den 2022-09-13 14:03: from:from:reply-to:date:date:message-id:message-id:to:to:cc: mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references Thanks to my friend who didnt need a credit, and helped me out in reaching this solution. i have no frinds, but it might be related https://gitlab.com/fumail/fuglu/-/issues/262 with my conservative list of signed headers it pass Indeed, it's because you set the following headers in dkim signing headers:- from : subject : date : to : message-id Although not sure why you've added some space, as per standards I think only colon separated list its the compliant format like the following:- from:subject:date:to:message-id Anyhow this is my final update, the previous headers set which I included wasnt perfect as cc header was causing a trouble, given it can fail at some point e.g. when replying more than one time to the same recipient through a mailing list, and mind me OX and iRedMail, I had to check your signing headers set, hopefully you are ok for me to present it here as the optimal one to avoid DKIM failures:- OX:- Date:From:To:In-Reply-To:References:Subject:From IRM:- x-mailer:message-id:in-reply-to:to:references:date:subject :mime-version:content-transfer-encoding:content-type:from iRedMail seems to be the best headers set given it includes X-Mailer header, which enhances signature validity, when client uses specific mail client app, although it can be faked yet one must know which client app the sender would use and if was able to have information to this length I guess signature validity would be an easy task to break it further. Also, I was advised by a friend to duplicate the signing headers in order to disallow spoofing signature further, while I couldnt see how nor populate a proof of concept, I removed it but if someone understand it, I would appreciate their elaboration, surely with thanks :) Good luck. Zakaria.
Re: The end of Dovecot Director?
On 2022-10-21 10:54, Zhang Huangbin wrote: On Oct 21, 2022, at 5:51 PM, Zhang Huangbin wrote: If mailbox is in Maildir format (and stored on shared storage like NFS), accessing it from different server may corrupt Dovecot index files and mailbox becomes unaccessible. Director perfectly avoids this issue. To be clear: Accessing same mailbox from different IMAP servers __at the same time__. Zhang Huangbin, founder of: - iRedMail: Open source email server solution: https://www.iredmail.org/ - Spider: Lightweight, on-premises Email Archiving Software: https://spiderd.io Thanks :)
Re: The end of Dovecot Director?
the problem that prevents most load balancers from handling the backend imap/pop traffic is that the load balancer needs to be aware of the context of each connection. which all boils down to the index files (only a single dovecot server can access a set of index files concurrently, else the indexes will get corrupted) As someone else asked on this thread, what prevents two clients, both being directed to the same server, from fighting over index files? Wouldn't file locks over NFS prevent this problem? And if so, doesn't that also prevent two dovecot installations from fighting over index files? What is a way to test your system to know if dovecot is using the default fcntl file locks over NFS4 and they actually work? Or is it better/safer to use dotlock on NFS4 without director?
Re: The end of Dovecot Director?
I setup load-balance cluster for clients with HAProxy + KeepAlived + Dovecot Director running in frontend servers, so sad we have to find an alternative to replace Director in such case. It's not about "small/medium" servers, but the demand of imap/pop3/lmtp proxy service, especially in load-balance cluster. Curious, trying to understand.. Why would not a true load balancer not be an attractive option for those that need to load balance services across multiple front ends? It is the model we use with most of our ISP's and scales very well. The choice of load balancer is important, but with HA load balancers, you are assured that you don't have a single point of failure, and you can spread loads more granularly, eg POP, IMAP and other services. Not to mention, you can use the same load balancer from many other traffic shaping solutions. the problem that prevents most load balancers from handling the backend imap/pop traffic is that the load balancer needs to be aware of the context of each connection. which all boils down to the index files (only a single dovecot server can access a set of index files concurrently, else the indexes will get corrupted) in more usual HTTP case, you'd probably use some sort of cookie based session affinity to keep connections from a particular user going to the same backend http server. but in the IMAP/POP case most load balancers don't really know anything about the connection and are just blindly forwarding them to the backend nodes. director (or the custom nginx LB setups) get to handle part of the IMAP/POP transaction and get a bit of context (knowing which user the connection is for) to then make additional decisions about which backend imap node to send the connection through to (preventing the index corruption problem). you could use IP based affinity on pop/imap connections for a context-unaware load balancer, but if you end up with a lot of NAT users your connections will end up being unbalanced across the backend servers. and connections from something like a webmail server will all end up going to the same backend server (since they'd all come from the same IP address). you could also just have a dumb load balancer sitting in front and just randomly sending the connections to any backend imap server, but each backend imap server would have to maintain its own copy of the indexes. workable, but not particularly efficient, especially if you have large a large number of backend imap servers (though, with a small setup with only 2 or 3 backend imap servers for redundancy instead of performance, probably acceptable) you'd still want some sort of load balanced director or nginx pool as well, in order to handle redundancy at that level. but that's a much easier task, as you don't have to worry about the session context at that point. (we have hardware load balancers in front of the director nodes)
Re: The end of Dovecot Director?
> Op 21 okt. 2022 om 19:42 heeft Brendan Braybrook het
> volgende geschreven:
>
> On 2022-10-21 04:29, spi wrote:
>>> Am 21.10.22 um 13:14 schrieb Amol Kulkarni:
>>> Nginx has an mail proxy for pop, imap, smtp.
>>> Can it be used instead of director ?
>> Nginx can authenticate imap/smtp (and probably pop3) users. If you that, you
>> can define a backend server the session is routed to. Currently I use that
>> approach to authenticate users by client certificates and route them to the
>> appriopriate backend (well, I only have one ;-).
>
> we've recently switched to director, but we used to use nginx for this as
> well (we started using nginx before director existed). if you load balance
> the nginx proxies themselves, you can easily handle hundreds of thousands of
> concurrent imap connections with them.
>
> in debian/ubuntu, i don't think the nginx packages include the mail proxy
> bits. iirc, we had to compile nginx ourselves with the mail proxy bits
> included.
>
> the nginx config is pretty simple, you have to pre-specifiy the capabilities
> for each protocol and set up some sort of way for nginx to auth and get which
> backend node to send to as spi notes (in this example, it's an http call):
>
> mail {
> auth_http localhost:8080/cgi-bin/auth;
> proxy_pass_error_message on;
>
> pop3_capabilities "TOP" "UIDL" "RESP-CODES" "PIPELINING" "AUTH-RESP-CODE"
> "USER" "SASL PLAIN" "SASL PLAIN LOGIN";
> server {
>listen 110;
>protocol pop3;
>proxyon;
> }
>
> imap_capabilities "IMAP4rev1" "LITERAL+" "SASL-IR" "LOGIN-REFERRALS" "IDLE";
> server {
>listen 143;
>protocol imap;
>proxyon;
> }
> }
>
> localhost:8080/cgi-bin/auth then just auths the user/pass that nginx gets
> from the incoming request and returns success and the next hop for nginx to
> proxy to.
>
> the only real difficulty is that you then need to write your own state system
> into your cgi auth script to ensure that users get sent to the same backend
> imap server if they already have an existing connection and have some way to
> safely fail over to other backend imap servers should one go down. (it's nice
> to have director handle this state stuff for you)
Although Director does not do health checks and down servers automatically. I
was working on an open source program for that (as an alternative to Dovemon),
but that plan is canceled with this announcement :)
Re: The end of Dovecot Director?
On 2022-10-21 04:29, spi wrote:
Am 21.10.22 um 13:14 schrieb Amol Kulkarni:
Nginx has an mail proxy for pop, imap, smtp.
Can it be used instead of director ?
Nginx can authenticate imap/smtp (and probably pop3) users. If you that,
you can define a backend server the session is routed to. Currently I
use that approach to authenticate users by client certificates and route
them to the appriopriate backend (well, I only have one ;-).
we've recently switched to director, but we used to use nginx for this
as well (we started using nginx before director existed). if you load
balance the nginx proxies themselves, you can easily handle hundreds of
thousands of concurrent imap connections with them.
in debian/ubuntu, i don't think the nginx packages include the mail
proxy bits. iirc, we had to compile nginx ourselves with the mail proxy
bits included.
the nginx config is pretty simple, you have to pre-specifiy the
capabilities for each protocol and set up some sort of way for nginx to
auth and get which backend node to send to as spi notes (in this
example, it's an http call):
mail {
auth_http localhost:8080/cgi-bin/auth;
proxy_pass_error_message on;
pop3_capabilities "TOP" "UIDL" "RESP-CODES" "PIPELINING"
"AUTH-RESP-CODE" "USER" "SASL PLAIN" "SASL PLAIN LOGIN";
server {
listen 110;
protocol pop3;
proxyon;
}
imap_capabilities "IMAP4rev1" "LITERAL+" "SASL-IR" "LOGIN-REFERRALS"
"IDLE";
server {
listen 143;
protocol imap;
proxyon;
}
}
localhost:8080/cgi-bin/auth then just auths the user/pass that nginx
gets from the incoming request and returns success and the next hop for
nginx to proxy to.
the only real difficulty is that you then need to write your own state
system into your cgi auth script to ensure that users get sent to the
same backend imap server if they already have an existing connection and
have some way to safely fail over to other backend imap servers should
one go down. (it's nice to have director handle this state stuff for you)
Re: The end of Dovecot Director?
Nginx is an excellent suggestion for the purpose. However I do not like German client certificates. That is far too much "proof" of identification 18/21++ on a public network with nowhere to hide and those of us who are not German citizens and do not have the advantage of a friendly local police jurisdiction with massive international clout and an assumed legitimacy for all the online surveillance, policing, and copping with unfounded sex charges etc. being pressed online. Not that I care much for alcohol, but the analogy that comes to mind with such "proof" of identity presented across the internet as a public certificate is that of "public drunkenness," versus, say, "drinking privately in one's quarters," i.e., making an encrypted connection, and only then within the encrypted channel establishing identity and authorization with a username and password or other means of authentication. On Friday, October 21, 2022 3:29:36 AM AKDT, spi wrote: Am 21.10.22 um 13:14 schrieb Amol Kulkarni: Nginx has an mail proxy for pop, imap, smtp. Can it be used instead of director ? Nginx can authenticate imap/smtp (and probably pop3) users. If you that, you can define a backend server the session is routed to. Currently I use that approach to authenticate users by client certificates and route them to the appriopriate backend (well, I only have one ;-). -- Cheers spi
Re: The end of Dovecot Director?
On 2022-10-20 22:19, Zhang Huangbin wrote: On Oct 21, 2022, at 4:19 AM, Antonio Leding wrote: My understanding is that Director is targeted toward large enterprise mail installations that will incorporate several servers for a given function. In such an environment, Director would be the fore-person\traffic-cop keeping things organized & squared-away. Director is used when you setup frontend servers in a load-balance cluster, proxy imap/pop3/lmtp/managesieve requests to backend Dovecot servers. I setup load-balance cluster for clients with HAProxy + KeepAlived + Dovecot Director running in frontend servers, so sad we have to find an alternative to replace Director in such case. It's not about "small/medium" servers, but the demand of imap/pop3/lmtp proxy service, especially in load-balance cluster. Zhang Huangbin, founder of: - iRedMail: Open source email server solution: https://www.iredmail.org/ - Spider: Lightweight, on-premises Email Archiving Software: https://spiderd.io Curious, trying to understand.. Why would not a true load balancer not be an attractive option for those that need to load balance services across multiple front ends? It is the model we use with most of our ISP's and scales very well. The choice of load balancer is important, but with HA load balancers, you are assured that you don't have a single point of failure, and you can spread loads more granularly, eg POP, IMAP and other services. Not to mention, you can use the same load balancer from many other traffic shaping solutions. -- "Catch the Magic of Linux..." Michael Peddemors, President/CEO LinuxMagic Inc. Visit us at http://www.linuxmagic.com @linuxmagic A Wizard IT Company - For More Info http://www.wizard.ca "LinuxMagic" a Registered TradeMark of Wizard Tower TechnoServices Ltd. 604-682-0300 Beautiful British Columbia, Canada This email and any electronic data contained are confidential and intended solely for the use of the individual or entity to which they are addressed. Please note that any views or opinions presented in this email are solely those of the author and are not intended to represent those of the company.
Re: The end of Dovecot Director?
To be clear, you are removing the Director... --- Tom On 2022-10-21 13:28, Aki Tuomi wrote: To be clear, we are not removing proxying features from Dovecot either. Just the director ring feature. Aki On 21/10/2022 14:14 EEST Amol Kulkarni wrote: Nginx has an mail proxy for pop, imap, smtp. Can it be used instead of director ? On Fri, 21 Oct 2022 at 16:21, wrote: > On 2022-10-21 10:51, Zhang Huangbin wrote: > >> On Oct 21, 2022, at 5:23 PM, hi@zakaria.website wrote: > >> > >> I was wondering if one can achieve the same implementation with > >> haproxy without dovecot director? > > > > The most important part of Director is it makes sure same mail user > > always proxied to same backend IMAP server. > > > > If mailbox is in Maildir format (and stored on shared storage like > > NFS), accessing it from different server may corrupt Dovecot index > > files and mailbox becomes unaccessible. Director perfectly avoids this > > issue. > > > > HAProxy can proxy mail user from same client IP to same backend IMAP > > server, but not same mail user from different IPs. > > > > Quote (https://doc.dovecot.org/admin_manual/director/dovecotdirector/): > > > > "Director can be used by Dovecot’s IMAP/POP3/LMTP proxy to keep a > > temporary user -> mail server mapping. As long as user has simultaneous > > connections, the user is always redirected to the same server. Each > > proxy server is running its own director process, and the directors are > > communicating the state to each others. Directors are mainly useful for > > setups where all of the mail storage is seen by all servers, such as > > with NFS or a cluster filesystem." > > > > > > Zhang Huangbin, founder of: > > - iRedMail: Open source email server solution: > > https://www.iredmail.org/ > > - Spider: Lightweight, on-premises Email Archiving Software: > > https://spiderd.io > > Aha makes sense, although I was not able to see how can index files be > corrupted when its if will going to be updated, its in same manner as > from different connection, e.g. opening email account from different app > clients, with different connections, does not corrupt the index files? > > Also, Is it the issue Director resolving as well its with maintaining > the logged in dovecot connection to same backend? Anyhow, thanks for > your valuable efforts in clearing this :) > > I wondered if there is any other solution to avoid corrupting index > files? Perhaps if dovecot offer database indexing as well as login > sessions, it seems that this would eliminate Director requirement, and > offer better high availability, as for now userdb/authdb is only > available per my knowledge, and using database cluster resolves the > issue with user and auth queries during simultaneous connections to a > different backends. > > Otherwise, it seems in large enterprise deployment with high > availability a Director implementation will be needed, hopefully we will > find an alternative solution by the time Dovecot 3 is released. > > I might need to get my head around building dovecot with customised > modules and review the code which was removed and return it back, if > anyone is planning to this, and well off ahead of me, please let me > know, we might be able to help one another. > > With thanks. > > Zakaria. >
Re: The end of Dovecot Director?
Am 21.10.22 um 13:14 schrieb Amol Kulkarni: Nginx has an mail proxy for pop, imap, smtp. Can it be used instead of director ? Nginx can authenticate imap/smtp (and probably pop3) users. If you that, you can define a backend server the session is routed to. Currently I use that approach to authenticate users by client certificates and route them to the appriopriate backend (well, I only have one ;-). -- Cheers spi
Re: The end of Dovecot Director?
To be clear, we are not removing proxying features from Dovecot either. Just the director ring feature. Aki > On 21/10/2022 14:14 EEST Amol Kulkarni wrote: > > > Nginx has an mail proxy for pop, imap, smtp. > Can it be used instead of director ? > > > On Fri, 21 Oct 2022 at 16:21, wrote: > > On 2022-10-21 10:51, Zhang Huangbin wrote: > > >> On Oct 21, 2022, at 5:23 PM, hi@zakaria.website wrote: > > >> > > >> I was wondering if one can achieve the same implementation with > > >> haproxy without dovecot director? > > > > > > The most important part of Director is it makes sure same mail user > > > always proxied to same backend IMAP server. > > > > > > If mailbox is in Maildir format (and stored on shared storage like > > > NFS), accessing it from different server may corrupt Dovecot index > > > files and mailbox becomes unaccessible. Director perfectly avoids this > > > issue. > > > > > > HAProxy can proxy mail user from same client IP to same backend IMAP > > > server, but not same mail user from different IPs. > > > > > > Quote (https://doc.dovecot.org/admin_manual/director/dovecotdirector/): > > > > > > "Director can be used by Dovecot’s IMAP/POP3/LMTP proxy to keep a > > > temporary user -> mail server mapping. As long as user has simultaneous > > > connections, the user is always redirected to the same server. Each > > > proxy server is running its own director process, and the directors are > > > communicating the state to each others. Directors are mainly useful for > > > setups where all of the mail storage is seen by all servers, such as > > > with NFS or a cluster filesystem." > > > > > > > > > Zhang Huangbin, founder of: > > > - iRedMail: Open source email server solution: > > > https://www.iredmail.org/ > > > - Spider: Lightweight, on-premises Email Archiving Software: > > > https://spiderd.io > > > > Aha makes sense, although I was not able to see how can index files be > > corrupted when its if will going to be updated, its in same manner as > > from different connection, e.g. opening email account from different app > > clients, with different connections, does not corrupt the index files? > > > > Also, Is it the issue Director resolving as well its with maintaining > > the logged in dovecot connection to same backend? Anyhow, thanks for > > your valuable efforts in clearing this :) > > > > I wondered if there is any other solution to avoid corrupting index > > files? Perhaps if dovecot offer database indexing as well as login > > sessions, it seems that this would eliminate Director requirement, and > > offer better high availability, as for now userdb/authdb is only > > available per my knowledge, and using database cluster resolves the > > issue with user and auth queries during simultaneous connections to a > > different backends. > > > > Otherwise, it seems in large enterprise deployment with high > > availability a Director implementation will be needed, hopefully we will > > find an alternative solution by the time Dovecot 3 is released. > > > > I might need to get my head around building dovecot with customised > > modules and review the code which was removed and return it back, if > > anyone is planning to this, and well off ahead of me, please let me > > know, we might be able to help one another. > > > > With thanks. > > > > Zakaria. > >
Re: The end of Dovecot Director?
Nginx has an mail proxy for pop, imap, smtp. Can it be used instead of director ? On Fri, 21 Oct 2022 at 16:21, wrote: > On 2022-10-21 10:51, Zhang Huangbin wrote: > >> On Oct 21, 2022, at 5:23 PM, hi@zakaria.website wrote: > >> > >> I was wondering if one can achieve the same implementation with > >> haproxy without dovecot director? > > > > The most important part of Director is it makes sure same mail user > > always proxied to same backend IMAP server. > > > > If mailbox is in Maildir format (and stored on shared storage like > > NFS), accessing it from different server may corrupt Dovecot index > > files and mailbox becomes unaccessible. Director perfectly avoids this > > issue. > > > > HAProxy can proxy mail user from same client IP to same backend IMAP > > server, but not same mail user from different IPs. > > > > Quote (https://doc.dovecot.org/admin_manual/director/dovecotdirector/): > > > > "Director can be used by Dovecot’s IMAP/POP3/LMTP proxy to keep a > > temporary user -> mail server mapping. As long as user has simultaneous > > connections, the user is always redirected to the same server. Each > > proxy server is running its own director process, and the directors are > > communicating the state to each others. Directors are mainly useful for > > setups where all of the mail storage is seen by all servers, such as > > with NFS or a cluster filesystem." > > > > > > Zhang Huangbin, founder of: > > - iRedMail: Open source email server solution: > > https://www.iredmail.org/ > > - Spider: Lightweight, on-premises Email Archiving Software: > > https://spiderd.io > > Aha makes sense, although I was not able to see how can index files be > corrupted when its if will going to be updated, its in same manner as > from different connection, e.g. opening email account from different app > clients, with different connections, does not corrupt the index files? > > Also, Is it the issue Director resolving as well its with maintaining > the logged in dovecot connection to same backend? Anyhow, thanks for > your valuable efforts in clearing this :) > > I wondered if there is any other solution to avoid corrupting index > files? Perhaps if dovecot offer database indexing as well as login > sessions, it seems that this would eliminate Director requirement, and > offer better high availability, as for now userdb/authdb is only > available per my knowledge, and using database cluster resolves the > issue with user and auth queries during simultaneous connections to a > different backends. > > Otherwise, it seems in large enterprise deployment with high > availability a Director implementation will be needed, hopefully we will > find an alternative solution by the time Dovecot 3 is released. > > I might need to get my head around building dovecot with customised > modules and review the code which was removed and return it back, if > anyone is planning to this, and well off ahead of me, please let me > know, we might be able to help one another. > > With thanks. > > Zakaria. >
Re: The end of Dovecot Director?
On 2022-10-21 10:51, Zhang Huangbin wrote: On Oct 21, 2022, at 5:23 PM, hi@zakaria.website wrote: I was wondering if one can achieve the same implementation with haproxy without dovecot director? The most important part of Director is it makes sure same mail user always proxied to same backend IMAP server. If mailbox is in Maildir format (and stored on shared storage like NFS), accessing it from different server may corrupt Dovecot index files and mailbox becomes unaccessible. Director perfectly avoids this issue. HAProxy can proxy mail user from same client IP to same backend IMAP server, but not same mail user from different IPs. Quote (https://doc.dovecot.org/admin_manual/director/dovecotdirector/): "Director can be used by Dovecot’s IMAP/POP3/LMTP proxy to keep a temporary user -> mail server mapping. As long as user has simultaneous connections, the user is always redirected to the same server. Each proxy server is running its own director process, and the directors are communicating the state to each others. Directors are mainly useful for setups where all of the mail storage is seen by all servers, such as with NFS or a cluster filesystem." Zhang Huangbin, founder of: - iRedMail: Open source email server solution: https://www.iredmail.org/ - Spider: Lightweight, on-premises Email Archiving Software: https://spiderd.io Aha makes sense, although I was not able to see how can index files be corrupted when its if will going to be updated, its in same manner as from different connection, e.g. opening email account from different app clients, with different connections, does not corrupt the index files? Also, Is it the issue Director resolving as well its with maintaining the logged in dovecot connection to same backend? Anyhow, thanks for your valuable efforts in clearing this :) I wondered if there is any other solution to avoid corrupting index files? Perhaps if dovecot offer database indexing as well as login sessions, it seems that this would eliminate Director requirement, and offer better high availability, as for now userdb/authdb is only available per my knowledge, and using database cluster resolves the issue with user and auth queries during simultaneous connections to a different backends. Otherwise, it seems in large enterprise deployment with high availability a Director implementation will be needed, hopefully we will find an alternative solution by the time Dovecot 3 is released. I might need to get my head around building dovecot with customised modules and review the code which was removed and return it back, if anyone is planning to this, and well off ahead of me, please let me know, we might be able to help one another. With thanks. Zakaria.
Re: lmtp userdb can't resolve users
Question are you using a db like postgresql or mysql etc
when running virtual mailboxes it is just simply a better solution
my setup is as follows
i use a django project to drive it
here are the basic's
in the dovecot.conf :
passdb {
args = /usr/local/etc/dovecot/dovecot-pgsql.conf
driver = sql
}
mail_plugins = " virtual notify replication fts fts_lucene "
service lmtp {
process_limit=1000
vsz_limit = 512m
client_limit=1
unix_listener /usr/home/postfix.local/private/dovecot-lmtp {
group = postfix
mode = 0600
user = postfix
}
}
mail_location = maildir:~/
&
# cat dovecot-pgsql.conf
driver = pgsql
connect = host=localhost port=5433 dbname=scom_billing user=pgsql
password=xxx
default_pass_scheme = PLAIN
password_query = SELECT username as user, password FROM email_users
WHERE username = '%u' and password <> 'alias' and status = True and
destination = '%u'
user_query = SELECT home, uid, gid FROM email_users WHERE username =
'%u' and password <> 'alias' and status = True and destination = '%u'
#iterate_query = SELECT user, password FROM email_users WHERE username =
'%u' and password <> 'alias' and status = True and destination = '%u'
iterate_query = SELECT "username" as user, domain FROM email_users WHERE
status = True and alias_flag = False
Happy Friday !!!
Thanks - paul
Paul Kudla
Scom.ca Internet Services
004-1009 Byron Street South
Whitby, Ontario - Canada
L1N 4S3
Toronto 416.642.7266
Main 1.866.411.7266
Fax 1.888.892.7266
Email p...@scom.ca
On 10/21/2022 3:18 AM, George Asenov wrote:
Hello,
I use postfix with dovecot as an lmtp LDA with unix users and multiple
domain names and mailboxes in Maildir format placed in domain directory
like:
/home/mainuser/homes/u...@domain.tld/Maildir
or
/home/mainuser/domains/domain2.tld/homes/u...@domain2.tld/Maildir
which have the main user as a group and u...@domain.tld/u...@domain2.tld
as owner
Postfix have
virtual_alias_maps = hash:/etc/postfix/virtual
there in virtual file there is map like:
u...@domain.tld user-dom...@domain.tld
and in /etc/passwd
there are actually 2 users with the same home dir and
same UID/GID (only the username is different)
and in postfix
mailbox_transport = lmtp:unix:private/dovecot-lmtp
The issue is that when postfix passes the email for local delivery to
dovecot lmtp it sends the username as user-domain@serverhostname.tld
but dovecot is configured with
!include auth-system.conf.ext
can't resolve this username thus fails to deliver.
I've found a workaround in the net to use custom userdb just for lmtp
like this:
protocol lmtp {
mail_plugins = $mail_plugins sieve
userdb {
driver = passwd-file
args = username_format=%n /etc/passwd
}
}
which works but produce some warnings because there is the root user (ID
0) and actually is a dirty workaround
Is there more elegant solution??
Re: The end of Dovecot Director?
> On Oct 21, 2022, at 5:51 PM, Zhang Huangbin wrote: > > If mailbox is in Maildir format (and stored on shared storage like NFS), > accessing it from different server may corrupt Dovecot index files and > mailbox becomes unaccessible. Director perfectly avoids this issue. To be clear: Accessing same mailbox from different IMAP servers __at the same time__. Zhang Huangbin, founder of: - iRedMail: Open source email server solution: https://www.iredmail.org/ - Spider: Lightweight, on-premises Email Archiving Software: https://spiderd.io
Re: The end of Dovecot Director?
> On Oct 21, 2022, at 5:23 PM, hi@zakaria.website wrote: > > I was wondering if one can achieve the same implementation with haproxy > without dovecot director? The most important part of Director is it makes sure same mail user always proxied to same backend IMAP server. If mailbox is in Maildir format (and stored on shared storage like NFS), accessing it from different server may corrupt Dovecot index files and mailbox becomes unaccessible. Director perfectly avoids this issue. HAProxy can proxy mail user from same client IP to same backend IMAP server, but not same mail user from different IPs. Quote (https://doc.dovecot.org/admin_manual/director/dovecotdirector/): "Director can be used by Dovecot’s IMAP/POP3/LMTP proxy to keep a temporary user -> mail server mapping. As long as user has simultaneous connections, the user is always redirected to the same server. Each proxy server is running its own director process, and the directors are communicating the state to each others. Directors are mainly useful for setups where all of the mail storage is seen by all servers, such as with NFS or a cluster filesystem." Zhang Huangbin, founder of: - iRedMail: Open source email server solution: https://www.iredmail.org/ - Spider: Lightweight, on-premises Email Archiving Software: https://spiderd.io
Re: The end of Dovecot Director?
Steff Majeur (Do 20 Okt 2022 11:24:49 CEST):
> I recently stumbled upon the following commit on the Dovecot core Github
> repository:
> https://github.com/dovecot/core/commit/4a187116dc2311804be22724007d357323005358
>
> Apparently, Dovecot Director is going to be removed in the next major version
> of Dovecot and the commercial Dovecot cluster architecture will be its
> successor:
> https://github.com/dovecot/documentation/blob/a85b742ec4fc2744db30a6943b3c25f004e46720/source/admin_manual/cluster/index.rst
>
> This would be a huge blow for many organizations around the world that are
> currently using Dovecot with Director in a shared storage environment.
We - the communitiy - are free to continue development of the director.
Especially large organizations should re-think their ideas of getting
free software for free.
Best regards from Dresden/Germany
Viele Grüße aus Dresden
Heiko Schlittermann
--
SCHLITTERMANN.de internet & unix support -
Heiko Schlittermann, Dipl.-Ing. (TU) - {fon,fax}: +49.351.802998{1,3} -
gnupg encrypted messages are welcome --- key ID: F69376CE -
signature.asc
Description: PGP signature
Re: The end of Dovecot Director?
On 2022-10-21 06:19, Zhang Huangbin wrote: On Oct 21, 2022, at 4:19 AM, Antonio Leding wrote: My understanding is that Director is targeted toward large enterprise mail installations that will incorporate several servers for a given function. In such an environment, Director would be the fore-person\traffic-cop keeping things organized & squared-away. Director is used when you setup frontend servers in a load-balance cluster, proxy imap/pop3/lmtp/managesieve requests to backend Dovecot servers. I setup load-balance cluster for clients with HAProxy + KeepAlived + Dovecot Director running in frontend servers, so sad we have to find an alternative to replace Director in such case. It's not about "small/medium" servers, but the demand of imap/pop3/lmtp proxy service, especially in load-balance cluster. Zhang Huangbin, founder of: - iRedMail: Open source email server solution: https://www.iredmail.org/ - Spider: Lightweight, on-premises Email Archiving Software: https://spiderd.io Hi, I was wondering if one can achieve the same implementation with haproxy without dovecot director? Load balancing all requests to pop3, imap, managesieve and lmtp services from specified frontend servers i.e. webmail to specified backend servers and using NFS mount filesystem/syncing data across all servers to access emails with high availability? Not sure whats the big deal director is offering? Is it just a native functionality providing a feature to find which backend server have X emails available and chooses to load from e.g. its content i.e. like checks which first server that doesnt return http 404 response equivalent in IMAP/POP3/LMTP/ManageSieve? Sometime ago I used Varnish caching directors to implement high availability using 404 response status in http web server, and it seems great if we can have this feature in dovecot too, as it offers high availability with delayed-syncing/partial-syncing across unknown selected servers, I managed to use Varnish too in dovecot proxy service i.e. the webmail, yet it requires NFS mount or high available file system all servers can have through immediate access to e.g. maildir? Any helpful input that would clear the picture for me in regards dovecot director, would be ver much appreciated. With thanks. Zakaria.
lmtp userdb can't resolve users
Hello,
I use postfix with dovecot as an lmtp LDA with unix users and multiple
domain names and mailboxes in Maildir format placed in domain directory
like:
/home/mainuser/homes/u...@domain.tld/Maildir
or
/home/mainuser/domains/domain2.tld/homes/u...@domain2.tld/Maildir
which have the main user as a group and u...@domain.tld/u...@domain2.tld
as owner
Postfix have
virtual_alias_maps = hash:/etc/postfix/virtual
there in virtual file there is map like:
u...@domain.tld user-dom...@domain.tld
and in /etc/passwd
there are actually 2 users with the same home dir and
same UID/GID (only the username is different)
and in postfix
mailbox_transport = lmtp:unix:private/dovecot-lmtp
The issue is that when postfix passes the email for local delivery to
dovecot lmtp it sends the username as user-domain@serverhostname.tld
but dovecot is configured with
!include auth-system.conf.ext
can't resolve this username thus fails to deliver.
I've found a workaround in the net to use custom userdb just for lmtp
like this:
protocol lmtp {
mail_plugins = $mail_plugins sieve
userdb {
driver = passwd-file
args = username_format=%n /etc/passwd
}
}
which works but produce some warnings because there is the root user (ID
0) and actually is a dirty workaround
Is there more elegant solution??
RE: The end of Dovecot Director?
> servers. > > > > I setup load-balance cluster for clients with HAProxy + KeepAlived + > Dovecot Director running in frontend servers, so sad we have to find an > alternative to replace Director in such case. The code is still available you just need to build it yourself. I think they will develop a newer version, but maybe this 'older' module can be still used. > > > > It's not about "small/medium" servers, but the demand of > imap/pop3/lmtp proxy service, especially in load-balance cluster. > > I agree. I would even state that moving towards a containerized environment you do not have one huge server that does it all, but multiple sperate containers.
Re: The end of Dovecot Director?
El 21/10/22 a les 7:19, Zhang Huangbin ha escrit: On Oct 21, 2022, at 4:19 AM, Antonio Leding wrote: My understanding is that Director is targeted toward large enterprise mail installations that will incorporate several servers for a given function. In such an environment, Director would be the fore-person\traffic-cop keeping things organized & squared-away. Director is used when you setup frontend servers in a load-balance cluster, proxy imap/pop3/lmtp/managesieve requests to backend Dovecot servers. I setup load-balance cluster for clients with HAProxy + KeepAlived + Dovecot Director running in frontend servers, so sad we have to find an alternative to replace Director in such case. It's not about "small/medium" servers, but the demand of imap/pop3/lmtp proxy service, especially in load-balance cluster. It's used also to backend a 3rd party mailbox/IMAP for an account. -- Narcis Garcia __ I'm using this dedicated address because personal addresses aren't masked enough at this mail public archive. Public archive administrator should fix this against automated addresses collectors.
Re: The end of Dovecot Director?
You still need in some sense one coherent file system to store and retrieve the mail messages. Although a load-balance cluster would still be quite useful for rejecting the bulk of unauthorized connections. I am sure in many cases a small/medium server can in fact sit and function quite adequately behind a large enterprise load balancing firewall and proxy, given the typical quantities of spam "out there" and the large number of bad connections typically attempted on any given system. On Thursday, October 20, 2022 9:19:59 PM AKDT, Zhang Huangbin wrote: On Oct 21, 2022, at 4:19 AM, Antonio Leding wrote: My understanding is that Director is targeted toward large enterprise mail installations that will incorporate several servers for a given function. In such an environment, Director would be the fore-person\traffic-cop keeping things organized & squared-away. Director is used when you setup frontend servers in a load-balance cluster, proxy imap/pop3/lmtp/managesieve requests to backend Dovecot servers. I setup load-balance cluster for clients with HAProxy + KeepAlived + Dovecot Director running in frontend servers, so sad we have to find an alternative to replace Director in such case. It's not about "small/medium" servers, but the demand of imap/pop3/lmtp proxy service, especially in load-balance cluster. Zhang Huangbin, founder of: - iRedMail: Open source email server solution: https://www.iredmail.org/ - Spider: Lightweight, on-premises Email Archiving Software: https://spiderd.io
