[Dovecot] dsync and active-active fault tolerance mail servers
I would like to set up an active-active fault tolerance couple of mail servers.
Is the following procedure correct?
Let's suppose I want to set up two servers for acme.com, mail1.acme.com and
mail2.acme.com
I declare both servers as MX in DNS configuration.
Mailbox are stored in maildir with a single UID/GID owner for all the maildir
files
Userbase is MySQL, user tables are replicated using MySQL tools; in this case,
the replica is one way, that is mail1.acme.com holds the master copy of MySQL
tables and mail2.acme.com hold the copy.
The replica is done via dsync with a procedure like this executed on
mail1.acme.com server:
foreach mailbox {
dsync mirror mailbox
}
Where mailbox is the full mail address, used also as a login name for
authentication.
My goal is to allow users to connect the mail servers they like and have all
their mail (a 10/15 minute of sync delay is acceptable).
This is the theory, now some questions.
1) Use Linux user/group owner of the files and directories are the same on both
hosts (they are both foobar), but have different numeric ID, in one host foobat
has UID 1234 and on the other host foobar has UID 2345. Is this an issue?
2) Mail location path is the same on both hosts, but where does dsync get it?
Does dsync get all the information from the userbase defined in dovecot.conf
(and other included files)?
3) Have I to scan the entire userbase and execute dsync for each user, or is
there a trick to do for every user with a single command?
4) How often is good to run the synchronization? 5 minutes? 10? 15?
5) What is the correct command line to invoke dsync in this scenario?
6) Does anyone else tried this? :)
Thank you.
Ciao,
luigi
--
/
+--[Luigi Rosa]--
\
(1) Everything depends.
(2) Nothing is always.
(3) Everything is sometimes.
Re: [Dovecot] Dovecot 2.0 and BSDI 4.X
The Doctor put forth on 8/20/2010 9:24 PM: On Thu, Aug 19, 2010 at 06:22:09PM +0100, Timo Sirainen wrote: On Wed, 2010-08-18 at 23:37 -0600, The Doctor wrote: Aug 18 23:25:31 doctor dovecot: log: Error: net_accept() failed: Invalid argument http://hg.dovecot.org/dovecot-2.0/rev/565f18727209 fixes this? Wind River discontinued BSD/OS (BSDi) in 2003, 7 years ago. This version of BSDi The Doctor is running thus hasn't seen a patch for _at least_ 7 years. Seven year old gcc environment? No wonder things don't work. Is there any real legitimate reason either of you are wasting your time with trying to make Dovecot work on a 7+ year old _DEAD_ operating system? Why haven't you moved this box up to the latest version of FreeBSD or OpenBSD? -- Stan
Re: [Dovecot] Authentication woes - Couldn't drop privileges: Mail access not allowed for root
Hi Timo, thanks again for pointing out the missing userdb configuration entry yesterday. Now that both Dovecot 2.0.0 servers are running fine, I use the following command to backup mail data from serverA to serverB in a shell script running on serverB, looping over user names: /usr/local/dovecot-2.0/bin/dsync -v -R -u $user backup \ ssh -i /home/$user/.ssh/id_dsa $u...@servera /usr/local/dovecot-2.0/bin/dsync I made a small mistake during setup and had one of the id_dsa files owned by root instead of $user. This was deliberate, because the sync script is run by root and I thought that the SSH key should thus be owned by root aswell. The resulting error message was: Permission denied (publickey,keyboard-interactive). dsync-local(foouser): Error: read() from worker server failed: EOF dsync-local(foouser): Fatal: dsync backup: Looks like you're trying to run backup in wrong direction. Source is empty and destination is not. The first line is generated by SSH, because serverA allows key-based authentication only and the key file on serverB was unreadable as dsync seems to drop root privileges in this case (at least that's my guess). Do you think one should add an example for key-based authentication to http://wiki2.dovecot.org/Tools/Dsync ? -Ralph
Re: [Dovecot] pigeonhole sieve imapflags bug
[oops, apparently I didn't reply this to the list]
Tim Traver wrote:
Hi all,
ok, I just instealled the dovecot 2.0.0 release, and the latest
mercurial pigeonhole source.
All is well, except for a particular sieve rule that may not be working
right.
It is a simple rule that looks like this :
require [fileinto,imap4flags];
# rule:[Past Due Notices]
if allof (header :contains Subject Account Past Due Notice)
{
fileinto Billing;
setflag \\Seen;
stop;
}
Anyways, what happens is the message that matches that rule does indeed
get moved to the Billing folder, but it does not get marked as read.
BUT! if I change the order of the actions to mark it as read first, then
move it to the Billing folder, it does both of those actions...
Is that the way it is supposed to work? or is that a bug?
Yes, this is the way it is supposed to work. The
setflag/addflag/removeflag operate on an internal variable with flags.
The fileinto and keep commands use the current value of this variable to
assign flags. Modifications to the internal flag variable therefore only
have an effect on subsequent keep and fileinto actions and not ones
already executed. Read RFC5232 (http://tools.ietf.org/html/rfc5232) for
more info.
Regards,
Stephan
[Dovecot] sieve_after scripts with virtual domains
Hi all,
first of all I am new to the list, so hello everyone :)
I recently encountered an annoying problem with sieve, I am using dovecot
version 1.2.13 on centos 5.5, the packages installed are the following:
dovecot-sieve-0.1.17-5.el5
dovecot-managesieve-0.11.11-0_4.el5
dovecot-1.2.13-1_111.el5
the mail server is configured with virtual hosts (mysql), also the sieve plugin
and managesieve are active. Users can create their own filters through
roundcube and everything works fine.
I also put two script to be executed before all sieve scripts in sieve_before,
they just move any SPAM or VIRUS tagged message to a Junk folder. The problem
is that while this works for my domain (that is also the server domain) it
doesn't for the others domains, it seems the sieve_before scripts get ignored.
I made many tests with the domains I am hosting, if I send a spam message (I am
using the spamassassin test messages) it gets marked as SPAM, but gets only
moved to the junk folder if the recipient is @cslash.net (that is my domain).
here is some config extracts:
=== dovecot.conf ===
plugin {
sieve = ~/.dovecot.sieve
sieve_dir = ~/sieve
sieve_before = /srv/mail/sieve-before
sieve_after = /srv/mail/sieve-after
}
===
/srv/mail $ cat sieve-before/spam.sieve
require fileinto;
if header :contains X-Spam-Flag YES {
fileinto Junk;
}
P.S.
The sieve scripts are all compiled.
Re: [Dovecot] Dovecot 2.0 and BSDI 4.X
On 21.8.2010, at 3.24, The Doctor wrote:
Aug 18 23:25:31 doctor dovecot: master: Warning: service(anvil):
process_limit reached, client connections are being dropped
I'm not really sure about this.. This still happens with the above
change? Anything else logged before it? Do you even have anvil process?
Yes, the anvil process is a postfix process.
Dovecot has its own anvil process too (yeah, I stole the name from Postfix).
listen = *
..
service imap-login {
inet_listener imaps {
address = 204.209.81.1
}
Better to set the address in the listen setting instead of explicitly
for different inet_listeners.
Had to resort back to inet_listener .
Question how is the auth process handled in 2 that differs from 1?
What does that have to do with anything?
Because in Dovecot 1 I had
auth_executable = /usr/contrib/libexec/dovecot/dovecot-auth
and in Dovecot 2 I have
executable = /usr/dovecot/libexec/dovecot/auth
Only the binary name changed from dovecot-auth to auth.
Re: [Dovecot] dsync and active-active fault tolerance mail servers
On 21.8.2010, at 9.05, Luigi Rosa wrote:
The replica is done via dsync with a procedure like this executed on
mail1.acme.com server:
foreach mailbox {
dsync mirror mailbox
}
1) Use Linux user/group owner of the files and directories are the same on
both hosts (they are both foobar), but have different numeric ID, in one host
foobat has UID 1234 and on the other host foobar has UID 2345. Is this an
issue?
No. Just don't have your userdb specify the UID as a number (or at all, just
set a global mail_uid = foobar in config file).
2) Mail location path is the same on both hosts, but where does dsync get it?
Does dsync get all the information from the userbase defined in dovecot.conf
(and other included files)?
dsync reads the config the same way all other standalone programs (including
dovecot master) do it: it executes doveconf, which parses the config. Then if
you give -u parameter to dsync it also does userdb lookup from auth process.
3) Have I to scan the entire userbase and execute dsync for each user, or is
there a trick to do for every user with a single command?
Currently you need to execute dsync separately for each user. It's also more
reliable since I think there are still some rare random crashes in dsync. If
you find these, let me know and I'll try to fix them. :) Also you should
probably be running a few dsyncs in parallel to finish the sync in less time.
4) How often is good to run the synchronization? 5 minutes? 10? 15?
Dunno. But see http://dovecot.org/list/dovecot/2010-August/051714.html
5) What is the correct command line to invoke dsync in this scenario?
Maybe simply: dsync mirror -u u...@domain mail2.acme.com
Re: [Dovecot] Dovecot 2.0 and BSDI 4.X
On Sat, Aug 21, 2010 at 01:40:00PM +0100, Timo Sirainen wrote:
On 21.8.2010, at 3.24, The Doctor wrote:
Aug 18 23:25:31 doctor dovecot: master: Warning: service(anvil):
process_limit reached, client connections are being dropped
I'm not really sure about this.. This still happens with the above
change? Anything else logged before it? Do you even have anvil process?
Yes, the anvil process is a postfix process.
Dovecot has its own anvil process too (yeah, I stole the name from Postfix).
listen = *
..
service imap-login {
inet_listener imaps {
address = 204.209.81.1
}
Better to set the address in the listen setting instead of explicitly
for different inet_listeners.
Had to resort back to inet_listener .
Question how is the auth process handled in 2 that differs from 1?
What does that have to do with anything?
I am just wondering what else has changed?
IT seems that the authentication is caught
in a non-ending loop.
Because in Dovecot 1 I had
auth_executable = /usr/contrib/libexec/dovecot/dovecot-auth
and in Dovecot 2 I have
executable = /usr/dovecot/libexec/dovecot/auth
Only the binary name changed from dovecot-auth to auth.
--
Member - Liberal International This is doc...@nl2k.ab.ca Ici doc...@nl2k.ab.ca
God, Queen and country! Never Satan President Republic! Beware AntiChrist
rising!
http://twitter.com/rootnl2k http://www.facebook.com/dyadallee
Australia!! Vote to throw the REpublicans to the sharks!
Re: [Dovecot] dsync and active-active fault tolerance mail servers
Timo Sirainen said the following on 21/08/2010 14.54: Maybe simply: dsync mirror -u u...@domain mail2.acme.com Timo, thank you for all your answers. I will set up a replicated active-active configuration in the following weeks and I will let you know if problems arise. Ciao, luigi -- / +--[Luigi Rosa]-- \ I'm glad I was not born before tea. --Sidney Smith (1771-1845)
Re: [Dovecot] 1.2.13 QRESYNC crash.
On Thu, 2010-08-19 at 18:37 +0100, Timo Sirainen wrote: On Wed, 2010-08-18 at 22:27 +0100, David Woodhouse wrote: Aug 18 22:07:31 twosheds IMAP(dwmw2): : Panic: file mail-index-transaction.c: line 637 (mail_index_transaction_lookup): assertion failed: (seq = t-first_new_seq seq = t-last_new_seq) A00131 SELECT lists.bluez (QRESYNC (1154090296 1861 1:* (1,120,1578,2064,2226,2280,2298 1,120,12037,12523,12685,12739,12757))) Thanks, fixed: http://hg.dovecot.org/dovecot-1.2/rev/70fa6178380e Hm, looking at RFC5162 again I realise that SELECT command isn't actually valid. The '1:*' for the known-uids is not permitted. From the formal syntax in §6: capability =/ QRESYNC select-param= QRESYNC SP ( uidvalidity SP mod-sequence-value [SP known-uids] [SP seq-match-data] ) ;; conforms to the generic select-param ;; syntax defined in [IMAPABNF] seq-match-data = ( known-sequence-set SP known-uid-set ) uidvalidity = nz-number known-uids = sequence-set ;; sequence of UIDs, * is not allowed known-sequence-set = sequence-set ;; set of message numbers corresponding to ;; the UIDs in known-uid-set, in ascending order. ;; * is not allowed. known-uid-set = sequence-set ;; set of UIDs corresponding to the messages in ;; known-sequence-set, in ascending order. ;; * is not allowed. §3.1 says: If the list of known UIDs was also provided, the server should only report flag changes and expunges for the specified messages. If the client did not provide the list of UIDs, the server acts as if the client has specified 1:maxuid, where maxuid is the mailbox's UIDNEXT value minus 1. So instead of giving the known-uid set '1:*', the client should actually have omitted the optional known-uid parameter completely. It *should* have sent this command: A00131 SELECT lists.bluez (QRESYNC (1154090296 1861 (1,120,1578,2064,2226,2280,2298 1,120,12037,12523,12685,12739,12757))) Dovecot doesn't like that though: A00131 BAD Error in IMAP command SELECT: Invalid QRESYNC parameters -- David WoodhouseOpen Source Technology Centre david.woodho...@intel.com Intel Corporation
[Dovecot] DSYNC needs a lot more documentation
Started looking into the dsync utility and the doc are seriously incomplete. I can of course scour the internet looking for the missing information but that doesn't fix the problem with the docs. I might try to rewrite the docs myself once I figure it out.
Re: [Dovecot] DSYNC needs a lot more documentation
On 21.8.2010, at 16.24, Marc Perkel wrote: Started looking into the dsync utility and the doc are seriously incomplete. I can of course scour the internet looking for the missing information but that doesn't fix the problem with the docs. I might try to rewrite the docs myself once I figure it out. Or you could mention some of the things you think are incomplete.
Re: [Dovecot] DSYNC needs a lot more documentation
On 8/21/2010 9:16 AM, Timo Sirainen wrote: On 21.8.2010, at 16.24, Marc Perkel wrote: Started looking into the dsync utility and the doc are seriously incomplete. I can of course scour the internet looking for the missing information but that doesn't fix the problem with the docs. I might try to rewrite the docs myself once I figure it out. Or you could mention some of the things you think are incomplete. Ok - when I type dsync at the command line it says: usage: dsync [-C alt char] [-m mailbox] [-u user] [-frRv] mirror local mail_location | [user@]host | remote dsync command However the man page mentions nothing about any remote commands. There is a reference to ssh in one example but there isn't any kind of overview as to how this all ties in. Does dsync pick up information from dovecot.conf or dovecot to know where the email is an what format it is in? Does dovecot need to be running on both ends? Does this run continuously once you start it or does it need to be run once a minute? The information in the man page isn't complete enough for me to even figure out what to ask. When you write software you never have to learn it so you don't have the perspective of someone who never heard of it before and wondering what is this? I can probable figure it out if I read enough of the message from the dovecot lists but the docs should be complete enough so I don't have to do that unless I'm doing something very weird. So - ServerA is running dovecot. On ServerB I want to have a live copy so that if the drives on ServerA die I can recover on ServerB? Does it do that? I want to run dovecot on two servers so that if either server fails the other seamlessly takes over and when the other comes back up they resync as if nothing had happened. Is that possible? If so - how? If it just does backups, how is it different than rsync? Anyhow - just letting you know from the perspective of someone who knows nothing and exploring it for the first time. I wanted to say this before I learned anything more about it.
[Dovecot] sieve rule help
Dear All,
I am using dovecot-2.0.0 version and all my sieve
filters are working perfectly except the following one.
I am trying to flag message coming from jagd...@example.net mail id.
require
[fileinto,envelope,reject,vacation,imap4flags,relational,comparator-i;ascii-numeric,regex,body,date];
if address :contains From jagd...@example.net
{
keep :flags [Flagged] ;
}
Is it possible to make it work without using addflag to add a flag ?
Re: [Dovecot] sieve rule help
On Aug 21, 2010, at 10:01 AM, Piyush Joshi wrote:
Dear All,
I am using dovecot-2.0.0 version and all my sieve
filters are working perfectly except the following one.
I am trying to flag message coming from jagd...@example.net mail id.
require
[fileinto
,envelope
,reject,vacation,imap4flags,relational,comparator-i;ascii-
numeric,regex,body,date];
if address :contains From jagd...@example.net
{
keep :flags [Flagged] ;
}
Is it possible to make it work without using addflag to add a flag ?
From a recent thread (pigeonhole sieve imapflags bug):
if allof (header :contains Subject Account Past Due Notice)
{
fileinto Billing;
setflag \\Seen;
stop;
}
Reading http://www.faqs.org/rfcs/rfc5232.html maybe:
if address :contains From jagd...@example.net
{
keep :flags \\Flagged ;
}
// Brad
[Dovecot] imap, imaps, ports; config for secure-only service
Hello, having difficulty setting up a 'secure-only' service on a
non-standard port.
Objective is to open a secure Dovecot service on an internet-visible
port, while also using an insecure service for hosts on an internal
network (so that one particular client which is not SSL/TLS-capable
can continue to use the service).
Checking wiki1 and wiki2, I think that port 143 can be used for a
service in both encrypted and unencrypted operations. (Wiki2
describes how port 143 can be used with or without STARTTLS.) I've
managed to run both TB/Vista, and KMail/Debian on 143 from local
machines, and they seem to work fine. (Though they do only work if
the clients are set to use 'unencrypted logins' such as KMail's 'clear
text' setting, and TB's 'don't use secure authentication'.) Dovecot's
SSL certificate on the server expires in 2028.
But I cannot get my 'secure-only service' working at all. I've chosen
port 10943 for that service, and have been testing from both KMail on
the server, and TB/Vista on another local machine.
Here's the config:
protocol imap {
# inet users, and also users presented from offsite via an SSH tunnel,
# will use IMAP (any capability) on 143.
listen = *:143
# 143 will not be reachable from outside the internal network (except
# via an SSH tunnel)
#
# External users will be allowed only to use port 10943 (as forwarded
# from the gateway), and will have to use SSL
ssl_listen = *:10943
}
Other settings include:
#other settings
protocols = imap imaps
disable_plaintext_auth = no
ssl_disable = no
ssl_cert_file = /etc/ssl/certs/dovecot.pem
ssl_key_file = /etc/ssl/private/dovecot.pem
ssl_verify_client_cert = no
ssl_cipher_list = ALL:!LOW
ssl_require_client_cert = no
ssl_username_from_cert = no
mail_location = maildir:~/Maildir
#log and debug settings
mail_debug = yes
auth_debug = yes
Log from a successful session on port 143:
Logs (using port 143, KMail on the server)
14:43:22 auth(default): client in:
AUTH^I1^IPLAIN^Iservice=IMAP^Isecured^Ilip=127.0.0.1^Irip=127.0.0.1^Iresp=hidden
14:43:22 auth(default): client out: OK^I1^Iuser=ron
14:43:22 auth(default): master in: REQUEST^I1^I4057^I1
14:43:22 auth(default): master out:
USER^I1^Iron^Isystem_user=ron^Iuid=1001^Igid=1001^Ihome=/home/ron
14:43:22 IMAP(ron): Effective uid=1001, gid=1001
14:43:22 IMAP(ron): maildir: data=/home/ron/Maildir
14:43:22 IMAP(ron): maildir: root=/home/ron/Maildir,
index=/home/ron/Maildir, control=, inbox=
14:43:22 imap-login: Login: user=ron, method=PLAIN,
rip=127.0.0.1, lip=127.0.0.1, TLS
Unsuccessful session
Logs (using 10943, KMail on the server) - absolutely no sign of any
authentication, and the client reports a timeout.
17:03:30 imap-login: Disconnected: Inactivity: rip=127.0.0.1,
lip=127.0.0.1, TLS handshake
17:04:06 imap-login: Disconnected: rip=127.0.0.1, lip=127.0.0.1,
TLS handshake
17:05:08 imap-login: Disconnected: Inactivity: rip=127.0.0.1,
lip=127.0.0.1, TLS handshake
143 works fine, so Dovecot is (of course) perfectly good at managing
the auth and service, so I must have something wrong in the config. I
cannot see it. Could anyone share a config they've used for
*secure-only* IMAP service on an internet-visible port? I'd like to
compare the settings.
regards, Ron
Re: [Dovecot] DSYNC needs a lot more documentation
On 2010-08-21 12:51 PM, Marc Perkel m...@perkel.com wrote: When you write software you never have to learn it so you don't have the perspective of someone who never heard of it before and wondering what is this? Mark, is this another case of your absolute failure to even *try* to google the answer for yourself? You do know that man pages are not the only place (and quite often definitely not the best place) to find documentation for any given software? http://www.lmgtfy.com/?q=dsync+dovecot+wiki -- Best regards, Charles
Re: [Dovecot] DSYNC needs a lot more documentation
On Sat, Aug 21, 2010 at 11:00 AM, Charles Marcus cmar...@media-brokers.com wrote: On 2010-08-21 12:51 PM, Marc Perkel m...@perkel.com wrote: When you write software you never have to learn it so you don't have the perspective of someone who never heard of it before and wondering what is this? Mark, is this another case of your absolute failure to even *try* to google the answer for yourself? You do know that man pages are not the only place (and quite often definitely not the best place) to find documentation for any given software? http://www.lmgtfy.com/?q=dsync+dovecot+wiki -- Best regards, Charles Haha! That is an awesome link thanks!
Re: [Dovecot] DSYNC needs a lot more documentation
On Aug 21, 2010, at 11:42 AM, Brandon Lamb wrote: On Sat, Aug 21, 2010 at 11:00 AM, Charles Marcus cmar...@media-brokers.com wrote: On 2010-08-21 12:51 PM, Marc Perkel m...@perkel.com wrote: When you write software you never have to learn it so you don't have the perspective of someone who never heard of it before and wondering what is this? Mark, is this another case of your absolute failure to even *try* to google the answer for yourself? You do know that man pages are not the only place (and quite often definitely not the best place) to find documentation for any given software? http://www.lmgtfy.com/?q=dsync+dovecot+wiki -- Best regards, Charles Haha! That is an awesome link thanks! Indeed!
Re: [Dovecot] DSYNC needs a lot more documentation
On 8/21/2010 11:00 AM, Charles Marcus wrote: On 2010-08-21 12:51 PM, Marc Perkelm...@perkel.com wrote: When you write software you never have to learn it so you don't have the perspective of someone who never heard of it before and wondering what is this? Mark, is this another case of your absolute failure to even *try* to google the answer for yourself? You do know that man pages are not the only place (and quite often definitely not the best place) to find documentation for any given software? http://www.lmgtfy.com/?q=dsync+dovecot+wiki You are missing the point. When documentation is done right then you don't have to google it unless you are doing something tricky.
Re: [Dovecot] DSYNC needs a lot more documentation
On 8/21/2010 11:00 AM, Charles Marcus wrote: On 2010-08-21 12:51 PM, Marc Perkelm...@perkel.com wrote: When you write software you never have to learn it so you don't have the perspective of someone who never heard of it before and wondering what is this? Mark, is this another case of your absolute failure to even *try* to google the answer for yourself? You do know that man pages are not the only place (and quite often definitely not the best place) to find documentation for any given software? http://www.lmgtfy.com/?q=dsync+dovecot+wiki Besides - the wiki has the same incomplete information as the man page.
Re: [Dovecot] DSYNC needs a lot more documentation
On 08/21/2010 09:58 PM Marc Perkel wrote: Besides - the wiki has the same incomplete information as the man page. Yeah, the wiki shows the manual page. ;-) But now it's time to tell us, waht you are missing / what's incomplete. (see also MID:f559a442-1e7f-4d78-a674-15e2843a3...@iki.fi) Regards, Pascal -- The trapper recommends today: 5e1f1e55.1023...@localdomain.org
Re: [Dovecot] DSYNC needs a lot more documentation
On 2010-08-21 3:56 PM, Marc Perkel m...@perkel.com wrote: You are missing the point. When documentation is done right then you don't have to google it unless you are doing something tricky. True - but 2.0 is brand new, so instead of posting to the list mostly a vague complaint with a vague offer of a future update to the docs, it would have been better to just not post until you had some updates to offer, or at a minimum, provide specific details of what you think is missing - Timo has always been very quick to remedy missing/incomplete documentation as far as I can see... Besides - the wiki has the same incomplete information as the man page. Point taken (I haven't installed 2.0 yet so can't look at the man page)... -- Best regards, Charles
