Re: [Dovecot] dsync deleted my mailbox - what did I do wrong?
On 8/4/2014 5:20 μμ, Nikolaos Milas wrote: Still no subfolder sync (but individual messages on existing subfolders get sync'ed): Given that I can't make dsync work properly, until this thread provides info that might lead to correct operation, which I hope, I am thinking of trying syncing using lsyncd unison (instead of dsync). Before moving on, I would like to kindly ask list members: 1. Are there any admins around who have setup dsync replication and see new subfolder replication work correctly? 2. Is lsyncd unison expected to work correctly given the particular architecture of Dovecot? (I would think it should work.) Anyone has tried it? Thanks, Nick
Re: [Dovecot] dsync deleted my mailbox - what did I do wrong?
Hello Many different dsync issues have come up in this thread. Ill try to answer them as best as I can. 1) dsync backup -R The conclusion reached in the thread was correct. Instead of the backup option, doveadm import would be better suited for merging old mails into an existing mailbox. 2) Maildir + INBOX + backup/sync/replicate In the test scenarios where the INBOX on one side was to be completely removed, e.g. doveadm backup -R the dsync failed and nothing was synced to the target. This is because before moving the source mails to the mailbox, dsync cleans out the old ones ( -R preserves nothing) and in Maildir the INBOX can not be removed. This is a feature/not easily solvable, because in Maildir INBOX is different from other folders. 3) dsync replication / doveadm sync not working as expected. These came in pretty late in the thread and I did not get a full picture of what kind of setups and parameters were used. I suspect these might be a configuration issue. I think trying with different configurations and going through the documentation, such as it is, once more, is your best bet. Use -D and -v to make dsync more verbose, so you do not miss any error messages. br, Teemu Huovila
Re: [Dovecot] dsync deleted my mailbox - what did I do wrong?
On 8/4/2014 12:38 μμ, Teemu Huovila wrote: 3) dsync replication / doveadm sync not working as expected. These came in pretty late in the thread and I did not get a full picture of what kind of setups and parameters were used. I suspect these might be a configuration issue. I think trying with different configurations and going through the documentation, such as it is, once more, is your best bet. Use -D and -v to make dsync more verbose, so you do not miss any error messages. Thank you for the reply; I am focusing on the 3rd part, since this is the one I can provide feedback about. My current configuration is exactly as suggested on the wiki2 and I list it below for your reference. Neither using replication nor using dsync from CLI leads to subfolders getting replicated, as I have explained. As an example, if a user creates subfolder boxtest e.g. under Inbox on either side, it never gets created on the other side. Running dsync with -Dv does not reveal any errors. For example, here is the output of command: # dsync -fDv -u imaptester mirror ssh -l root vmail1.example.com dsync -u imaptester Mailbox boxtest under Inbox (on vmail server) -containing one message- should get replicated (created) on vmail1 server, but it does not. If you can spot anything that can help on tracing the problem, please help. Otherwise, I can't see what is causing the erratic replication. Thanks, Nick * ... dsync-local(imaptester): Debug: brain M: in box 'INBOX.boxtest' recv_state=mailbox send_state=mailbox dsync-local(imaptester): Debug: brain M: out box 'INBOX.boxtest' recv_state=mailbox send_state=mailbox changed=0 dsync-local(imaptester): Debug: brain M: out state=sync_mails changed=0 dsync-local(imaptester): Debug: brain M: in state=sync_mails dsync-local(imaptester): Debug: brain M: in box 'INBOX.boxtest' recv_state=mailbox send_state=mailbox dsync-local(imaptester): Debug: brain M: out box 'INBOX.boxtest' recv_state=attributes send_state=changes changed=1 dsync-local(imaptester): Debug: brain M: out state=sync_mails changed=1 dsync-local(imaptester): Debug: brain M: in state=sync_mails dsync-local(imaptester): Debug: brain M: in box 'INBOX.boxtest' recv_state=attributes send_state=changes dsync-local(imaptester): Debug: brain M: out box 'INBOX.boxtest' recv_state=changes send_state=mail_requests changed=1 dsync-local(imaptester): Debug: brain M: out state=sync_mails changed=1 dsync-local(imaptester): Debug: brain M: in state=sync_mails dsync-local(imaptester): Debug: brain M: in box 'INBOX.boxtest' recv_state=changes send_state=mail_requests dsync-local(imaptester): Debug: brain M: Import INBOX.boxtest: Import change GUID=1396119018.M550517P3113.vmail.example.com,S=1169,W=1194 UID=1 hdr_hash= dsync-local(imaptester): Debug: brain M: out box 'INBOX.boxtest' recv_state=changes send_state=mail_requests changed=1 dsync-local(imaptester): Debug: brain M: out state=sync_mails changed=1 dsync-local(imaptester): Debug: brain M: in state=sync_mails dsync-local(imaptester): Debug: brain M: in box 'INBOX.boxtest' recv_state=changes send_state=mail_requests dsync-local(imaptester): Debug: brain M: Import INBOX.boxtest: Last common UID=1 dsync-local(imaptester): Debug: brain M: out box 'INBOX.boxtest' recv_state=mail_requests send_state=mails changed=1 dsync-local(imaptester): Debug: brain M: out state=sync_mails changed=1 dsync-local(imaptester): Debug: brain M: in state=sync_mails dsync-local(imaptester): Debug: brain M: in box 'INBOX.boxtest' recv_state=mail_requests send_state=mails dsync-local(imaptester): Debug: brain M: out box 'INBOX.boxtest' recv_state=mail_requests send_state=mails changed=0 dsync-local(imaptester): Debug: brain M: out state=sync_mails changed=0 dsync-local(imaptester): Debug: brain M: in state=sync_mails dsync-local(imaptester): Debug: brain M: in box 'INBOX.boxtest' recv_state=mail_requests send_state=mails dsync-local(imaptester): Debug: brain M: out box 'INBOX.boxtest' recv_state=mails send_state=done changed=1 dsync-local(imaptester): Debug: brain M: out state=sync_mails changed=1 dsync-local(imaptester): Debug: brain M: in state=sync_mails dsync-local(imaptester): Debug: brain M: in box 'INBOX.boxtest' recv_state=mails send_state=done dsync-local(imaptester): Debug: brain M: Import INBOX.boxtest: Saved UIDs: dsync-local(imaptester): Debug: brain M: Import INBOX.boxtest: Finish update: min_next_uid=2 min_first_recent_uid=2 min_highest_modseq=2 min_highest_pvt_modseq=0 dsync-local(imaptester): Debug: brain M: out box 'INBOX.boxtest' recv_state=recv_last_common send_state=done changed=1 dsync-local(imaptester): Debug: brain M: out state=sync_mails changed=1 dsync-local(imaptester): Debug: brain M: in state=sync_mails dsync-local(imaptester): Debug: brain M: in box 'INBOX.boxtest' recv_state=recv_last_common send_state=done dsync-local(imaptester): Debug: brain M: out
Re: [Dovecot] dsync deleted my mailbox - what did I do wrong?
On 04/08/2014 03:00 PM, Nikolaos Milas wrote: Neither using replication nor using dsync from CLI leads to subfolders getting replicated, as I have explained. As an example, if a user creates subfolder boxtest e.g. under Inbox on either side, it never gets created on the other side. I cant find any errors, but I might be missing something obvious. I only have a few suggestions for things to check. 1) You listed the config for one host (vmail i assume). Is the configuration similar on the vmail1 side? Especially, can the command dsync -u user find the correct location for the users mails? 2) For the replicator plugin scenario, does doveadm have access to auth, i.e. does doveadm user '*' work on both sides? 3) Are the dovecot instances running on different hosts (dovecot --hostdomain is different)? 4) Instead of dsync mirror, try using the v2.2 syntax doveadm sync. Also, i _think_ you need to execute dsync-server on the other side, so your full command becomes: doveadm sync -u imaptester ssh -l root vmail1.example.com doveadm dsync-server -u imaptester Sadly, there is no man-page for doveadm sync yet. br, Teemu Huovila
Re: [Dovecot] dsync deleted my mailbox - what did I do wrong?
On 8/4/2014 4:47 μμ, Teemu Huovila wrote:
I cant find any errors, but I might be missing something obvious. I only have a
few suggestions for things to check.
1) You listed the config for one host (vmail i assume). Is the configuration
similar on the vmail1 side? Especially, can the
command dsync -u user find the correct location for the users mails?
Yes, mail is replicated in existing folders, but new subfolders don't
get replicated.
The config I listed was on the vmail (master) side, indeed. Config on
the vmail1 (replicated) side is identical except the replication parts.
I list it at the end of this mail for your reference.
2) For the replicator plugin scenario, does doveadm have access to auth, i.e.
does doveadm user '*' work on both sides?
Yes. Everything is fine: 317 identical accounts on each side (read from
replicated LDAP).
3) Are the dovecot instances running on different hosts (dovecot --hostdomain
is different)?
Yes: vmail.example.com vs vmail1.example.com
4) Instead of dsync mirror, try using the v2.2 syntax doveadm sync. Also,
i_think_ you need to execute dsync-server on the
other side, so your full command becomes:
doveadm sync -u imaptester ssh -l root vmail1.example.com doveadm dsync-server
-u imaptester
Sadly, there is no man-page for doveadm sync yet.
OK, I ran the command:
doveadm -Dv sync -u imaptester ssh -l root vmail1.example.com doveadm
dsync-server -u imaptester
The output is similar. Still no subfolder sync (but individual messages
on existing subfolders get sync'ed):
...
dsync-local(imaptester): Debug: brain M: in box 'INBOX.boxtest'
recv_state=mailbox send_state=mailbox
dsync-local(imaptester): Debug: brain M: out box 'INBOX.boxtest'
recv_state=mailbox send_state=mailbox changed=0
dsync-local(imaptester): Debug: brain M: out state=sync_mails changed=0
dsync-local(imaptester): Debug: brain M: in state=sync_mails
dsync-local(imaptester): Debug: brain M: in box 'INBOX.boxtest'
recv_state=mailbox send_state=mailbox
dsync-local(imaptester): Debug: brain M: out box '' recv_state=mailbox
send_state=mailbox changed=1
dsync-local(imaptester): Debug: brain M: out state=master_send_mailbox
changed=1
dsync-local(imaptester): Debug: brain M: in state=master_send_mailbox
dsync-local(imaptester): Debug: brain M: out state=sync_mails changed=1
dsync-local(imaptester): Debug: brain M: in state=sync_mails
...
Please suggest any other ideas!
Thanks,
Nick
== vmail1 Config file
==
# cat /etc/dovecot/dovecot.conf
# 2.2.12: dovecot.conf
protocols = imap pop3
login_greeting = VMAIL1 POP/IMAP Srv XAPITI XPICTOY
mail_location = maildir:~/Maildir/
mail_gid = 5000
mail_uid = 5000
auth_mechanisms = plain login
auth_username_format = %Lu
auth_verbose = yes
disable_plaintext_auth = no
mail_plugins = quota
protocol imap {
imap_client_workarounds = delay-newmail
mail_plugins = quota imap_quota
}
protocol pop3 {
mail_max_userip_connections = 3
mail_plugins = quota
pop3_client_workarounds = outlook-no-nuls oe-ns-eoh
pop3_uidl_format = %08Xu%08Xv
}
protocol lda {
auth_socket_path = /var/run/dovecot/auth-master
info_log_path =
log_path =
mail_plugins = quota
postmaster_address = sysad...@example.com
sendmail_path = /usr/lib/sendmail
}
userdb {
args = /etc/dovecot/dovecot-usrdb-ldap.conf
driver = ldap
}
passdb {
args = /etc/dovecot/dovecot-passdb-ldap.conf
driver = ldap
}
plugin {
quota = maildir:User quota
quota_rule = *:storage=4G
quota_rule2 = Trash:storage=+3%%
quota_warning = storage=75%% quota-warning 75 %u
quota_warning2 = storage=90%% quota-warning 90 %u
}
service quota-warning {
executable = script /opt/mail1.sh
user = vmail
unix_listener quota-warning {
user = vmail
}
}
service auth {
unix_listener /var/spool/postfix/private/auth {
group = postfix
mode = 0660
user = postfix
}
unix_listener auth-master {
group = vmail
mode = 0660
user = vmail
}
user = root
}
service imap-login {
service_count = 1
vsz_limit = 64 M
}
service pop3-login {
service_count = 1
vsz_limit = 64 M
}
ssl_ca = /etc/pki/tls/certs/chain-2228.pem
ssl_cert = /etc/pki/tls/certs/cert-2228.pem
ssl_key = /etc/pki/tls/private/key-2228.pem
syslog_facility = local1
Re: [Dovecot] dsync deleted my mailbox - what did I do wrong?
Hey Jiri. Thanks for getting back. On Friday, April 4, 2014 4:48:48 PM CEST, Jiri Bourek wrote: - where did I fail (ignoring the backups, please. That's .. something I know) From the man page: backup - Backup mails from default mail location to location2 (or vice versa, if -R parameter is given). No changes are ever done to the source location. Any changes done in destination are discarded. Yeah, maybe. That's what I thought _after_ the fact (i.e. that was what I hinted at with 'one way sync'). But see below. The last sentence describes what happened to you: all new mail on the new machine is a change and was discarded (by deleting new mail.) If I'm not mistaken, this is correct behaviour for backup mode - you get exact copy of the source side (maildir:/tmp/mail_backup) on destination side (d...@darklajid.de) That would be sort of okay. Except that isn't what happened: - The target mailbox was killed completely - Nothing was restored If what you're suggesting here is true I'd expect a clean copy of my source - even if it destroys all other changes. That did NOT happen though. It nuked the target and didn't restore a thing. Plus, dsync mirror did exactly the same: Nuked the (live) mailbox once more, same error message, not a single message restored (but also no modification to the source). - Can I use dsync ... for backups? I don't think that this is a good idea after yesterday night? AFAIK you can safely use it to make the backup. I'm not sure if it can be reliably used to restore data (don't think so but I'm not an expert.) I'd use doveadm import for that. That'd be my experience at this point as well, of course. :-) The bigger question is if this is well-known / correct and if this should be documented in a better fashion. Was I really that naive to expect that to work (in that case: ignore the documentation request) or could that happen again? Ben
Re: [Dovecot] dsync deleted my mailbox - what did I do wrong?
The last sentence describes what happened to you: all new mail on the
new machine is a change and was discarded (by deleting new mail.) If
I'm not mistaken, this is correct behaviour for backup mode - you get
exact copy of the source side (maildir:/tmp/mail_backup) on
destination side (d...@darklajid.de)
That would be sort of okay. Except that isn't what happened:
- The target mailbox was killed completely
- Nothing was restored
If what you're suggesting here is true I'd expect a clean copy of my
source - even if it destroys all other changes. That did NOT happen
though. It nuked the target and didn't restore a thing.
True - if we move from problem is dsync deleted new mail to problem
is dsync was unable to restore the backup, the described behaviour
looks like a bug to me too. It may have something to do with the maildir
format, I recall some discussion regarding folder INBOX, which needs
special handling (because it's physically stored in maildir root,
whereas every other folder is stored in folder-named subdirectory)
That said, I tried something along what you did and it failed for me
too. So I deleted the mailbox completely, recreated it, tried again and
this time the restore succeeded. It seems the easiest possible way to
reproduce the faulty behaviour is:
1. create mailbox for testing, here t...@example.com
2. create IMAP folder under INBOX
( namespace inbox { separator = / } )
# doveadm mailbox create -u t...@example.com INBOX/test
3. attempt to restore from backup
# doveadm backup -u t...@example.com -R maildir:/mnt/mail-backups/test/
which yields (on Dovecot 2.2.12)
dsync(t...@example.com): Error: Mailbox INBOX sync: mailbox_delete
failed: INBOX can't be deleted.
Another try shows that IMAP folder created somewhere else (not under
INBOX) isn't a problem:
# doveadm mailbox create -u t...@example.com testtest
# doveadm backup -u t...@example.com -R maildir:/mnt/mail-backups/test/
This yields no output, folder testtest is deleted (as expected), INBOX
is populated from backup.
Another try, this time I used mbox instead of maildir by specifying
-o mail_location=mbox:/path/test/mail
to doveadm. Worked without error even with INBOX/test folder (which got
deleted during restore)
No idea if this can be considered as a bug, or the test does something
that is not supposed to be done in the first place (Although different
results with different storage format suggests a bug to me.)
Plus, dsync mirror did exactly the same: Nuked the (live) mailbox once
more, same error message, not a single message restored (but also no
modification to the source).
I was doing some trial and error testing with doveadm sync (should be
the same as dsync mirror.) If used on a mailbox which has seen some
changes, this sync's behaviour is just strange.
Or - to be more precise - it seems strange on first sight, but when you
think about it, it does what is supposed to do. The sync mode is (AFAIK)
designed to keep single mailbox synchronized on two hosts. If you
created new mailbox on the new host, then had some mail delivered to it
and after some time decided to add mail from old host, then you don't
have single mailbox - you have two mailboxes with the same name.
In other words this scenario is probably something dsync wasn't designed
to be used for and there's no surprise mirror mode can't handle it.
And again - I'm no expert, so it's entirely possible everything I wrote
here is complete and utter nonsense Let's hope someone more
knowledgeable corrects me if that is the case.
Re: [Dovecot] dsync deleted my mailbox - what did I do wrong?
On 7/4/2014 4:01 μμ, Jiri Bourek wrote: I was doing some trial and error testing with doveadm sync (should be the same as dsync mirror.) If used on a mailbox which has seen some changes, this sync's behaviour is just strange. I can confirm (on 2.2.12) that the behavior is the same using replication (mirroring). Creating a folder on either end, e.g. as a subfolder of Inbox, does not create an identical new folder on the other end. I would expect that the folder is replicated! Manual dsync from the CLI will not replicate folders as well. I will agree with Jiri that is a strange (wrong?) behavior in dsync. Folders should be replicated, whether new or not. If not, how message moves between older and newer folders can be replicated on the fly? Please advise on how to handle this situation because if folders are created/deleted/moved by users, dsync may lead to a chaos! Nick
Re: [Dovecot] dsync deleted my mailbox - what did I do wrong?
Hi - where did I fail (ignoring the backups, please. That's .. something I know) From the man page: backup - Backup mails from default mail location to location2 (or vice versa, if -R parameter is given). No changes are ever done to the source location. Any changes done in destination are discarded. The last sentence describes what happened to you: all new mail on the new machine is a change and was discarded (by deleting new mail.) If I'm not mistaken, this is correct behaviour for backup mode - you get exact copy of the source side (maildir:/tmp/mail_backup) on destination side (d...@darklajid.de) - Can I use dsync ... for backups? I don't think that this is a good idea after yesterday night? AFAIK you can safely use it to make the backup. I'm not sure if it can be reliably used to restore data (don't think so but I'm not an expert.) I'd use doveadm import for that.
