Another item.
Again, it may be a 2.3.13 issue and I'm now on 2.3.17. But...I had
problem when using the "-N" parameter for dsync. So - I just have (had):
replication_dsync_parameters = -d -l 30 -U -x INBOX/virtual -x
INBOX/shared
Now that things are working - I wanted to have my other namespaces sync
as well. So I went to:
replication_dsync_parameters = -d -l 30 -U -n INBOX -n INBOX/Archives -n
INBOX/Lists -x INBOX/virtual -x INBOX/shared
This appears to be working (the sync is just starting)...but I'm seeing
lock errors in the logs such as:
Dec 3 14:34:24 bubba dovecot:
doveadm([email protected])<31785><TV+0LlGbqmEpfAAAnbWsyw>: Error:
Couldn't lock /var/mail/amfes.com/dmiller/.dovecot-sync.lock:
fcntl(/var/mail/amfes.com/dmiller/.dovecot-sync.lock, write-lock,
F_SETLKW) locking failed: Timed out after 30 seconds (WRITE lock held by
pid 31373)
Checking the pid in question I see it's actively syncing a folder in my
mailbox. So I'm guessing, purely guessing, that by having multiple
namespaces explicitly directed to sync Dovecot is trying to start a sync
process for each of those namespaces - but all of them share a common
lock and therefore only one operation is allowed at a time.
Am I correct, and whether or not I am - how can I correct these errors?
Do I dare try going back to just "-N"?
--
Daniel
------ Original Message ------
From: "Daniel Miller" <[email protected]>
To: [email protected]
Sent: 12/3/2021 2:16:28 PM
Subject: Replication weirdness
First, I have to say this. After configuring everything correctly - and
that means *everything* correctly - Dovecot replication Just Works. I'm
not sure how (yes I do - Timo & Co. Magic) - but it does. Real-time new
sync is near instantaneous.
Now the problem. Or the background for the problem. My primary server
uses sdbox for primary storage, mdbox for archival storage, and
fts-solr. I spun up a second server, using sdbox, mdbox, and
fts-flatcurve. My namespaces are as defined below. As best I can tell
(based on diff comparing two 'doveconf -n' outputs) my namespaces are
the same on both servers.
namespace archives {
list = children
location = mdbox:/var/mail/%d/%n/Archives/mdbox
mailbox Unsorted {
auto = no
special_use = \Archive
}
prefix = INBOX/Archives/
separator = /
subscriptions = no
type = private
}
namespace inbox {
alias_for =
hidden = no
inbox = yes
list = yes
location =
mailbox "Deleted Messages" {
auto = no
autoexpunge = 30 days
special_use = \Trash
}
mailbox Drafts {
auto = subscribe
special_use = \Drafts
}
mailbox Sent {
auto = subscribe
special_use = \Sent
}
mailbox Trash {
auto = subscribe
autoexpunge = 30 days
special_use = \Trash
}
prefix = INBOX/
separator = /
subscriptions = no
type = private
}
namespace lists {
list = children
location = mdbox:/var/mail/%d/%n/Lists/mdbox
prefix = INBOX/Lists/
separator = /
subscriptions = no
type = private
}
namespace subscriptions {
hidden = yes
list = no
location =
prefix =
separator = /
subscriptions = yes
type = private
}
namespace usershares {
list = yes
location = sdbox:/var/mail/%%d/%%n/sdbox:NO-NOSELECT
prefix = INBOX/shared/%%d/%%n/
separator = /
subscriptions = no
type = shared
}
namespace virtual {
list = children
location = virtual:/var/mail/%d/%n/virtual
mailbox Flagged {
comment = All my flagged messages
special_use = \Flagged
}
prefix = INBOX/virtual/
separator = /
subscriptions = no
}
I also have:
plugin {
mailbox_alias_new = Sent Messages
mailbox_alias_new2 = Sent Items
mailbox_alias_new3 = Deleted Messages
mailbox_alias_old = Sent
mailbox_alias_old2 = Sent
mailbox_alias_old3 = Trash
}
This setup worked fine with my single server. Then I enabled
replication - just on the primary. Dsync went to work (it seemed to
take forever for the initial sync but that's what happens with large
mailboxes and slow internet connections).
The problem came up with certain subfolders. And I believe it only
happens with subfolders that have spaces in their names. I had two
user's mailboxes (under Sent), one of which had a "Sent Messages"
symlink alias for "Sent", that started generating tens or hundreds of
duplicates during sync. Fortunately those subfolders only had a few
mails in them. But I had trees looking like:
[...] (below is under /var/mail/domain/user/sdbox/mailboxes/Sent/)
Proposal
Requests-temp-c6e003375e64a961c93d00009db5accb-temp-1-temp-f80b1a00ce9aa961a86-temp-2
Proposal
Requests-temp-c6e003375e64a961c93d00009db5accb-temp-1-temp-f80b1a00ce9aa961a86-temp-3
Proposal
Requests-temp-c6e003375e64a961c93d00009db5accb-temp-2-temp-023fa4271c9ca9611ade0400b88bfabe
Proposal
Requests-temp-c6e003375e64a961c93d00009db5accb-temp-2-temp-023fa4271c9ca9611ad-temp-1
Proposal Requests-temp-c6e003375e64a961c93d00009db5accb-temp-2-temp-1
Proposal Requests-temp-c6e003375e64a961c93d00009db5accb-temp-2-temp-2
Proposal
Requests-temp-c6e003375e64a961c93d00009db5accb-temp-2-temp-2-temp-1-temp-1
Proposal Requests-temp-c6e003375e64a961c93d00009db5accb-temp-2-temp-3
Proposal Requests-temp-c6e003375e64a961c93d00009db5accb-temp-2-temp-4
Proposal Requests-temp-c6e003375e64a961c93d00009db5accb-temp-2-temp-5
Proposal
Requests-temp-c6e003375e64a961c93d00009db5accb-temp-2-temp-e2aa0f35c99ba961356500009db5accb
[...]
I kept stopping, cleaning up the folders, and re-starting - and they
kept regenerating. I tried renaming the folders to eliminate the spaces
and I think that helped in one case - for the others I just moved the
folders outside of the mail area completely to let the sync finish.
Now that it's been stable for a day or two - I enabled sync in the
other direction. And after setting *all* the required parameters
instead of just most of them...it's working. But...I'm nervous about
moving the problem folders back over. I will say, if it makes any
difference, my primary server *was* running version 2.3.13 and I just
updated it to 2.3.17. The remote is also 2.3.17.
--
Daniel
