On Thu, 16 Dec 2004, Jeremy Kitchen wrote:
>just curious, but why is binc trying to lock the mailbox? isn't the maildir
>format supposed to eliminate the need for locking?
Hi, Jeremy.
Good question. Actually the answer is that yes, Maildir was designed so no
locking would be needed. And yes, Binc IMAP locks the maildir... The
reason is that Dr. Bernstein, who designed this mailbox format, only had
qmail-pop3d and qmail-local in mind (and, quite frankly, it was designed
for mailboxes with few messages also). For both these apps, no locking is
required. Unfortunately, IMAP4 has other requirements when assigning
unique identifiers to each message that force us to use the lock after
all.
In the POP3 protocol, the unique name of a message qualifies as a UID. So
no matter what messages come and go and it what order, POP3 can just look
at Maildir messages' file names. Clients can sync their local storage with
the POP3 server by comparing the UIDs it advertises, disregarding their
textual format.
In IMAP, the unique identifiers are assigned in strictly ascending order
on a per mailbox basis, starting at a fixed number (like 1) and increasing
one after one. So the first message seen by the IMAP server gets the UID
1. The next gets UID 2. A message can then only get UID 4 if the server
has first announced the arrival and then the expunge of UID 3. These UID
numbers are unique for a message, and shared among all concurrent clients.
So, when two Binc IMAP server instances simultaneously open a maildir and
start assigning UIDs, they have to be strictly synchronized, or they may
end up assigning the same UID to different messages. The first instance
locks, scans and assignes, then writes its updated UID list to the cache
file and releases the lock. The last instance locks the mailbox, checks
the cache file left by the first instance to see what UIDs the first
instance assigned, scans, assigns and finally releases the lock again.
This avoids all concurrency problems like - what happens if someone with a
haywire "rm" command deletes messages while Binc is scanning. Since only
one instance can scan one mailbox at a time, external access poses no
problem whatsoever.
Hope this answers your question.
Andy :-)
PS: And yes, I know it's been mentioned before, but the perfect mailbox
format has yet to be discovered. For example, one that allows snappy
access even with 300000 messages, one that allows setting of flags
without race conditions (such as the renaming in Maildir, which is
very inefficient with big mailboxes with many accessors, such as a
shared mailing list), one that allows moving, adding and deleting
messages fast, and in a crash-safe way. One that deals with modern
principles such as indexing, shared mailboxes and high concurrency.
That would be something. :)
--
Andreas Aardal Hanssen | http://www.andreas.hanssen.name/gpg
Author of Binc IMAP | "It is better not to do something
http://www.bincimap.org/ | than to do it poorly."
