Hi, Charlie,
On Tue, 18 Mar 2003, Charlie Brady wrote:
>On Tue, 18 Mar 2003, Andreas Aardal Hanssen wrote:
>> On Mon, 17 Mar 2003, Charlie Brady wrote:
>> >We need to decide what we are doing and why we are doing it.
>> The first couple of lines of the IMAPdir draft state what we are trying to
>> do.
>That concerns me greatly. I thought that we were here because we wanted a
>secure, reliable, manageable IMAP daemon. But it seems that this project
>has taken the turn to define an abstract data structure.
Binc IMAP is a daemon that is designed to support multiple backends,
although this is not possible today.
The work on the daemon is very important, and it is progressing quite
quickly. But the number of posts regarding folder paths on this mailing
list suggests that it is something that we need to address properly.
Now, if we consider Binc IMAP as the only client of the structure used to
store folders, then we will end up with something that is no better than
Maildir++ - a format designed around one server. With "Trash" reserved,
with inaccurate(!) quota support, with all mailboxes under INBOX unless
the client sets up a namespace and the server interprets INBOX and
INBOX.INBOX as the same mailbox, and which allows IMAP servers to break
the IMAP standard by moving, deleting _or_ marking messages that are asked
by the client to be marked as deleted.
It's not a new mailbox format, it's just a thing that allows Courier-IMAP
to show multi level mailboxes to its clients.
Now - I am no fan of Maildir++, but since Maildir++ is the only thing we
have had so far that does more or less what we want, this is what Binc
supports today. It serves only as a way to represent a multi level Maildir
hierarchy.
But Maildir++ can not do what I want it to - it _is_ a Maildir _only_
structure. It also defines root level mailboxes in IMAP to be subfolders
of INBOX in the file system.
Our work with IMAPdir is an essential part of the development of Binc. But
we are not designing a structure that is only for Binc, but one that can
apply to the entire IMAP community because it seems everyone is doing it
their own way.
So - mailbox format independent, with no special cases except INBOX when
it comes to working mailboxes. Inspired by IMAP, but protocol independent.
>> Other than that, the goal of this specification is to find a way to
>> store multi level IMAP mailboxes on a file system in an unambiguous way.
>Hasn't Maildir++/Courier already done that (for better or worse)?
Unambiguous? INBOX and INBOX.INBOX both point to ~/Maildir, but
~/Maildir/.INBOX is inaccessible. Mailbox format independent? It's
Maildir only. With no special cases? INBOX needs to be translated to '.'
when working with the file system, but to INBOX when the server displays
it to the client.
>> Absolutely. Every server is free to choose a delimiter. However, the
>> IMAPdir specification suggests (perhaps suggests is too loose) the
>> seperators / or \, and that's no coincidence.
>> / and \ are natural delimiters in a folder hierarchy, and everyone's used
>> to it.
>They're not natural, they just happen to have been used by a variety of
>operating systems.
You're splitting hairs. \ and / are traditionally hierarchy seperators in
Windows and Unix, there's not need to argue about that.
>> '.' is not a delimiter, it's just a character that is allowed in
>> any file name. Now most shells allow escaped characters too.
>I don't think this is relevant.
>I think we should focus just on the hierarchy separator used by bincimap,
>and the file storage used by bincimap.
Well there it seems we will forever disagree. There is no reason to define
a format that only works for one application.
We will not save any time focusing on this being a Binc-only structure, if
time is of any concern.
>> The '.' comes, from what I understand, from NNTP. The news protocol
>> exports its news groups list using '.' as the delimiter, probably an
>> artifact from older days when having '/' in a file name was considered a
>> problem. There is no reason for other protocols not to export '/' or '\'
>> as the delimiter.
>The NNTP history is interesting. Other than that, I don't think we gain
>anything by discussing other protocols.
Are you at all paying attention to this discussion? The point of this
section is that the '.' is not traditionally a delimiter - it's nothing
more than an artifact from early protocol design.
Yet you say we should not allow it in IMAP mailbox names, because it is
used as a soft delimiter by Maildir++ when storing a folder name. It's
_not_ a delimiter traditionally, and users will enjoy being allowed to use
it in file names.
Since / and \ _are_ traditional delimiters, users are _more likely_ to
accept that \ and / are not allowed in a mailbox name.
>> If you read one of my previous mails on this list, you will see that this
>> approach does not work very well. You need to consider the facts of
>> UIDVALIDITY with delete and rename (renaming a superior also means that
>> inferiors should be renamed, taking UIDVALIDITY values into
>> consideration), deleting a superior does not imply deleting its inferiors
>> and that this approach would need a recursive descent traversal while the
>> one suggested by Maildir++ and IMAPdir does not.
>As I've said in previous emails, the UIDVALIDITY issue isn't
>insurmountable, and recursive descent traversal isn't difficult. But I
>doubt that I'll convince you to try using a directory tree, and it isn't
>something that needs to be urgently resolved - I can move/rename
>directories as required to fit in.
If foo is an mbox, foo/bar is a maildir and foo/bar/foo is an mbox, how
would you represent this structure with your suggested scheme?
The answer is - you _can't_. And no, Maildir is not the only mailbox
format around. Allowing mail users to use different mailbox formats
together in the same structure is a huge gain with IMAPdir.
A practical example is that many universities still use the Unix spool
mbox style INBOX, that is /var/mail/<user> or /var/spool/mail/<user>.
Being allowed in an environment like that to use Maildir for all other
mailboxes is a great feature of IMAPdir.
Also, if for some reason a "top secret" folder called "work/NSAproof" is
only accessible via ODBC or something, then a spec file could reside at
any level in the same structure. Now - if the root folder is a Maildir,
and the IMAP user decides to delete the Maildir, then the other mailboxes
are left _intact_.
>> This approach also fails for mailbox formats other than Maildir or other
>> directory based formats. You can not create the same structure using, say,
>> mbox.
>You can, except that mbox has the natural limitation that it can't have
>subfolders.
Of course it can! There is _no natural limitation_. Just use the spec that
IMAPdir suggests.
Here is a quote from Mark Crispin, who is an undisputable fan of mbox:
"So basically you're using a single dot as a soft hierarchy delimiter in
the filenames, so you can have "dual use names" with flat-file formats
such as traditional UNIX and mbx format. Seems like a reasonable way to
do it.
-- Mark --"
>> >We start getting ourselves, and probably others, confused if we try to
>> >allow either . or / to be used as part of a folder name. The only simple
>> To this I do not agree. If '.' is not the hierarchy delimiter, then there
>> is no reason to disallow its use in mailbox names.
>There is. '.' is defined in Maildir++ to be the standin character for the
>heirarchy separator. If we need to map back from the file system to IMAP
>(during LIST), then we need to know whether '.' is literal or is a standin
>character. Is "my.stuff" a folder called "stuff" within "my", or a folder
>called "my.stuff"? Well that probably depends on exactly which IMAP daemon
>is exporting it at this moment. I don't think that is satisfactory.
No - it is absolutely not dependent of which daemon exports it, any more
than it is dependent on wether it is Gimp, Abobe Photoshop or Paint Shop
Pro that shows a PNG file.
The IMAPdir structure will provide the exact mapping of file names to
mailbox names and any client that does not follow this convention is in
breach with IMAPdir, and on its own.
>> With '/'. If someone
>> wants to make a mailbox called
>> "Before/after year 2000"
>> Then this should naturally be allowed when a '.' is the delimiter.
>You've confused me. If '/' is the separator, then "Before/after year 2000"
As the last sentence says "when '.' is the delimiter", '/' is in this
example _not_ the seperator. Delimiter <=> Seperator.
In that example, the folder is on the root level, and it is called
"Before/after year 2000". Not "Before" being the parent of "after year..".
>> "2nd. world war"
>> should be allowed when '/' is the delimiter.
>How would you store it in the file system? How would you list that folder?
If '/' is the delimiter, as in this example, then it would be stored
exacly as it is, or as bash would display it:
./2nd.\ world\ war/
>> Not only does this have an
>> easy logical implementation, but it is makes sense to users too.
>Most users won't see the strings that are used in the IMAP protocol, and
>won't see the file system storage.
You're mixing up a mailbox representation and its name again. The user
is less likely to accept not being able to use '.' than '/' or '\'.
>> The reason for the exception with the single dot at the start of a mailbox
>> name is to allow Maildir++ users to convert painlessly.
>If you want the usage to be painless for users, and you don't want to have
>a conversion process which renames folders, then you will need to use the
>leading period consistently. Or have the whole prefix configuration, and
>let the local admin decide whether to rename folders or leave them alone
>and use a period prefix. Either will be much better than having any
>ambiguity or special case.
The admin is not required to rename anything. It's up to each and every
user wether they want to rename their folders or not.
>> The side effect
>> for Maildir++ users is that they can not have a root level folder named
>> "cur", "new" or "tmp" containing mails. They will have to live with this
>> until the admin converts their depositories to true IMAPdir.
>Do you have an algorithm for converting to "true IMAPdir"? That might help
>me understand exactly what you are proposing.
We are discussing this in this very thread!
First we need to agree that there is no reason whatsoever to disallow '.'
in a mailbox name.
Then we need to agree on a one-to-one map of a mailbox name to a file name
and back. And there must be no inconsistency.
Finally we need to agree that IMAPdir is _not_ Maildir++. It's not
one-to-one compatible because it supports _more_ than Maildir++ does.
Conversion from Maildir++ to IMAPdir is completely painless. Conversion
back to Maildir++ would require a script that puts dots in front of all
Maildir subfolders' file names, and converts the characters that IMAPdir
allows but Maildir++ does not allow, and puts "maildirfolder" files inside
every subfolder, then makes sure that '.' is a Maildir.
>> >OTOH, we have a file and directory hierarchy, and we wish to export a
>> >subset of it by IMAP. There is no requirement that I can see that all
>> >files and directories can be referred to via an IMAP folder name. Don't
>> >invent that requirement.
>> This you will have to explain to me. The representation of the mailbox
>> name _must_ map one to one with the name of the mailbox as exported by
>> this structure.
>Each IMAP folder name must map unambigously to a maildir directory path,
>and that directory path must map back to the same IMAP folder name. But
^^^^^^^^^^^^^^
This is a major point.
>not every directory must have a folder name. Consider /tmp/foo, for
>instanec. If that makes sense to you, then you should then consider ~/foo,
>then ~/Maildir/foo. None of these needs to have an IMAP folder
>representation.
/tmp/foo isn't exactly inside the IMAPdir, unless you symlink it, so it's
completely uninteresting. Only entries at the root level inside an IMAPdir
are mapped.
>> In that sense, yes - in IMAPdir you _must_ be able to
>> refer to each mailbox name's file representation with an IMAP mailbox.
>That is only possible if you define certain file representations as not
>part of IMAPdir - which is the same thing that I am saying.
No, we just need to define what _is_ an IMAPdir entry. And if we use
Caskey's escape characters, they all are.
>> The file system says something about the structure to an IMAPdir compliant
^^
>> client.
>There are no IMAPdir clients.
What role does this statement play in this thread? Are you saying that the
design should play a different role for the IMAP community until somebody
starts using it?
>> This client must be required to represent this structure exactly
>> the way IMAPdir defines it, to the client.
>Why? Why does IMAPdir matter, except in so far as it helps us (you) build
>an IMAP daemon which works with existing IMAP clients?
Off topic. With no structure, there is no IMAP folder hierarchy.
>> I have a refinement. A single dot at the start of a filename refers to a
>> single dot at the start of an IMAP mailbox.
>This conflicts with Courier/Maildir++.
In what way? Courier/Maildir++ clients will not be able to interpret the
contents of IMAPdir - true. That is one of the side effects of IMAPdir not
being equivalent, but a replacement for, Maildir++.
>> July.2nd -> July/2nd
>> This is completely unambiguous.
>Perhaps, but I don't think it is compatible with Courier.
>This one worries me too:
>..foo -> ./foo
>By that you mean "foo" within ".", do you?
Yes - exactly. '.' is a regular character just like "a" "b" and "c".
If we follow Caskey's suggested escapechar-approach, then the exception is
that a leading '.' need not be escaped in the file system to represent a
true '.' in a mailbox name.
Andy :-)
--
Andreas Aardal Hanssen | http://www.andreas.hanssen.name/gpg
Author of Binc IMAP | Nil desperandum
