On Wed, 19 Mar 2003, Andreas Aardal Hanssen wrote: > Hi, Charlie,
Hi Andreas, > On Tue, 18 Mar 2003, Charlie Brady wrote: > >On Tue, 18 Mar 2003, Andreas Aardal Hanssen wrote: > >> On Mon, 17 Mar 2003, Charlie Brady wrote: > >> >We need to decide what we are doing and why we are doing it. > >> The first couple of lines of the IMAPdir draft state what we are trying to > >> do. > >That concerns me greatly. I thought that we were here because we wanted a > >secure, reliable, manageable IMAP daemon. But it seems that this project > >has taken the turn to define an abstract data structure. > > Binc IMAP is a daemon that is designed to support multiple backends, > although this is not possible today. And we are talking about a single backend just now. > The work on the daemon is very important, and it is progressing quite > quickly. But the number of posts regarding folder paths on this mailing > list suggests that it is something that we need to address properly. Sure. > Now, if we consider Binc IMAP as the only client of the structure used to > store folders, then we will end up with something that is no better than > Maildir++ - a format designed around one server. With "Trash" reserved, > with inaccurate(!) quota support, with all mailboxes under INBOX unless > the client sets up a namespace and the server interprets INBOX and > INBOX.INBOX as the same mailbox, and which allows IMAP servers to break > the IMAP standard by moving, deleting _or_ marking messages that are asked > by the client to be marked as deleted. Please don't confuse the issues between Courier IMAP and Maildir++. Not all of the issues you mention are implicit in the Maildir++ file system format as defined here: http://www.inter7.com/courierimap/README.maildirquota.html If you are arguing for throwing out Maildir++ as a "standard" and defining a better system of hierarchical folder storage, then I certainly won't stand in your way. But if you argue that way, you should abandon all of Maildir++, and argue from a fresh starting position. > It's not a new mailbox format, it's just a thing that allows Courier-IMAP > to show multi level mailboxes to its clients. And to manage quotas (however poorly) - that does need to be seen as an added feature over maildir. > Now - I am no fan of Maildir++, but since Maildir++ is the only thing we > have had so far that does more or less what we want, What does a nested set of maildirs not provide what bincimap needs? > this is what Binc supports today. It serves only as a way to represent a > multi level Maildir hierarchy. Does this mean that the only parts of Maildir++ you wish to use are the algorithm to collapse nested folder names to Maildir directory names within a single directory? > But Maildir++ can not do what I want it to - it _is_ a Maildir _only_ > structure. It also defines root level mailboxes in IMAP to be subfolders > of INBOX in the file system. The document quoted above does, but that is out of place in that document. Does the document define an on-disk storage format, or does it define the operation of an IMAP daemon? Which of those things are we trying to remain compatible with? > Our work with IMAPdir is an essential part of the development of Binc. But > we are not designing a structure that is only for Binc, but one that can > apply to the entire IMAP community because it seems everyone is doing it > their own way. Which is not too much of a problem if one can be easily transformed into the other. To my thinking it is more important that we have competition in the IMAP daemon space (so we can choose the one which does the job we need best) than that there be the possibility to seemlessly switch from one daemon to another. Unless other IMAP daemon development teams are buying into a new backend format, I don't think there's any point in trying to anticipate their needs. They will decide those for themselves. Bincimap will only be able to establish a de facto standard if it quickly becomes a robust and widely deployed IMAP daemon. IMO it will only do that if we focus on exactly what it needs in order to be that. > So - mailbox format independent, with no special cases except INBOX when > it comes to working mailboxes. Inspired by IMAP, but protocol independent. I don't think it can be protocol independent, because the hierarchy separator is only an IMAP concept, and the hierarchy separator is intrinsic to our choice of transform from folder name to directory name (and vice versa). > >> Other than that, the goal of this specification is to find a way to > >> store multi level IMAP mailboxes on a file system in an unambiguous way. > >Hasn't Maildir++/Courier already done that (for better or worse)? > > Unambiguous? INBOX and INBOX.INBOX both point to ~/Maildir, but > ~/Maildir/.INBOX is inaccessible. Mailbox format independent? It's > Maildir only. With no special cases? INBOX needs to be translated to '.' > when working with the file system, but to INBOX when the server displays > it to the client. INBOX.INBOX and INBOX being equivalent is not implicit in the Maildir++ file layout, but is a consequence of Courier implementing 'INBOX' as a namespace. I presume that bincimap will be sensible enough not to have "INBOX" as the default namespace, so bincimap won't suffer from this problem (although there will probably be some Courier incompatibilities as a result of that decision). INBOX is and will always be a special case (rfc 2060 required). ~/Maildir/.INBOX being inaccessible is correct according to rfc 2060. Maildir only I have no problem with, but I also see no problem with replacing a maildir in the Maildir++ scheme with am mbox of the same name and having things work OK. Out of this lot, all I see that you *might* want to address is ~/Maildir/.INBOX being inaccessible via IMAP. Why do you want to address that? If you have such a maildir, and you want it visible via IMAP, rename it in the filesystem. There's no need for code in bincimapd to deal with that maildir. I can't see why or how you'd have such a maildir anyway. > >> Absolutely. Every server is free to choose a delimiter. However, the > >> IMAPdir specification suggests (perhaps suggests is too loose) the > >> seperators / or \, and that's no coincidence. > >> / and \ are natural delimiters in a folder hierarchy, and everyone's used > >> to it. > >They're not natural, they just happen to have been used by a variety of > >operating systems. > > You're splitting hairs. \ and / are traditionally hierarchy seperators in > Windows and Unix, there's not need to argue about that. That's true, but I don't see the usefulness of discussing them in this context. IMO we should be concerning ourselves only with IMAP folder references and storage in the file system. > >> '.' is not a delimiter, it's just a character that is allowed in > >> any file name. Now most shells allow escaped characters too. > >I don't think this is relevant. > >I think we should focus just on the hierarchy separator used by bincimap, > >and the file storage used by bincimap. > > Well there it seems we will forever disagree. There is no reason to define > a format that only works for one application. There is. It means that you can finish that application. > We will not save any time focusing on this being a Binc-only structure, if > time is of any concern. Time is of concern, and I disagree with you. I think that the only applications we should be concerned about are bincimap and any mail delivery agents. The mail delivery agents will either deliver to a single named maildir (and won't be concerned about hierarchical relationships) or will deliver to a single named Maildir++ (i.o.w, they will be maildir++ quota aware, but won't be concerted with hierarchical relationships beyond that). Any other concerns should be considered later. The big one to consider will be other mail user agents which do direct maildir access. I don't think they're worth worrying about, because that is what IMAP is for (my opinion). > Are you at all paying attention to this discussion? To every word. > The point of this section is that the '.' is not traditionally a > delimiter - it's nothing more than an artifact from early protocol > design. > > Yet you say we should not allow it in IMAP mailbox names, because it is > used as a soft delimiter by Maildir++ when storing a folder name. It's used as the IMAP hierarchy delimiter by Courier. Therefore it doesn't appear within a folder name in a Courier installation. You've said that Courier compatibility is a requirement, therefore this issue can't be ignored. > It's _not_ a delimiter traditionally, and users will enjoy being > allowed to use it in file names. Yep. > Since / and \ _are_ traditional delimiters, users are _more likely_ to > accept that \ and / are not allowed in a mailbox name. No disagreement from me there. The simplest way to implement this, IMO, is to use '/' as the hierarchy separator, and store nested maildirs. > >As I've said in previous emails, the UIDVALIDITY issue isn't > >insurmountable, and recursive descent traversal isn't difficult. But I > >doubt that I'll convince you to try using a directory tree, and it isn't > >something that needs to be urgently resolved - I can move/rename > >directories as required to fit in. > > If foo is an mbox, foo/bar is a maildir and foo/bar/foo is an mbox, how > would you represent this structure with your suggested scheme? > > The answer is - you _can't_. Correct(*). But then I don't believe that you need to. Firstly I'd argue that there isn't a need to support the mbox format. But even assuming that you do have mbox support, all you need to do is say that mbox folders can't have subfolders. IMAP's protocol definition allows you to "just say no". Why do you need to support foo as an mbox, foo/bar is a maildir and foo/bar/foo is an mbox? How would you create that structure via IMAP? Can you provide a practical situation where it is *necessary* to provide that support? (*) No, incorrect. I would convert both the mboxen to maildirs, and then have a nested structure, foo/bar/foo within foo/bar within foo. Hence I would provide the support that you asked for. > And no, Maildir is not the only mailbox > format around. Allowing mail users to use different mailbox formats > together in the same structure is a huge gain with IMAPdir. That is speculation. What is the real world benefit of providing different mailbox formats? How does the user, via an IMAP client, create mailboxes with differing formats? If we accept that it is useful for mail users to use different mailbox formats, why do the mailboxes need to live in the same structure? > A practical example is that many universities still use the Unix spool > mbox style INBOX, that is /var/mail/<user> or /var/spool/mail/<user>. > Being allowed in an environment like that to use Maildir for all other > mailboxes is a great feature of IMAPdir. INBOX is always, and will always be a special case. > Also, if for some reason a "top secret" folder called "work/NSAproof" is > only accessible via ODBC or something, then a spec file could reside at > any level in the same structure. Now - if the root folder is a Maildir, > and the IMAP user decides to delete the Maildir, then the other mailboxes > are left _intact_. I don't quite understand this example, but in any case it does seem contrived. In any case, we should cross this bridge when we come to it. [BTW, have you done any reading on Extreme Programming, e.g. http://www.extremeprogramming.org/? Worth reading, and highly relevant to this discussion. Solve today's problems today. Solve tomorrow's problems tomorrow, if you have time.] > >> This approach also fails for mailbox formats other than Maildir or other > >> directory based formats. You can not create the same structure using, say, > >> mbox. > >You can, except that mbox has the natural limitation that it can't have > >subfolders. > > Of course it can! There is _no natural limitation_. Just use the spec that > IMAPdir suggests. > > Here is a quote from Mark Crispin, who is an undisputable fan of mbox: > > "So basically you're using a single dot as a soft hierarchy delimiter in > the filenames, so you can have "dual use names" with flat-file formats > such as traditional UNIX and mbx format. Seems like a reasonable way to > do it. > > -- Mark --" OK. So we have a tradeoff to consider. Providing support for hierarchies of mbox files requires a character (other than /) to be set aside as a hierarchy limiter. Having a character set aside means adding complications in the name mapping of files/directories to IMAP folders. Is it important that we provide support for hierarchies of mbox files? [Mark Crispin seems to say "no", since, AIUI, uw-imap doesn't have such support.] > >There is. '.' is defined in Maildir++ to be the standin character for the > >heirarchy separator. If we need to map back from the file system to IMAP > >(during LIST), then we need to know whether '.' is literal or is a standin > >character. Is "my.stuff" a folder called "stuff" within "my", or a folder > >called "my.stuff"? Well that probably depends on exactly which IMAP daemon > >is exporting it at this moment. I don't think that is satisfactory. > > No - it is absolutely not dependent of which daemon exports it, any more > than it is dependent on wether it is Gimp, Abobe Photoshop or Paint Shop > Pro that shows a PNG file. That's an invalid analogy. Those applications display a set of pixels. We are talking about interpretation of directory names as IMAP folders. RFC 2060 tells us "The interpretation of mailbox names is implementation-dependent". I agree with RFC 2060. > The IMAPdir structure will provide the exact mapping of file names to > mailbox names and any client that does not follow this convention is in > breach with IMAPdir, and on its own. OK. > As the last sentence says "when '.' is the delimiter", '/' is in this > example _not_ the seperator. Delimiter <=> Seperator. > > In that example, the folder is on the root level, and it is called > "Before/after year 2000". Not "Before" being the parent of "after year..". I don't see how this can be stored in any scheme we've mentioned. But that's not fatal. Whatever isn't/can't be supported, we just say "NO". IMAP allows that. > >> "2nd. world war" > >> should be allowed when '/' is the delimiter. > >How would you store it in the file system? How would you list that folder? > > If '/' is the delimiter, as in this example, then it would be stored > exacly as it is, or as bash would display it: > > ./2nd.\ world\ war/ You can't store "/" in a *nix file system. > >> Not only does this have an > >> easy logical implementation, but it is makes sense to users too. > >Most users won't see the strings that are used in the IMAP protocol, and > >won't see the file system storage. > > You're mixing up a mailbox representation and its name again. The user > is less likely to accept not being able to use '.' than '/' or '\'. True, although it hasn't stopped people from using Courier. > >> The reason for the exception with the single dot at the start of a mailbox > >> name is to allow Maildir++ users to convert painlessly. > >If you want the usage to be painless for users, and you don't want to have > >a conversion process which renames folders, then you will need to use the > >leading period consistently. Or have the whole prefix configuration, and > >let the local admin decide whether to rename folders or leave them alone > >and use a period prefix. Either will be much better than having any > >ambiguity or special case. > > The admin is not required to rename anything. It's up to each and every > user wether they want to rename their folders or not. If the admin changes system software which would change the user's view of their data, the admin should do system wide migration so that the data appear not to have changed to the user. > >> The side effect > >> for Maildir++ users is that they can not have a root level folder named > >> "cur", "new" or "tmp" containing mails. They will have to live with this > >> until the admin converts their depositories to true IMAPdir. > >Do you have an algorithm for converting to "true IMAPdir"? That might help > >me understand exactly what you are proposing. > > We are discussing this in this very thread! I don't see anything in this thread about not having "cur", "new" or "tmp". That's what I don't understand. > First we need to agree that there is no reason whatsoever to disallow '.' > in a mailbox name. If '.' is the hierarchy separator used by the imap daemon, then there's no way for a client to create or request a mailbox which contains a period. The periods are hierarchy separators. OK, we therefore use '/' as the hierarchy separator. If we do so, we don't allow '/' in mailbox names. > Then we need to agree on a one-to-one map of a mailbox name to a file name > and back. And there must be no inconsistency. Agreed. There should be no fuzziness about leading periods. > Finally we need to agree that IMAPdir is _not_ Maildir++. It's not > one-to-one compatible because it supports _more_ than Maildir++ does. Not only that, it is incompatible with Maildir++. The interpretation of a leading period in a maildir name is an incompatibility. Moreover, I consider the incompatibility unnecessary, if not gratuitous. Here's what I recommend instead. When the imap daemon converts from folder name to maildir directory name, it prepends a configuration prefix string. In the case of an installation which is compatible with an existing Courier implementation, that string will be "./Maildir/.". In the case of a new installation which wishes to store its maildir folders without the leading prefix, the string will be "./Maildir/". No need for any special case code, no incompatibility with existing Courier installations, no migration necessary, and no enforced policy of leading dots in mailbox file/directories. > Conversion from Maildir++ to IMAPdir is completely painless. Either the users will have the pain of having their mailboxes change name, or an admin will need to run a script to remove the leading period on each folder name system wide. > Conversion back to Maildir++ would require a script that puts dots in > front of all Maildir subfolders' file names, Not needed if both with and without are compatible with bincimap, and only a runtime configuration option need change. > and converts the characters that IMAPdir allows but Maildir++ does not allow, What would you convert them to? > and puts "maildirfolder" files inside every subfolder, then makes sure > that '.' is a Maildir. OK. > >> This you will have to explain to me. The representation of the mailbox > >> name _must_ map one to one with the name of the mailbox as exported by > >> this structure. > >Each IMAP folder name must map unambigously to a maildir directory path, > >and that directory path must map back to the same IMAP folder name. But > > This is a major point. > > >not every directory must have a folder name. Consider /tmp/foo, for > >instanec. If that makes sense to you, then you should then consider ~/foo, > >then ~/Maildir/foo. None of these needs to have an IMAP folder > >representation. > > /tmp/foo isn't exactly inside the IMAPdir, unless you symlink it, so it's > completely uninteresting. Only entries at the root level inside an IMAPdir > are mapped. You've missed my point, which is that files and directories inside the directory hierarchy can be uninteresting (and therefore ignored). If I copy .pinerc into ~/Maildir, there is no reason that it must be interpreted as a mailbox and exported by the imap daemon. The same argument applies to ~/Maildir/foo. Just because I was silly enough to put it there does not mean that the IMAP daemon *has* to make it visible via IMAP. Ditto for ~/Maildir/INBOX, ~/Maildir/.INBOX, etc. > >> In that sense, yes - in IMAPdir you _must_ be able to > >> refer to each mailbox name's file representation with an IMAP mailbox. > >That is only possible if you define certain file representations as not > >part of IMAPdir - which is the same thing that I am saying. > > No, we just need to define what _is_ an IMAPdir entry. And if we use > Caskey's escape characters, they all are. Why do they need to be? > >> The file system says something about the structure to an IMAPdir compliant > ^^ > >> client. > >There are no IMAPdir clients. > > What role does this statement play in this thread? Are you saying that the > design should play a different role for the IMAP community until somebody > starts using it? The design should satisfy bincimapd's community. Nothing else at this stage. > >> This client must be required to represent this structure exactly > >> the way IMAPdir defines it, to the client. > >Why? Why does IMAPdir matter, except in so far as it helps us (you) build > >an IMAP daemon which works with existing IMAP clients? > > Off topic. With no structure, there is no IMAP folder hierarchy. Correct. The structure needs to be designed to support an IMAP folder hierarchy. No more and no less. Satisfying "IMAPdir" is a circular argument, and therefore, not useful. > >> I have a refinement. A single dot at the start of a filename refers to a > >> single dot at the start of an IMAP mailbox. > >This conflicts with Courier/Maildir++. > > In what way? Users will see their "foo" mailbox appear one day as ".foo". This is undesirable (I think we will all agree). > If we follow Caskey's suggested escapechar-approach, then the exception is > that a leading '.' need not be escaped in the file system to represent a > true '.' in a mailbox name. Who does this special case help? -- Charlie
