Re: RENAME and imap compliance
RENAME, on the other hand, is broken on almost all servers. Maybe. But it is not impossible to fix RENAME in servers staying complaint with IMAP4rev1. And I don't buy the argument that a server can't store 4bytes UIDVALIDITY somewhere when mailbox is deleted/renamed. The Cyrus server (including the one running on imap.andrew.cmu.edu) advertises that it does not implement RENAME in a compliant fashion NO_ATOMIC_RENAME capability. This used to be a problem in a single implementations. Several servers derived from Cyrus don't have this issue. Alexey
Re: RENAME and imap compliance
Andreas Aardal Hanssen writes: And I don't buy the argument that a server can't store 4bytes UIDVALIDITY somewhere when mailbox is deleted/renamed. Do you understand the problem with UIDVALIDITY and RENAME? Not only do you have to store your 4 bytes, but you will have to store all UIDVALIDITY values that any folder has had ever, after renames, forever. Which sounds bad, but is it really? The amoung of mail grows over time too, and after a few years a biggish server might have many gigabytes of old mail stored and maybe 100kbytes of old UIDVALIDITY name/value pairs. --Arnt
Re: RENAME and imap compliance
Andreas Aardal Hanssen [EMAIL PROTECTED] writes: On Wed, 22 Jan 2003, Alexey Melnikov wrote: RENAME, on the other hand, is broken on almost all servers. Maybe. But it is not impossible to fix RENAME in servers staying complaint with IMAP4rev1. Compliant is one thing, but bumping UIDVALIDITY for source and destination mailboxes when renaming means that most offline clients have to re-scan the folder and download headers. Which means that RENAME in practise will be _slower_ than create, copy, delete. So why do we need RENAME? Not all clients are offline clients, so not all clients need to perform the re-scan. And I don't buy the argument that a server can't store 4bytes UIDVALIDITY somewhere when mailbox is deleted/renamed. Do you understand the problem with UIDVALIDITY and RENAME? Not only do you have to store your 4 bytes, but you will have to store all UIDVALIDITY values that any folder has had ever, after renames, forever. This is to prevent the situation where the same UID/UIDVALIDITY points to two different messages at two different times. Assuming the UIDVALIDITY value must grow, as the specification says, I guess you only have to store the highest last used one. This seems like a rather trivial thing to implement, compared with all other complex things in IMAP. But I agree that the current situation is no good. Either clarify the document, or if doesn't seem possible to do so, remove RENAME.
Re: RENAME and imap compliance
Andreas Aardal Hanssen wrote: On Wed, 22 Jan 2003, Alexey Melnikov wrote: RENAME, on the other hand, is broken on almost all servers. Maybe. But it is not impossible to fix RENAME in servers staying complaint with IMAP4rev1. Compliant is one thing, but bumping UIDVALIDITY for source and destination mailboxes when renaming means that most offline clients have to re-scan the folder and download headers. Once again, you only have to bump UIDVALIDITY if the current value is smaller than the biggest. Which means that RENAME in practise will be _slower_ than create, copy, delete. So why do we need RENAME? And I don't buy the argument that a server can't store 4bytes UIDVALIDITY somewhere when mailbox is deleted/renamed. Do you understand the problem with UIDVALIDITY and RENAME? Not only do you have to store your 4 bytes, but you will have to store all UIDVALIDITY values that any folder has had ever, after renames, forever. No you don't. You just store the biggest UIDVALIDITY ever for the mailbox. This is to prevent the situation where the same UID/UIDVALIDITY points to two different messages at two different times. Alexey
Re: RENAME and imap compliance
On Wed, 22 Jan 2003, Alexey Melnikov wrote: I am taking this offline to clarify some stuff... Andreas Aardal Hanssen wrote: Which means that RENAME in practise will be _slower_ than create, copy, delete. Please, explain how this follows. I wrote Compliant is one thing, but bumping UIDVALIDITY for source and destination mailboxes when renaming means that most offline clients have to re-scan the folder and download headers. Which means that RENAME in practise will be _slower_ than create, copy, delete. So why do we need RENAME? Most clients have to re-scan all headers when UIDVALIDITY bumps. This does not apply to - for instance - Pine. This scan will anyway often be slower than submitting CREATE, then COPY, then DELETE. If the IMAP-server gets a RENAME, sure it's fast for the _server_, but the _client_ will be very slow. So - why do we need RENAME? An online client will not bother too much with having to rescan the folder, although doing a COPY on the server side will necessarily be faster, unless ofcourse the server is seriously slow. Andy So why do we need RENAME? Alexey -- Andreas Aardal Hanssen - Binc IMAP http://www.bincimap.andreas.hanssen.name/
Re: RENAME and imap compliance
Simon Josefsson wrote: Andreas Aardal Hanssen [EMAIL PROTECTED] writes: On Wed, 22 Jan 2003, Alexey Melnikov wrote: RENAME, on the other hand, is broken on almost all servers. Maybe. But it is not impossible to fix RENAME in servers staying complaint with IMAP4rev1. Compliant is one thing, but bumping UIDVALIDITY for source and destination mailboxes when renaming means that most offline clients have to re-scan the folder and download headers. Which means that RENAME in practise will be _slower_ than create, copy, delete. So why do we need RENAME? Not all clients are offline clients, so not all clients need to perform the re-scan. So, if the server doesn't bump UIDVALIDITY, the disconnected client doesn't have to rescan. And finally, if the server doesn't support UIDPLUS, the disconnected client has to perform rescan after COPY anyway. And I don't buy the argument that a server can't store 4bytes UIDVALIDITY somewhere when mailbox is deleted/renamed. Do you understand the problem with UIDVALIDITY and RENAME? Not only do you have to store your 4 bytes, but you will have to store all UIDVALIDITY values that any folder has had ever, after renames, forever. This is to prevent the situation where the same UID/UIDVALIDITY points to two different messages at two different times. Assuming the UIDVALIDITY value must grow, as the specification says, I guess you only have to store the highest last used one. This seems like a rather trivial thing to implement, compared with all other complex things in IMAP. Exactly. But I agree that the current situation is no good. Either clarify the document, or if doesn't seem possible to do so, remove RENAME. I agree. I was just arguing than the first can and should be done. I know several clients that use RENAME. Alexey
Re: RENAME and imap compliance
Andreas Aardal Hanssen wrote: If they do need to grow, server would have to remember the last UIDVALIDITY for deleted mailboxes, so RENAME could check if the UIDVALIDITY must be changed. I don't like that behaviour. It's very unnecessary and requires permanent space for deleted mailboxes. Exactly! Which is why the RFC recommends the use of UNIX time (time(NULL)), in which case it is unnecessary to remember any previous values. Unless your server is running NTP client and local time goes back. Alexey
Re: RENAME and imap compliance
On Wed, 22 Jan 2003, Alexey Melnikov wrote: Andreas Aardal Hanssen wrote: If they do need to grow, server would have to remember the last UIDVALIDITY for deleted mailboxes, so RENAME could check if the UIDVALIDITY must be changed. I don't like that behaviour. It's very unnecessary and requires permanent space for deleted mailboxes. Exactly! Which is why the RFC recommends the use of UNIX time (time(NULL)), in which case it is unnecessary to remember any previous values. Unless your server is running NTP client and local time goes back. Alexey What happens if the alternative UIDVALIDITY log file gets messed up? Andy -- Andreas Aardal Hanssen - Binc IMAP http://www.bincimap.andreas.hanssen.name/
Re: RENAME and imap compliance
Andreas Aardal Hanssen writes: What happens if the alternative UIDVALIDITY log file gets messed up? If the server's persistent storage is messed up, every guarantee breaks. One example among dozens: If the server's UIDNEXT storage is hand-edited, the UID grows guarantee is broken. --Arnt
Re: RENAME and imap compliance
Andreas Aardal Hanssen writes: On Wed, 22 Jan 2003, Arnt Gulbrandsen wrote: Andreas Aardal Hanssen writes: What happens if the alternative UIDVALIDITY log file gets messed up? If the server's persistent storage is messed up, every guarantee breaks. One example among dozens: If the server's UIDNEXT storage is hand-edited, the UID grows guarantee is broken. Exactly - so if you set the time on your computer, do you agree that conditions may be broken? Is that relevant? I actually don't, I think it's too careless, but I don't see that as relevant. What I'm saying is that storing the maximum UIDVALIDITY for each past mailbox does not introduce any new failure mode. If an IMAP server runs into file system corruption, any of the following can conceivably happen: - UIDNEXT can shrink - folders can be spontaneously renamed - messages or flags can change, silently invalidating client caches - the UID of a message can change and if the UIDVALIDITY name/value pairs are stored in a file, - the RENAME/UIDVALIDITY guarantee can be broken If you can accept the first four, the last one should be okay too. So when we discuss RENAME, we can't start with what-ifs that don't apply exclusively to IMAP. Agreed. We should, however, try to make the IMAP features consistent with each other. --Arnt
Re: RENAME and imap compliance
Hi Andreas, --On Wednesday, January 22, 2003 12:58:39 PM +0100 Andreas Aardal Hanssen [EMAIL PROTECTED] wrote: | Compliant is one thing, but bumping UIDVALIDITY for source and | destination mailboxes when renaming means that most offline clients have | to re-scan the folder and download headers. | | Which means that RENAME in practise will be _slower_ than | create, copy, delete. So why do we need RENAME? | | Most clients have to re-scan all headers when UIDVALIDITY bumps. This does | not apply to - for instance - Pine. This scan will anyway often be slower | than submitting CREATE, then COPY, then DELETE. That's not true. When doing CREATE/COPY/DELETE the client still has to scan the copy destination mailbox to determine the UIDs assigned to the copied messages unless the server supports UIDPLUS (and quite a few do not). And in the case of a hierarchy rename the client would have to do CREATE/COPY/DELETE for each mailbox in the hierarchy (I would argue that users expect the hierarchical rename behaviour rather than having just the top-level mailbox renamed without the children). -- Cyrus Daboo
Re: RENAME and imap compliance
On Wed, 22 Jan 2003, Cyrus Daboo wrote: It seems to me that the problem here is the requirement to keep the UIDVALIDITY the same when doing RENAME. Where is that required? What's required is that the last-assigned UID has to be saved, unless the UIDVALIDITY changes. I propose the following solution: remove that constraint and require the renamed mailbox to get a new UIDVALIDITY value as if it were simply being created. That's a way of solving the problem, since then the last-assigned UID becomes moot. The only problem with this is the hierarchical rename because the uidvalidities on the child mailboxes will also be changed and I think it may be too much to have those new uidvalidities returned as untagged responses - particularly if the hierarchy is large. Correct. And since this should be atomic, this can effectively lock the entire hierarchy for a substantial period of time while all this is being done. Its worth noting that the current rename behaviour is only of benefit to the client actually doing the rename. Other clients that connect after the rename will have no knowledge of what happened and will be forced to do a full resync of their caches for the new state. Yup. This is another issue that leads me to question the value of a RENAME in IMAP. -- Mark -- http://staff.washington.edu/mrc Science does not emerge from voting, party politics, or public debate.
Re: RENAME and imap compliance
Mark Crispin wrote: On Tue, 21 Jan 2003, Ken Murchison wrote: Not only doesn't this do the right thing with UIDVALIDITY (a flaw that almost every server has), but Cyrus doesn't even Cyrus maintains UIDVALIDITY. do the RENAME in an atomic fashion Does Cyrus change the UIDVALIDITY of each of the mailboxes that are renamed? No. (also, does Cyrus still fail to rename the inferiors?). No. Cyrus renames the entire tree. That's good news. When did that change? v2.0.0 (about 3 years ago) -- Kenneth Murchison Oceana Matrix Ltd. Software Engineer 21 Princeton Place 716-662-8973 x26 Orchard Park, NY 14127 --PGP Public Key--http://www.oceana.com/~ken/ksm.pgp
