Re: Why is a message immutable?
On Wed, 2003-03-19 at 09:43, John Milan wrote: This is exactly why I raised my initial objection, and after discussing this with Mark, he and I agree that messages that are modified in any way that would cause a modification of the ENVELOPE or BODYSTRUCTURE are different messages, and thus the UID of the message MUST be changed. Shouldn't UID be changed if _anything_ changes in message header or body? Why am I beating a horse that that died back in 1998? Because it wasn't beaten properly! I think this is a huge hole in the IMAP spec, and other than several references stating 'messages MUST be immutable', I have yet to see any problem explaining why it's so obviously impossible for messages to be mutable. Because of current caching rules. Two possible ways to support updating messages in backwards compatible way: 1. Change the UID, but add some way for new clients to find out what it's old UIDs were. 2. Change UIDVALIDITY if something is updated, and add some way for new clients to invalidate cache only for changed messages. I'd say 1. is more realistic.
Re: Why is a message immutable?
In IMAP, a client is free to cache any part of a message for as long as it wants. That implies that once a server has sent e.g. a message's ENVELOPE to a client, that message can't be changed any more, or the client's cache would hold incorrect data. The only way for a server to invalidate the client's cache is to send an EXPUNGE for the relevant message and issue a new UID. --Arnt
RE: Why is a message immutable?
In IMAP, a client is free to cache any part of a message for as long as it wants. That implies that once a server has sent e.g. a message's ENVELOPE to a client, that message can't be changed any more, or the client's cache would hold incorrect data. It implies nothing of the sort. A client's decision to cache any part of a message for eternity is purely on the shoulders of the client. It has no bearing on how the server should store the data. Again, think of a browser. I could set the caching such that the first time I visit a web site, that web page is cached forever, but I claim that wouldn't be very useful. The only way for a server to invalidate the client's cache is to send an EXPUNGE for the relevant message and issue a new UID. With current clients, I absolutely agree. But, as I mentioned into another reply, I think enabling updates for current clients is not necessary. What is necessary is that they don't break. Newer clients will be able to make use of an update capability. JM
RE: Why is a message immutable?
Shouldn't UID be changed if _anything_ changes in message header or body? I don't understand why? It's certainly not necessary. The main concern seems to be client caching rules, and perhaps this is what's broken. Because of current caching rules. Two possible ways to support updating messages in backwards compatible way: Here's where I think explicitly stating messages are immutable is actually a disservice: just because clients have been implmented this way until now doesn't mean future generations of clients need to be implemented this way forever. This is software after all, things change. 1. Change the UID, but add some way for new clients to find out what it's old UIDs were. 2. Change UIDVALIDITY if something is updated, and add some way for new clients to invalidate cache only for changed messages. I think there is another passive option that uses currently defined FLAGs that will be safely ignored by current clients. That is, if a message changes, it switches the \Unseen flag. True, current clients will not be smart enough to flush their caches and FETCH again, but newer clients will be. I don't think enabling this feature for all IMAP clients built until now is feasible, rather, the goal would simply be not to break all IMAP clients built until now. JM
Re: Why is a message immutable?
At 2003-03-19 07:32:25 -0800, [EMAIL PROTECTED] wrote: Newer clients will be able to make use of an update capability. To what end? -- ams
Re: Why is a message immutable?
It implies nothing of the sort. A client's decision to cache any part of a message for eternity is purely on the shoulders of the client. It has no bearing on how the server should store the data. While this is true, a client's ability to cache the message if it so chooses is promised by the IMAP protocol. Again, think of a browser. I could set the caching such that the first time I visit a web site, that web page is cached forever, but I claim that wouldn't be very useful. The web browser analogy does not hold, mainly because HTTP either makes no promises about the client's ability to cache the data, or it makes very specific promises via certain headers. On the other hand, IMAP makes very strong promises about the client's ability to cache the data. The only way for a server to invalidate the client's cache is to send an EXPUNGE for the relevant message and issue a new UID. With current clients, I absolutely agree. But, as I mentioned into another reply, I think enabling updates for current clients is not necessary. What is necessary is that they don't break. Newer clients will be able to make use of an update capability. Perhaps I am misunderstanding. Are your proposing the ability to update a message without changing the UID, or not? If so, then how do you propose that existing clients not break? How do you update a message that an existing client has cached? If not, then I don't think there's any disagreement: essentially you're proposing some sort of optimized append (append-based-on ...). john - Original Message - From: John Milan [EMAIL PROTECTED] To: Arnt Gulbrandsen [EMAIL PROTECTED] Cc: [EMAIL PROTECTED] Sent: Wednesday, March 19, 2003 7:32 AM Subject: RE: Why is a message immutable? In IMAP, a client is free to cache any part of a message for as long as it wants. That implies that once a server has sent e.g. a message's ENVELOPE to a client, that message can't be changed any more, or the client's cache would hold incorrect data. It implies nothing of the sort. A client's decision to cache any part of a message for eternity is purely on the shoulders of the client. It has no bearing on how the server should store the data. Again, think of a browser. I could set the caching such that the first time I visit a web site, that web page is cached forever, but I claim that wouldn't be very useful. The only way for a server to invalidate the client's cache is to send an EXPUNGE for the relevant message and issue a new UID. With current clients, I absolutely agree. But, as I mentioned into another reply, I think enabling updates for current clients is not necessary. What is necessary is that they don't break. Newer clients will be able to make use of an update capability. JM
RE: Why is a message immutable?
Well, I think what makes IMAP really interesing is its extended session. Consider the IDLE command, this is really making use of an active connection to send new events. This is quite powerful and used to good effect as a means to immediately notify users of incoming mail. POP simply can't do this, but IMAP can. Now imagine there is a class of data out there whose perceived value diminishes as a function of time, say stock prices. I could very easily imagine an IMAP bridge that gathers stock prices, and a client that's interested in stock prices. This *could* be implemented today, but the cost is quite high, for if the IMAP server sent *new* stock price messages every 15 minutes for a basket of ticker symbols, the client cache quickly becomes unreasonable. BUT, if each ticker symbol, by virtue of being unique in their space, had a similar mapping in the IMAP server's space, then the IMAP server could send UPDATEs for the latest stock prices. You can imagine a window in MS Outlook, possibly on their summary page, that provides the real time stock data to the user. That's off the top of my head, anyway. I imaging this would also have huge impact for workflow systems, or any sort of project management where distribution lists and 'active' data are valuable. JM -Original Message- From: Abhijit Menon-Sen [mailto:[EMAIL PROTECTED] Sent: Wednesday, March 19, 2003 7:38 AM To: John Milan Cc: [EMAIL PROTECTED] Subject: Re: Why is a message immutable? At 2003-03-19 07:32:25 -0800, [EMAIL PROTECTED] wrote: Newer clients will be able to make use of an update capability. To what end? -- ams
RE: Why is a message immutable?
Sigh. Snort. Let's say you decide (for some as yet unexplained reason) that you want to change the header. The ENVELOPE then changes for the message, invalidating any cached values, client or server, for that message. That would be the idea. Because you don't know that what clients have cached the envelope, you now need to notify clients that the envelope has changed. Which clients have fetched the ENVELOPE? Don't know! Don't care! The server keeps track of the latest information, the clients grab the updates as needed. snip possible implementation that obviously wouldn't work But all of this aside. Internet mail messages are exactly that -- messages. They are composed by the issuer in the form the issuer chooses the message to take. What reasonable circumstances would you have for changing the content of the message that was handed to you in good faith by the delivery system? You've never seen a message change on its way to a receiver? Haven't you ever asked anyone to 'look over this message' before you sent it? There are quite a few people who wish they could have modified a message they sent to their bosses in haste. What part of the message are you planning to modify? I'm proposing basically everything except the UID and INTERNALDATE right now, but I suspect there will be other parts that should be immutable. Computed values obviously make no sense. For instance, RFC822.SIZE must only be changed as a result of an update, it can never be set by a client directly. Macros data items like ALL, FULL etc. also wouldn't make sense. But the very well defined, and quite usefull, section definitions allowing for full or partial FETCHes should be UPDATe-able, in my opinion. Why as a client would I *EVER* want my server gratuitously changing a message without my explicit permission and direction. I would never want something gratuitous in the sense of being unwarranted (free is another story :). However, I would like to know if the server has something *new* for me, much like it tells me I have a new message. We can and do manipulate meta-data about a message. I would suggest that annotating a message so that it can be interpretted differently by a client that chooses to support such annotations is a better way to alter the view or content of a message. I think annotations fall apart if there are alot of them and their individual perceived value is close to nil. I think there is a broad range of data that falls in this category. Think stock prices. There are alot of closing day prices for IBM between now and, say, 1935 that I simply don't care about. I want to know the latest information, because that has the most value. I'm also not advocating that every and all messages are updateable by anyone at anytime. I don't like chaos any more than you do. But one can easily imaging a permission scheme like the basic filesystem's read/write system that can be supported by ACLs, but one argument at a time. JM
RE: Why is a message immutable?
Why not imagine a new protocol to go with it? Well, I suppose I could. What would it need? It would need to establish a SESSION with the ability to AUTHENTICATE, probably supporting multiple AUTHENTICATION implementations for LOGIN. It would need the ability to LIST any directories used to organize data. It would certainly need the ability to CREATE inboxes for incoming data, and if you CREATE, you most likely need DELETE as well. It would be quite useful to STORE metadata flags with the data. And, of course, what use would the data be if you couldn't FETCH it? Because it could handle new events, it would be nice to communicate them with the client via something like IDLE and unrequested server responses. It would also be very helpful to SEARCH on the data, however, in order for search to know what to look for, it would need to define equivalents for ENVELOPEs, HEADERs, BODYs and their parts. It could even use ACLs to provide a basic permission system. With ACLs, it could richly support COPYs and MOVEs from one inbox to another. And, of course, you'd want a way to clean things up, so something like EXPUNGE would be useful. And finally, it would need a way to UPDATE data. And I'd like it to be a standard so it may be more readily accepted. What I don't like imagining is introducing yet another protocol to the world. I'd rather stand on the shoulders of giants. JM
RE: Why is a message immutable?
I want to throw a few ideas (and some strong opinions!) into this discussion: (1). There are IMAP servers that, in certain cases, do not assign new UIDs to messages that are changed. This is because the servers have no way of telling that a message has been changed. This makes the servers non-compliant and creates a regrettable situation. But it is a fact of life. (2). I happen to be in the habit of editing messages that I receive. I don't do it often but I certainly do it. The most common case is with messages that have generic or non-descriptive subjects. I may add text to the subject field in order to help me identify a message more easily in case I need to go back later and look for it. Another case is when I read an encrypted message; I may choose to re-store it in plaintext form. (3). Saying that people do not or should not edit received messages strikes me as rather irrelevant because whatever we are discussing here would presumably apply to all messages and there is one category of message that people inevitably do edit, drafts. (4). Adding any capability to IMAP to allow messages to be changed in any piecemeal fashion would, in my opinion, make a nightmare of the protocol. (Not saying that anyone is suggesting that.) Providing a way to change a message by completely replacing it could be reasonable. (5). IMAP could, in principle, be modified to allow a client to be notified that a message has been changed by using a per-message UIDVALIDITY. However I see no way to implement such a thing in a way that would be compatible for clients that don't support it. In other words, such a thing could not be an extension and therefore would be unacceptable. (6). If we allow messages to be changed, how much do we allow them to be changed? Suppose a message is changed beyond all recognition? Suppose one message is essentially replaced by another? At what point do we insist that it be given a new UID -- assuming we must because at some point it essentially becomes a new message? To me, this seems such a morass that it is not worth discussing. My conclusion: messages SHOULD be immutable and that's that. (7). The fact that messages are immutable should not be a big deal. You can change a message in any way at all as long as you retire its UID and give it a new one. If you need a link between the original and the changed version then that can be done outside of IMAP by using a thread id or something similar in the headers. IMAP cannot be perfect, at least because it has to be used in conditions that are imperfect. But IMAP is awfully good and thinking about changing it often serves just to reinforce how good it is. Pete
Re: Why is a message immutable?
John, Offline IMAP use with UIDs for cache resynchronization was first mentioned in RFC 1733, which was published in December 1994. The concept may be older, but at least it's eight years old. So, what are you suggesting, precisely? To me it sounds as if you want to break compatibility with eight years of RFC-mandated practice. A google search for 'imap offline' returns 29,000 hits, so it's quite a lot of practice, too. Have I misunderstood? --Arnt
RE: Why is a message immutable?
I'll take the last first :) In fact, the Exchange server can and will do just that. If a client modifies a message on the Exchange server (you can do that for messages that you authored if they've not yet been sent), when the client saves their changes on the message, the server assigns a new UID to the message and sends an expunge for the old message. OK, then other then the IMAP specification saying you needed a new UID, what technical reason is there for creating a new UID? Because that's the mechanism that the IMAP server defined for indicating to clients that a message has been modified. You're right. When IMAP was designed back in the early 90's, a different mechanism could have been designed along the lines of what you're proposing (marking a message as being unseen when it's been modified), but I'm pretty certain that just marking a message as being unseen after its existance isn't sufficient (UIDValidity plays into this as well). I would make use of unrequested, untagged server responses carrying flag information for a particular UID. The client could receive, at any time, the following: S: * 23 FETCH (FLAGS (\Unseen)) Nothing more. Certainly not the entire message or even its header. This would be a signal to the client that, if it has message 23 in its cache, it should remove it. The client could then determine if and when it wants to fetch the 'new' data for UID 23. Perhaps when the user actively selects the inbox the message resides in. Or perhaps on a timer, or perhaps when it gets the unrequested message. It would be up to the client. Now the rest: Ok, so you're saying that it's better to do: S: * 23 FETCH (FLAGS (\Unseen)) Then (assuming 23 messages in the mailbox and the last one's the one that's modified): S: * 23 EXPUNGE S: * EXISTS 23 ? I fail to see any real difference between the two approaches (although I'm pretty sure that your use of the \Unseen flag is likely to have significant issues in a multi-accessed mailbox case - the \Unseen flag is defined as being seen by only one client, so if two clients have the same mailbox opened simultaneously, this doesn't work. The *23 EXPUNGE/* EXISTS 23 paradigm doesn't have this problem. The current IMAP specification says that you change the UID, send an expunge for the old message and an EXISTS for the new one. So effectively you've created a new message. Hmmm... you may have effectively created a new message, but you've definitely duplicated data and took another hit on the cache. This isn't something you'd want to do for updates coming in at high frequency, any more than you'd want many annotations. That's happens with your proposal as well - you're just replacing one mechanism of notifying the client to update the cache with another. Federal law used to say cars on the highway couldn't go faster than 55. Things can change. Yes, but... Imagine a somewhat different scenario that's more relevant to computers. Imagine a hypothetical system service called Open Handle that's implemented on the x86 interrupt 21h. Further imagine that the documentation for this API says: Call AH = 3DH AL Access code (see text) DS:DX Pointer to pathname Return Carry set: AX 1 = Invalid function code 2 = File not found 3 = Path not found 4 = Too many open files 5 = Access denied 12 = Invalid access Carry not set: No error Remember, this is just a hypothetical, nobody in their right mind would ever write such an API definition. Now imagine that they decided that it was necessary to add a couple of new error codes to deal with new scenarios (for example if they wanted to add file sharing semantics to the system they wanted to add a distinguished error code for the file's in use by another process to the API assigning a new error code of 32 for that case(remember, this is just a hypothetical, it would NEVER happen in real life). But when they added the new value returned in AX, they discovered that lo and behold - people had written applications that assumed that the ONLY legal values in AX would be 1,2,3,4,5, and 12! And further, they discovered that the people had used the error codes as indexes into an array with 12 entries in it. And the new error code of 32 caused them to go beyond the end of that table because they knew that the only possible return codes were 1,2,3,4,5 and 12. And the apps crashed horribly. What would the system service developers do in such a hypothetical scenario? Well, they'd probably add a table to the system that mapped the new error codes to the old error codes, and add a new system service to get the real return code. If you have access to the NT source code (and I know that there are non MS people on the list that do have access to it), you can look at the file dostab.asm in the NTVDM tree if you want to see an example of such a
RE: Why is a message immutable?
On Wed, 19 Mar 2003, John Milan wrote: I would make use of unrequested, untagged server responses carrying flag information for a particular UID. The client could receive, at any time, the following: S: * 23 FETCH (FLAGS (\Unseen)) Presumably you mean S: * 23 FETCH (FLAGS ()) since there is a \Seen flag but no \Unseen flag. How does the client determine the difference between this and some client doing tag STORE 23 -FLAGS (\Seen) What happens when some other client sets the \Seen flag before this client got its cache updated? Are you suggesting that the server keep track of clients, so that it knows whether or not to present the \Seen flag to that client? If so, how? Federal law used to say cars on the highway couldn't go faster than 55. Things can change. Non-sequitor. There was no good reason for the late unlamented 55 MPH speed limit in the USA. There are excellent reasons for a guarantee that message data is immutable. OK, then other then the IMAP specification saying you needed a new UID, what technical reason is there for creating a new UID? The technical reason is that this tells the client that the message data is different. -- Mark -- http://staff.washington.edu/mrc Science does not emerge from voting, party politics, or public debate.
RE: Why is a message immutable?
On Wed, 2003-03-19 at 19:03, John Milan wrote: Well, I think what makes IMAP really interesing is its extended session. Consider the IDLE command, this is really making use of an active connection to send new events. This is quite powerful and used to good effect as a means to immediately notify users of incoming mail. IDLE isn't actually required for sending new events. Except for EXPUNGE, server is free to send them any time it wishes.
