On Fri, 2002-01-11 at 22:01, Sam Varshavchik wrote: > Jeffrey Stedfast writes: > > > On Fri, 2002-01-11 at 17:28, Adrian Hosey wrote: > >> On 11 Jan 2002, Jeffrey Stedfast wrote: > >> : > >> : The UID STORE command was setting the \Deleted flag on a very large set > >> : of messages. This is a perfectly valid operation. > >> > >> It's technically legal, but it's not wise to try and do it all with one > >> command. Your example of a large non-contiguous set of UIDs can be broken > >> into multiple smaller UID STOREs. "Be conservative in what you send." > > > > Granted we will have to do this anyway, but it is still broken behavior > > for courier-imapd to impose token size limits. > > Actually, what's really broken here is RFC 2060. IMAP is a fundamentally > broken protocol that fails to take into account the harsh realities of the > world at large.
this is debatable. I'll grant you that it's not perfect, but it's probably one of the better mail protocols around (unfortunately). > > As someone else already noted, the UW-IMAP server imposes a fixed limit on > the size of the entire IMAP command (it just fgets() the next command off > the wire, and chews it up). yea, this is BADLY broken. But to be honest, limiting to 16k isn't a whole lot better (it's only twice as good ;-) > > Now, when the author of RFC 2060 cannot himself write an implementation that > strictly adheres to his own write-up, to the letter, what does that tell > you?[1] > > >> Dynamically growing buffers is fine, but there still needs to be a > >> limit. > > > > The only limit that you need is the "system is out of resources" limit. > > Congratulations: a single hostile user is now able to completely DOS the box > out of its resources. > > The following question is not really meant to be confrontational, but: have > you ever written a server -- for a relatively complicated protocol -- that > is designed to be publicly accessible? No, but I'm sure I could. I do, however, have plenty of experience writing mail software ;-) > > > You could, as an option, write a generic token interface that could > > optionally save token data to disk if it was larger than a size you > > wanted to store in memory. > > With or without an upper limit on the size of the token? If without any > limit, then a single hostile user can now fill your entire server's disk. > Wonderful. > > And if you do impose an upper limit on the token size, when spooled to disk, > then you failed to satisfy your own initial requirements of strict adherence > to RFC 2060. > > >> To do otherwise invites denial of service. > > > > Not if you do it right. > > I'm curious. Please explain how to do this "right": by that meaning > properly implementing RFC 2060, up to your own standards, while preventing > DOS attacks. No matter how you turn things around, if you actually think > things through you will have no choice but agree that you have to impose > maximum limits on these kinds of things, so the original "bug", as you > stated, is a red herring. Why can't you do incremental parsing of the token? You could have say a 4k buffer, do a read(), parse as much as you can, keep state, and if you haven't encountered the end of the token - read in another ~4k (4096 - remaining bytes from previous read). Parse as much as you can, etc etc. When parsing a uid-set, you could do something creative like setting a mode bit on the message files whos uids are encountered when parsing the uid-set (this wouldn't use *any* memory or disk space). Once you complete the parsing of the uid-set, you could scan the directory and set whatever flags were requested in the next token on any message with that particular bit set (probably a +x bit). Yes, this idea hasn't been completely thought out. You'd still have to consider another client trying to set flags on an intersecting set of messages (or even a non-intersecting set really...or maybe even multiple clients trying to change flags at all). Perhaps you could lock the files or something as a solution, I don't know. Maybe you can make it so only a single client can have READ-WRITE access to the folder at a time, this'd solve the multiple client thing. Okay, so what if 1 client has READ-WRITE access and we have multiple clients (that can only get READ-ONLY access) requesting info and also using uid-sets? Okay, so perhaps one client can set u+x bit, another client can set the g+x bit and you can set the o+x bit for a third client and any client after that would just be locked out of the folder until the other requests are finished being processed. OR: you could just queue all the clients right from the start... Wow, just listen to me pour out ideas that might actually work! *gasp*! And all this is just rolling off the tip of my tongue as I write this! Imagine what one could do if they were to sit down and really think about the problem, he/she might actually be able to come up with a really good solution! Oh no! What have I done!? I have provoked thought! Disclaimer: my +x idea may not be the perfect or ideal solution, there will always be a race condition; but I think it's a prime example of "thinking out of the box". Remember: just because you haven't thought of a way to do it, doesn't make it impossible. Addendum: *except* when it *is* impossible ;-) > > Let me make a wild guess: the IMAP client in Evolution will happily spew > forth infinitely-sized IMAP commands; people are bitching because Evolution > goes tits-up on huge folders because of this; and you want everyone else to > fix this for you? Not at all, I've already mentioned that I know we will have to modify Evolution to accept the limitations of the IMAP servers in use. I was hoping, however, that one of my favorite IMAP server implementations would at least consider making their server more robust. But alas, apparently this will be up to me to do some weekend when I get some spare time. This is really less about you fixing a "bug" to make Evolution work than it is for you to fix your software to be better. Whether you fix the bug or not, I still have to fix Evolution to work around server limitations. We can't expect all of our users to use the fixed version of Courier imapd, can we? :-) So the notion that I am expecting everyone else to fix this for me is sheer folly! (I am loving this expression!) Also note (in a later part of this email I think) that I have fixed interoperability issues before, and ones that were far more obviously "server bugs" than this one. > > Well, it just so happens that I'm writing this message using my own IMAP > client. As a side note it's never going to see the light of day because > I'll rewrite it from scratch, when I have some free time -- but the point is > that it will happily handle these kinds of large folders situations without > choking. It doesn't matter whether it's the UW-IMAP server ot the > Courier-IMAP server it's talking to. And that's not necessarily becuase I > happen to have some casual knowledge of their innards. Good for you. I would argue that it's not true - had you 0 knowledge of the limitations of imap servers - it is doubtful that you would have limited your commands to 1k since rfc2060 says to do no such thing. > > It's because from reading RFC 2060 it was always patently clear from me, > right from the very beginning, that expecting to fire multi-megabyte tokens > at the IMAP server, and expecting that to work in real life, is sheer folly. I bet you also read rfc822 and it was "patently clear, right from the beginning", that email messages would never be longer than 4k. Oh wait, there are messages thrown around *all* the time that are >4k. Back in the '90's, I'm sure a 4k message was a LARGE message. Who knows, maybe some mail server implementations limited their email-message-buffer to 4k! "But you need an upper limit." Just like Bill Gates knew that no one would ever need more than 640k. Hah! Sheer folly I say! (okay, I'm sure you didn't read rfc822 and really think that, I was just poking fun. In a few years, a multi-megabyte token will be considered tiny - I just replaced my 6.4 gig drive that died with an 80 gig drive, and I thought 6.4 gig was huge! how will I ever fill 80gigs!?!?) > Putting my client hat on, I feel confident that I'm in the safe zone if I > limiting my whole commands to less than a 1,000 bytes. So, in a similar > situation, my IMAP client will simply issue multiple commands to go through > the entire message list, a couple of hundred messages at a time, and do what > it needs to do. What if someone were to implement a server that didn't allow tokens to be more than say 20 bytes? What then? By your mode of thinking, this is perfectly acceptable. After all, I believe that all the defined IMAP protocol tokens would fit within a 20 byte buffer, so wouldn't that be just fine and dandy? You can even pack a small uid-set into 20 bytes... Even your superb client couldn't handle this server implementation... > > And, as a side note, when I put my server hat on, I conclude that I should > be safe limiting tokens to 16K bytes, and parsing IMAP commands one token at > a time, thus imposing no actual upper limit on the size of the overall IMAP > command[2]. It's surely better than a single 8k buffer that the entire command must squeeze into, but it's hardly acceptable IMHO. > > > Just because other servers impose limits doesn't mean you should. If the > > IMAP specification does not impose limits, then neither should you. If > > The IMAP specification is fundamentally broken. And this is not the only > broken part, BTW. Actual implementations have to make judgement calls like > this. Thinking rationally about it you have no choice but come to that > conclusion. Anything else is sheer folly. > > > you impose limitations, then you are not compliant with the > > specification IMHO. > > Go bitch to Crispin. Should I also use this argument next time someone complains about Evolution's imap implementation? We have worked around so many bugs in a variety of IMAP implementations, the code is riddled with comments like this one from just yesterday: /* This is a kludge around a bug in Exchange * 5.5 that sometimes claims multiple messages * have the same UID. See bug #17694 for * details. */ Here's a log of the type of transaction that this works around: -------------------- sending : A00052 UID FETCH 262763:* (FLAGS RFC822.SIZE) received: * 9 FETCH (FLAGS (\Recent) RFC822.SIZE 2065 UID 262766) received: * 10 FETCH (FLAGS (\Recent) RFC822.SIZE 2065 UID 262766) received: * 11 FETCH (FLAGS (\Recent) RFC822.SIZE 2565 UID 262768) received: * 12 FETCH (FLAGS (\Recent) RFC822.SIZE 1877 UID 262770) received: A00052 OK FETCH completed. -------------------- Isn't that just beautiful? ;-) Or how about this one from earlier last week? -------------------- B00021 UID FETCH 10576:* (FLAGS RFC822.SIZE BODY.PEEK[HEADER]) ... UID 10575) B00022 UID FETCH 10576:* (FLAGS RFC822.SIZE BODY.PEEK[HEADER]) ... UID 10576) -------------------- And the list goes on...and on (I'm sure I don't have to add another "and on" ;-) > > >> A quick look at the UWash imapd code and > >> it looks like that limits the size of the _entire_ command buffer to > >> 8K. Have you tried your large STORE operation against a UWash server? > >> Maybe it works, maybe I'm reading the code wrong. > > > > No, you are right. But again, just because someone else does it doesn't > > make it right. > > Prediction: you explain to Crispin the fact that Evolution will choke on > large UW-IMAP folders, and what the reason is, and he'll tell you that > you're full of crap[3]. *sigh* I give up. I was hoping that I could at least convince you to rethink your implementation to try and find ways of increasing the limitation of token sizes that you accept if not going the whole 9 yards and implementing it with no limitation. Let me apologize for any feelings I may have hurt (it was definitely not my intention to do so). But *please*, do consider an implementation that doesn't limit clients to token sizes. It's just that having to "guess" what the max allowable token size is on a server is never a Good Thing (tm). While you are probably right that minimizing token sizes to 1k on the client-side will float most anywhere, you can never ever be 100% certain of that. And if you go and try to convince those guys that a less-than-1k buffer is too limiting, and they make the same argument you are advocating "but there must be an upper bound", then we will all be hurting badly. Jeff -- Jeffrey Stedfast Evolution Hacker - Ximian, Inc. [EMAIL PROTECTED] - www.ximian.com _______________________________________________ courier-users mailing list [EMAIL PROTECTED] Unsubscribe: https://lists.sourceforge.net/lists/listinfo/courier-users
