Re: qmail-pop3d bug
On Sun, Mar 11, 2001 at 07:37:06PM +, Mark Delany wrote: [...] A more sensible strategy might be to introduce a new "info" flag (say '3' equals POP wire size) on the filename, eg, a 10,000 byte email has a name something like this: Maildir/new/980195114.16740.geex:2,RS3,1 From reading URL:http://cr.yp.to/proto/maildir.html, it is not clear to me that this would be the proper format for such an 'info' extension. I would worry that MUAs and other software dealing with maildir (scripts!) would expect info semantics in the 2, series to be at the end of the filenames. Optimally the wire-size is calculated when the mail is written to Maildir/tmp/ and then applied as an "info" flag when the file is moved to Maildir/new/. A possible complication with this approach is that my reading of Maildir infers that "info" can only be set when the file moves from Maildir/new/ to Maildir/cur/. No, this is not what that document says. It says "When you move a file from new to cur, you have to change it's name [...]" You *have* to change the name when the file move from new/ to cur/ , but there is no specification of other cases; in fact, lots of MUA's will change info when the file has been in cur/ for a while: mutt, for example, moves the file from new/ to cur/, adds :2, and only modifies that to be 2,S after the user has read the message (it is no longer 'N'ew). Vince.
Re: qmail-pop3d bug
A more sensible strategy might be to introduce a new "info" flag (say '3' equals POP wire size) on the filename, eg, a 10,000 byte email has a name something like this: Maildir/new/980195114.16740.geex:2,RS3,1 From reading URL:http://cr.yp.to/proto/maildir.html, it is not clear to me that this would be the proper format for such an 'info' extension. I would worry that MUAs and other software dealing with maildir (scripts!) would expect info semantics in the 2, series to be at the end of the filenames. Indeed, and given that "info is morally equivalent to the Status field used by mbox readers" I suspect that the my suggested syntax is beyond the original intent. Optimally the wire-size is calculated when the mail is written to Maildir/tmp/ and then applied as an "info" flag when the file is moved to Maildir/new/. A possible complication with this approach is that my reading of Maildir infers that "info" can only be set when the file moves from Maildir/new/ to Maildir/cur/. No, this is not what that document says. It says "When you move a file from new to cur, you have to change it's name [...]" You stopped quoting before the most important part! Here's the complete sentence. "When you move a file from new to cur, you have to change its name from uniq to uniq:info." To me that implies that a file in new cannot have an "info" section. You *have* to change the name when the file move from new/ to cur/ , but there is no specification of other cases; in fact, lots of MUA's will change info when the file has been in cur/ for a while: mutt, for example, moves the file from new/ to cur/, adds :2, and only modifies that to be 2,S after the user has read the message (it is no longer 'N'ew). Right, but that's my point. To specify another case. Regards.
Re: qmail-pop3d bug
On Mon, Mar 12, 2001 at 11:03:33AM +, Mark Delany wrote: To me that implies that a file in new cannot have an "info" section. You're right. I didn't think the original point throuhgh. Regards, Vince.
Re: qmail-pop3d bug
Peter van Dijk wrote: Not if it's calculated as the file is written to the Maildir. True, but that hurts writing performance. Have you tested this? It doesn't seem that qmail has ever been CPU bound -- and if the CPU has spare cycles while writing, then counting lines and adding bytes for CR/LF won't hurt writing performance at all. -- Michael T. Babcock (PGP: 0xBE6C1895) http://www.fibrespeed.net/~mbabcock/
Re: qmail-pop3d bug
On Mon, Mar 12, 2001 at 08:55:47AM -0500, Michael T. Babcock wrote: Peter van Dijk wrote: Not if it's calculated as the file is written to the Maildir. True, but that hurts writing performance. Have you tested this? It doesn't seem that qmail has ever been CPU bound -- and if the CPU has spare cycles while writing, then counting lines and adding bytes for CR/LF won't hurt writing performance at all. 'hurts writing performance' (although you are right :) also means 'increases resource use', in whatever way. Measuring in wall-clock time, the performance hit is probably negligible indeed. Greetz, Peter.
Re: qmail-pop3d bug
John R. Levine wrote: Pop3d just reports the file sizes, while it's clear from the RFC that it's supposed to report the wire size of each message, i.e., the size using cr/lf as a line terminator, so the sizes it reports are too small. Peter van Dijk replied: Yes. This behaviour is known. Fixing it, however, involves a *huge* performance downgrade of qmail-pop3d. Scott Gifford pondered: A solution I have considered is storing the messages in wire format. Especially for POP/IMAP-only clients, seems like it could be a medium-sized performance win, since the line-conversion is done only once, regardless of how many times the message is downloaded. If the message were kept in wire-format from SMTP through delivery, no line conversion would be required at either end, and a larger performance gain would be possible. Has anybody tried this, or anything like it? Something like that, yes. My last job involved building custom SMTP and POP3 servers from scratch. I stored messages in a quasi-maildir-ish folder scheme, and left the CRLF linebreaks intact. Since there were no local users on this box, there was no need to worry about local MUA's, and everything ran smoothly. I would think that it probably wouldn't be too much trouble to rig an MUA to look for CRLFs, so that it could coexist in this environment if need be. ---Kris Kelley
Re: qmail-pop3d bug
On Sat, Mar 10, 2001 at 05:47:29PM -0500, Scott Gifford wrote: Peter van Dijk [EMAIL PROTECTED] writes: On Sat, Mar 10, 2001 at 01:12:13PM -0500, John R Levine wrote: The usual mailbox vs. maildir war has flared up on inet-access, and points out a bug in qmail-pop3d. When you do a LIST command, it gives you the size of each message. Pop3d just reports the file sizes, while it's clear from the RFC that it's supposed to report the wire size of each message, i.e., the size using cr/lf as a line terminator, so the sizes it reports are too small. [ ... ] Yes. This behaviour is known. Fixing it, however, involves a *huge* performance downgrade of qmail-pop3d. A solution I have considered is storing the messages in wire format. Especially for POP/IMAP-only clients, seems like it could be a medium-sized performance win, since the line-conversion is done only once, regardless of how many times the message is downloaded. If the message were kept in wire-format from SMTP through delivery, no line conversion would be required at either end, and a larger performance gain would be possible. Has anybody tried this, or anything like it? Have not tried it, but it sounds like it would definitely work. Greetz, Peter.
Re: qmail-pop3d bug
On Sat, Mar 10, 2001 at 01:12:13PM -0500, John R Levine wrote: The usual mailbox vs. maildir war has flared up on inet-access, and points out a bug in qmail-pop3d. When you do a LIST command, it gives you the size of each message. Pop3d just reports the file sizes, while it's clear from the RFC that it's supposed to report the wire size of each message, i.e., the size using cr/lf as a line terminator, so the sizes it reports are too small. I gather nobody's ever reported this as a bug, and I expect that the only thing that uses the size is the "don't download bigger than size X" option for which it's close enough, but it's still wrong. If I mis-remember correctly, qpopper may have a similar problem in that the stated size does not necessarily match the size sent down the wire. How so? Because qpopper adds X-UIDL and Status: headers to the out-going message (perhaps it includes this in the size calc but I haven't looked at the code in such a long time, or perhaps it only adds those headers when the mail is re-written). I use courier-imap, and its POP daemon does get the sizes right, presumably by reading the files and adding the number of \n characters. A more sensible strategy might be to introduce a new "info" flag (say '3' equals POP wire size) on the filename, eg, a 10,000 byte email has a name something like this: Maildir/new/980195114.16740.geex:2,RS3,1 Optimally the wire-size is calculated when the mail is written to Maildir/tmp/ and then applied as an "info" flag when the file is moved to Maildir/new/. A possible complication with this approach is that my reading of Maildir infers that "info" can only be set when the file moves from Maildir/new/ to Maildir/cur/. Regards.
Re: qmail-pop3d bug
Yes. This behaviour is known. Fixing it, however, involves a *huge* performance downgrade of qmail-pop3d. Not if it's calculated as the file is written to the Maildir. 'Usually, during the AUTHORIZATION state of the POP3 session, the POP3 server can calculate the size of each message in octets when it opens the maildrop. . simply counts each occurance of this character in a message as two octets.' Typical of those RFCs authors that, consciously or otherwise, used a single implementation to guide much of their thinking on protocol design. POP3 is not the only standard that suffers as a consequence - consider SMTP and DNS? We shouldn't have to live with short-sightedness forever. Regards.
Re: qmail-pop3d bug
On Sun, Mar 11, 2001 at 07:51:47PM +, Mark Delany wrote: Yes. This behaviour is known. Fixing it, however, involves a *huge* performance downgrade of qmail-pop3d. Not if it's calculated as the file is written to the Maildir. True, but that hurts writing performance. 'Usually, during the AUTHORIZATION state of the POP3 session, the POP3 server can calculate the size of each message in octets when it opens the maildrop. . simply counts each occurance of this character in a message as two octets.' Typical of those RFCs authors that, consciously or otherwise, used a single implementation to guide much of their thinking on protocol design. POP3 is not the only standard that suffers as a consequence - consider SMTP and DNS? The implementors had to make a choice. Line format is in fact the obvious choice for message size. Using server-implementation size is however not a big deviation. We shouldn't have to live with short-sightedness forever. That's why we have QMTP :) Greetz, Peter.
Re: qmail-pop3d bug
On Sun, Mar 11, 2001 at 07:37:06PM +, Mark Delany wrote: [snip] I use courier-imap, and its POP daemon does get the sizes right, presumably by reading the files and adding the number of \n characters. A more sensible strategy might be to introduce a new "info" flag (say '3' equals POP wire size) on the filename, eg, a 10,000 byte email has a name something like this: Maildir/new/980195114.16740.geex:2,RS3,1 Putting the linecount in there makes more sense. Some MUAs might be happy about that, and it still allows easy calculation of wiresize (add number of lines to physical size). More info, less bytes :) Optimally the wire-size is calculated when the mail is written to Maildir/tmp/ and then applied as an "info" flag when the file is moved to Maildir/new/. Yes. Mind the performance penalty tho. A possible complication with this approach is that my reading of Maildir infers that "info" can only be set when the file moves from Maildir/new/ to Maildir/cur/. That's what the spec says, indeed. A delivery process is not supposed to know anything, so :info is not needed in new/. Greetz, Peter.
Re: qmail-pop3d bug
Putting the linecount in there makes more sense. Some MUAs might be happy about that, and it still allows easy calculation of wiresize (add number of lines to physical size). More info, less bytes :) Optimally the wire-size is calculated when the mail is written to Maildir/tmp/ and then applied as an "info" flag when the file is moved to Maildir/new/. Yes. Mind the performance penalty tho. Not a bad idea. The performance penalty would be tiny, reading buffers that are about to be written out won't cause an extra page fault. A possible complication with this approach is that my reading of Maildir infers that "info" can only be set when the file moves from Maildir/new/ to Maildir/cur/. That's what the spec says, indeed. A delivery process is not supposed to know anything, so :info is not needed in new/. Gee, we find that even Dan isn't infallible. In retrospect, there's all sorts of hints that the delivery process could leave. -- John R. Levine, IECC, POB 727, Trumansburg NY 14886 +1 607 387 6869 [EMAIL PROTECTED], Village Trustee and Sewer Commissioner, http://iecc.com/johnl, Member, Provisional board, Coalition Against Unsolicited Commercial E-mail
Re: qmail-pop3d bug
On Sun, Mar 11, 2001 at 06:05:47PM -0500, John R. Levine wrote: [snip] Yes. Mind the performance penalty tho. Not a bad idea. The performance penalty would be tiny, reading buffers that are about to be written out won't cause an extra page fault. True. A possible complication with this approach is that my reading of Maildir infers that "info" can only be set when the file moves from Maildir/new/ to Maildir/cur/. That's what the spec says, indeed. A delivery process is not supposed to know anything, so :info is not needed in new/. Gee, we find that even Dan isn't infallible. In retrospect, there's all sorts of hints that the delivery process could leave. The spec is not 100% specific everywhere. And indeed, hints can be put in lots of places :) Greetz, Peter.
Re: qmail-pop3d bug
On Sun, Mar 11, 2001 at 06:05:47PM -0500, John R. Levine wrote: Putting the linecount in there makes more sense. Some MUAs might be happy about that, and it still allows easy calculation of wiresize (add number of lines to physical size). More info, less bytes :) Optimally the wire-size is calculated when the mail is written to Maildir/tmp/ and then applied as an "info" flag when the file is moved to Maildir/new/. Yes. Mind the performance penalty tho. Not a bad idea. Agreed. Line count is probably a more useful number as the other values can be derived. I retract my POPsize suggestion in favour of line count. The performance penalty would be tiny, reading buffers that are about to be written out won't cause an extra page fault. I also agree that it's an acceptable CP cost to scan a buffer just prior to writing. CP is cheap and plentiful on most qmail systems. A possible complication with this approach is that my reading of Maildir infers that "info" can only be set when the file moves from Maildir/new/ to Maildir/cur/. That's what the spec says, indeed. A delivery process is not supposed to know anything, so :info is not needed in new/. Gee, we find that even Dan isn't infallible. In retrospect, there's all sorts of hints that the delivery process could leave. Yep. And it probably wouldn't be too hard to change the standard though I note that, eg, mutt totally ignores any existing "info" values. But I'm willing to bet that they will change code if they see a good reason and they will be especially interested in a change that lets them know line count without scanning. Regards.
qmail-pop3d bug
The usual mailbox vs. maildir war has flared up on inet-access, and points out a bug in qmail-pop3d. When you do a LIST command, it gives you the size of each message. Pop3d just reports the file sizes, while it's clear from the RFC that it's supposed to report the wire size of each message, i.e., the size using cr/lf as a line terminator, so the sizes it reports are too small. I gather nobody's ever reported this as a bug, and I expect that the only thing that uses the size is the "don't download bigger than size X" option for which it's close enough, but it's still wrong. I use courier-imap, and its POP daemon does get the sizes right, presumably by reading the files and adding the number of \n characters. Regards, John Levine, [EMAIL PROTECTED], Primary Perpetrator of "The Internet for Dummies", Information Superhighwayman wanna-be, http://iecc.com/johnl, Sewer Commissioner Finger for PGP key, f'print = 3A 5B D0 3F D9 A0 6A A4 2D AC 1E 9E A6 36 A3 47
Re: qmail-pop3d bug
On Sat, Mar 10, 2001 at 01:12:13PM -0500, John R Levine wrote: The usual mailbox vs. maildir war has flared up on inet-access, and points out a bug in qmail-pop3d. When you do a LIST command, it gives you the size of each message. Pop3d just reports the file sizes, while it's clear from the RFC that it's supposed to report the wire size of each message, i.e., the size using cr/lf as a line terminator, so the sizes it reports are too small. Yes, this is known. I gather nobody's ever reported this as a bug, and I expect that the only thing that uses the size is the "don't download bigger than size X" option for which it's close enough, but it's still wrong. I use courier-imap, and its POP daemon does get the sizes right, presumably by reading the files and adding the number of \n characters. Yes. This behaviour is known. Fixing it, however, involves a *huge* performance downgrade of qmail-pop3d. I have studied the wording in RFC1939 heavily (section 11 "Message Format" specifically) and I think it is unclear. 'Usually, during the AUTHORIZATION state of the POP3 session, the POP3 server can calculate the size of each message in octets when it opens the maildrop. . simply counts each occurance of this character in a message as two octets.' The concept is obvious. The design decision made in qmail-pop3d however, is understandable, and I (as one of a few users who are aware of this 'bug') can perfectly live with it. The only other maildir MDA+pop3 implementation that I have played with is Cistron's. Their Maildir MDA counts the number of lines (it's passing the message through anyway) and adds a Lines: header. The pop3d opens each message (something qmail-pop3d doesn't have to do right now) and reads the headers to find the Lines: line. It then uses this to calculate the LF-CRLF overhead. This is not as expensive as counting the number of lines from the pop3d itself, but it does take away a lot of the performance benefit of Maildir. Greetz, Peter.
Re: qmail-pop3d bug
On Sat, Mar 10, 2001 at 09:21:46PM +0100, Peter van Dijk wrote: [snip] 'Usually, during the AUTHORIZATION state of the POP3 session, the POP3 server can calculate the size of each message in octets when it opens the maildrop. . simply counts each occurance of this character in a message as two octets.' Note that the lack of counting those extra line-terminators means some progress bars will proceed slightly past 100% when downloading messages from qmail-pop3d. Funny, but not annoying. Greetz, Peter.
Re: qmail-pop3d bug
Peter van Dijk [EMAIL PROTECTED] writes: On Sat, Mar 10, 2001 at 01:12:13PM -0500, John R Levine wrote: The usual mailbox vs. maildir war has flared up on inet-access, and points out a bug in qmail-pop3d. When you do a LIST command, it gives you the size of each message. Pop3d just reports the file sizes, while it's clear from the RFC that it's supposed to report the wire size of each message, i.e., the size using cr/lf as a line terminator, so the sizes it reports are too small. [ ... ] Yes. This behaviour is known. Fixing it, however, involves a *huge* performance downgrade of qmail-pop3d. A solution I have considered is storing the messages in wire format. Especially for POP/IMAP-only clients, seems like it could be a medium-sized performance win, since the line-conversion is done only once, regardless of how many times the message is downloaded. If the message were kept in wire-format from SMTP through delivery, no line conversion would be required at either end, and a larger performance gain would be possible. Has anybody tried this, or anything like it? -ScottG.
