Re: bkgdtest: a test for the extra spaces at the end of lines
On 10Aug2010 02:25, Vincent Lefevre vinc...@vinc17.org wrote: | On 2010-08-09 09:36:02 -0700, Michael Elkins wrote: | This was buried deep in a thread, so I am reposting for wider publication. | | Several people have complained that when selected text in the pager | that there is extra space at the end of pasted blocks. This test | program is to help diagnose the problem. | | Note that screen has bce support (I'm using TERM=screen-bce), but | I sometimes get spaces at the end of lines in it (at least when | switching window). The problem seems to come from screen, though. | I reported it here: | | http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=578729 I can imagine it happening if the terminal _displaying_ the screen session does not have bce... -- Cameron Simpson c...@zip.com.au DoD#743 http://www.cskk.ezoshosting.com/cs/
Further question(s) about locking, viewing inbox, etc.
Thanks for all the help so far. :-) I've been thinking (now that's something!) about how mutt interacts with mail delivery programs and I'm realising that I don't *really* understand how it all works regarding locking etc. Take a trivially simple case, mail for user chris is delivered to /var/mail/chris by, say, postfix. I run mutt which looks at /var/mail/chris and displays the index of messages there. What happens (as regards locks etc.) when mutt is displaying the inbox index and postfix wants to deliver another message? Obviously mutt doesn't keep the file /var/mail/chris permanently locked as otherwise postfix would never be able to append the new message (if it was obeying the locking protocol). So what *does* happen? Does mutt open the file, scan the contents and then close it periodically? This would seem to make sense as it often does take a while for mutt to notice that postfix has appended a new message. However I don't then understand what's going wrong when my python script appends a message to /var/mail/chris (or wherever). What happens (in the non NFS situation) is that mutt complains Mailbox was externally modified. Flags may be wrong. I can also provoke this message if I run another copy of mutt and add/delete messages. This error doesn't seem to relate to locking at all, it seems to be that mutt is detecting some *other* change to the mbox file in question. Can someone tell me what exactly mutt is checking to decide that Mailbox was externally modified. Flags may be wrong.? Is it a check to see if all messages up to the new appended one are unchanged, or is it something simpler than this like a simple length check? -- Chris Green
Re: Problem with mutt and mailbox changes (currently best 1.5.18)
On Tue, Aug 10, 2010 at 06:58:14AM +0800, Charles Jie wrote: Before the bug is fixed, I suggest going back to mutt 1.5.18. I found it has an extra benefit - it opens mailbox much faster than 1.5.20. I don't know why, but you can benchmark it. (I have some mailboxes of size 1-200MB. The time-saving for them is obvious.) The ticket has not yet been assigned, so we probably shouldn't hold our breath. That makes your work-around quite attractive. Now I just have to figure out whether I can pin an apt-get install (on ubuntu) to the older version, or if it's easier to fetch old source. I logged into the flea tracker, and there's a field which allows Add/Change #3424 (Mutt falsely detects its own mbox writes as new mail.), but I don't see any simple me too option, which could perhaps have helped us vote the problem toward higher visibility. Erik -- Linux runs on 1.13 percent of desktops, 20.8 percent of servers, and on 91 percent of the world's 500 fastest supercomputers. - http://web.eweek.com/t?ctl=777CB:D3FDA57AA7175079D12B9A56271F01FD;
Re: Further question(s) about locking, viewing inbox, etc.
On Tue, Aug 10, 2010 at 09:28:14AM +0100, Chris G wrote: Thanks for all the help so far. :-) I've been thinking (now that's something!) about how mutt interacts with mail delivery programs and I'm realising that I don't *really* understand how it all works regarding locking etc. Take a trivially simple case, mail for user chris is delivered to /var/mail/chris by, say, postfix. I run mutt which looks at /var/mail/chris and displays the index of messages there. What happens (as regards locks etc.) when mutt is displaying the inbox index and postfix wants to deliver another message? Obviously mutt doesn't keep the file /var/mail/chris permanently locked as otherwise postfix would never be able to append the new message (if it was obeying the locking protocol). So what *does* happen? Does mutt open the file, scan the contents and then close it periodically? This would seem to make sense as it often does take a while for mutt to notice that postfix has appended a new message. However I don't then understand what's going wrong when my python script appends a message to /var/mail/chris (or wherever). What happens (in the non NFS situation) is that mutt complains Mailbox was externally modified. Flags may be wrong. I can also provoke this message if I run another copy of mutt and add/delete messages. This error doesn't seem to relate to locking at all, it seems to be that mutt is detecting some *other* change to the mbox file in question. I now know! I went and looked at the Python mailbox class. When it writes *anything* to an mbox file what it actually does is to create a new file, copy the contents of the old mbox file to it, adds the new data and then renames the new mbox file over the old mbox file. This accounts for the errors I have been seeing. The problem has nothing whatsoever to do with locking, it's the replacement of the file that does it. There is a sort of logic to what python does, one *might* be deleting or adding a message into the middle of the mbox in which case the copy, rename sequence is the only sensible way to do it. However the documentation might have told me that and I'd not have wasted lots of my (and the list's) time trying to work out what was wrong. I can almost certainly fix the problem by simply not using the Python mailbox class and by just appending new messages to the mbox using straightforward file write calls (with appropriate locking of course). Can someone tell me what exactly mutt is checking to decide that Mailbox was externally modified. Flags may be wrong.? Is it a check to see if all messages up to the new appended one are unchanged, or is it something simpler than this like a simple length check? -- Chris Green -- Chris Green
Re: Problem with mutt and mailbox changes (currently best 1.5.18)
On Tue, Aug 10, 2010 at 06:36:38PM +1000, Erik Christiansen wrote: I logged into the flea tracker, and there's a field which allows Add/Change #3424 (Mutt falsely detects its own mbox writes as new mail.), but I don't see any simple me too option, which could perhaps have helped us vote the problem toward higher visibility. FYI, I plan to tackle this bug soon. The discussion has not got unnoticed. :) me
Re: Further question(s) about locking, viewing inbox, etc.
On Tue, Aug 10, 2010 at 10:11:35AM +0100, Chris G wrote: There is a sort of logic to what python does, one *might* be deleting or adding a message into the middle of the mbox in which case the copy, rename sequence is the only sensible way to do it. However the documentation might have told me that and I'd not have wasted lots of my (and the list's) time trying to work out what was wrong. When Mutt needs to delete a message out of the middle of a mbox, it does not replace the file at all. Instead, all the messages after the deleted one are first copied to a temporary file, then mutt truncates the file at the start of the deleted message, then copies the remaining messages from the temporary file back into the same open file. Another open Mutt will still get the flags may be wrong error in this case, because mbox has no easy way of matching up messages (message-ids may not be unique). me
Re: Problem with mutt and mailbox changes (currently best 1.5.18)
On Tue, Aug 10, 2010 at 06:40:18AM -0700, Michael Elkins wrote: FYI, I plan to tackle this bug soon. The discussion has not got unnoticed. :) Grateful thanks, Michael. If you'll need another beta tester, just yell out when the hard work's done. ;-) Erik
Re: Further question(s) about locking, viewing inbox, etc.
On Tue, Aug 10, 2010 at 10:11:35AM +0100, Chris G wrote: There is a sort of logic to what python does, one *might* be deleting or adding a message into the middle of the mbox Unless you're expunging messages, or changing their status, it does not make sense to do that. In other words, if you're only delivering a message, the only sensible behavior is to lock the spool and append the message. I don't think there's any compelling reason to insert a message into the middle of the spool... Mail clients need to deal with sorting messages anyway. in which case the copy, rename sequence is the only sensible way to do it. Even in the case of expunging messages, it actually isn't... There's another way that's sensible, and better-performing, but also more complicated to implement and potentially devastatingly destructive if you don't do it right: expunge in place. The idea is to: 1. Copy, in chunks, data from undeleted messages which are later in sequence over messages which are deleted 2. Continue, copying later messages over moved/deleted chunks until all the undeleted message data is sequential 3. Truncate the file. The risk here is that if you have a power outage or similar event while your mail program is writing the file, it guarantees corruption of the mail spool. My argument is that this is still sensible to implement, because: - With typical usage patterns, usually earlier messages in an inbox are old, and the user is keeping them around for some reason. - Messages at the end of the file are new, and require action. Most typically, these are the ones that get deleted. - On modern hardware, with MMIO, performing the operations above will be extremely fast for the common case described above - As a result, the risk of expunging the spool file during a power outage is extremely low... And filesystem corruption is somewhat likely to occur, regardless of implementation, under those conditions. - On a well-run busy mail server, the system should be tied to a UPS which shuts down the mail system when an imminent power failure is detected (i.e. either immediately, or when the battery is getting low). The performance benefits make this worth considering. There's more risk than with the traditional method, but there's always *some* risk of filesystem corruption under any sort of failure mode, and with this method there's still really not very much risk, especially if you're doing all this on a well-run mail server. Were I to implement such a thing, I'd also implement the traditional method side-by-side. I'd provide an API call to allow selecting one of them explicitly, and also make the algorithm selectable by environment variable so that users could override the default behavior of applications using the library. -- Derek D. Martinhttp://www.pizzashack.org/ GPG Key ID: 0xDFBEAD02 -=-=-=-=- This message is posted from an invalid address. Replying to it will result in undeliverable mail due to spam prevention. Sorry for the inconvenience. pgpSw1Qy1Qlxz.pgp Description: PGP signature
