Re: [Mailman-Developers] Problem with MM after power outage

2003-10-09 Thread Barry Warsaw
On Mon, 2003-09-15 at 14:50, Peter C. Norton wrote: On Fri, Sep 12, 2003 at 05:56:00PM -0400, Barry Warsaw wrote: On Fri, 2003-09-12 at 17:40, Harald Meland wrote: Hence, I think it makes more sense to have the default be do fsync(2), and let any performance-conscious site decide whether

Re: [Mailman-Developers] Problem with MM after power outage

2003-09-12 Thread Barry Warsaw
Catching up... On Tue, 2003-08-19 at 18:45, Simone Piunno wrote: MTAs usually provide a configuration setting to enable cache flush for each transaction (by use of fsync()), but this is disabled by default because of the severe impact in performance. Indeed. I could add an optional flag

Re: [Mailman-Developers] Problem with MM after power outage

2003-09-12 Thread Brad Knowles
At 9:39 AM -0400 2003/09/12, Barry Warsaw wrote: Indeed. I could add an optional flag to enable fsync'ing in the _ext_write() calls in Switchboard.py, but I wouldn't want to enable it by default. It really kills performance. For 2.1.3, I'll add a flag to Switchboard.py but disable it by

Re: [Mailman-Developers] Problem with MM after power outage

2003-09-12 Thread Barry Warsaw
On Fri, 2003-09-12 at 12:28, Brad Knowles wrote: I would suggest making this flag visible on certain OSes. For example, turned off by default but visible on Linux, and not even visible (without hacking) on OSes that don't need it. Hmm, I don't know what make it visible on some OSs

Re: [Mailman-Developers] Problem with MM after power outage

2003-09-12 Thread Brad Knowles
At 12:56 PM -0400 2003/09/12, John A. Martin wrote: And, moreover, the choice should depend upon the file system and file system options. As you know, all Linux boxen do not necessarily only run ext2 even by distribution default. It's easy enough to check the type of filesystem to be used,

Re: [Mailman-Developers] Problem with MM after power outage

2003-09-12 Thread Peter C. Norton
On Fri, Sep 12, 2003 at 09:58:18PM +0200, Brad Knowles wrote: At 12:56 PM -0400 2003/09/12, John A. Martin wrote: And, moreover, the choice should depend upon the file system and file system options. As you know, all Linux boxen do not necessarily only run ext2 even by distribution

Re: [Mailman-Developers] Problem with MM after power outage

2003-09-12 Thread Harald Meland
[Peter C. Norton] On Fri, Sep 12, 2003 at 09:58:18PM +0200, Brad Knowles wrote: At 12:56 PM -0400 2003/09/12, John A. Martin wrote: And, moreover, the choice should depend upon the file system and file system options. As you know, all Linux boxen do not necessarily only run ext2 even

Re: [Mailman-Developers] Problem with MM after power outage

2003-09-12 Thread Barry Warsaw
On Fri, 2003-09-12 at 17:40, Harald Meland wrote: Although I haven't done any testing as to how much performance is lost by fsync(2)ing, I suspect that the sites who actually *need* this lost performance are (much) more likely to read the upgrade notes than your average Mailman site admin.

Re: [Mailman-Developers] Problem with MM after power outage

2003-09-12 Thread Harald Meland
[Barry Warsaw] Except that when I did some very simple tests, I saw a 97% hit in performance with fsync turned on. Ouch. That's pretty severe, all right. Even though I would *guess* that most casual Mailman sites would pull through an effective halving of performance without any problems,

Re: [Mailman-Developers] Problem with MM after power outage

2003-09-12 Thread Barry Warsaw
On Fri, 2003-09-12 at 12:42, Brad Knowles wrote: At 12:35 PM -0400 2003/09/12, Barry Warsaw wrote: Hmm, I don't know what make it visible on some OSs would mean. ;) So that no hacking is required to make it something the user can see and modify. We'd still be doing the dangerous

Re: [Mailman-Developers] Problem with MM after power outage

2003-09-12 Thread Barry Warsaw
On Fri, 2003-09-12 at 15:58, Brad Knowles wrote: It's easy enough to check the type of filesystem to be used, and whether chatter +S has been run on the particular directory structure. Maybe there should be a FAQ or README.LINUX items discussing the issue, and suggesting filesystem

Re: [Mailman-Developers] Problem with MM after power outage

2003-08-25 Thread Barry Warsaw
On Tue, 2003-08-19 at 15:17, Simone Piunno wrote: On Tuesday 19 August 2003 04:08, Barry Warsaw wrote: After power was restored and the machine came up, Mailman (2.1.2) is now spitting out errors: File /usr/local/mailman/Mailman/Queue/Switchboard.py, line 246, in _ext_read

Re: [Mailman-Developers] Problem with MM after power outage

2003-08-25 Thread Simone Piunno
On Monday 25 August 2003 10:15, Brad Knowles wrote: ok, but how do you make sure the file is really on disk instead of, We close the file before we rename it. Do you fsync() the directory after the close and before the rename? According to python documentation (os module), this

Re: [Mailman-Developers] Problem with MM after power outage

2003-08-25 Thread Simone Piunno
On Monday 25 August 2003 22:11, Simone Piunno wrote: Do you fsync() the directory after the close and before the rename? Ah, I've found what you were meaning... this is from PosixFilesystem.py (ZODB implementation): import os from posix import fsync def sync_directory(self,dir):

Re: [Mailman-Developers] Problem with MM after power outage

2003-08-22 Thread Simone Piunno
On Wednesday 20 August 2003 00:06, you wrote: ok, but how do you make sure the file is really on disk instead of, e.g., half on disk and half on cache? We close the file before we rename it. This would be ok if the underlying operating system flushed the disk cache upon close(), but I'm

Re: [Mailman-Developers] Problem with MM after power outage

2003-08-19 Thread Simone Piunno
On Tuesday 19 August 2003 04:08, Barry Warsaw wrote: After power was restored and the machine came up, Mailman (2.1.2) is now spitting out errors: File /usr/local/mailman/Mailman/Queue/Switchboard.py, line 246, in _ext_read dict = marshal.load(fp) ValueError: bad marshal data

Re: [Mailman-Developers] Problem with MM after power outage

2003-08-18 Thread Barry Warsaw
On Fri, 2003-08-15 at 12:52, Bill Bradford wrote: Looks like the problem resolved itself after a few minutes. Bill On Fri, Aug 15, 2003 at 09:58:39AM -0500, Bill Bradford wrote: After power was restored and the machine came up, Mailman (2.1.2) is now spitting out errors: Traceback

[Mailman-Developers] Problem with MM after power outage

2003-08-15 Thread Bill Bradford
After power was restored and the machine came up, Mailman (2.1.2) is now spitting out errors: Traceback (most recent call last): File /usr/local/mailman/bin/qrunner, line 270, in ? main() File /usr/local/mailman/bin/qrunner, line 230, in main qrunner.run() File

Re: [Mailman-Developers] Problem with MM after power outage

2003-08-15 Thread Bill Bradford
Looks like the problem resolved itself after a few minutes. Bill On Fri, Aug 15, 2003 at 09:58:39AM -0500, Bill Bradford wrote: After power was restored and the machine came up, Mailman (2.1.2) is now spitting out errors: Traceback (most recent call last): File