Re: [Mailman-Developers] re: Much cpu/memory load

2002-11-04 Thread Barry A. Warsaw
> "DM" == Dan Mick <[EMAIL PROTECTED]> writes: DM> Sigh. This isn't making life any easier trying to debug the DM> "Archiver eats my machine" problem: DM> self._open_index_file_as_stdout(arcdir, hdr) DM> Any tricks for using pdb with a program that wants to steal

Re: [Mailman-Developers] re: Much cpu/memory load

2002-11-04 Thread Barry A. Warsaw
Here's a patch that eliminates the MailList object from the article index pickle. It also undoes the incomplete hack in MailList for not pickling the lock object (that should have been a major clue). I've done some moderate testing of this patch, and I think it's safe, and probably fixes the maj

Re: [Mailman-Developers] re: Much cpu/memory load

2002-11-04 Thread Barry A. Warsaw
Dan found what is probably the majority of the problem. In the i18n patch, Article instances in HyperArch.py grew an _mlist instance so that means a unique instance of the MailList is pickled and unpickled in every Article in the archive. Ug. I'm working on a fix for that and will see about the

Re: [Mailman-Developers] re: Much cpu/memory load

2002-11-01 Thread Dan Mick
Sigh. This isn't making life any easier trying to debug the "Archiver eats my machine" problem: self._open_index_file_as_stdout(arcdir, hdr) Any tricks for using pdb with a program that wants to steal stdout? (it's the first invocation of _update_simple_index, the one for the Date inde

Re: [Mailman-Developers] re: Much cpu/memory load

2002-11-01 Thread Dan Mick
Dan Mick wrote: Barry, wasn't there some Python instrumentation trick for using the GC to find unreferenced objects and complain, or invoking GC manually in a loop to try to alleviate such problems?was it in the gc module?... Adding this: --- /usr/local/src/mailman/Mailman/Archiver/piperm

Re: [Mailman-Developers] re: Much cpu/memory load

2002-11-01 Thread Dan Mick
Dan Mick wrote: Simone Piunno wrote: On Thu, Oct 31, 2002 at 07:03:50PM -0800, Dan Mick wrote: The problem's affecting me badly right now too. So far it seems that the holdup is in the 'date.html' index file processing; everything else is finished, but the "bin/qrunner -r Arch -o" process

Re: [Mailman-Developers] re: Much cpu/memory load

2002-11-01 Thread Dan Mick
Simone Piunno wrote: On Thu, Oct 31, 2002 at 07:03:50PM -0800, Dan Mick wrote: The problem's affecting me badly right now too. So far it seems that the holdup is in the 'date.html' index file processing; everything else is finished, but the "bin/qrunner -r Arch -o" process has lost its littl

Re: [Mailman-Developers] re: Much cpu/memory load

2002-11-01 Thread Danny Terweij
From: "Simone Piunno" <[EMAIL PROTECTED]> > > The problem's affecting me badly right now too. So far it seems that the > Ok, I believe this is because of my patch for i18n in archives. > Do you feel like trying this patch? > It should speed up things a lot > Index: HyperArch.py > ==

Re: [Mailman-Developers] re: Much cpu/memory load

2002-11-01 Thread Danny Terweij
From: "Dan Mick" <[EMAIL PROTECTED]> > Barry A. Warsaw wrote: > > A quick fix for the Archiver problems, until I can debug them more is > > to add the following in ArchRunner.py, just under the "class > > ArchRunner" line: > The problem's affecting me badly right now too. So far it seems that th

Re: [Mailman-Developers] re: Much cpu/memory load

2002-11-01 Thread Simone Piunno
On Thu, Oct 31, 2002 at 07:03:50PM -0800, Dan Mick wrote: > The problem's affecting me badly right now too. So far it seems that the > holdup is in the 'date.html' index file processing; everything else is > finished, but the "bin/qrunner -r Arch -o" process has lost its little mind > trying t

Re: [Mailman-Developers] re: Much cpu/memory load

2002-10-31 Thread Dan Mick
Barry A. Warsaw wrote: A quick fix for the Archiver problems, until I can debug them more is to add the following in ArchRunner.py, just under the "class ArchRunner" line: class ArchRunner(Runner): QDIR = mm_cfg.ARCHQUEUE_DIR SLEEPTIME = mm_cfg.minutes(10) def _dispose(self, mlist, m

Re: [Mailman-Developers] re: Much cpu/memory load

2002-10-31 Thread Danny Terweij
From: "Barry A. Warsaw" <[EMAIL PROTECTED]> > A quick fix for the Archiver problems, until I can debug them more is > to add the following in ArchRunner.py, just under the "class > ArchRunner" line: Thanx, i'll try that tomorrow. (Now here 22:39) I'll let you know what then happens :) Danny.

Re: [Mailman-Developers] re: Much cpu/memory load

2002-10-31 Thread Barry A. Warsaw
A quick fix for the Archiver problems, until I can debug them more is to add the following in ArchRunner.py, just under the "class ArchRunner" line: class ArchRunner(Runner): QDIR = mm_cfg.ARCHQUEUE_DIR SLEEPTIME = mm_cfg.minutes(10) def _dispose(self, mlist, msg, msgdata): This at

Re: [Mailman-Developers] re: Much cpu/memory load

2002-10-31 Thread Danny Terweij
From: "Andrew D. Clark" <[EMAIL PROTECTED]> > I've had the same problem. ArchRunner seems to have serious performance > problems if you've got heavy list activity and are archiving to both > pipermail and mbox. I "fixed" it by setting ARCHIVE_TO_MBOX=1. This will > only archive to mbox, so your

[Mailman-Developers] re: Much cpu/memory load

2002-10-31 Thread Andrew D. Clark
I've had the same problem. ArchRunner seems to have serious performance problems if you've got heavy list activity and are archiving to both pipermail and mbox. I "fixed" it by setting ARCHIVE_TO_MBOX=1. This will only archive to mbox, so your pipermail archives won't be updated. You should