Geoff Shang wrote:
>
>The problem we're seeing is that if the list gets any way busy, the memory
>usage of the python process that runs Mailman skyrockets, dragging the
>system to a crawl.
Which Python process? mailmanctl should not be affected. Beyond that,
there are 8 qrunner processes. Are they all affected, or just one or
two?
>We're a little clueless as to how to debug this further, so any help would
>be appreciated.
There is an issue that affects memory usage in the qrunners. They keep
a cache of list objects in memory to reduce disk IO. The cache is
supposed to free the space used by a list object when there are no
more references to that object, but it turns out there is a
self-reference in the list objects, so the cache simply grows until it
holds a copy of each list.
There are other issues in that large messages can cause the runners
that handle it to grow, and Python's memory management is such that
Python itself never gives freed memory back to the OS. Memory can be
freed within Python and it will be available for reuse within that
process, but it is not given back to the OS.
I recommend disabling the list cache within the qrunners. This was done
for the now defunct 2.2 branch, but has not been done on the 2.1
branch.
The attached Runner.patch.txt file contains a patch to do this. I
suggest you try the patch and see if that helps.
--
Mark Sapiro <[email protected]> The highway is for gamblers,
San Francisco Bay Area, California better use your sense - B. Dylan
=== modified file 'Mailman/Queue/Runner.py'
--- Mailman/Queue/Runner.py 2008-05-08 03:42:28 +0000
+++ Mailman/Queue/Runner.py 2008-12-19 20:59:20 +0000
@@ -20,7 +20,6 @@
import time
import traceback
-import weakref
from cStringIO import StringIO
from Mailman import mm_cfg
@@ -198,22 +197,18 @@
if keepqueued:
self._switchboard.enqueue(msg, msgdata)
- # Mapping of listnames to MailList instances as a weak value dictionary.
- _listcache = weakref.WeakValueDictionary()
-
def _open_list(self, listname):
- # Cache the open list so that any use of the list within this process
- # uses the same object. We use a WeakValueDictionary so that when the
- # list is no longer necessary, its memory is freed.
- mlist = self._listcache.get(listname)
- if not mlist:
- try:
- mlist = MailList.MailList(listname, lock=False)
- except Errors.MMListError, e:
- syslog('error', 'error opening list: %s\n%s', listname, e)
- return None
- else:
- self._listcache[listname] = mlist
+ # We no longer cache the list instances. Because of changes to
+ # MailList.py needed to avoid not reloading an updated list, caching
+ # is not as effective as it once was. Also, with OldStyleMemberships
+ # as the MemberAdaptor, there was a self-reference to the list which
+ # kept all lists in the cache. Changing this reference to a
+ # weakref.proxy created other issues.
+ try:
+ mlist = MailList.MailList(listname, lock=False)
+ except Errors.MMListError, e:
+ syslog('error', 'error opening list: %s\n%s', listname, e)
+ return None
return mlist
def _log(self, exc):
------------------------------------------------------
Mailman-Users mailing list [email protected]
http://mail.python.org/mailman/listinfo/mailman-users
Mailman FAQ: http://wiki.list.org/x/AgA3
Security Policy: http://wiki.list.org/x/QIA9
Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/
Unsubscribe:
http://mail.python.org/mailman/options/mailman-users/archive%40jab.org
