On 9/29/2011 8:58 AM, Andrew Case wrote:
>
> When I did this I saw that maybe 1-5% of the time a user was still omitted
> from the list (user was silently removed). I think that because these are
> processed by the queue runner on a different host and because the
> timestamp check is being done on an NFS stored file, there is potential
> that the qrunner for this doesn't yet have an updated mtime for that file
> (or even a small ntp time drift could cause this). When I commented out
> the caching part of the code in MailList.py this bug never seems to show
> up:
> #if mtime < self.__timestamp:
> # # File is not newer
> # return None, None
Actually, that is not the cache. It is just the test for whether the
current list object, cached or whatever, needs to be reloaded from disk.
I think that your configuration with NFS and possible time jitter
between servers makes the bug more likely.
> So I think there may still be a race condition here, but the chances of it
> are unlikely that human interaction would trigger this. If however, you
> have a script that is subscribing users (one after another), this could
> still come up. I actually happen to have such a script, but I run it on
> the same host as the qrunners, so I haven't experienced this before.
It can happen even where everything is on a single host, but as I said,
I think your configuration makes it more likely.
> In my case I think it's probably not worth keeping the performance gain
> that the caching adds for sake of consistency.
Attached is a patch to remove list caching from the qrunners. This patch
has the additional advantage of limiting the growth of the qrunners over
time. Old entries were supposed to be freed from the cache, but a self
reference in the default MemberAdaptor prevented this from occurring.
For reasons of trying not to be disruptive this patch and the bug fix I
sent earlier were never applied to the 2.1 branch. I think this was a
mistake, and I will apply them for Mailman 2.1.15.
> Attached is the modified stress test I'm using.
Thanks.
--
Mark Sapiro <m...@msapiro.net> The highway is for gamblers,
San Francisco Bay Area, California better use your sense - B. Dylan
=== modified file 'Mailman/Queue/Runner.py'
--- Mailman/Queue/Runner.py 2008-05-08 03:42:28 +0000
+++ Mailman/Queue/Runner.py 2008-12-19 20:59:20 +0000
@@ -20,7 +20,6 @@
import time
import traceback
-import weakref
from cStringIO import StringIO
from Mailman import mm_cfg
@@ -198,22 +197,18 @@
if keepqueued:
self._switchboard.enqueue(msg, msgdata)
- # Mapping of listnames to MailList instances as a weak value dictionary.
- _listcache = weakref.WeakValueDictionary()
-
def _open_list(self, listname):
- # Cache the open list so that any use of the list within this process
- # uses the same object. We use a WeakValueDictionary so that when the
- # list is no longer necessary, its memory is freed.
- mlist = self._listcache.get(listname)
- if not mlist:
- try:
- mlist = MailList.MailList(listname, lock=False)
- except Errors.MMListError, e:
- syslog('error', 'error opening list: %s\n%s', listname, e)
- return None
- else:
- self._listcache[listname] = mlist
+ # We no longer cache the list instances. Because of changes to
+ # MailList.py needed to avoid not reloading an updated list, caching
+ # is not as effective as it once was. Also, with OldStyleMemberships
+ # as the MemberAdaptor, there was a self-reference to the list which
+ # kept all lists in the cache. Changing this reference to a
+ # weakref.proxy created other issues.
+ try:
+ mlist = MailList.MailList(listname, lock=False)
+ except Errors.MMListError, e:
+ syslog('error', 'error opening list: %s\n%s', listname, e)
+ return None
return mlist
def _log(self, exc):
_______________________________________________
Mailman-Developers mailing list
Mailman-Developers@python.org
http://mail.python.org/mailman/listinfo/mailman-developers
Mailman FAQ: http://wiki.list.org/x/AgA3
Searchable Archives:
http://www.mail-archive.com/mailman-developers%40python.org/
Unsubscribe:
http://mail.python.org/mailman/options/mailman-developers/archive%40jab.org
Security Policy: http://wiki.list.org/x/QIA9
