Revision: 8214
http://svn.sourceforge.net/mailman/?rev=8214&view=rev
Author: bwarsaw
Date: 2007-05-09 20:08:48 -0700 (Wed, 09 May 2007)
Log Message:
-----------
Forward port part of Mark's r8204 fixes:
- Queue/Runner.py. Queue/Switchboard.py
Now that we have .bak queue entries for recovery, it is no longer the
case that an unparseable message is lost. In this case, and in case
of other exceptions when dequeueing, I added a preservation feature
to move the .bak file to qfiles/shunt as a .psv file and write an
appropriate log entry. It is also possible for an attempt to shunt
a message to fail. One example that occurred in practice (bug 1656289)
was caused by a huge message that threw a MemoryError in processing and
then threw another MemoryError in the attempt to pickle the message for
the shunt queue. In this case as well, I log and attempt to preserve
the original queue entry by renaming.
Ports of other non-GUI parts of r8204 coming up...
Modified Paths:
--------------
branches/exp-elixir-branch/Mailman/Queue/Runner.py
branches/exp-elixir-branch/Mailman/Queue/Switchboard.py
branches/exp-elixir-branch/docs/NEWS.txt
Modified: branches/exp-elixir-branch/Mailman/Queue/Runner.py
===================================================================
--- branches/exp-elixir-branch/Mailman/Queue/Runner.py 2007-05-09 20:54:30 UTC
(rev 8213)
+++ branches/exp-elixir-branch/Mailman/Queue/Runner.py 2007-05-10 03:08:48 UTC
(rev 8214)
@@ -92,16 +92,16 @@
# Ask the switchboard for the message and metadata objects
# associated with this filebase.
msg, msgdata = self._switchboard.dequeue(filebase)
- except email.Errors.MessageParseError, e:
- # It's possible to get here if the message was stored in the
- # pickle in plain text, and the metadata had a _parsemsg key
- # that was true, /and/ if the message had some bogosity in
- # it. It's almost always going to be spam or bounced spam.
- # There's not much we can do (and we didn't even get the
- # metadata, so just log the exception and continue.
+ except Exception, e:
+ # This used to just catch email.Errors.MessageParseError,
+ # but other problems can occur in message parsing, e.g.
+ # ValueError, and exceptions can occur in unpickling too.
+ # We don't want the runner to die, so we just log and skip
+ # this entry, but preserve it for analysis.
self._log(e)
- log.error('Ignoring unparseable message: %s', filebase)
- self._switchboard.finish(filebase)
+ log.error('Skipping and preserving unparseable message: %s',
+ filebase)
+ self._switchboard.finish(filebase, preserve=True)
continue
try:
self._onefile(msg, msgdata)
@@ -116,9 +116,21 @@
self._log(e)
# Put a marker in the metadata for unshunting
msgdata['whichq'] = self._switchboard.whichq()
- new_filebase = self._shunt.enqueue(msg, msgdata)
- log.error('SHUNTING: %s', new_filebase)
- self._switchboard.finish(filebase)
+ # It is possible that shunting can throw an exception, e.g. a
+ # permissions problem or a MemoryError due to a really large
+ # message. Try to be graceful.
+ try:
+ new_filebase = self._shunt.enqueue(msg, msgdata)
+ log.error('SHUNTING: %s', new_filebase)
+ self._switchboard.finish(filebase)
+ except Exception, e:
+ # The message wasn't successfully shunted. Log the
+ # exception and try to preserve the original queue entry
+ # for possible analysis.
+ self._log(e)
+ log.error('SHUNTING FAILED, preserving original entry: %s',
+ filebase)
+ self._switchboard.finish(filebase, preserve=True)
# Other work we want to do each time through the loop
Utils.reap(self._kids, once=True)
self._doperiodic()
Modified: branches/exp-elixir-branch/Mailman/Queue/Switchboard.py
===================================================================
--- branches/exp-elixir-branch/Mailman/Queue/Switchboard.py 2007-05-09
20:54:30 UTC (rev 8213)
+++ branches/exp-elixir-branch/Mailman/Queue/Switchboard.py 2007-05-10
03:08:48 UTC (rev 8214)
@@ -149,12 +149,28 @@
msg = email.message_from_string(msg, Message.Message)
return msg, data
- def finish(self, filebase):
+ def finish(self, filebase, preserve=False):
bakfile = os.path.join(self.__whichq, filebase + '.bak')
try:
- os.unlink(bakfile)
+ if preserve:
+ psvfile = os.path.join(config.SHUNTQUEUE_DIR,
+ filebase + '.psv')
+ # Create the directory if it doesn't yet exist.
+ # Copied from __init__.
+ omask = os.umask(0) # rwxrws---
+ try:
+ try:
+ os.mkdir(config.SHUNTQUEUE_DIR, 0770)
+ except OSError, e:
+ if e.errno <> errno.EEXIST: raise
+ finally:
+ os.umask(omask)
+ os.rename(bakfile, psvfile)
+ else:
+ os.unlink(bakfile)
except EnvironmentError, e:
- elog.exception('Failed to unlink backup file: %s', bakfile)
+ elog.exception('Failed to unlink/preserve backup file: %s',
+ bakfile)
def files(self, extension='.pck'):
times = {}
Modified: branches/exp-elixir-branch/docs/NEWS.txt
===================================================================
--- branches/exp-elixir-branch/docs/NEWS.txt 2007-05-09 20:54:30 UTC (rev
8213)
+++ branches/exp-elixir-branch/docs/NEWS.txt 2007-05-10 03:08:48 UTC (rev
8214)
@@ -4,7 +4,7 @@
Here is a history of user visible changes to Mailman.
-2.2 alpha 1 (XX-XXX-200X)
+3.0 alpha 1 (XX-XXX-200X)
Configuration
@@ -104,13 +104,18 @@
Bug fixes and other patches
+ - Queue runner processing is improved to log and preserve for analysis in
+ the shunt queue certain bad queue entries that were previously logged
+ but lost. Also, entries are preserved when an attempt to shunt throws
+ an exception (1656289).
+
- The processing of Topics regular expressions has changed. Previously the
Topics regexp was compiled in verbose mode but not documented as such
- which caused some confusion. Also, the documentation indicated that
topic
- keywords could be entered one per line, but these entries were not
handled
- properly. Topics regexps are now compiled in non-verbose mode and multi-
- line entries are 'ored'. Existing Topics regexps will be converted when
- the list is updated so they will continue to work.
+ which caused some confusion. Also, the documentation indicated that
+ topic keywords could be entered one per line, but these entries were not
+ handled properly. Topics regexps are now compiled in non-verbose mode
+ and multi- line entries are 'ored'. Existing Topics regexps will be
+ converted when the list is updated so they will continue to work.
- The List-Help, List-Subscribe, and List-Unsubscribe headers were
incorrectly suppressed in messages that Mailman sends directly to
This was sent by the SourceForge.net collaborative development platform, the
world's largest Open Source development site.
_______________________________________________
Mailman-checkins mailing list
[email protected]
Unsubscribe:
http://mail.python.org/mailman/options/mailman-checkins/archive%40jab.org
