On Tue, May 9, 2017 at 12:58 PM, Mark Sapiro <[email protected]> wrote:
> On 05/09/2017 07:54 AM, Matt Morgan wrote: > > > > I do see this in the error log: > > > > Apr 27 20:58:43 2017 (517) Uncaught runner exception: 'ascii' codec can't > > decode > > byte 0xb5 in position 26: ordinal not in range(128) > > Apr 27 20:58:43 2017 (517) Traceback (most recent call last): > > File "/usr/local/mailman/Mailman/Queue/Runner.py", line 119, in > _oneloop > > self._onefile(msg, msgdata) > > File "/usr/local/mailman/Mailman/Queue/Runner.py", line 190, in > _onefile > > keepqueued = self._dispose(mlist, msg, msgdata) > > File "/usr/local/mailman/Mailman/Queue/ArchRunner.py", line 77, in > > _dispose > > mlist.ArchiveMail(msg) > > File "/usr/local/mailman/Mailman/Archiver/Archiver.py", line 216, in > > ArchiveMail > > h.processUnixMailbox(f) > > File "/usr/local/mailman/Mailman/Archiver/pipermail.py", line 596, in > > processUnixMailbox > > self.add_article(a) > > File "/usr/local/mailman/Mailman/Archiver/pipermail.py", line 640, in > > add_article > > author = fixAuthor(article.decoded['author']) > > File "/usr/local/mailman/Mailman/Archiver/pipermail.py", line 63, in > > fixAuthor > > while i>0 and (L[i-1][0] in lowercase or > > UnicodeDecodeError: 'ascii' codec can't decode byte 0xb5 in position 26: > > ordinal not in range(128) > > > > Apr 27 20:58:43 2017 (517) SHUNTING: > > 1493351921.551243+8e185be2446d37cbda2f99d888a5a7b7e0643d85 > > > > Is that the error I'm looking for? > > > Yes. > > > > That ArchRunner line looks relevant. > > It's a reasonably common series of entries in the log (10ish times a > day), > > though the timing doesn't line up with the date/time on the messages that > > are missing. Is that normal? Is there a way to verify that the error is > > specific to any given missing message? > > > Every one of those messages is in Mailman's shunt queue with names like > 1493351921.551243+8e185be2446d37cbda2f99d888a5a7b7e0643d85.pck. You can > see the messages in a couple of ways. > > In Mailman's directory (/usr/local/mailman in your case) > > bin/show_qfiles qfiles/shunt/* > Just FYI, in case anyone's reading this in the list archives in future, you may need a "../" in front of "qfiles/shunt/*" there. > > will show them all or you can view individual ones by name, or you can run > > bin/dumpdb qfiles/shunt/xxx.pck > > where xxx.pck is an individual name. The latter will also show the > message metadata which will indicate, among other things, the queue the > message came from which in the case of these should be 'archive'. > I subscribed to the list so that I could track this more easily. A message from yesterday was kept out of the archives in the same way. Here are the headers (anonymized a little): Delivered-To: [email protected] Received: by 10.25.153.69 with SMTP id b66csp370037lfe; Wed, 10 May 2017 12:55:08 -0700 (PDT) X-Received: by 10.84.128.65 with SMTP id 59mr10521582pla.121.1494446108067; Wed, 10 May 2017 12:55:08 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1494446108; cv=none; d=google.com; s=arc-20160816; b=FRn0ZWGg2oTxTU3+KArFayyfI4JJZKZiZsezc25C6hsUgjDqdAn5CAj4aBKIlfkR7Z TQYcmBDk8KoNPaLphRrgPXmCg6rV1OLnU6EzcFdJoMxW3kLuG5tbhJFJSl4ypxh7TQDe zJ3GY1KRWu7AWdaESXevJLyuMwwjJ3VTBwQUwMsXdNju3F3DfekHHuDJdlNWWVrFvLXM 96rGZi9T1Cihg2GW5bWvaNQAxkbmlOIlcsTB3SSx0Yq9P+aOFvQwDBq+U8KDYRCZcoi9 LI5WiMmivYJ0zEToSnd+EJZ+bn4Yd9xFt7KWJubjdc55lSRzsIrN1hjnaQeCD8Eq3RAP w12g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d= google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:reply-to:list-subscribe :list-help:list-post:list-archive:list-unsubscribe:list-id :precedence:subject:message-id:in-reply-to:to:references:date :mime-version:from:dkim-signature:arc-authentication-results; bh=cizuuK18iQqPuWzF8CUKeju438HlKLJbPg8FiN9HnfU=; b=kmeTsmGWUerbMCwqenno45KatPObxGKdigUWDNtufBWJKM2eNTK23S1qRIK2ele8Gh tdJayjTyXIS432QUS40Ruc0/SCk6RI/MYns3khWDZNxfFYSCYppHBOwtM84i4sX6zILk CQwSj3uVFdj6QbBM4TiUots9hgA3piHzwUINJn7SYatHcrJWeNSBDgAbRZDWJNsOTl0m lOcQVDRZ+xSMb8Xe8QFnSQmSy2xCaI5ixYytUSIL5KO06dwGurnajE4D1I+vwfCgZGcb nmUwNUwF6eWzvkWBL0M8UNbrcsdRq6f//ezkIrLr5Gmn4uTwON7dm/HL9gcYnbehS0Sp 0SeA== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) [email protected]; spf=neutral (google.com: 206.180.233.162 is neither permitted nor denied by best guess record for domain of [email protected]) smtp.mailfrom= [email protected]; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=me.com Return-Path: <[email protected]> Received: from cool.conservation-us.org (cool.tdl.com. [206.180.233.162]) by mx.google.com with ESMTPS id q185si4018820pga.417.2017.05.10.12.55.06 (version=TLS1 cipher=AES128-SHA bits=128/128); Wed, 10 May 2017 12:55:08 -0700 (PDT) Received-SPF: neutral (google.com: 206.180.233.162 is neither permitted nor denied by best guess record for domain of [email protected]) client-ip=206.180.233.162; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) [email protected]; spf=neutral (google.com: 206.180.233.162 is neither permitted nor denied by best guess record for domain of [email protected]) smtp.mailfrom= [email protected]; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=me.com Received: from localhost ([127.0.0.1] helo=cool.tdl.com) by cool.conservation-us.org with esmtp (Exim 4.69) (envelope-from < [email protected]>) id 1d8XhD-0000aU-GX; Wed, 10 May 2017 12:54:51 -0700 Received: from pv35p22im-ztdg05131101.me.com ([17.133.189.43]) by cool.conservation-us.org with esmtp (Exim 4.69) (envelope-from < [email protected]>) id 1d8XhC-0000aM-7S for [email protected]; Wed, 10 May 2017 12:54:50 -0700 Received: from process-dkim-sign-daemon.pv35p22im-ztdg05131101.me.com by pv35p22im-ztdg05131101.me.com (Oracle Communications Messaging Server 7.0.5.38.0 64bit (built Feb 26 2016)) id < [email protected]> for [email protected]; Wed, 10 May 2017 19:54:49 +0000 (GMT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=me.com; s=4d515a; t=1494446089; bh=iMRS0rztCQMUAhGFAT7khfPTdH9Fy/59g3wJDcQzaxM=; h=From:Content-type:MIME-version:Subject:Date:To:Message-id; b=P994EyGUDRT5hcWBUpFuaXhGcIwIv5y+2Fc2iyA9I1vQsqyq4rg57CptHkrr/t3A7 yf4tU+IIbYu0Jq/F9xtVnPT3u/B5s2MfZvIORNOvKxxx1bm0BwXpGqwwLTAE73gfxO y8akmoAfaoUJlbQ+Wptu8Y7uNLPBaMyuUoomX8Bcm7UiNW9lWBLtIgCAR0pAJ0StqP Fm1N9Rqn4omlJ5z7uQ1V/wdMgO5QRAY0NT2punAB5sGs3eEJJavfWZsiGDbNBIq56v 9mYCxgyId1spAvb4HgVbKNd28V7C3y5OreWQsPdTp0HS218dODJQekkAv6dTd5RxSc FMEP1hpTefwhg== Received: from icloud.com ([127.0.0.1]) by pv35p22im-ztdg05131101.me.com (Oracle Communications Messaging Server 7.0.5.38.0 64bit (built Feb 26 2016)) with ESMTPSA id <[email protected]> for [email protected]; Wed, 10 May 2017 19:54:46 +0000 (GMT) X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:,, definitions=2017-05-10_16:,, signatures=0 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0 clxscore=1034 suspectscore=0 malwarescore=0 phishscore=0 adultscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1701120000 definitions=main-1705100136 From: XXXXXXXX <[email protected]> MIME-version: 1.0 (Mac OS X Mail 10.3 \(3273\)) Date: Wed, 10 May 2017 16:54:44 -0300 References: <CACJwyVrjx-m1VZ= [email protected]> To: Conservators in Private Practice Discussion < [email protected]> In-reply-to: <CACJwyVrjx-m1VZ= [email protected]> Message-id: <[email protected]> X-Mailer: Apple Mail (2.3273) Subject: Re: [CIPPNEWS] closing private conservation studio X-BeenThere: [email protected] X-Mailman-Version: 2.1.23 Precedence: list List-Id: Conservators in Private Practice Discussion < cippnews.cool.conservation-us.org> List-Unsubscribe: < http://cool.conservation-us.org/mailman/options/cippnews>, <mailto: [email protected]?subject=unsubscribe> List-Archive: <http://cool.conservation-us.org/mailman/private/cippnews/ > List-Post: <mailto:[email protected]> List-Help: <mailto:[email protected] ?subject=help> List-Subscribe: < http://cool.conservation-us.org/mailman/listinfo/cippnews>, <mailto: [email protected]?subject=subscribe> Reply-To: Conservators in Private Practice Discussion < [email protected]> Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Errors-To: [email protected] Sender: "cippnews" <[email protected]> Here is the section from the error log (same issue, I think): May 10 12:54:51 2017 (517) SHUNTING: 1494446090.4214+cdb2b1813d6f36d682d0f74cfc581e6046735197 May 10 12:58:26 2017 (517) Uncaught runner exception: 'ascii' codec can't decode byte 0xb5 in position 26: ordinal not in range(128) May 10 12:58:26 2017 (517) Traceback (most recent call last): File "/usr/local/mailman/Mailman/Queue/Runner.py", line 119, in _oneloop self._onefile(msg, msgdata) File "/usr/local/mailman/Mailman/Queue/Runner.py", line 190, in _onefile keepqueued = self._dispose(mlist, msg, msgdata) File "/usr/local/mailman/Mailman/Queue/ArchRunner.py", line 77, in _dispose mlist.ArchiveMail(msg) File "/usr/local/mailman/Mailman/Archiver/Archiver.py", line 216, in ArchiveMail h.processUnixMailbox(f) File "/usr/local/mailman/Mailman/Archiver/pipermail.py", line 596, in processUnixMailbox self.add_article(a) File "/usr/local/mailman/Mailman/Archiver/pipermail.py", line 640, in add_article author = fixAuthor(article.decoded['author']) File "/usr/local/mailman/Mailman/Archiver/pipermail.py", line 63, in fixAuthor while i>0 and (L[i-1][0] in lowercase or UnicodeDecodeError: 'ascii' codec can't decode byte 0xb5 in position 26: ordinal not in range(128) Here is the output from the dumpdb command (anonymized a little): [----- start pickle file -----] <----- start object 1 -----> >From [email protected] Wed May 10 12:54:50 2017 Received: from pv35p22im-ztdg05131101.me.com ([17.133.189.43]) by cool.conservation-us.org with esmtp (Exim 4.69) (envelope-from <[email protected]>) id 1d8XhC-0000aM-7S for [email protected]; Wed, 10 May 2017 12:54:50 -0700 Received: from process-dkim-sign-daemon.pv35p22im-ztdg05131101.me.com by pv35p22im-ztdg05131101.me.com (Oracle Communications Messaging Server 7.0.5.38.0 64bit (built Feb 26 2016)) id <[email protected]> for [email protected]; Wed, 10 May 2017 19:54:49 +0000 (GMT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=me.com; s=4d515a; t=1494446089; bh=iMRS0rztCQMUAhGFAT7khfPTdH9Fy/59g3wJDcQzaxM=; h=From:Content-type:MIME-version:Subject:Date:To:Message-id; b=P994EyGUDRT5hcWBUpFuaXhGcIwIv5y+2Fc2iyA9I1vQsqyq4rg57CptHkrr/t3A7 yf4tU+IIbYu0Jq/F9xtVnPT3u/B5s2MfZvIORNOvKxxx1bm0BwXpGqwwLTAE73gfxO y8akmoAfaoUJlbQ+Wptu8Y7uNLPBaMyuUoomX8Bcm7UiNW9lWBLtIgCAR0pAJ0StqP Fm1N9Rqn4omlJ5z7uQ1V/wdMgO5QRAY0NT2punAB5sGs3eEJJavfWZsiGDbNBIq56v 9mYCxgyId1spAvb4HgVbKNd28V7C3y5OreWQsPdTp0HS218dODJQekkAv6dTd5RxSc FMEP1hpTefwhg== Received: from icloud.com ([127.0.0.1]) by pv35p22im-ztdg05131101.me.com (Oracle Communications Messaging Server 7.0.5.38.0 64bit (built Feb 26 2016)) with ESMTPSA id <[email protected]> for [email protected]; Wed, 10 May 2017 19:54:46 +0000 (GMT) X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:,, definitions=2017-05-10_16:,, signatures=0 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0 clxscore=1034 suspectscore=0 malwarescore=0 phishscore=0 adultscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1701120000 definitions=main-1705100136 From: XXXXXXXXXX <[email protected]> Content-type: text/plain; charset=utf-8 Content-transfer-encoding: quoted-printable MIME-version: 1.0 (Mac OS X Mail 10.3 \(3273\)) Date: Wed, 10 May 2017 16:54:44 -0300 References: <CACJwyVrjx-m1VZ=s68r5cv-dihv3vchfxq2a6qhwddk5jzd...@mail.gmail.com> To: Conservators in Private Practice Discussion <[email protected]> In-reply-to: <CACJwyVrjx-m1VZ=s68r5cv-dihv3vchfxq2a6qhwddk5jzd...@mail.gmail.com> Message-id: <[email protected]> X-Mailer: Apple Mail (2.3273) Subject: Re: [CIPPNEWS] closing private conservation studio X-BeenThere: [email protected] X-Mailman-Version: 2.1.23 Precedence: list List-Id: Conservators in Private Practice Discussion <cippnews.cool.conservation-us.org> List-Unsubscribe: <http://cool.conservation-us.org/mailman/options/cippnews>, <mailto:[email protected]?subject=unsubscribe> List-Archive: <http://cool.conservation-us.org/mailman/private/cippnews/> List-Post: <mailto:[email protected]> List-Help: <mailto:[email protected]?subject=help> List-Subscribe: <http://cool.conservation-us.org/mailman/listinfo/cippnews>, <mailto:[email protected]?subject=subscribe> X-List-Received-Date: Wed, 10 May 2017 19:54:50 -0000 Send me list of what=E2=80=99s still available! love XXX <----- start object 2 -----> { '_parsemsg': False, 'add_header': { 'Reply-To': 'Conservators in Private Practice Discussion <[email protected]>'}, 'from_is_list': 0, 'lang': 'en', 'listname': 'cippnews', 'original_sender': '[email protected]', 'origsubj': 'Re: [CIPPNEWS] closing private conservation studio', 'pipeline': [ 'ToUsenet', 'AfterDelivery', 'Acknowledge', 'WrapMessage', 'ToOutgoing'], 'received_time': 1494446090.4214, 'recips': [ hundreds of recipients follow ], 'stripped_subject': <email.header.Header instance at 0x1ed6f80>, 'tolist': 1, 'version': 3, 'whichq': '/usr/local/mailman/qfiles/archive'} [----- end pickle file -----] > The issue appears to be a bug in Mailman. but I need more info to be > sure including the headers from one of the shunted messages. > > The exception is thrown when the archiver is trying to canonicalize the > author's display name into "Last, First" form and I think the actual > exception in this case is in checking if a "word" of the name begins > with a lowercase letter. Since the word is already unidode in this case, > it is apparently Python's string.lowercase that contains \xb5 in > position 26. > > What do you get if you invoke Python interactively on this server and do > > import string > string.lowercase > > I get 'abcdefghijklmnopqrstuvwxyz' > I get the same thing! Does that make any sense? xxxx@yyyyy:/usr/local/mailman/logs# python Python 2.7.5 (default, May 29 2013, 02:28:51) [GCC 4.8.0] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import string >>> string.lowercase 'abcdefghijklmnopqrstuvwxyz' Thanks as always, Matt ------------------------------------------------------ Mailman-Users mailing list [email protected] https://mail.python.org/mailman/listinfo/mailman-users Mailman FAQ: http://wiki.list.org/x/AgA3 Security Policy: http://wiki.list.org/x/QIA9 Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/ Unsubscribe: https://mail.python.org/mailman/options/mailman-users/archive%40jab.org
