Re: [Mailman-Users] Messages shunted with "TypeError: decoding Unicode is not supported"

[Josh Clark]

Thanks again, Mark, this is all *very* appreciated.

I'm interested in how many of these addresses are u'...', and how  
they got to be that way.



Of the 1639 e-mail addresses, 349 are u'...' I don't have any great  
ideas about what might be out of the ordinary about my installation.  
I've never made any mods to the Mailman scripts. I *have* used custom  
html templates (charset=utf-8), but I don't think that this would  
affect the character encoding on the server side, would it? If you'd  
like to take a look:



However, the majority of subscriptions come via a script that I set up  
to send subscribe commands via email to the list. It accepts an email  
address in a web form and sends it along as a subscribe command to  
moxiemail-requests. Here's a sample (with domains changed to  
example.com):


--
Return-Path: <[EMAIL PROTECTED]>
X-Original-To: [EMAIL PROTECTED]
Delivered-To: [EMAIL PROTECTED]
Received: by houdini.example.com (Postfix, from userid 536)
id 26E3A312869A; Tue, 28 Oct 2008 14:38:07 -0500 (CDT)
From: [EMAIL PROTECTED]
To: [EMAIL PROTECTED]
Subject: subscribe [EMAIL PROTECTED]
Mime-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Type: text/plain; charset=US-ASCII; format=flowed
Message-Id: <[EMAIL PROTECTED]>
Date: Tue, 28 Oct 2008 14:38:07 -0500 (CDT)

subscribe [EMAIL PROTECTED]
--

I'm happy to do any tests or troubleshooting that might be helpful to  
get at the root problem.


The attached patch to Decorate.py (applied after removing the other  
one) will work around the problem.


Excellent, thanks! A quick question before I try it: I've now tried to  
send this message a few times; is it possible that these attempts were  
successfully mailed to some portion of the list (e.g., some of the non- 
unicode addresses)? If it's an indicator, the last address that showed  
up with the "bin/withlist" commands that you suggested was also the  
same address that showed up in the message footer for the debug patch  
that you originally sent (and, I presume, the address that caused the  
error).


(If it's likely that some significant number of people have gotten  
several copies of the message, I just want to add an apology for the  
repeats before mailing it again. Either way, I'll give the patch a  
spin in the next 24 hours.)


Thanks again for all of the speedy, detailed help with this.

All best,
Josh
--
Mailman-Users mailing list
Mailman-Users@python.org
http://mail.python.org/mailman/listinfo/mailman-users
Mailman FAQ: http://wiki.list.org/x/AgA3
Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/
Unsubscribe: 
http://mail.python.org/mailman/options/mailman-users/archive%40jab.org

Security Policy: http://wiki.list.org/x/QIA9

Re: [Mailman-Users] Messages shunted with "TypeError: decoding Unicode is not supported"

[Mark Sapiro]

Josh Clark wrote:

>> Try applying the attached Decorate.patch.txt patch to
>> Mailman/Handlers/Decorate.py and restart mailman.
>>
>> Then try the post and report what you find in Mailman's 'debug' log.
>
>Thanks again, Mark. Here's what shows up in the debug log (email  
>address anonymized to protect the innocent; it was an unremarkable  
>yahoo.com account).
>
>$ cat debug
>Oct 28 05:14:39 2008 (22325) msg_footer type: 
>footer type: 
>footer: ___
>You are subscribed to this mailing list as [EMAIL PROTECTED]
>To edit your subscription options or to unsubscribe, visit:
>http://mailman.example.com/mailman/options/moxiemail/address%40example.com
>
>lcset: us-ascii


That narrows it down a bit. What is happening is one of the strings
being interpolated into the footer is unicode. The footer is

msg_footer = """___
You are subscribed to this mailing list as %(user_address)s
To edit your subscription options or to unsubscribe, visit:
%(user_optionsurl)s"""

and somehow this '[EMAIL PROTECTED]' is unicode rather than a string.

Try the following:

Remove the patch from Decorate.py as it isn't needed any more. Then do

$ bin/withlist moxiemail
Loading list moxiemail (unlocked)
The variable `m' is the moxiemail MailList instance
>>> for mem in m.getMembers():
... print repr(m.getMemberCPAddress(mem))
...

The first line above represents a prompt and a withlist command. The
next lines are output from withlist ending with a '>>> ' python
prompt. You type what follows the prompt and get a '... ' prompt
because the 'for' is incomplete. You type the next line indented, and
then just  to the next prompt. This should be followed by a list of

'[EMAIL PROTECTED]' and/or u'[EMAIL PROTECTED]' like addresses
ending with a '>>> ' prompt. Type control-D at this prompt to quit.

I'm interested in how many of these addresses are u'...', and how they
got to be that way. The above will answer the first question.

The attached patch to Decorate.py (applied after removing the other
one) will work around the problem, but the real question is how the
'case preserved' addresses got to be unicode in the first place.

-- 
Mark Sapiro <[EMAIL PROTECTED]>The highway is for gamblers,
San Francisco Bay Area, Californiabetter use your sense - B. Dylan

--- f:/test-mailman-2.1/Mailman/Handlers/Decorate.py2008-06-11 
15:09:34.0 -0700
+++ f:/test-mailman/Mailman/Handlers/Decorate.py2008-10-28 
08:53:30.15625 -0700
@@ -98,8 +98,18 @@
 # TK: Try to keep the message plain by converting the header/
 # footer/oldpayload into unicode and encode with mcset/lcset.
 # Try to decode qp/base64 also.
-uheader = unicode(header, lcset, 'ignore')
-ufooter = unicode(footer, lcset, 'ignore')
+if isinstance(header, str):
+uheader = unicode(header, lcset, 'ignore')
+elif isinstance(header, unicode):
+uheader = header
+else:
+assert False, 'Adding msg_header: header not string or unicode'
+if isinstance(footer, str):
+ufooter = unicode(footer, lcset, 'ignore')
+elif isinstance(footer, unicode):
+ufooter = footer
+else:
+assert False, 'Adding msg_footer: footer not string or unicode'
 try:
 oldpayload = unicode(msg.get_payload(decode=True), mcset)
 frontsep = endsep = u''
--
Mailman-Users mailing list
Mailman-Users@python.org
http://mail.python.org/mailman/listinfo/mailman-users
Mailman FAQ: http://wiki.list.org/x/AgA3
Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/
Unsubscribe: 
http://mail.python.org/mailman/options/mailman-users/archive%40jab.org

Security Policy: http://wiki.list.org/x/QIA9

[Mailman-Users] Messages shunted with "TypeError: decoding Unicode is not supported"

[Josh Clark]

Hi,

I'm having trouble sending a message from one of my lists using  
Mailman 2.1.11. The message itself is entirely ascii and sent with the  
US-ASCII charset, but it's failing with a "unicode is not supported"  
error. My (extremely unsophisticated) reading of the error log leads  
me to believe that the problem ise happening when the footer is  
applied to the message, but the footer text is also ascii.


Any ideas or suggestions would be much appreciated!

From the error log:

Oct 27 05:15:16 2008 (2734) Uncaught runner exception: decoding  
Unicode is not supported

Oct 27 05:15:16 2008 (2734) Traceback (most recent call last):
  File "/usr/local/mailman/Mailman/Queue/Runner.py", line 120, in  
_oneloop

self._onefile(msg, msgdata)
  File "/usr/local/mailman/Mailman/Queue/Runner.py", line 191, in  
_onefile

keepqueued = self._dispose(mlist, msg, msgdata)
  File "/usr/local/mailman/Mailman/Queue/OutgoingRunner.py", line 74,  
in _dispose

self._func(mlist, msg, msgdata)
  File "/usr/local/mailman/Mailman/Handlers/SMTPDirect.py", line 159,  
in process

deliveryfunc(mlist, msg, msgdata, envsender, refused, conn)
  File "/usr/local/mailman/Mailman/Handlers/SMTPDirect.py", line 292,  
in verpdeliver

Decorate.process(mlist, msgcopy, msgdata)
  File "/usr/local/mailman/Mailman/Handlers/Decorate.py", line 102,  
in process

ufooter = unicode(footer, lcset, 'ignore')
TypeError: decoding Unicode is not supported

Oct 27 05:15:16 2008 (2734) SHUNTING:  
1225102514.5674801+43301f942e02e78abe431e7148223e1c0165fe58



Many thanks for your help. All best,
Josh


--
Mailman-Users mailing list
Mailman-Users@python.org
http://mail.python.org/mailman/listinfo/mailman-users
Mailman FAQ: http://wiki.list.org/x/AgA3
Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/
Unsubscribe: 
http://mail.python.org/mailman/options/mailman-users/archive%40jab.org

Security Policy: http://wiki.list.org/x/QIA9