On approximately 11/5/2008 12:38 PM, came the following characters from
the keyboard of Barry Warsaw:
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
On Oct 30, 2008, at 6:17 PM, Andrew McNamara wrote:
That's a tricker case, but I think it should use bytes internally. One of
the early goals of email was that be able to cope with malformed MIME -
this includes incorrectly encoded messages. So I think it must keep a
bytes representation internally.
However - charset encoding is part of the MIME spec, so users have a
reasonable expectation that the mime lib will present them with unicode.
So the API needs to be unicode.
The latter doesn't though, and it needs a lot of work (we tried and
failed
at pycon).
Yes, it's hard. I think we're going to have to break the API.
I did make a start on a new API for email to work better with bytes and
unicode. I didn't get that far before other work intruded. My current
thinking is that you need separate APIs where appropriate to access
email content as unicodes (or decoded data in general). For example,
normally headers and their values would be bytes, but there would be an
API to retrieve the decoded values as unicodes.
Similarly, where get_payload() now takes a 'decoded' option, there would
be a separate API for retrieving the decoded payload. This is a bit
trickier because depending on the content-type, you might want a
unicode, or an image, or a sound file, etc.
Another tricky issue is how to set these things. We have to get in the
habit of writing
message[b'Subject'] = b'Hello'
but that's really gross, and of course email_from_string() would have to
become email_from_bytes(). Maybe the API accepts unicode strings but
only if they are ASCII?
There are lots of other problems with the email package, and while it's
made my life much better on the whole, it is definitely in need of
improvement. Unfortunately, I don't see myself having much time to
attack it in the near future. Maybe we can make it a Pycon sprint
(instead of spending all that time on the bzr experiment ;), or, if
someone else wants to lead the dirty work, I would definitely pitch in
with my thoughts on API and implementation.
I would find
message[b'Subject'] = b'Hello'
to be totally gross.
While RFC Email is all ASCII, except if 8bit transfer is legal, there
are internal encoding provided that permit the expression of Unicode in
nearly any component of the email, except for header identifiers. But
there are never Unicode characters in the transfer, as they always get
encoded (there can be UTF-8 byte sequences, of course, if 8bit transfer
is legal; if it is not, then even UTF-8 byte sequences must be further
encoded).
Depending on the level of email interface, there should be no interface
that cannot be expressed in terms of Unicode, plus an encoding to use
for the associated data. Even 8-bit binary can be translated into a
sequence of Unicode codepoints with the same numeric value, for example.
That isn't particularly, efficient, though, so providing a few
interfaces that accept binary blobs to encode in various ways would be
handy. Of course binary data should allow specification of an
associated encoding also.
I haven't looked at the details of the Python libraries yet, but it is a
subject I eventually want to get familiar with, as I've written Perl
scripts to read and write email, and have tweaked a couple open source
email clients a bit. The Python POP, IMAP, SMTP and NNTP sound like
they raise the level of abstraction a bit, and should make it even
easier to read and write email.
So many projects, so many ideas, but limited time :( Helping with this
would be something I would really enjoy, but I'm significantly
backlogged at present.
Maybe I should outline what would be nice to see, before delving into
the interfaces. This could be helpful if you invent a new interface;
maybe some of the ideas would help avoid designs that require the above
grossness. Alternately, perhaps viewing these comments as an extremely
high-level set of expectations could help view the existing interfaces
in a way that can achieve these goals, without major rework, even if
there are few warts.
I'll speak in terms of creating and sending a message, but receiving
should be similar, and simpler (because encoding choices were already
made, and only need to be decoded).
It would be nice to specify, at "message creation" time, the preferred
types of encodings that should be used, and then not have to think about
the encodings any more, and just provide Unicode at the interfaces (or
binary for certain blobs).
It is not clear that the message should be encoded on the fly, but
rather after negotiation with the server, after determining if 8bit
transfer is legal.
Once the message is complete, it should be retrievable as a blob
(perhaps pickle is appropriate, or just any of the possible email
bytestreams that could be sent) that can be re-instantiated later.
Once the message is sendable, the actual binary bytestream sent should
be available for retrieval. This could be used as the "sent" log, if
one is desired (usually is for email clients). The saved bytestream
should be able to be used to re-instantiate an equivalent message object
later.
--
Glenn -- http://nevcal.com/
===========================
A protocol is complete when there is nothing left to remove.
-- Stuart Cheshire, Apple Computer, regarding Zero Configuration Networking
_______________________________________________
Python-3000 mailing list
Python-3000@python.org
http://mail.python.org/mailman/listinfo/python-3000
Unsubscribe:
http://mail.python.org/mailman/options/python-3000/archive%40mail-archive.com
