[issue8898] The email package should defer to the codecs module for all aliases

[STINNER Victor]

STINNER Victor  added the comment:

This issue is not newcomer friendly, I remove the easy keyword.

--
keywords:  -easy
nosy: +vstinner

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue8898] The email package should defer to the codecs module for all aliases

[R. David Murray]

Changes by R. David Murray rdmur...@bitdance.com:


--
assignee: r.david.murray - 
components: +email -Library (Lib)
nosy: +barry

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8898
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue8898] The email package should defer to the codecs module for all aliases

[Michele Orrù]

Michele Orrù maker...@gmail.com added the comment:

Any idea about how to unittest mime.aliases?

Also, since I've just created a new file, are there some buracratic issues? I 
mean, do I have to add something at the top of the file?
(I'm just signing the Contributor Agreement)

--
Added file: http://bugs.python.org/file22146/issue8898_2.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8898
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue8898] The email package should defer to the codecs module for all aliases

[Marc-Andre Lemburg]

Marc-Andre Lemburg m...@egenix.com added the comment:

Michele Orrù wrote:
 
 Michele Orrù maker...@gmail.com added the comment:
 
 Any idea about how to unittest mime.aliases?

Test the APIs you probably created for accessing it.

 Also, since I've just created a new file, are there some buracratic issues? I 
 mean, do I have to add something at the top of the file?
 (I'm just signing the Contributor Agreement)

You just need to put the usual copyright line at the top of
the file, together with the sentence from the agreement.

Apart from that, you also need to make sure that the other build
setups include the new file (PCbuild, Makefile.pre.in, etc.). If you
don't know how to do this, you can ask someone else to take
care of this, since it usually requires domain knowledge (e.g.
to add the file to the Windows builds).

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8898
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue8898] The email package should defer to the codecs module for all aliases

[R. David Murray]

R. David Murray rdmur...@bitdance.com added the comment:

Your new file isn't in the patch.  I'm imagining it is a table and a couple 
methods, so I think perhaps putting it either in charset or in utils would be 
better than creating a new file.

As for testing it, what I'd love to see is a test that downloads the current 
IANA table (there are routines in test.support for doing this in a way that 
respects the test suite's 'resources' settings), pulls out the preferred MIME 
aliases, and makes sure that all of them are mapped to some canonical Python 
codec.  Then you can invert that and make sure all of the results returned by 
that test map back to the correct MIME alias.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8898
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue8898] The email package should defer to the codecs module for all aliases

[R. David Murray]

R. David Murray rdmur...@bitdance.com added the comment:

Prompted on IRC, I see I missed the file because it was so short.

This still isn't what I'm looking for.  We are assuming that email is going to 
use the codec eventually so that it is not a bad thing to have charset 
pre-populate the codec cache.  So what I'm looking for is:

try:
python_name = codecs.lookup(input_charset).name
mime_name = ALIASES.get(python_name, input_charset)
except LookupError:
mime_name = input_charset

MAL's idea was to implement the ALIASES step via a two-way mapping in the 
encodings module (python-canonical-name = MIME-preferred-name).  That would 
be fine, too, but the email.charset logic should look like the above however 
the table is implemented.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8898
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue8898] The email package should defer to the codecs module for all aliases

[R. David Murray]

R. David Murray rdmur...@bitdance.com added the comment:

The second line in that try: block should have been:

  mime_name = ALIASES.get(python_name, python_name)

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8898
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue8898] The email package should defer to the codecs module for all aliases

[Michele Orrù]

Changes by Michele Orrù maker...@gmail.com:


Added file: http://bugs.python.org/file22153/issue8898_3.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8898
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue8898] The email package should defer to the codecs module for all aliases

[Éric Araujo]

Éric Araujo mer...@netwok.org added the comment:

 email (silently) failed to encode a string

Is this silent error another bug to fix?

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8898
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue8898] The email package should defer to the codecs module for all aliases

[R. David Murray]

R. David Murray rdmur...@bitdance.com added the comment:

Not in email5.  The RFC says that if the charset parameter isn't known you just 
pass it through.  In email6 we will be making a more careful distinction 
between errors that should be passed silently per the RFC, and ones that should 
be noisy because the API in question is being used to create the message 
ab-initio.  (In email5 the exact same machinery is used to create a message 
from parsed source as is used to create a message programatically, resulting in 
the silent passing of certain errors that should really be noisy.)

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8898
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue8898] The email package should defer to the codecs module for all aliases

[R. David Murray]

R. David Murray rdmur...@bitdance.com added the comment:

What is not-a-charset?

I apparently misunderstood what normalize_encodings does.  It isn't doing a 
lookup in the codecs registry and returning the canonical name for the codec.  
Does that mean we actually have to fetch the codec in order to get the 
canonical name?  I suspect so, and that is probably OK, since in most cases the 
codec is eventually going to get called while processing the email that 
triggered the ALIASES lookup.

I also notice that there is a table of aliases in the codec module 
documentation, so that will need to be updated as well.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8898
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue8898] The email package should defer to the codecs module for all aliases

[Marc-Andre Lemburg]

Marc-Andre Lemburg m...@egenix.com added the comment:

R. David Murray wrote:
 
 R. David Murray rdmur...@bitdance.com added the comment:
 
 What is not-a-charset?

 I apparently misunderstood what normalize_encodings does.  It isn't doing a 
 lookup in the codecs registry and returning the canonical name for the codec. 
  Does that mean we actually have to fetch the codec in order to get the 
 canonical name?  I suspect so, and that is probably OK, since in most cases 
 the codec is eventually going to get called while processing the email that 
 triggered the ALIASES lookup.
 
 I also notice that there is a table of aliases in the codec module 
 documentation, so that will need to be updated as well.

As far as the aliases.py part of the patch goes, I'm fine with that
since it corrects a few real bugs and adds the missing Latin-N
codec names.

Regarding using this table in the email package, I'm not really
clear on what you want to achieve.

If you are looking for a way to determine whether Python has a codec
installed for a certain charset name, then codecs.lookup() will
tell you this (and it also applies all the aliasing and normalization
needed).

If you want to avoid the actual codec module import (codecs.lookup()
imports the module), you can mimic the logic used by the lookup function
of the encodings package. Not sure, whether that's worth it, though,
since it is rather likely that you're going to use the codec you've
just looked up soon after the test and codecs.lookup() caches the
found codecs.

If you want to convert an arbitrary encoding name to a registered
standard IANA MIME charset name, then the aliases.py module is not
going to be of much help, since we are using our own canonical
names which do not necessarily map to the MIME charset names.

You'd have to add a new mime_alias map to the email package
for that. I'd suggest to use the same approach as for the
aliases.py module, which is to first normalize the encoding
name using normalize_encoding() and then running that through
the mime_alias map.

Hope that helps.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8898
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue8898] The email package should defer to the codecs module for all aliases

[Michele Orrù]

Michele Orrù maker...@gmail.com added the comment:

+1

What do you think? Ezio, David?

--
title: The email package should defer to the codecs module for  all aliases - 
The email package should defer to the codecs module for all aliases

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8898
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue8898] The email package should defer to the codecs module for all aliases

[R. David Murray]

R. David Murray rdmur...@bitdance.com added the comment:

Well, my thought was to avoid having multiple charset alias lists in the 
stdlib, and reusing the one in codecs, which is larger than the one in email, 
seemed to make sense.  This came up because a bug was reported where email 
(silently) failed to encode a string because the charset alias, while present 
in codecs, wasn't present in the email ALIASES table.

I suppose that as an alternative I could add full support for the IANA aliases 
list to email.  Email is the most likely place to run in to variant charset 
aliases anyway.

If that's the way we go, then this issue should be changed over to covering 
just updating codecs with the missing aliases, and a new issue opened for 
adding full IANA alias support to email.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8898
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue8898] The email package should defer to the codecs module for all aliases

[Michele Orrù]

Michele Orrù maker...@gmail.com added the comment:

In that case, I could still take care of it; it would be really easy to do.

So, it's up to you to tell me what is the best design choice. (:

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8898
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue8898] The email package should defer to the codecs module for all aliases

[Marc-Andre Lemburg]

Marc-Andre Lemburg m...@egenix.com added the comment:

R. David Murray wrote:
 
 R. David Murray rdmur...@bitdance.com added the comment:
 
 Well, my thought was to avoid having multiple charset alias lists in the 
 stdlib, and reusing the one in codecs, which is larger than the one in email, 
 seemed to make sense.  This came up because a bug was reported where email 
 (silently) failed to encode a string because the charset alias, while present 
 in codecs, wasn't present in the email ALIASES table.
 
 I suppose that as an alternative I could add full support for the IANA 
 aliases list to email.  Email is the most likely place to run in to variant 
 charset aliases anyway.
 
 If that's the way we go, then this issue should be changed over to covering 
 just updating codecs with the missing aliases, and a new issue opened for 
 adding full IANA alias support to email.

I think it would be useful to have a mapping from the Python
canoncial name (the one the encodings package uses) to the
preferred MIME name as referenced in the IANA list:

http://www.iana.org/assignments/character-sets

This mapping could also be added to the encodings package
together with a function that translates a given encoding
name to its canoncial Python name (codec_module_name())
and another one to translate it to the preferred MIME name
according to the above list (encoding_mime_name()).

Note that we don't support all the aliases mentioned in the IANA
list because many of the are outdated and some have proved to be
wrong (the aliased encodings are actually different in a few
places). There are also a few encodings in the list which we
don't support at all.

Since we only rarely get requests for supporting new aliases or
encodings, I think it's safe to say that the existing set
is fairly complete from a practical point of view.

--
title: The email package should defer to the codecs module for all aliases - 
The email package should defer to the codecs module for   all aliases

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8898
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue8898] The email package should defer to the codecs module for all aliases

[R. David Murray]

R. David Murray rdmur...@bitdance.com added the comment:

I agree that since we get very few requests to add aliases our current tables 
are probably what we want.  So adding the MIME_preferred_name mapping 
*somewhere* is indeed what I would like to see happen.  It doesn't matter to me 
whether it is in the codecs module or the email module.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8898
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue8898] The email package should defer to the codecs module for all aliases

[Michele Orrù]

Michele Orrù maker...@gmail.com added the comment:

After discussing on IRC, it figured out that the best choice would be to use 
normalize_encoding plus ALIAS, as the attached patch does.

--
Added file: http://bugs.python.org/file22094/issue8898_normalize.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8898
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue8898] The email package should defer to the codecs module for all aliases

[Marc-Andre Lemburg]

Marc-Andre Lemburg m...@egenix.com added the comment:

R. David Murray wrote:
 
 R. David Murray rdmur...@bitdance.com added the comment:
 
 euc_jp and euc_kr seem to be backward (that is, codecs translates them to the 
 _ version, instead of translating the _ version to the - version).  I worry 
 that there might be other deviations from the standard email names.  I would 
 suggest we pull the list of preferred MIME names from the IANA charset 
 registry and make a test out of them in the email package.  If changing the 
 name returned by codecs is determined to not be acceptable, then those 
 entries will need to remain in the charset module ALIASES table and the 
 codecs-check logic adjusted accordingly.
 
 Unfortunately the IANA registry does not list MIME names for all of the 
 charsets in common use, and the canonical names are not always the ones 
 commonly used in email.  Hopefully the codecs registry is using the most 
 common name for those, and hopefully if there are differences it won't break 
 any user code, since any reasonable email code should be coping with the 
 aliases in any case.

The way I understand the patch was that the email package will
start to use the encoding aliases for determining the codec
name instead of its own list. That is: only for decoding the
input data, not for creating a correct MIME encoding name in
output data.

--
title: The email package should defer to the codecs module for all aliases - 
The email package should defer to the codecs module for   all aliases

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8898
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue8898] The email package should defer to the codecs module for all aliases

[R. David Murray]

R. David Murray rdmur...@bitdance.com added the comment:

Well, it turns out that back when I opened this issue I misunderstood what the 
ALIASES table was used for.  it *is* used before doing a codecs lookup, but it 
is also used to convert whatever charset name the programmer specifies into the 
standard MIME name for the codec when generating emails.

Clearly the email module needs to base its transformation on the IANA table.  I 
think the ideal would be to have a program that pulls the IANA table and 
generates the ALIASES table.  On the other hand, codecs should already have all 
of those aliases (this theoretical program could be used to ensure that), so 
another alternative is to use codecs to look up the python canonical name for 
the charset, and have the email ALIASES table just map the ones where that 
isn't the preferred MIME name into the MIME name.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8898
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue8898] The email package should defer to the codecs module for all aliases

[Michele Orrù]

Michele Orrù maker...@gmail.com added the comment:

Well, actually encodings.aliases links to the encoding _module name_, as
described in the doc:
 Encoding Aliases Support
This module is used by the encodings package search function to
map encodings names to module names.

So I've adjusted your snippet according to this, as you can see in the
attachment.

I've also slightly changed the imports as pep8 says:

Yes: import os
import sys

No: import sys, os


Anyway, running the test failed for two encodings, there are two bugs there,
indeed.
- mcbs has something broken in its imports;
- tactis module is not present.

Since they are really easy to fix, I haven't yet reported to the bugtraker.
Let me know what should I do.
Post on bugs.python.org bug and patch? Any new test specifically for the
email module?

--
Added file: http://bugs.python.org/file22057/unnamed

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8898
___brbrdiv class=gmail_quoteblockquote class=gmail_quote 
style=margin:0 0 0 .8ex;border-left:1px #ccc 
solid;padding-left:1exbr Regarding the tests, I don#39;t see tests for 
the aliases anywhere, so something like:br



for alias, codec_name in encodings.aliases.items():br
    self.assertEqual(codecs.lookup(alias).name, codec_name)br
could be added somewhere to check that all the aliases in the dict map to the 
correct codec.br/blockquoteWell, actually encodings.aliases links to the 
encoding _module name_, as described in the doc:brquot;quot;quot; Encoding 
Aliases Supportbr

    This module is used by the encodings package search function tobr    
map encodings names to module names.brquot;quot;quot;brSo I#39;ve 
adjusted your snippet according to this, as you can see in the attachment./div

div class=gmail_quotebrI#39;ve also slightly changed the imports as pep8 
says:brquot;quot;quot;brYes: import osbr import 
sysbrbrNo:  import sys, osdivquot;quot;quot;/div

divbr/divdivAnyway, running the test failed for two encodings, there 
are two bugs there, indeed./divdiv- mcbs has something broken in its 
imports;/divdiv- tactis module is not present./divdivbr/div

divSince they are really easy to fix, I haven#39;t yet reported to the 
bugtraker. Let me know what should I do./divdivPost on a 
href=http://bugs.python.org;bugs.python.org/a bug and patch? Any new test 
specifically for the email module?/div

divbr/div/div-- brMichele Orrùbr
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue8898] The email package should defer to the codecs module for all aliases

[Michele Orrù]

Changes by Michele Orrù maker...@gmail.com:


Added file: http://bugs.python.org/file22058/fail_tactis.txt

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8898
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue8898] The email package should defer to the codecs module for all aliases

[Michele Orrù]

Changes by Michele Orrù maker...@gmail.com:


Added file: http://bugs.python.org/file22059/issue8898_withtests.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8898
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue8898] The email package should defer to the codecs module for all aliases

[Michele Orrù]

Changes by Michele Orrù maker...@gmail.com:


Added file: http://bugs.python.org/file22060/fail_mcbs.txt

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8898
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue8898] The email package should defer to the codecs module for all aliases

[Marc-Andre Lemburg]

Marc-Andre Lemburg m...@egenix.com added the comment:

Michele Orrù wrote:
 
 Michele Orrù maker...@gmail.com added the comment:
 
 Well, actually encodings.aliases links to the encoding _module name_, as
 described in the doc:
  Encoding Aliases Support
 This module is used by the encodings package search function to
 map encodings names to module names.
 
 So I've adjusted your snippet according to this, as you can see in the
 attachment.
 
 I've also slightly changed the imports as pep8 says:
 
 Yes: import os
 import sys
 
 No: import sys, os
 
 
 Anyway, running the test failed for two encodings, there are two bugs there,
 indeed.
 - mcbs has something broken in its imports;

mbcs is only available on Windows.

 - tactis module is not present.

I'm not sure what happened here: either the alias entry is wrong
or the codec module was not committed.

In either case, no one has complained about this encoding not working,
so we can probably just remove it from the alias table. See
http://bugs.python.org/issue1251921 for a similar report and
discussion.

--
title: The email package should defer to the codecs module for all aliases - 
The email package should defer to the codecs module for   all aliases

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8898
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue8898] The email package should defer to the codecs module for all aliases

[Michele Orrù]

Michele Orrù maker...@gmail.com added the comment:

So, what do you prefer? Add a check for sys.platform, or just skip it?

discussion on python-dev. So I'm +1 for just skipping it for now (with a XXX
comment on the right maybe).

--
title: The email package should defer to the codecs module for  all aliases - 
The email package should defer to the codecs module for all aliases
Added file: http://bugs.python.org/file22064/unnamed

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8898
___div class=gmail_quoteblockquote class=gmail_quote style=margin:0 0 0 
.8ex;border-left:1px #ccc solid;padding-left:1ex;div class=imbr
gt; - mcbs has something broken in its imports;br
br
/divmbcs is only available on Windows.br/blockquotedivSo, what do you 
prefer? Add a check for sys.platform, or just skip it?/divblockquote 
class=gmail_quote style=margin:0 0 0 .8ex;border-left:1px #ccc 
solid;padding-left:1ex;


div class=imbr
gt; - tactis module is not present.br
br
/divI#39;m not sure what happened here: either the alias entry is wrongbr
or the codec module was not committed.br
br
In either case, no one has complained about this encoding not working,br
so we can probably just remove it from the alias table. Seebr
a href=http://bugs.python.org/issue1251921; 
target=_blankhttp://bugs.python.org/issue1251921/a for a similar report 
andbr
discussion.br
br/blockquotedivI don#39;t have such autority, and probably such a 
choice will require a discussion on python-dev. So I#39;m +1 for just skipping 
it for now (with a XXX comment on the right maybe)./divdivbr

/divdivbr/div/div-- brMichele Orrùbr
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue8898] The email package should defer to the codecs module for all aliases

[Michele Orrù]

Michele Orrù maker...@gmail.com added the comment:

Sorry, I was told that email the bugtracker could not work properly.


  - mcbs has something broken in its imports;

 mbcs is only available on Windows.
So, what do you prefer? Add a check for sys.platform, or just skip it?

  - tactis module is not present.

 I'm not sure what happened here: either the alias entry is wrong
 or the codec module was not committed.

 In either case, no one has complained about this encoding not working,
 so we can probably just remove it from the alias table. See
 http://bugs.python.org/issue1251921 for a similar report and
 discussion.

I don't have such autority, and probably such a choice will require a 
discussion on python-dev. So I'm +1 for just skipping it for now (with a XXX 
comment on the right maybe).

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8898
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue8898] The email package should defer to the codecs module for all aliases

[Michele Orrù]

Changes by Michele Orrù maker...@gmail.com:


Removed file: http://bugs.python.org/file22057/unnamed

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8898
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue8898] The email package should defer to the codecs module for all aliases

[Marc-Andre Lemburg]

Marc-Andre Lemburg m...@egenix.com added the comment:

Michele Orrù wrote:
 
 Michele Orrù maker...@gmail.com added the comment:
 
 Sorry, I was told that email the bugtracker could not work properly.
 
 
 - mcbs has something broken in its imports;
 
 mbcs is only available on Windows.

 So, what do you prefer? Add a check for sys.platform, or just skip it?

The test suite provides ways to implement known failures on
specific platforms, so I'd suggest to use those mechanisms.
I've never used those, so can't comment on how much work it is
to use them.

If that's too difficult, just use sys.platform.

 - tactis module is not present.
 
 I'm not sure what happened here: either the alias entry is wrong
 or the codec module was not committed.
 
 In either case, no one has complained about this encoding not working,
 so we can probably just remove it from the alias table. See
 http://bugs.python.org/issue1251921 for a similar report and
 discussion.
 
 I don't have such autority, and probably such a choice will require a 
 discussion on python-dev. So I'm +1 for just skipping it for now (with a XXX 
 comment on the right maybe).

Given the old discussion on the other ticket, it's fine to
remove the alias entry:

# tactis codec
'tis260' : 'tactis',

--
title: The email package should defer to the codecs module for all aliases - 
The email package should defer to the codecs module for   all aliases

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8898
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue8898] The email package should defer to the codecs module for all aliases

[Michele Orrù]

Michele Orrù maker...@gmail.com added the comment:

unittest.skip* are decorators, so useless in this case; also, AFAIS
Lib/test/ uses sys.platform.

I would suggest to put a try statement in encodings.mbcs, and raise an
error in case the imported modules imported are not found.
But this is another story.

--
title: The email package should defer to the codecs module for  all aliases - 
The email package should defer to the codecs module for all aliases
Added file: http://bugs.python.org/file22065/issue8898_skip.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8898
___diff -r cc60d0283fad Lib/email/charset.py
--- a/Lib/email/charset.py  Fri May 20 16:55:06 2011 +0200
+++ b/Lib/email/charset.py  Sun May 22 14:18:05 2011 +0200
@@ -10,6 +10,7 @@
 ]
 
 from functools import partial
+from codecs import lookup
 
 import email.base64mime
 import email.quoprimime
@@ -63,36 +64,6 @@
 'utf-8':   (SHORTEST,  BASE64, 'utf-8'),
 }
 
-# Aliases for other commonly-used names for character sets.  Map
-# them to the real ones used in email.
-ALIASES = {
-'latin_1': 'iso-8859-1',
-'latin-1': 'iso-8859-1',
-'latin_2': 'iso-8859-2',
-'latin-2': 'iso-8859-2',
-'latin_3': 'iso-8859-3',
-'latin-3': 'iso-8859-3',
-'latin_4': 'iso-8859-4',
-'latin-4': 'iso-8859-4',
-'latin_5': 'iso-8859-9',
-'latin-5': 'iso-8859-9',
-'latin_6': 'iso-8859-10',
-'latin-6': 'iso-8859-10',
-'latin_7': 'iso-8859-13',
-'latin-7': 'iso-8859-13',
-'latin_8': 'iso-8859-14',
-'latin-8': 'iso-8859-14',
-'latin_9': 'iso-8859-15',
-'latin-9': 'iso-8859-15',
-'latin_10':'iso-8859-16',
-'latin-10':'iso-8859-16',
-'cp949':   'ks_c_5601-1987',
-'euc_jp':  'euc-jp',
-'euc_kr':  'euc-kr',
-'ascii':   'us-ascii',
-}
-
-
 # Map charsets to their Unicode codec strings.
 CODEC_MAP = {
 'gb2312':  'eucgb2312_cn',
@@ -103,6 +74,8 @@
 'us-ascii':None,
 }
 
+# Aliases defined by the user
+ALIASES = dict()
 
 
 # Convenience functions for extending the above mappings
@@ -220,9 +193,12 @@
 input_charset = str(input_charset, 'ascii')
 except UnicodeError:
 raise errors.CharsetError(input_charset)
-input_charset = input_charset.lower()
-# Set the input charset after filtering through the aliases
-self.input_charset = ALIASES.get(input_charset, input_charset)
+# Set the input charset after filtering through its aliases defined in
+# codecs library
+try:
+self.input_charset = lookup(input_charset).name
+except LookupError:
+self.input_charset = ALIASES.get(input_charset, input_charset)
 # We can try to guess which encoding and conversion to use by the
 # charset_map dictionary.  Try that first, but let the user override
 # it.
diff -r cc60d0283fad Lib/encodings/aliases.py
--- a/Lib/encodings/aliases.py  Fri May 20 16:55:06 2011 +0200
+++ b/Lib/encodings/aliases.py  Sun May 22 14:18:05 2011 +0200
@@ -254,7 +254,7 @@
 # hp_roman8 codec
 'roman8' : 'hp_roman8',
 'r8' : 'hp_roman8',
-'csHPRoman8' : 'hp_roman8',
+'cshproman8' : 'hp_roman8',
 
 # hz codec
 'hzgb'   : 'hz',
@@ -298,6 +298,7 @@
 'iso_ir_157' : 'iso8859_10',
 'l6' : 'iso8859_10',
 'latin6' : 'iso8859_10',
+'latin_6': 'iso8859_10',
 
 # iso8859_11 codec
 'thai'   : 'iso8859_11',
@@ -308,6 +309,7 @@
 'iso_8859_13': 'iso8859_13',
 'l7' : 'iso8859_13',
 'latin7' : 'iso8859_13',
+'latin_7': 'iso8859_13',
 
 # iso8859_14 codec
 'iso_8859_14': 'iso8859_14',
@@ -316,11 +318,13 @@
 'iso_ir_199' : 'iso8859_14',
 'l8' : 'iso8859_14',
 'latin8' : 'iso8859_14',
+'latin_8': 'iso8859_14',
 
 # iso8859_15 codec
 'iso_8859_15': 'iso8859_15',
 'l9' : 'iso8859_15',
 'latin9' : 'iso8859_15',
+'latin_9': 'iso8859_15',
 
 # iso8859_16 codec
 'iso_8859_16': 'iso8859_16',
@@ -328,6 +332,7 @@
 'iso_ir_226' : 'iso8859_16',
 'l10': 'iso8859_16',
 'latin10': 'iso8859_16',
+'latin_10'   : 'iso8859_16',
 
 # iso8859_2 codec
 'csisolatin2': 'iso8859_2',
@@ -336,6 +341,7 @@
 'iso_ir_101' : 'iso8859_2',
 'l2' : 'iso8859_2',
 'latin2' : 'iso8859_2',
+'latin_2': 'iso8859_2',
 
 # iso8859_3 codec
 'csisolatin3': 'iso8859_3',
@@ -344,6 +350,7 @@
 'iso_ir_109' : 'iso8859_3',
 'l3' : 'iso8859_3',
   

[issue8898] The email package should defer to the codecs module for all aliases

[Ezio Melotti]

Ezio Melotti ezio.melo...@gmail.com added the comment:

Something like:
if name == 'mbcs' and not sys.platform.startswith('win'):
continue
should be enough.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8898
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue8898] The email package should defer to the codecs module for all aliases

[Michele Orrù]

Changes by Michele Orrù maker...@gmail.com:


Removed file: http://bugs.python.org/file22065/issue8898_skip.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8898
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue8898] The email package should defer to the codecs module for all aliases

[Michele Orrù]

Changes by Michele Orrù maker...@gmail.com:


Added file: http://bugs.python.org/file22066/issue8898_skip.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8898
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue8898] The email package should defer to the codecs module for all aliases

[Ezio Melotti]

Ezio Melotti ezio.melo...@gmail.com added the comment:

I suggest to:
  1) remove the alias for tactis;
  2) add the aliases for latin_* and the tests for the aliases;
  3) fix the email.charset to use the new aliases instead of its own dict.

2) and 3) should go on 3.3 only, 1) could be considered a bug and fixed on 
2.7/3.2 too, but since the codec is already missing, removing the alias won't 
change anything (i.e. it will raise a LookupError with or without alias).

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8898
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue8898] The email package should defer to the codecs module for all aliases

[Marc-Andre Lemburg]

Marc-Andre Lemburg m...@egenix.com added the comment:

Ezio Melotti wrote:
 
 Ezio Melotti ezio.melo...@gmail.com added the comment:
 
 I suggest to:
   1) remove the alias for tactis;
   2) add the aliases for latin_* and the tests for the aliases;
   3) fix the email.charset to use the new aliases instead of its own dict.
 
 2) and 3) should go on 3.3 only, 1) could be considered a bug and fixed on 
 2.7/3.2 too, but since the codec is already missing, removing the alias won't 
 change anything (i.e. it will raise a LookupError with or without alias).

+1

--
title: The email package should defer to the codecs module for all aliases - 
The email package should defer to the codecs module for   all aliases

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8898
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue8898] The email package should defer to the codecs module for all aliases

[Michele Orrù]

Michele Orrù maker...@gmail.com added the comment:

In the sense that the alias for 'tactis' should be removed also in 2.7 and 3.2?

--
title: The email package should defer to the codecs module for  all aliases - 
The email package should defer to the codecs module for all aliases

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8898
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue8898] The email package should defer to the codecs module for all aliases

[Michele Orrù]

Changes by Michele Orrù maker...@gmail.com:


Removed file: http://bugs.python.org/file22064/unnamed

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8898
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue8898] The email package should defer to the codecs module for all aliases

[R. David Murray]

R. David Murray rdmur...@bitdance.com added the comment:

euc_jp and euc_kr seem to be backward (that is, codecs translates them to the _ 
version, instead of translating the _ version to the - version).  I worry that 
there might be other deviations from the standard email names.  I would suggest 
we pull the list of preferred MIME names from the IANA charset registry and 
make a test out of them in the email package.  If changing the name returned by 
codecs is determined to not be acceptable, then those entries will need to 
remain in the charset module ALIASES table and the codecs-check logic adjusted 
accordingly.

Unfortunately the IANA registry does not list MIME names for all of the 
charsets in common use, and the canonical names are not always the ones 
commonly used in email.  Hopefully the codecs registry is using the most common 
name for those, and hopefully if there are differences it won't break any user 
code, since any reasonable email code should be coping with the aliases in any 
case.

Ezio, if you want to steal this one from me, that's fine by me.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8898
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue8898] The email package should defer to the codecs module for all aliases

[R. David Murray]

R. David Murray rdmur...@bitdance.com added the comment:

Hmm.  Must have misread.  Looks like all the common charsets do have MIME 
entries in the IANA table.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8898
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue8898] The email package should defer to the codecs module for all aliases

[R. David Murray]

R. David Murray rdmur...@bitdance.com added the comment:

On second thought the resolution order ought to be swapped anyway: if the user 
has added an ALIAS, they are going to want that used, not the one from codecs.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8898
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue8898] The email package should defer to the codecs module for all aliases

[Michele Orrù]

Michele Orrù maker...@gmail.com added the comment:

The attached patch adds aliases for latin_N in encodings.aliases, and fixes 
email.charset behaviour according to codecs.lookup, as requested.
Tested on (Arch) Linux.

Am I supposed to add any unittest? I'm wavering about where they should be 
placed (in encodings or email?).

--
keywords: +patch
nosy: +ezio.melotti, maker
Added file: http://bugs.python.org/file22049/issue8898.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8898
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue8898] The email package should defer to the codecs module for all aliases

[Michele Orrù]

Changes by Michele Orrù maker...@gmail.com:


--
nosy: +eric.araujo

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8898
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue8898] The email package should defer to the codecs module for all aliases

[Michele Orrù]

Changes by Michele Orrù maker...@gmail.com:


Removed file: http://bugs.python.org/file22049/issue8898.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8898
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue8898] The email package should defer to the codecs module for all aliases

[Michele Orrù]

Changes by Michele Orrù maker...@gmail.com:


Added file: http://bugs.python.org/file22053/issue8898.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8898
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue8898] The email package should defer to the codecs module for all aliases

[Ezio Melotti]

Ezio Melotti ezio.melo...@gmail.com added the comment:

The patch looks ok to me.
AFAIU the lookup will take care to normalize the name and return latin_N.  This 
also implies that other names (like 'latin-N', 'LaTiN~~N' and so on) will be 
normalized to latin_N and then accepted.

Regarding the tests, I don't see tests for the aliases anywhere, so something 
like:
for alias, codec_name in encodings.aliases.items():
self.assertEqual(codecs.lookup(alias).name, codec_name)
could be added somewhere to check that all the aliases in the dict map to the 
correct codec.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8898
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue8898] The email package should defer to the codecs module for all aliases

[R. David Murray]

R. David Murray rdmur...@bitdance.com added the comment:

Too late for 3.2, will implement for 3.3.

--
title: The email package should defer to the codecs module for  all aliases - 
The email package should defer to the codecs module for all aliases
versions: +Python 3.3 -Python 3.2

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8898
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue8898] The email package should defer to the codecs module for all aliases

[Shashwat Anand]

Shashwat Anand anand.shash...@gmail.com added the comment:

from email.charset.ALIASES most of them failed to be recognize by codecs module.


 for i in email.charset.ALIASES.keys():
... try:
... codecs.lookup(i)
... except LookupError:
... print(Not recognized by codecs : alias {} mapped to {}.format(i, 
email.charset.ALIASES[i]))
... 
... 
Not recognized by codecs : alias latin-8 mapped to iso-8859-14
Not recognized by codecs : alias latin-9 mapped to iso-8859-15
Not recognized by codecs : alias latin-2 mapped to iso-8859-2
Not recognized by codecs : alias latin-3 mapped to iso-8859-3
codecs.CodecInfo object for encoding iso8859-1 at 0x10160af58
Not recognized by codecs : alias latin-6 mapped to iso-8859-10
Not recognized by codecs : alias latin-7 mapped to iso-8859-13
Not recognized by codecs : alias latin-4 mapped to iso-8859-4
Not recognized by codecs : alias latin-5 mapped to iso-8859-9
codecs.CodecInfo object for encoding euc_jp at 0x1016260b8
Not recognized by codecs : alias latin-10 mapped to iso-8859-16
codecs.CodecInfo object for encoding ascii at 0x101626120
Not recognized by codecs : alias latin_10 mapped to iso-8859-16
codecs.CodecInfo object for encoding iso8859-1 at 0x10160aae0
Not recognized by codecs : alias latin_2 mapped to iso-8859-2
Not recognized by codecs : alias latin_3 mapped to iso-8859-3
Not recognized by codecs : alias latin_4 mapped to iso-8859-4
Not recognized by codecs : alias latin_5 mapped to iso-8859-9
Not recognized by codecs : alias latin_6 mapped to iso-8859-10
Not recognized by codecs : alias latin_7 mapped to iso-8859-13
Not recognized by codecs : alias latin_8 mapped to iso-8859-14
Not recognized by codecs : alias latin_9 mapped to iso-8859-15
codecs.CodecInfo object for encoding cp949 at 0x101626390
codecs.CodecInfo object for encoding euc_kr at 0x101626530


So basically apart from latin-1 all the latin* failed to be recognized by 
codecs.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8898
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue8898] The email package should defer to the codecs module for all aliases

[Marc-Andre Lemburg]

Marc-Andre Lemburg m...@egenix.com added the comment:

Shashwat Anand wrote:
 
 Shashwat Anand anand.shash...@gmail.com added the comment:
 
 from email.charset.ALIASES most of them failed to be recognize by codecs 
 module.
 
 
 for i in email.charset.ALIASES.keys():
 ... try:
 ... codecs.lookup(i)
 ... except LookupError:
 ... print(Not recognized by codecs : alias {} mapped to 
 {}.format(i, email.charset.ALIASES[i]))
 ... 
 ... 
 Not recognized by codecs : alias latin-8 mapped to iso-8859-14
 Not recognized by codecs : alias latin-9 mapped to iso-8859-15
 Not recognized by codecs : alias latin-2 mapped to iso-8859-2
 Not recognized by codecs : alias latin-3 mapped to iso-8859-3
 codecs.CodecInfo object for encoding iso8859-1 at 0x10160af58
 Not recognized by codecs : alias latin-6 mapped to iso-8859-10
 Not recognized by codecs : alias latin-7 mapped to iso-8859-13
 Not recognized by codecs : alias latin-4 mapped to iso-8859-4
 Not recognized by codecs : alias latin-5 mapped to iso-8859-9
 codecs.CodecInfo object for encoding euc_jp at 0x1016260b8
 Not recognized by codecs : alias latin-10 mapped to iso-8859-16
 codecs.CodecInfo object for encoding ascii at 0x101626120
 Not recognized by codecs : alias latin_10 mapped to iso-8859-16
 codecs.CodecInfo object for encoding iso8859-1 at 0x10160aae0
 Not recognized by codecs : alias latin_2 mapped to iso-8859-2
 Not recognized by codecs : alias latin_3 mapped to iso-8859-3
 Not recognized by codecs : alias latin_4 mapped to iso-8859-4
 Not recognized by codecs : alias latin_5 mapped to iso-8859-9
 Not recognized by codecs : alias latin_6 mapped to iso-8859-10
 Not recognized by codecs : alias latin_7 mapped to iso-8859-13
 Not recognized by codecs : alias latin_8 mapped to iso-8859-14
 Not recognized by codecs : alias latin_9 mapped to iso-8859-15
 codecs.CodecInfo object for encoding cp949 at 0x101626390
 codecs.CodecInfo object for encoding euc_kr at 0x101626530
 
 
 So basically apart from latin-1 all the latin* failed to be recognized by 
 codecs.

We need to add aliases for those codecs. The current aliases
list only supports the format latinN for N in 1-10.

--
title: The email package should defer to the codecs module for all aliases - 
The email package should defer to the codecs module for   all aliases

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8898
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue8898] The email package should defer to the codecs module for all aliases

[Shashwat Anand]

Shashwat Anand anand.shash...@gmail.com added the comment:

We need to add aliases for those codecs. The current aliases
list only supports the format latinN for N in 1-10.

latinN means latin1 to latin10 ? 
But latin_1 is a recognized alias.

 codecs.lookup('latin_1')
codecs.CodecInfo object for encoding iso8859-1 at 0x10160aae0

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8898
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue8898] The email package should defer to the codecs module for all aliases

[Marc-Andre Lemburg]

Marc-Andre Lemburg m...@egenix.com added the comment:

Shashwat Anand wrote:
 
 Shashwat Anand anand.shash...@gmail.com added the comment:
 
 We need to add aliases for those codecs. The current aliases
 list only supports the format latinN for N in 1-10.
 
 latinN means latin1 to latin10 ? 

Yes. We should add aliases for the format latin_N as well.

 But latin_1 is a recognized alias.
 
 codecs.lookup('latin_1')
 codecs.CodecInfo object for encoding iso8859-1 at 0x10160aae0

Yes, since that's the native name of the dedicated Python codec
for ISO-8859-1.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8898
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com