subject:"Codecs"



Change by admin :


--
github: None -> 38224

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue215308] codecs base classes need documentation



Change by admin :


___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue223642] Add site-codecs package to enable drop-in codec support



Change by admin :


--
github: None -> 33517

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue401877] Completed codecs API docs



Change by admin :


___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue401477] Fixes of ReadStream.readline() in UTF-16 and -LE codecs



Change by admin :


___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue223642] Add site-codecs package to enable drop-in codec support



Change by admin :


--
github: None -> 33517

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue401877] Completed codecs API docs



Change by admin :


--
github: None -> 33341

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue215308] codecs base classes need documentation



Change by admin :


--
github: None -> 33193

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue401477] Fixes of ReadStream.readline() in UTF-16 and -LE codecs



Change by admin :


--
github: None -> 33086

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44723] Codec name normalization breaks custom codecs

2022-01-24 Thread Inada Naoki



Change by Inada Naoki :


--
nosy: +methane

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44723] Codec name normalization breaks custom codecs



Change by Gregory P. Smith :


--
nosy: +gregory.p.smith

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37751] In codecs, function 'normalizestring' should convert both spaces and hyphens to underscores.



Gregory P. Smith  added the comment:

note that Bodo's own followup issue about the breaking change for python-iconv 
was filed as https://bugs.python.org/issue44723

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37751] In codecs, function 'normalizestring' should convert both spaces and hyphens to underscores.



Gregory P. Smith  added the comment:

(note: this might not be the true cause of that issue; though it sounds 
potentially related - I haven't investigated far enough yet)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37751] In codecs, function 'normalizestring' should convert both spaces and hyphens to underscores.



Gregory P. Smith  added the comment:

https://bugs.python.org/issue46508 filed to track fixing the acceptance and use 
of garbage codec values regression that this caused.

--
nosy: +gregory.p.smith

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45921] codecs module doesn't support iso-8859-6-i, iso-8859-6-e, iso-8859-8-i or iso-8859-8-i

2021-11-29 Thread Mark Sapiro



Mark Sapiro  added the comment:

The mailman-us...@python.org list received a post with the From: header 
containing a Hebrew display name RFC 2047 encoded with the iso-8859-8-i charset 
which threw a LookupError: unknown encoding: iso-8859-8-i exception in 
processing and shunted the message. The message body also had the charset 
declared as iso-8859-8-i although it contained only ascii. Unfortunately, I 
don't have the original message so I can't say what MUA created it or how 
common this usage is.

I do think that just adding these as aliases for the non-annotated encodings is 
an appropriate response.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45921] codecs module doesn't support iso-8859-6-i, iso-8859-6-e, iso-8859-8-i or iso-8859-8-i

2021-11-29 Thread Marc-Andre Lemburg



Marc-Andre Lemburg  added the comment:

Even though these are IANA recognized encodings, we need to apply he same logic 
as we do for all new encodings, which essentially boils down to: Are these 
encoding in wider spread use today ?

Reading through the RFC 1556, it seems that the added -i or -e are just 
indications for applications on how to interpret BIDI information: either 
implicit by looking at the order of characters in the stream or explicit via 
control characters embedded in the stream. They are not new encodings, with new 
mappings.

If that's a correct interpretation, we could add those as aliases for the 
non-annotated encodings.

After more than 20 years with Unicode support in Python and the world moving 
towards UTF-8, I have become fairly reluctant towards adding more encoding 
support to Python.

If people are still using unsupported encodings, it's probably better to point 
them to other dedicated tools for converting text to UTF-8, e.g. iconv, than 
extending the pretty extensive support we already have in Python.

--
nosy: +lemburg

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45921] codecs module doesn't support iso-8859-6-i, iso-8859-6-e, iso-8859-8-i or iso-8859-8-i

2021-11-29 Thread Erlend E. Aasland



Change by Erlend E. Aasland :


--
versions: +Python 3.11 -Python 3.6, Python 3.7, Python 3.8, Python 3.9

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45921] codecs module doesn't support iso-8859-6-i, iso-8859-6-e, iso-8859-8-i or iso-8859-8-i

2021-11-28 Thread Mark Sapiro



New submission from Mark Sapiro :

iso-8859-6-i, iso-8859-6-e, iso-8859-8-i and iso-8859-8-i are all IANA 
recognized character sets per 
https://www.iana.org/assignments/character-sets/character-sets.xhtml. These are 
all unrecognized by codecs.lookup().

--
components: Library (Lib)
messages: 407240
nosy: msapiro
priority: normal
severity: normal
status: open
title: codecs module doesn't support iso-8859-6-i, iso-8859-6-e, iso-8859-8-i 
or iso-8859-8-i
type: behavior
versions: Python 3.10, Python 3.6, Python 3.7, Python 3.8, Python 3.9

___
Python tracker 
<https://bugs.python.org/issue45921>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25880] codecs should raise specific UnicodeDecodeError/UnicodeEncodeError rather than just UnicodeError

2021-11-26 Thread Irit Katriel


Change by Irit Katriel :


--
title: u'..'.encode('idna') → UnicodeError: label empty or too long -> codecs 
should raise specific UnicodeDecodeError/UnicodeEncodeError rather than just 
UnicodeError
versions: +Python 3.11 -Python 2.7, Python 3.4

___
Python tracker 
<https://bugs.python.org/issue25880>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45277] typo in codecs documentation

2021-10-04 Thread Pablo Galindo Salgado



Pablo Galindo Salgado  added the comment:


New changeset 0b568530ecb6ccc696771acf8018ec6fff1a0a5f by Pablo Galindo (Miss 
Islington (bot)) in branch '3.10':
bpo-45277: Fix typo in codecs doc (GH-28555)
https://github.com/python/cpython/commit/0b568530ecb6ccc696771acf8018ec6fff1a0a5f


--
nosy: +pablogsal

___
Python tracker 
<https://bugs.python.org/issue45277>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45277] typo in codecs documentation



Change by Terry J. Reedy :


--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45277] typo in codecs documentation



miss-islington  added the comment:


New changeset 2a7d985bb3a1d85d63f135956750b330408702e6 by Miss Islington (bot) 
in branch '3.9':
bpo-45277: Fix typo in codecs doc (GH-28555)
https://github.com/python/cpython/commit/2a7d985bb3a1d85d63f135956750b330408702e6


--

___
Python tracker 
<https://bugs.python.org/issue45277>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45277] typo in codecs documentation



miss-islington  added the comment:


New changeset 371ca3b54d355476cc735cb932e80f5cd16310da by Miss Islington (bot) 
in branch '3.10':
bpo-45277: Fix typo in codecs doc (GH-28555)
https://github.com/python/cpython/commit/371ca3b54d355476cc735cb932e80f5cd16310da


--

___
Python tracker 
<https://bugs.python.org/issue45277>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45277] typo in codecs documentation



Change by miss-islington :


--
nosy: +miss-islington
nosy_count: 3.0 -> 4.0
pull_requests: +26939
pull_request: https://github.com/python/cpython/pull/28556

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45277] typo in codecs documentation



Change by miss-islington :


--
pull_requests: +26940
pull_request: https://github.com/python/cpython/pull/28557

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45277] typo in codecs documentation



Terry J. Reedy  added the comment:


New changeset 4c0fc65cd8a6d4c18330505576ccd4b46abeec1c by Terry Jan Reedy in 
branch 'main':
bpo-45277: Fix typo in codecs doc (GH-28555)
https://github.com/python/cpython/commit/4c0fc65cd8a6d4c18330505576ccd4b46abeec1c


--

___
Python tracker 
<https://bugs.python.org/issue45277>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45277] typo in codecs documentation



Change by Terry J. Reedy :


--
keywords: +patch
pull_requests: +26938
stage:  -> patch review
pull_request: https://github.com/python/cpython/pull/28555

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45277] typo in codecs documentation

2020-06-26 Thread శ్రీనివాస్ రెడ్డి తాటిపర్తి



Terry J. Reedy  added the comment:

Thanks for the report.  I will fix this.

--
nosy: +terry.reedy

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45277] typo in codecs documentation

2021-09-24 Thread HugoThiolliere


New submission from HugoThiolliere :

There is a typo in 
https://docs.python.org/3/library/codecs.html#encodings-and-unicode

The first sentence in the last paragraph before the table reads : "There’s 
another encoding that is able to encoding the full range of Unicode characters"

When it should read "There’s another encoding that is able to encode the full 
range of Unicode characters"

--
assignee: docs@python
components: Documentation
messages: 402545
nosy: Gronahak, docs@python
priority: normal
severity: normal
status: open
title: typo in codecs documentation
type: enhancement
versions: Python 3.9

___
Python tracker 
<https://bugs.python.org/issue45277>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44723] Codec name normalization breaks custom codecs

2021-07-23 Thread Bodo Graumann

New submission from Bodo Graumann :

This is a follow up on https://bugs.python.org/issue37751 concerning
normalization of codec names.

First of all, the changes made therein are not documented correctly.
In the implementation
| Normalization works as follows: all non-alphanumeric
| characters except the dot used for Python package names are
| collapsed and replaced with a single underscore, e.g. ' -;#'
| becomes '_'. Leading and trailing underscores are removed.”
Cf.
[encodings/__init__.py](https://github.com/python/cpython/blob/bb3e0c240bc60fe08d332ff5955d54197f79751c/Lib/encodings/__init__.py#L47-L50)

The documentation however only states that:
| Search functions are expected to take one argument, being the encoding name
in all lower case letters with hyphens and spaces converted to underscores
Cf. https://docs.python.org/3/library/codecs.html#codecs.register

Secondly, this change breaks lots of iconv codecs with the python-iconv
binding. E.g. `ASCII//TRANSLIT` is now normalized to `ascii_translit`, which
iconv does not understand. Codec names which use hyphens also break and iinm
not all of them have aliases in iconv without hyphens.
Cf. [python-iconv #4](https://github.com/bodograumann/python-iconv/issues/4)

How about first looking up the given name and only then, if the given name
could not be found, looking for the codec by its normalized name?

--
components: Unicode
messages: 398042
nosy: bodograumann, ezio.melotti, vstinner
priority: normal
severity: normal
status: open
title: Codec name normalization breaks custom codecs
type: behavior
versions: Python 3.9

___
Python tracker
<https://bugs.python.org/issue44723>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37751] In codecs, function 'normalizestring' should convert both spaces and hyphens to underscores.

2021-06-13 Thread STINNER Victor



STINNER Victor  added the comment:

> The codecs api feels extremely well-fitting for integrating iconv in python 
> and any alternative I can think of seems unsatisfactory.

This issue is now closed, would you mind to open a new issue?

--

___
Python tracker 
<https://bugs.python.org/issue37751>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37751] In codecs, function 'normalizestring' should convert both spaces and hyphens to underscores.

2021-06-13 Thread Bodo Graumann


Bodo Graumann  added the comment:

Unfortunately this is not quite finished yet.

First of all, the change is bigger than what is documented: “Changed in version 
3.9: Hyphens and spaces are converted to underscore.“

In reality, now
| Normalization works as follows: all non-alphanumeric
| characters except the dot used for Python package names are
| collapsed and replaced with a single underscore, e.g. '  -;#'
| becomes '_'. Leading and trailing underscores are removed.”
Cf. 
[encodings/__init__.py](https://github.com/python/cpython/blob/bb3e0c240bc60fe08d332ff5955d54197f79751c/Lib/encodings/__init__.py#L47-L50)

Secondly, this change breaks lots of iconv codecs with the python-iconv 
binding. E.g. `ASCII//TRANSLIT` is now normalized to `ascii_translit`, which 
iconv does not understand. Codec names which use hyphens also break and iinm 
not all of them have aliases in iconv without hyphens.
Cf. [python-iconv #4](https://github.com/bodograumann/python-iconv/issues/4)

The codecs api feels extremely well-fitting for integrating iconv in python and 
any alternative I can think of seems unsatisfactory.
Please advise.

--
nosy: +bodograumann

___
Python tracker 
<https://bugs.python.org/issue37751>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37751] In codecs, function 'normalizestring' should convert both spaces and hyphens to underscores.

2021-04-28 Thread STINNER Victor



STINNER Victor  added the comment:

Thanks Inada-san for documenting the change in codecs.register() doc!

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37751] In codecs, function 'normalizestring' should convert both spaces and hyphens to underscores.

2021-04-27 Thread Inada Naoki



Inada Naoki  added the comment:


New changeset cf9d65c5af7905d9e9945a297dbbf15d3bcace15 by Miss Islington (bot) 
in branch '3.9':
bpo-37751: Update `codecs.register()` doc. (GH-25643)
https://github.com/python/cpython/commit/cf9d65c5af7905d9e9945a297dbbf15d3bcace15


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37751] In codecs, function 'normalizestring' should convert both spaces and hyphens to underscores.

2021-04-27 Thread Inada Naoki



Inada Naoki  added the comment:


New changeset 5c84bb506aaca01f5f750116d8f7a41d41f8124d by Inada Naoki in branch 
'master':
bpo-37751: Update `codecs.register()` doc. (GH-25643)
https://github.com/python/cpython/commit/5c84bb506aaca01f5f750116d8f7a41d41f8124d


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37751] In codecs, function 'normalizestring' should convert both spaces and hyphens to underscores.

2021-04-27 Thread miss-islington



Change by miss-islington :


--
pull_requests: +24368
pull_request: https://github.com/python/cpython/pull/25677

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37751] In codecs, function 'normalizestring' should convert both spaces and hyphens to underscores.

2021-04-27 Thread STINNER Victor



STINNER Victor  added the comment:


New changeset 531c81038e28b6cfa0f9791467bf671c88c6f4c4 by Miss Islington (bot) 
in branch '3.9':
bpo-37751: Document codecs.lookup() change in What's New in Python 3.9 
(GH-23096) (GH-25659)
https://github.com/python/cpython/commit/531c81038e28b6cfa0f9791467bf671c88c6f4c4


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37751] In codecs, function 'normalizestring' should convert both spaces and hyphens to underscores.

2021-04-27 Thread miss-islington



Change by miss-islington :


--
nosy: +miss-islington
nosy_count: 8.0 -> 9.0
pull_requests: +24350
pull_request: https://github.com/python/cpython/pull/25659

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37751] In codecs, function 'normalizestring' should convert both spaces and hyphens to underscores.

2021-04-26 Thread Inada Naoki



Change by Inada Naoki :


--
pull_requests: +24336
pull_request: https://github.com/python/cpython/pull/25643

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37751] In codecs, function 'normalizestring' should convert both spaces and hyphens to underscores.

2021-04-24 Thread hai shi



hai shi  added the comment:

Thanks Marc-Andre for your supplement of PEP-100.
Thanks Inada, victor for your review and merge.

After PR-23096 merged, I suggest to close this bpo.
If there have any other questions, we can reopen it again.

--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37751] In codecs, function 'normalizestring' should convert both spaces and hyphens to underscores.

2021-04-23 Thread STINNER Victor



STINNER Victor  added the comment:


New changeset 32980fb669a6857276da18895fcc0cb6f6fbb544 by Hai Shi in branch 
'master':
bpo-37751: Document codecs.lookup() change in What's New in Python 3.9 
(GH-23096)
https://github.com/python/cpython/commit/32980fb669a6857276da18895fcc0cb6f6fbb544


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37751] In codecs, function 'normalizestring' should convert both spaces and hyphens to underscores.

2021-04-23 Thread Miro Hrončok


Change by Miro Hrončok :


--
nosy:  -hroncok

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37751] In codecs, function 'normalizestring' should convert both spaces and hyphens to underscores.

2021-04-23 Thread Marc-Andre Lemburg



Marc-Andre Lemburg  added the comment:

On 23.04.2021 07:47, Inada Naoki wrote:
> 
> Inada Naoki  added the comment:
> 
> I think it is too late. Python 3.9 has been released already.  Reverting the 
> change is also breaking change.
> 
> PEP 100 says:
> "Search functions are expected to take one argument, the encoding name in all 
> lower case letters and with hyphens and spaces converted to underscores"
> https://www.python.org/dev/peps/pep-0100/#codecs-coder-decoders-lookup
> 
> But codecs.register() says:
> "Search functions are expected to take one argument, being the encoding name 
> in all lower case letters".
> 
> I don't know historical reason why two document are inconsistent.
> https://docs.python.org/3/library/codecs.html#codecs.register

I guess just an oversight on my part.

PEP 100 is certainly what I meant and implemented. I should have also
made it clear in PEP 100 that I meant lower case ASCII letters.

--

___
Python tracker 
<https://bugs.python.org/issue37751>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37751] In codecs, function 'normalizestring' should convert both spaces and hyphens to underscores.

2021-04-23 Thread Marc-Andre Lemburg

Marc-Andre Lemburg  added the comment:

On 23.04.2021 03:37, akdor1154 wrote:
> 
> akdor1154  added the comment:
> 
> If I understand the target of this issue, this is a breaking change in python 
> 3.9 .
> 
> E.g. see https://github.com/SAP/PyHDB/issues/149
> 
> Ideally if we did not intend to break libraries then can this be fixed?
> Or if it is acceptable to have such a breaking change, can it be documented 
> as such? (maybe this is https://github.com/python/cpython/pull/23096 ? though 
> I would not interpret that as a breaking change at first glance)

This patch only restored the behavior we had before (and for many many
years). It's not breaking, it's in fact resolving a break which was
caused by earlier:

https://bugs.python.org/issue37751#msg349448

Please note that search functions determine how to map codec names
to codec implementations. The codec search function in the encodings
package uses one way to do this (and depends on how the package
is structured).

The approach taken by the encodings search function is listed here:
https://github.com/python/cpython/blob/master/Lib/encodings/__init__.py#L43

Other search functions can work in different ways.

Now, unfortunately, parts of this kind of normalization have also made
its way into the codecs module itself and into the Unicode
implementation and perhaps not always in a way which allows search
functions to use a different approach or which is consistent.

As I mentioned before, the safest way to go about this is to use
alnum only names for codecs, with the addition of underscores to
separate important parts.

The Python implementation should make sure that such names continue
to work when passed through any codec name normalization.

--
title: In codecs,  function 'normalizestring' should convert both spaces and 
hyphens to underscores. -> In codecs, function 'normalizestring' should convert 
both spaces and hyphens to underscores.

___
Python tracker 
<https://bugs.python.org/issue37751>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37751] In codecs, function 'normalizestring' should convert both spaces and hyphens to underscores.

2021-04-22 Thread Inada Naoki



Inada Naoki  added the comment:

codecs.register() was added in this commit.
https://github.com/python/cpython/commit/e2d67f98d1aade1059b2ff3278672b2ffbaf180e

And its docstring has been added in this commit.
https://github.com/python/cpython/commit/0ae2981dec3de96a1f7d63b0535992cf1462ac92

Both commits doesn't describe why normalization was differ from PEP 100.

--
nosy: +mark

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37751] In codecs, function 'normalizestring' should convert both spaces and hyphens to underscores.

2021-04-22 Thread Inada Naoki



Inada Naoki  added the comment:

I think it is too late. Python 3.9 has been released already.  Reverting the 
change is also breaking change.

PEP 100 says:
"Search functions are expected to take one argument, the encoding name in all 
lower case letters and with hyphens and spaces converted to underscores"
https://www.python.org/dev/peps/pep-0100/#codecs-coder-decoders-lookup

But codecs.register() says:
"Search functions are expected to take one argument, being the encoding name in 
all lower case letters".

I don't know historical reason why two document are inconsistent.
https://docs.python.org/3/library/codecs.html#codecs.register

--
nosy: +methane

___
Python tracker 
<https://bugs.python.org/issue37751>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37751] In codecs, function 'normalizestring' should convert both spaces and hyphens to underscores.

2021-04-22 Thread hai shi



hai shi  added the comment:

>Ideally if we did not intend to break libraries then can this be fixed?
Or if it is acceptable to have such a breaking change.

Hi, akdor1154, thanks for your notice. It was a bugfix in fact:) 
https://bugs.python.org/issue37751#msg349448

> maybe this is https://github.com/python/cpython/pull/23096 ? though I would 
> not interpret that as a breaking change at first glance

@victor ping :)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37751] In codecs, function 'normalizestring' should convert both spaces and hyphens to underscores.

2021-04-22 Thread akdor1154



akdor1154  added the comment:

If I understand the target of this issue, this is a breaking change in python 
3.9 .

E.g. see https://github.com/SAP/PyHDB/issues/149

Ideally if we did not intend to break libraries then can this be fixed?
Or if it is acceptable to have such a breaking change, can it be documented as 
such? (maybe this is https://github.com/python/cpython/pull/23096 ? though I 
would not interpret that as a breaking change at first glance)

--
nosy: +akdor1154

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue12508] Codecs Anomaly

2020-11-11 Thread Irit Katriel



Change by Irit Katriel :


--
versions: +Python 3.10, Python 3.8, Python 3.9 -Python 2.7, Python 3.2, Python 
3.3

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37751] In codecs, function 'normalizestring' should convert both spaces and hyphens to underscores.

2020-11-01 Thread hai shi



Change by hai shi :


--
pull_requests: +22010
stage: resolved -> patch review
pull_request: https://github.com/python/cpython/pull/23096

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41919] Move the codecs' register operation to testcases

2020-10-03 Thread hai shi



Change by hai shi :


--
keywords: +patch
pull_requests: +21522
stage:  -> patch review
pull_request: https://github.com/python/cpython/pull/22513

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41919] Move the codecs' register operation to testcases

2020-10-03 Thread hai shi



New submission from hai shi :

After PR22360 merged, we can move the codecs' register operation to testcases.

--
components: Tests
messages: 377863
nosy: shihai1991, vstinner
priority: normal
severity: normal
status: open
title: Move the codecs' register operation to testcases
type: enhancement
versions: Python 3.10

___
Python tracker 
<https://bugs.python.org/issue41919>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41115] Codecs should raise precise UnicodeDecodeError or UnicodeEncodeError

2020-06-26 Thread utkarsh



utkarsh  added the comment:

@thatiparthy These were the most logical changes, standard error messages, 
which were already there in the existing code, I just edited them as mentioned 
here. What part of your "work" do you think i copied?
Sent this PR to get familiar to the process mostly, i will close it if you feel 
insecure. No need to be rude.
thanks.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41115] Codecs should raise precise UnicodeDecodeError or UnicodeEncodeError


Srinivas  Reddy Thatiparthy(శ్రీనివాస్ రెడ్డి తాటిపర్తి) 
 added the comment:

@utk You could have taken some other easy issue from 
https://bugs.python.org/issue?status=1&@sort=-activity&@columns=id%2Cactivity%2Ctitle%2Ccreator%2Cstatus&@dispname=Easy%20issues&@startwith=0&@group=priority=6&@action=search&@filter=&@pagesize=50
 instead of copy pasting my work.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41115] Codecs should raise precise UnicodeDecodeError or UnicodeEncodeError

2020-06-26 Thread utkarsh



Change by utkarsh :


--
nosy: +utk
nosy_count: 8.0 -> 9.0
pull_requests: +20329
pull_request: https://github.com/python/cpython/pull/21170

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41115] Codecs should raise precise UnicodeDecodeError or UnicodeEncodeError

2020-06-26 Thread శ్రీనివాస్ రెడ్డి తాటిపర్తి


Change by Srinivas  Reddy Thatiparthy(శ్రీనివాస్ రెడ్డి తాటిపర్తి) 
:


--
keywords: +patch
pull_requests: +20323
stage: needs patch -> patch review
pull_request: https://github.com/python/cpython/pull/21165

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41115] Codecs should raise precise UnicodeDecodeError or UnicodeEncodeError

2020-06-25 Thread STINNER Victor



Change by STINNER Victor :


--
nosy:  -vstinner

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41115] Codecs should raise precise UnicodeDecodeError or UnicodeEncodeError

2020-06-25 Thread Walter Dörwald


Walter Dörwald  added the comment:

UnicodeEncodeError and UnicodeDecodeError are used to report un(en|de)codedable 
ranges in the source object, so it wouldn't make sense to use them for errors 
that have nothing to do with problems in the source object. Their constructor 
requires 5 arguments (encoding, object, start, end, reason), not just a simple 
message: e.g. UnicodeEncodeError("utf-8", "foo", 17, 23, "bad string").

But for reporting e.g. missing BOMs at the start it would be useful to use (0,  
0) as the offending range.

--
nosy: +doerwalter

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41115] Codecs should raise precise UnicodeDecodeError or UnicodeEncodeError

2020-06-25 Thread Emmanuel Arias



Emmanuel Arias  added the comment:

Hi,

IMO this can be mark as an easy issue.

@thatiparthy please, go ahead

--
nosy: +eamanu

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41115] Codecs should raise precise UnicodeDecodeError or UnicodeEncodeError

2020-06-25 Thread శ్రీనివాస్ రెడ్డి తాటిపర్తి


Srinivas  Reddy Thatiparthy(శ్రీనివాస్ రెడ్డి తాటిపర్తి) 
 added the comment:

This looks like an easy task. Shall I create a PR?

--
nosy: +thatiparthy

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41115] Codecs should raise precise UnicodeDecodeError or UnicodeEncodeError

2020-06-25 Thread Antoine Pitrou



New submission from Antoine Pitrou :

A number of codecs raise bare UnicodeError, rather than 
Unicode{Decode,Encode}Error. Example:

  File 
"/home/antoine/miniconda3/envs/pyarrow/lib/python3.7/encodings/utf_16.py", line 
67, in _buffer_decode
raise UnicodeError("UTF-16 stream does not start with BOM")

A more complete list can be found here:
https://gist.github.com/pitrou/60594b28d8e47edcdb97d9b15d5f9866

--
components: Library (Lib)
keywords: easy
messages: 372367
nosy: benjamin.peterson, ezio.melotti, lemburg, pitrou, serhiy.storchaka, 
vstinner
priority: normal
severity: normal
stage: needs patch
status: open
title: Codecs should raise precise UnicodeDecodeError or UnicodeEncodeError
type: behavior
versions: Python 3.7, Python 3.8, Python 3.9

___
Python tracker 
<https://bugs.python.org/issue41115>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37751] In codecs, function 'normalizestring' should convert both spaces and hyphens to underscores.



STINNER Victor  added the comment:

I created bpo-39337: codecs.lookup() ignores non-ASCII characters, whereas 
encodings.normalize_encoding() copies them.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37751] In codecs, function 'normalizestring' should convert both spaces and hyphens to underscores.



STINNER Victor  added the comment:

> Please note that external codec packages should not rely on the semantics of 
> the Python stdlib encodings package's search function.

latexcodec does register a search function.

> It's good practice to always only use ASCII lower case chars and the 
> underscore for codec names.

latexcodec uses encoding names like "latex+ascii" and their search function 
used "+" as a separator.

Don't worry, I just fixed latexcodec, my fix is already merged upstream! I 
simply changed the search function to split on "_" if the name contains "_".

* 
https://github.com/mcmtroffaes/latexcodec/commit/a30ae2cf061d7369b1aaa8179ddd1b486974fdad
* https://github.com/mcmtroffaes/latexcodec/pull/76
* https://github.com/mcmtroffaes/latexcodec/issues/75

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37751] In codecs, function 'normalizestring' should convert both spaces and hyphens to underscores.

2020-01-14 Thread Marc-Andre Lemburg



Marc-Andre Lemburg  added the comment:

Just to clarify: the change in the C implementation was the breaking change. 
The patch just restores the previous behavior: 
https://github.com/python/cpython/blob/master/Lib/encodings/__init__.py#L43

Please note that external codec packages should not rely on the semantics of 
the Python stdlib encodings package's search function. They should really 
register their own search function: 
https://docs.python.org/3.9/library/codecs.html#codecs.register

It's good practice to always only use ASCII lower case chars and the underscore 
for codec names.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37751] In codecs, function 'normalizestring' should convert both spaces and hyphens to underscores.



STINNER Victor  added the comment:

It seems quite easy to update latexcodec project to support Python 3.9. I 
proposed a solution there:
https://bugzilla.redhat.com/show_bug.cgi?id=1789613#c6

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37751] In codecs, function 'normalizestring' should convert both spaces and hyphens to underscores.

STINNER Victor  added the comment:

> The change is backwards incompatible and a backport would break things. See 
> for example how it breaks latexcodec:

I reopen the issue. I proposed PR 17997 to *document* the incompatible change 
in What's New in Python 3.8. IMO it's a deliberate change and it's correct.

I rely on Marc-Andre Lemburg who implemented codecs and encodings modules. He 
wrote: "Jordon is right. Conversion has to be to underscores, not hyphens.".

--
resolution: fixed -> 
status: closed -> open

___
Python tracker 
<https://bugs.python.org/issue37751>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37751] In codecs, function 'normalizestring' should convert both spaces and hyphens to underscores.



Change by STINNER Victor :


--
pull_requests: +17401
pull_request: https://github.com/python/cpython/pull/17997

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37751] In codecs, function 'normalizestring' should convert both spaces and hyphens to underscores.

2020-01-14 Thread Miro Hrončok


Miro Hrončok  added the comment:

The change is backwards incompatible and a backport would break things. See for 
example how it breaks latexcodec:

https://bugzilla.redhat.com/show_bug.cgi?id=1789613#c2

--
nosy: +hroncok

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39155] "utf8-sig" missing from codecs (inconsistency)

2019-12-29 Thread Peter Ludemann



New submission from Peter Ludemann :

In general, 'utf8' and 'utf-8' are interchangeable in the codecs (and in many 
parts of the Python library). However, 'utf8-sig' is missing ... and it happens 
to also be generated by lib2to3.tokenize.detect_encoding.

>>> import codecs
>>> codecs.getincrementaldecoder('utf-8-sig')()

>>> codecs.getincrementaldecoder('utf8-sig')()
Traceback (most recent call last):
  File "", line 1, in 
  File "/usr/lib/python3.6/codecs.py", line 987, in getincrementaldecoder
decoder = lookup(encoding).incrementaldecoder
LookupError: unknown encoding: utf8-sig

--
components: Unicode
messages: 358996
nosy: Peter Ludemann, ezio.melotti, vstinner
priority: normal
severity: normal
status: open
title: "utf8-sig" missing from codecs (inconsistency)
type: behavior
versions: Python 3.6, Python 3.7, Python 3.8

___
Python tracker 
<https://bugs.python.org/issue39155>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39154] "utf8-sig" missing from codecs (inconsistency)

2019-12-29 Thread Peter Ludemann



New submission from Peter Ludemann :

In general, 'utf8' and 'utf-8' are interchangeable in the codecs (and in many 
parts of the Python library). However, 'utf8-sig' is missing ... and it happens 
to also be generated by lib2to3.tokenize.detect_encoding.

>>> import codecs
>>> codecs.getincrementaldecoder('utf-8-sig')()

>>> codecs.getincrementaldecoder('utf8-sig')()
Traceback (most recent call last):
  File "", line 1, in 
  File "/usr/lib/python3.6/codecs.py", line 987, in getincrementaldecoder
decoder = lookup(encoding).incrementaldecoder
LookupError: unknown encoding: utf8-sig

--
components: Unicode
messages: 358994
nosy: Peter Ludemann, ezio.melotti, vstinner
priority: normal
severity: normal
status: open
title: "utf8-sig" missing from codecs (inconsistency)
type: behavior
versions: Python 3.6, Python 3.7, Python 3.8

___
Python tracker 
<https://bugs.python.org/issue39154>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38056] Overhaul Error Handlers section in codecs documentation

2019-10-12 Thread Ma Lin



Ma Lin  added the comment:

PR 15732 became an overhaul:

- replace/backslashreplace/surrogateescape were wrongly described as encoding 
only, in fact they can also be used in decoding.
- clarify the description of surrogatepass.
- add more descriptions to each handler.
- add two REPL examples.
- add indexes for Error Handler's name.
- add default parameter values in codecs.rst
- improve term "text encoding".

PR 15732 has a screenshot of the Error Handlers section.

--
components: +Unicode
nosy: +ezio.melotti, vstinner
title: Add examples for common text encoding Error Handlers -> Overhaul Error 
Handlers section in codecs documentation

___
Python tracker 
<https://bugs.python.org/issue38056>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37751] In codecs, function 'normalizestring' should convert both spaces and hyphens to underscores.

2019-08-21 Thread Jordon.X



Jordon.X <9651...@qq.com> added the comment:

Thanks vstinner. I also don't think it's necessary to backport to the old 
version. Close this issue is fine.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37751] In codecs, function 'normalizestring' should convert both spaces and hyphens to underscores.

2019-08-21 Thread STINNER Victor



STINNER Victor  added the comment:

Thanks for the fix Jordon Xu.

IMHO this change is not strictly a bugfix, but more like an enhancement. I 
close the issue.

If you consider that a backport to Python 3.7 and 3.8 is needed, please say so.

--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37751] In codecs, function 'normalizestring' should convert both spaces and hyphens to underscores.

2019-08-21 Thread STINNER Victor



STINNER Victor  added the comment:


New changeset 20f59fe1f7748ae899aceee4cb560e5e1f528a1f by Victor Stinner 
(Jordon Xu) in branch 'master':
bpo-37751: Fix codecs.lookup() normalization (GH-15092)
https://github.com/python/cpython/commit/20f59fe1f7748ae899aceee4cb560e5e1f528a1f


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37751] In codecs, function 'normalizestring' should convert both spaces and hyphens to underscores.

2019-08-12 Thread Marc-Andre Lemburg



Marc-Andre Lemburg  added the comment:

Jordon is right. Conversion has to be to underscores, not hyphens. I guess this 
bug was introduced when the normalization function was converted to C.

--
nosy: +lemburg

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37751] In codecs, function 'normalizestring' should convert both spaces and hyphens to underscores.



Jordon.X <9651...@qq.com> added the comment:

The design and code of the following four places need to be consistent,

No.1 https://github.com/python/peps/blob/master/pep-0100.txt#L292
No.2 https://github.com/python/cpython/blob/master/Python/codecs.c#L113
No.3 https://github.com/python/cpython/blob/master/Python/codecs.c#L53  
No.4 https://github.com/python/cpython/blob/master/Python/codecs.c#74

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37751] In codecs, function 'normalizestring' should convert both spaces and hyphens to underscores.

2019-08-03 Thread hai shi



hai shi  added the comment:

Hm, there is a bit misleading between 
desc(https://github.com/python/cpython/blob/master/Python/codecs.c#L53) and the 
code (https://github.com/python/cpython/blob/master/Python/codecs.c#L74).

--
nosy: +shihai1991

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37751] In codecs, function 'normalizestring' should convert both spaces and hyphens to underscores.



Change by Jordon.X <9651...@qq.com>:


--
keywords: +patch
pull_requests: +14838
stage:  -> patch review
pull_request: https://github.com/python/cpython/pull/15092

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37751] In codecs, function 'normalizestring' should convert both spaces and hyphens to underscores.



Jordon.X <9651...@qq.com> added the comment:

and I will try to fix it.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37751] In codecs, function 'normalizestring' should convert both spaces and hyphens to underscores.