[issue37751] In codecs, function 'normalizestring' should convert both spaces and hyphens to underscores.

Gregory P. Smith added the comment: note that Bodo's own followup issue about the breaking change for python-iconv was filed as https://bugs.python.org/issue44723 -- ___ Python tracker

[issue37751] In codecs, function 'normalizestring' should convert both spaces and hyphens to underscores.

Gregory P. Smith added the comment: (note: this might not be the true cause of that issue; though it sounds potentially related - I haven't investigated far enough yet) -- ___ Python tracker

[issue37751] In codecs, function 'normalizestring' should convert both spaces and hyphens to underscores.

Gregory P. Smith added the comment: https://bugs.python.org/issue46508 filed to track fixing the acceptance and use of garbage codec values regression that this caused. -- nosy: +gregory.p.smith ___ Python tracker

[issue37751] In codecs, function 'normalizestring' should convert both spaces and hyphens to underscores.

STINNER Victor added the comment: > The codecs api feels extremely well-fitting for integrating iconv in python > and any alternative I can think of seems unsatisfactory. This issue is now closed, would you mind to open a new issue? -- ___ Python

[issue37751] In codecs, function 'normalizestring' should convert both spaces and hyphens to underscores.

Bodo Graumann added the comment: Unfortunately this is not quite finished yet. First of all, the change is bigger than what is documented: “Changed in version 3.9: Hyphens and spaces are converted to underscore.“ In reality, now | Normalization works as follows: all non-alphanumeric |

[issue37751] In codecs, function 'normalizestring' should convert both spaces and hyphens to underscores.

STINNER Victor added the comment: Thanks Inada-san for documenting the change in codecs.register() doc! -- ___ Python tracker ___

[issue37751] In codecs, function 'normalizestring' should convert both spaces and hyphens to underscores.

Inada Naoki added the comment: New changeset cf9d65c5af7905d9e9945a297dbbf15d3bcace15 by Miss Islington (bot) in branch '3.9': bpo-37751: Update `codecs.register()` doc. (GH-25643) https://github.com/python/cpython/commit/cf9d65c5af7905d9e9945a297dbbf15d3bcace15 --

[issue37751] In codecs, function 'normalizestring' should convert both spaces and hyphens to underscores.

Inada Naoki added the comment: New changeset 5c84bb506aaca01f5f750116d8f7a41d41f8124d by Inada Naoki in branch 'master': bpo-37751: Update `codecs.register()` doc. (GH-25643) https://github.com/python/cpython/commit/5c84bb506aaca01f5f750116d8f7a41d41f8124d --

[issue37751] In codecs, function 'normalizestring' should convert both spaces and hyphens to underscores.

Change by miss-islington : -- pull_requests: +24368 pull_request: https://github.com/python/cpython/pull/25677 ___ Python tracker ___

[issue37751] In codecs, function 'normalizestring' should convert both spaces and hyphens to underscores.

STINNER Victor added the comment: New changeset 531c81038e28b6cfa0f9791467bf671c88c6f4c4 by Miss Islington (bot) in branch '3.9': bpo-37751: Document codecs.lookup() change in What's New in Python 3.9 (GH-23096) (GH-25659)

[issue37751] In codecs, function 'normalizestring' should convert both spaces and hyphens to underscores.

Change by miss-islington : -- nosy: +miss-islington nosy_count: 8.0 -> 9.0 pull_requests: +24350 pull_request: https://github.com/python/cpython/pull/25659 ___ Python tracker

[issue37751] In codecs, function 'normalizestring' should convert both spaces and hyphens to underscores.

Change by Inada Naoki : -- pull_requests: +24336 pull_request: https://github.com/python/cpython/pull/25643 ___ Python tracker ___

[issue37751] In codecs, function 'normalizestring' should convert both spaces and hyphens to underscores.

hai shi added the comment: Thanks Marc-Andre for your supplement of PEP-100. Thanks Inada, victor for your review and merge. After PR-23096 merged, I suggest to close this bpo. If there have any other questions, we can reopen it again. -- resolution: -> fixed stage: patch review ->

[issue37751] In codecs, function 'normalizestring' should convert both spaces and hyphens to underscores.

STINNER Victor added the comment: New changeset 32980fb669a6857276da18895fcc0cb6f6fbb544 by Hai Shi in branch 'master': bpo-37751: Document codecs.lookup() change in What's New in Python 3.9 (GH-23096) https://github.com/python/cpython/commit/32980fb669a6857276da18895fcc0cb6f6fbb544

[issue37751] In codecs, function 'normalizestring' should convert both spaces and hyphens to underscores.

Change by Miro Hrončok : -- nosy: -hroncok ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue37751] In codecs, function 'normalizestring' should convert both spaces and hyphens to underscores.

Marc-Andre Lemburg added the comment: On 23.04.2021 07:47, Inada Naoki wrote: > > Inada Naoki added the comment: > > I think it is too late. Python 3.9 has been released already. Reverting the > change is also breaking change. > > PEP 100 says: > "Search functions are expected to take

[issue37751] In codecs, function 'normalizestring' should convert both spaces and hyphens to underscores.

Marc-Andre Lemburg added the comment: On 23.04.2021 03:37, akdor1154 wrote: > > akdor1154 added the comment: > > If I understand the target of this issue, this is a breaking change in python > 3.9 . > > E.g. see https://github.com/SAP/PyHDB/issues/149 > > Ideally if we did not intend to

[issue37751] In codecs, function 'normalizestring' should convert both spaces and hyphens to underscores.

Inada Naoki added the comment: codecs.register() was added in this commit. https://github.com/python/cpython/commit/e2d67f98d1aade1059b2ff3278672b2ffbaf180e And its docstring has been added in this commit. https://github.com/python/cpython/commit/0ae2981dec3de96a1f7d63b0535992cf1462ac92 Both

[issue37751] In codecs, function 'normalizestring' should convert both spaces and hyphens to underscores.

Inada Naoki added the comment: I think it is too late. Python 3.9 has been released already. Reverting the change is also breaking change. PEP 100 says: "Search functions are expected to take one argument, the encoding name in all lower case letters and with hyphens and spaces converted to

[issue37751] In codecs, function 'normalizestring' should convert both spaces and hyphens to underscores.

hai shi added the comment: >Ideally if we did not intend to break libraries then can this be fixed? Or if it is acceptable to have such a breaking change. Hi, akdor1154, thanks for your notice. It was a bugfix in fact:) https://bugs.python.org/issue37751#msg349448 > maybe this is

[issue37751] In codecs, function 'normalizestring' should convert both spaces and hyphens to underscores.

akdor1154 added the comment: If I understand the target of this issue, this is a breaking change in python 3.9 . E.g. see https://github.com/SAP/PyHDB/issues/149 Ideally if we did not intend to break libraries then can this be fixed? Or if it is acceptable to have such a breaking change,

[issue37751] In codecs, function 'normalizestring' should convert both spaces and hyphens to underscores.

Change by hai shi : -- pull_requests: +22010 stage: resolved -> patch review pull_request: https://github.com/python/cpython/pull/23096 ___ Python tracker ___

[issue37751] In codecs, function 'normalizestring' should convert both spaces and hyphens to underscores.

STINNER Victor added the comment: I created bpo-39337: codecs.lookup() ignores non-ASCII characters, whereas encodings.normalize_encoding() copies them. -- ___ Python tracker

[issue37751] In codecs, function 'normalizestring' should convert both spaces and hyphens to underscores.

STINNER Victor added the comment: > Please note that external codec packages should not rely on the semantics of > the Python stdlib encodings package's search function. latexcodec does register a search function. > It's good practice to always only use ASCII lower case chars and the >

[issue37751] In codecs, function 'normalizestring' should convert both spaces and hyphens to underscores.

Marc-Andre Lemburg added the comment: Just to clarify: the change in the C implementation was the breaking change. The patch just restores the previous behavior: https://github.com/python/cpython/blob/master/Lib/encodings/__init__.py#L43 Please note that external codec packages should not

[issue37751] In codecs, function 'normalizestring' should convert both spaces and hyphens to underscores.

STINNER Victor added the comment: It seems quite easy to update latexcodec project to support Python 3.9. I proposed a solution there: https://bugzilla.redhat.com/show_bug.cgi?id=1789613#c6 -- ___ Python tracker

[issue37751] In codecs, function 'normalizestring' should convert both spaces and hyphens to underscores.

STINNER Victor added the comment: > The change is backwards incompatible and a backport would break things. See > for example how it breaks latexcodec: I reopen the issue. I proposed PR 17997 to *document* the incompatible change in What's New in Python 3.8. IMO it's a deliberate change and

[issue37751] In codecs, function 'normalizestring' should convert both spaces and hyphens to underscores.

Change by STINNER Victor : -- pull_requests: +17401 pull_request: https://github.com/python/cpython/pull/17997 ___ Python tracker ___

[issue37751] In codecs, function 'normalizestring' should convert both spaces and hyphens to underscores.

Miro Hrončok added the comment: The change is backwards incompatible and a backport would break things. See for example how it breaks latexcodec: https://bugzilla.redhat.com/show_bug.cgi?id=1789613#c2 -- nosy: +hroncok ___ Python tracker

[issue37751] In codecs, function 'normalizestring' should convert both spaces and hyphens to underscores.

Jordon.X <9651...@qq.com> added the comment: Thanks vstinner. I also don't think it's necessary to backport to the old version. Close this issue is fine. -- ___ Python tracker

[issue37751] In codecs, function 'normalizestring' should convert both spaces and hyphens to underscores.

STINNER Victor added the comment: Thanks for the fix Jordon Xu. IMHO this change is not strictly a bugfix, but more like an enhancement. I close the issue. If you consider that a backport to Python 3.7 and 3.8 is needed, please say so. -- resolution: -> fixed stage: patch review

[issue37751] In codecs, function 'normalizestring' should convert both spaces and hyphens to underscores.

STINNER Victor added the comment: New changeset 20f59fe1f7748ae899aceee4cb560e5e1f528a1f by Victor Stinner (Jordon Xu) in branch 'master': bpo-37751: Fix codecs.lookup() normalization (GH-15092) https://github.com/python/cpython/commit/20f59fe1f7748ae899aceee4cb560e5e1f528a1f --

[issue37751] In codecs, function 'normalizestring' should convert both spaces and hyphens to underscores.

Marc-Andre Lemburg added the comment: Jordon is right. Conversion has to be to underscores, not hyphens. I guess this bug was introduced when the normalization function was converted to C. -- nosy: +lemburg ___ Python tracker

[issue37751] In codecs, function 'normalizestring' should convert both spaces and hyphens to underscores.

Jordon.X <9651...@qq.com> added the comment: The design and code of the following four places need to be consistent, No.1 https://github.com/python/peps/blob/master/pep-0100.txt#L292 No.2 https://github.com/python/cpython/blob/master/Python/codecs.c#L113 No.3

[issue37751] In codecs, function 'normalizestring' should convert both spaces and hyphens to underscores.

hai shi added the comment: Hm, there is a bit misleading between desc(https://github.com/python/cpython/blob/master/Python/codecs.c#L53) and the code (https://github.com/python/cpython/blob/master/Python/codecs.c#L74). -- nosy: +shihai1991 ___

[issue37751] In codecs, function 'normalizestring' should convert both spaces and hyphens to underscores.

Change by Jordon.X <9651...@qq.com>: -- keywords: +patch pull_requests: +14838 stage: -> patch review pull_request: https://github.com/python/cpython/pull/15092 ___ Python tracker

[issue37751] In codecs, function 'normalizestring' should convert both spaces and hyphens to underscores.

Jordon.X <9651...@qq.com> added the comment: and I will try to fix it. -- ___ Python tracker ___ ___ Python-bugs-list mailing list

[issue37751] In codecs, function 'normalizestring' should convert both spaces and hyphens to underscores.

New submission from Jordon.X <9651...@qq.com>: In codecs.c, when _PyCodec_Lookup() call normalizestring(), both spaces and hyphens should be convered to underscores. Not convert spaces to hyphens. see:https://github.com/python/peps/blob/master/pep-0100.txt, Codecs (Coder/Decoders) Lookup