[issue20087] Mismatch between glibc and X11 locale.alias

Marc-Andre Lemburg added the comment: I believe we can close this old issue. The discussion was certainly a useful one. I guess we should stop updating the alias table automatically and instead add new aliases or change existing ones based on more research and using the X11 files as well as

[issue20087] Mismatch between glibc and X11 locale.alias

Marc-Andre Lemburg added the comment: Thanks, Serhiy. -- ___ Python tracker ___ ___

[issue20087] Mismatch between glibc and X11 locale.alias

Serhiy Storchaka added the comment: New changeset a55ac801f749a731250f3c7c1db7d546d22ae032 by Serhiy Storchaka in branch '2.7': [2.7] bpo-20087: Update locale alias mapping with glibc 2.27 supported locales. (GH-6708). (GH-6717)

[issue20087] Mismatch between glibc and X11 locale.alias

Serhiy Storchaka added the comment: New changeset 6049bda21b607acc90bbabcc604997e794e8aee1 by Serhiy Storchaka (Miss Islington (bot)) in branch '3.7': [3.7] bpo-20087: Update locale alias mapping with glibc 2.27 supported locales. (GH-6708) (GH-6713)

[issue20087] Mismatch between glibc and X11 locale.alias

Serhiy Storchaka added the comment: New changeset b1c70d0ffbb235def1deab62a744ffd9b5253924 by Serhiy Storchaka (Miss Islington (bot)) in branch '3.6': [3.6] bpo-20087: Update locale alias mapping with glibc 2.27 supported locales. (GH-6708) (GH-6714)

[issue20087] Mismatch between glibc and X11 locale.alias

Change by Serhiy Storchaka : -- pull_requests: +6409 ___ Python tracker ___ ___

[issue20087] Mismatch between glibc and X11 locale.alias

Change by miss-islington : -- pull_requests: +6407 ___ Python tracker ___

[issue20087] Mismatch between glibc and X11 locale.alias

Change by miss-islington : -- pull_requests: +6406 ___ Python tracker ___

[issue20087] Mismatch between glibc and X11 locale.alias

Serhiy Storchaka added the comment: New changeset cedc9b74202d8c1ae39bca261cbb45d42ed54d45 by Serhiy Storchaka in branch 'master': bpo-20087: Update locale alias mapping with glibc 2.27 supported locales. (ПР-6708)

[issue20087] Mismatch between glibc and X11 locale.alias

Serhiy Storchaka added the comment: Benjamin's patch did two things: 1) made the glibc alias table taking precedence over the X11 one; 2) updated the alias mapping with new glibc. The first part is controversial, but updating the alias mapping with new glibc is

[issue20087] Mismatch between glibc and X11 locale.alias

Change by Serhiy Storchaka : -- keywords: +patch pull_requests: +6401 stage: test needed -> patch review ___ Python tracker

[issue20087] Mismatch between glibc and X11 locale.alias

Licht Takeuchi added the comment: Hi all, The locale in the latest Ubuntu 18.04 contains en_IL as valid locale, but Python cannot resolve this. This makes test failure in pandas. https://github.com/pandas-dev/pandas/issues/20957 en_IL has significant impact because this

[issue20087] Mismatch between glibc and X11 locale.alias

Benjamin Peterson added the comment: New changeset 02371e0ed1ee82ec73e7d363bcf2ed40cde1397a by Benjamin Peterson in branch 'master': make the glibc alias table take precedence over the X11 one (#422) https://github.com/python/cpython/commit/02371e0ed1ee82ec73e7d363bcf2ed40cde1397a --

[issue20087] Mismatch between glibc and X11 locale.alias

Benjamin Peterson added the comment: New changeset df8280838f52d6ec45ba03ef734b0dec8a9c43fb by Benjamin Peterson in branch 'master': bpo-20087: Revert "make the glibc alias table take precedence over the X11 one (#422)" (#713)

[issue20087] Mismatch between glibc and X11 locale.alias

Changes by Benjamin Peterson : -- pull_requests: +633 ___ Python tracker ___ ___

[issue20087] Mismatch between glibc and X11 locale.alias

Changes by Serhiy Storchaka : -- pull_requests: -602 ___ Python tracker ___ ___

[issue20087] Mismatch between glibc and X11 locale.alias

Marc-Andre Lemburg added the comment: The main purpose of the alias table is to support normalization and this is used for getdefaultencoding() which was created to be able to determine the default encoding based on what X.org uses as default without doing temporary setlocale() tricks. Now,

[issue20087] Mismatch between glibc and X11 locale.alias

Changes by Larry Hastings : -- pull_requests: +602 ___ Python tracker ___ ___

[issue20087] Mismatch between glibc and X11 locale.alias

Benjamin Peterson added the comment: I'm still confused about what getlocale() is supposed to do. Why do we attempt to return an encoding anyway if the underlying setlocale call doesn't return one? Is getlocale() not supposed to a simple wrapper over the C locale? If not, how is one supposed

[issue20087] Mismatch between glibc and X11 locale.alias

Serhiy Storchaka added the comment: I'm feeling there is something wrong with the current locale design. See issues issue504219, issue10466, issue20088, issue25191, issue29571. -- ___ Python tracker

[issue20087] Mismatch between glibc and X11 locale.alias

Marc-Andre Lemburg added the comment: On 10.03.2017 08:37, Benjamin Peterson wrote: > > Do you believe this program should work? > > import locale, os > for l in open("/usr/share/i18n/SUPPORTED"): > alias, encoding = l.strip().split() > locale.setlocale(locale.LC_ALL, alias) > try:

[issue20087] Mismatch between glibc and X11 locale.alias

Benjamin Peterson added the comment: Do you believe this program should work? import locale, os for l in open("/usr/share/i18n/SUPPORTED"): alias, encoding = l.strip().split() locale.setlocale(locale.LC_ALL, alias) try: enc = locale.getlocale()[1] except ValueError:

[issue20087] Mismatch between glibc and X11 locale.alias

Serhiy Storchaka added the comment: The original issue is issue29571. The locale module returned encoding ISO8859-1 for locale en_IN (as in the X11 locale alias map), but glibc uses UTF-8 (as in glibc SUPPORT file). -- ___ Python tracker

[issue20087] Mismatch between glibc and X11 locale.alias

Marc-Andre Lemburg added the comment: On 09.03.2017 11:47, Serhiy Storchaka wrote: > > The SUPPORTED file from glibc is used for determining the default encoding > for locales that don't include it explicitly. For example en_IN uses UTF-8 > rather than ISO8859-1. No, the glibc locales don't

[issue20087] Mismatch between glibc and X11 locale.alias

Serhiy Storchaka added the comment: The SUPPORTED file from glibc is used for determining the default encoding for locales that don't include it explicitly. For example en_IN uses UTF-8 rather than ISO8859-1. -- ___ Python tracker

[issue20087] Mismatch between glibc and X11 locale.alias

Serhiy Storchaka added the comment: > Why is the X11 locale alias map used at all? It seems like it can only create > confusion with libc. Originally only the X11 locale alias map was used. The support of the glibc locale alias map was added 2.5 years ago (issue20079). --

[issue20087] Mismatch between glibc and X11 locale.alias

Marc-Andre Lemburg added the comment: On 09.03.2017 08:15, Benjamin Peterson wrote: > > "eo_XX" is just something that appears in the X11 locale.alias file. My > change doesn't add that; it was already there. (for Esperanto, which I > suppose explains the "XX") Yes, I know. That was an

[issue20087] Mismatch between glibc and X11 locale.alias

Benjamin Peterson added the comment: "eo_XX" is just something that appears in the X11 locale.alias file. My change doesn't add that; it was already there. (for Esperanto, which I suppose explains the "XX") Most of the changes you identify the glibc aliases taking precedence over the X11

[issue20087] Mismatch between glibc and X11 locale.alias

Marc-Andre Lemburg added the comment: Why was the PR merged while we were still discussing it ? -- ___ Python tracker ___

[issue20087] Mismatch between glibc and X11 locale.alias

Marc-Andre Lemburg added the comment: On 08.03.2017 10:37, Serhiy Storchaka wrote: > > The problem is that that table can get incorrect result for non-Linux > platforms (or for Linux with old glibc). Sure, it's a best effort approach. Also note that on today's systems you often don't have

[issue20087] Mismatch between glibc and X11 locale.alias

Marc-Andre Lemburg added the comment: On 08.03.2017 07:27, Benjamin Peterson wrote: > > Why is the X11 locale alias map used at all? It seems like it can only create > confusion with libc. Because it was the only such maintained mapping available at the time. It's also used for the X.org

[issue20087] Mismatch between glibc and X11 locale.alias

Serhiy Storchaka added the comment: The problem is that that table can get incorrect result for non-Linux platforms (or for Linux with old glibc). -- ___ Python tracker

[issue20087] Mismatch between glibc and X11 locale.alias

Marc-Andre Lemburg added the comment: On 08.03.2017 08:20, Serhiy Storchaka wrote: > > Serhiy Storchaka added the comment: > > Not all platforms use glibc 2.24 as libc. True. Many don't even use glibc. > Ideally most of entries should even not exist. We should ask libc for the > default

[issue20087] Mismatch between glibc and X11 locale.alias

Serhiy Storchaka added the comment: Not all platforms use glibc 2.24 as libc. Ideally most of entries should even not exist. We should ask libc for the default encoding if it is not included in the locale name. The aliases table should be used only for mapping commonly used but unsupported by

[issue20087] Mismatch between glibc and X11 locale.alias

Benjamin Peterson added the comment: Why is the X11 locale alias map used at all? It seems like it can only create confusion with libc. -- nosy: +benjamin.peterson ___ Python tracker

[issue20087] Mismatch between glibc and X11 locale.alias

Marc-Andre Lemburg added the comment: On 07.03.2017 18:23, Serhiy Storchaka wrote: > > Serhiy Storchaka added the comment: > >> 'cy_GB.ISO8859-1' to 'cy_GB.ISO8859-14' > > Looks as just fixing an error. The default West-European ISO8859-1 is changed > to Celtic cy_GB.ISO8859-14. This looks

[issue20087] Mismatch between glibc and X11 locale.alias

Serhiy Storchaka added the comment: > 'cy_GB.ISO8859-1' to 'cy_GB.ISO8859-14' Looks as just fixing an error. The default West-European ISO8859-1 is changed to Celtic cy_GB.ISO8859-14. This looks better option for Welsh. > 'tg_TJ.KOI8-C' to 'tg_TJ.KOI8-T' KOI8-C is not supported by Python,

[issue20087] Mismatch between glibc and X11 locale.alias

Marc-Andre Lemburg added the comment: I agree that it's reasonable to have glibc's aliases override the X.org ones, but this patch makes some pretty significant changes to Python's default assumptions with respect to default encodings for several locales. While some changes obviously make

[issue20087] Mismatch between glibc and X11 locale.alias

Changes by Serhiy Storchaka : -- stage: -> test needed versions: +Python 3.6, Python 3.7 -Python 3.4 ___ Python tracker ___

[issue20087] Mismatch between glibc and X11 locale.alias

Serhiy Storchaka added the comment: Needed a test for few common locales (en_IN, ru_RU) and maybe for unusual locales (uz_uz, uz_uz@cyrillic). I would prefer to have a separate issue that updates the aliases table to glibc 2.24. -- ___ Python

[issue20087] Mismatch between glibc and X11 locale.alias

Changes by Benjamin Peterson : -- pull_requests: +351 ___ Python tracker ___ ___

[issue20087] Mismatch between glibc and X11 locale.alias

Changes by Serhiy Storchaka storch...@gmail.com: -- versions: +Python 3.5 -Python 3.3 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue20087 ___ ___

[issue20087] Mismatch between glibc and X11 locale.alias

New submission from Serhiy Storchaka: The locale module uses locale alias table derived from X11 locale.alias file for mapping bare locale names without encodings to locale names with encodings. However sometimes glibc default encoding for a locale differs from that used in X11 locale.alias.

[issue20087] Mismatch between glibc and X11 locale.alias

Changes by Arfrever Frehtes Taifersar Arahesis arfrever@gmail.com: -- nosy: +Arfrever ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue20087 ___