[issue28180] Implementation of the PEP 538: coerce C locale to C.utf-8

[Matej Cepl]

Matej Cepl  added the comment:

Thank you very much for the hint. Do I have to include the patch for bpo-19977 
only (that would be easy), or also all twelve PRs for bpo-29240 (that would 
probably broke my will to do it)?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28180] Implementation of the PEP 538: coerce C locale to C.utf-8

[Nick Coghlan]

Nick Coghlan  added the comment:

The test cases for locale coercion *not* triggering still assume that 
bpo-19977, using surrogateescape on the standard streams in the POSIX locale, 
has been implemented (since that was implemented in Python 3.5).

Hence the various test cases complaining that they found "ascii:strict" (Py 3.4 
behaviour without bpo-19977) where they expected "ascii:surrogateescape" (the 
Py 3.5+ behaviour *with* bpo-19977).

To get a PEP 538 backport to work as intended on 3.4, you'd need to backport 
that earlier IO stream error handling change as well.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28180] Implementation of the PEP 538: coerce C locale to C.utf-8

[STINNER Victor]

STINNER Victor  added the comment:

RHEL 7.7 and RHEL 8 provides Python 3.6. PEP 538 was implemented in Python 3.7. 
PEP 538 feature was backported in RHEL 7.7 and RHEL 8 Python 3.6.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28180] Implementation of the PEP 538: coerce C locale to C.utf-8

[Matej Cepl]

Matej Cepl  added the comment:

> Python 3.4 is no longer supported upstream. Python 3 got tons of Unicode 
> fixes between Python 3.4 and Python 3.8.

Of course, I know that, but I just didn’t want to throw all my effort away, 
when I spent some hours on making it. And I guess, there may be somebody else 
who cares for 3.4 (ehm, RHEL-7 has 3.3, doesn’t it?).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28180] Implementation of the PEP 538: coerce C locale to C.utf-8

[STINNER Victor]

STINNER Victor  added the comment:

Python 3.4 is no longer supported upstream. Python 3 got tons of Unicode fixes 
between Python 3.4 and Python 3.8.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28180] Implementation of the PEP 538: coerce C locale to C.utf-8

[Matej Cepl]

Matej Cepl  added the comment:

I have tried to port this patch to Python 3.4 (still maintained by SUSE on 
SLE-12), but I have the hardest time to debug this. All affected tests end with 
errors like this:

[  493s] ==
[  493s] FAIL: test_test_PYTHONCOERCECLOCALE_not_set 
(test.test_c_locale_coercion.LocaleCoercionTests) (PYTHONCOERCECLOCALE=None, 
env_var='LC_CTYPE', nominal_locale='invalid.ascii')
[  493s] --
[  493s] Traceback (most recent call last):
[  493s]   File 
"/home/abuild/rpmbuild/BUILD/Python-3.4.10/Lib/test/test_c_locale_coercion.py", 
line 326, in _check_c_locale_coercion
[  493s] coercion_expected)
[  493s]   File 
"/home/abuild/rpmbuild/BUILD/Python-3.4.10/Lib/test/test_c_locale_coercion.py", 
line 219, in _check_child_encoding_details
[  493s] self.assertEqual(encoding_details, expected_details)
[  493s] AssertionError: {'fse[79 chars]cii:strict', 'stderr_info': 
'ascii:backslashre[45 chars]ict'} != {'fse[79 chars]cii:surrogateescape', 
'stderr_info': 'ascii:ba[63 chars]ape'}
[  493s]   {'fsencoding': 'ascii',
[  493s]'lang': '',
[  493s]'lc_all': '',
[  493s]'lc_ctype': 'invalid.ascii',
[  493s]'stderr_info': 'ascii:backslashreplace',
[  493s] -  'stdin_info': 'ascii:strict',
[  493s] ? ^^ ^
[  493s] 
[  493s] +  'stdin_info': 'ascii:surrogateescape',
[  493s] ?++ ^^^ ^^^
[  493s] 
[  493s] -  'stdout_info': 'ascii:strict'}
[  493s] ?  ^^ ^
[  493s] 
[  493s] +  'stdout_info': 'ascii:surrogateescape'}
[  493s] ? ++ ^^^ ^^^

yes, it is always a conflict between strict and surrogateescape. I probably 
don’t have time to finish debugging this, so I am just leaving this for 
posterity.

--
nosy: +mcepl
Added file: 
https://bugs.python.org/file48991/pep538_coerce_legacy_c_locale.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28180] Implementation of the PEP 538: coerce C locale to C.utf-8

[Nick Coghlan]

Nick Coghlan  added the comment:

Given that issue 32002 and issue 30672 track the known challenges in testing 
the expected locale coercion behaviour reliably, I'm going to go ahead and 
close this overall implementation issue (the feature is there, and works in a 
way we're happy with, we're just encountering some challenges clearly 
expressing those expectations as a regression test).

--
dependencies:  -PEP 538: Unexpected locale behaviour on *BSD (including Mac OS 
X)
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28180] Implementation of the PEP 538: coerce C locale to C.utf-8

[Xavier de Gaye]

Xavier de Gaye  added the comment:


New changeset 1588be66d7b0eeebc4614309cd0fc837ff52776a by xdegaye in branch 
'master':
bpo-28180: Fix the implementation of PEP 538 on Android (GH-4334)
https://github.com/python/cpython/commit/1588be66d7b0eeebc4614309cd0fc837ff52776a


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28180] Implementation of the PEP 538: coerce C locale to C.utf-8

[Xavier de Gaye]

Xavier de Gaye  added the comment:

PR 4334 added: fix the implementation of PEP 538 on Android.

The current implementation of PEP 538 fixes issue 28997 without the locale 
coercion for Android added by PR 4334, see msg305848.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28180] Implementation of the PEP 538: coerce C locale to C.utf-8

[Xavier de Gaye]

Change by Xavier de Gaye :


--
pull_requests: +4289
stage:  -> patch review

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28180] Implementation of the PEP 538: coerce C locale to C.utf-8

[Nick Coghlan]

Nick Coghlan added the comment:

The latest commit should get the Mac OS X buildbot back to green, but I had to 
disable some test cases to do it - see issue 30672 for details.

Issue 30565 is the one that covers silencing the locale coercion and locale 
compatibility warnings by default.

--
dependencies: +PEP 538: Unexpected locale behaviour on Mac OS X, PEP 538: 
default to skipping warning for implicit locale coercion?
stage: resolved -> 

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28180] Implementation of the PEP 538: coerce C locale to C.utf-8

[Nick Coghlan]

Nick Coghlan added the comment:


New changeset 7926516ff95ed9c8345ed4c4c4910f44ffbd5949 by Nick Coghlan in 
branch 'master':
bpo-28180: Standard stream & FS encoding differ on Mac OS X (GH-2208)
https://github.com/python/cpython/commit/7926516ff95ed9c8345ed4c4c4910f44ffbd5949


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28180] Implementation of the PEP 538: coerce C locale to C.utf-8

[Nick Coghlan]

Changes by Nick Coghlan :


--
pull_requests: +2252

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28180] Implementation of the PEP 538: coerce C locale to C.utf-8

[Nick Coghlan]

Nick Coghlan added the comment:

Ah, I finally understand Victor's comment on my initial attempt at fixing the 
tests on Mac OS X - the standard streams *don't* use the filesystem encoding, 
so they default to ASCII in the C locale, even on Mac OS X.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28180] Implementation of the PEP 538: coerce C locale to C.utf-8

[Nick Coghlan]

Nick Coghlan added the comment:

I've added dependencies for PEP 538 induced testing problems that have been 
broken out into their own issues.

I've also merged my attempt at fixing the tests on Mac OS X.

Something that's included in that patch is an implicit skip of the "LANG=UTF-8" 
case when checking external locale configuration. I expected that to behave the 
same way as "LC_CTYPE=UTF-8", but instead it's behaving more like "LC_CTYPE=C".

--
dependencies: +CODESET error on AMD64 FreeBSD 10.x Shared 3.x caused by the PEP 
538, Leak in test_c_locale_coercion

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28180] Implementation of the PEP 538: coerce C locale to C.utf-8

[Nick Coghlan]

Nick Coghlan added the comment:


New changeset 4563099d28e832aed22b85ce7e2a92236df03847 by Nick Coghlan in 
branch 'master':
bpo-28180: assume UTF-8 for Mac OS X PEP 538 tests (GH-2130)
https://github.com/python/cpython/commit/4563099d28e832aed22b85ce7e2a92236df03847


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28180] Implementation of the PEP 538: coerce C locale to C.utf-8

[STINNER Victor]

STINNER Victor added the comment:


New changeset eb52ac89929bb09b15c014ab8ff60eee685e86c7 by Victor Stinner in 
branch 'master':
bpo-28180: Fix test_capi.test_forced_io_encoding() (#2155)
https://github.com/python/cpython/commit/eb52ac89929bb09b15c014ab8ff60eee685e86c7


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28180] Implementation of the PEP 538: coerce C locale to C.utf-8

[STINNER Victor]

Changes by STINNER Victor :


--
pull_requests: +2205

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28180] Implementation of the PEP 538: coerce C locale to C.utf-8

[STINNER Victor]

Changes by STINNER Victor :


--
title: sys.getfilesystemencoding() should default to utf-8 -> Implementation of 
the PEP 538: coerce C locale to C.utf-8

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com