[issue34589] Py_Initialize() and Py_Main() should not enable C locale coercion

Nick Coghlan added the comment: I'm inclined to leave 3.7 alone unless/until we get an actual bug report for the current behaviour - if an embedder finds that it just straight up doesn't work for them, then they can either resort to tinkering with the private APIs (since they won't change

[issue34589] Py_Initialize() and Py_Main() should not enable C locale coercion

STINNER Victor added the comment: I fixed the issue in Python 3.8 with bpo-36443: commit d929f1838a8fba881ff0148b7fc31f6265703e3d Author: Victor Stinner Date: Wed Mar 27 18:28:46 2019 +0100 bpo-36443: Disable C locale coercion and UTF-8 Mode by default (GH-12589) I'm not sure if it

[issue34589] Py_Initialize() and Py_Main() should not enable C locale coercion

Nick Coghlan added the comment: I still think the current Python 3.7 behaviour makes the CPython runtime a badly behaved C library, since we may end up changing the active libc locale even when it isn't the main application, and a change hasn't been explicitly requested by the embedding

[issue34589] Py_Initialize() and Py_Main() should not enable C locale coercion

Change by Kubilay Kocak : -- nosy: +koobs ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue34589] Py_Initialize() and Py_Main() should not enable C locale coercion

STINNER Victor added the comment: C locale coercion is just one parameter used to select the Python filesystem enconding. There are many others which are read later during Python initialization, and they are inter-dependent.

[issue34589] Py_Initialize() and Py_Main() should not enable C locale coercion

Nick Coghlan added the comment: If I've understood https://bugs.python.org/issue35290 and its resolution in https://github.com/python/cpython/commit/f6e323ce322cf54b1a9e9252b13f93ebc28b5c24 correctly, that's a concrete example of why I consider the early coercion approach that avoids any

[issue34589] Py_Initialize() and Py_Main() should not enable C locale coercion

Nick Coghlan added the comment: The discussion with Victor on https://bugs.python.org/issue34914 highlighted the fact that it's OK to use 8-bit string comparisons to check for "-E", "-I", and a "-X coerce_legacy_c_locale=0" due to the fact that all encodings used as locale encodings are

[issue34589] Py_Initialize() and Py_Main() should not enable C locale coercion

Change by STINNER Victor : -- nosy: -vstinner ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue34589] Py_Initialize() and Py_Main() should not enable C locale coercion

Nick Coghlan added the comment: Correct - it won't change anything from 3.7.0, and even the original discrepancies relative to PEP 538 only affect applications that: 1. Are embedding a CPython runtime 2. *Aren't* already ensuring that they're running in a locale other than the C locale 3.

[issue34589] Py_Initialize() and Py_Main() should not enable C locale coercion

Ned Deily added the comment: Thanks, Victor and Nick, for the work. With the revert merged (GH-9430), this issue is no longer a release blocker for 3.7.1, correct? -- ___ Python tracker

[issue34589] Py_Initialize() and Py_Main() should not enable C locale coercion

Nick Coghlan added the comment: (Also, the patch is currently still written on the assumption that it won't be backported to 3.7.1. I haven't checked if it would apply cleanly to 3.7.x, but I suspect not, given the magnitude of the startup changes targeting 3.8.0 since 3.7.0 was released)

[issue34589] Py_Initialize() and Py_Main() should not enable C locale coercion

Nick Coghlan added the comment: https://github.com/python/cpython/pull/9257 should be complete now, although I may have gone overboard on the documentation updates (as it seems to be unclear, at least to Victor, why PYTHONCOERCECLOCALE exists in the first place) --

[issue34589] Py_Initialize() and Py_Main() should not enable C locale coercion

STINNER Victor added the comment: New changeset 06e7608207daab9fb82d13ccf2d3664535442f11 by Victor Stinner in branch 'master': Revert "bpo-34589: Add -X coerce_c_locale command line option (GH-9378)" (GH-9430) https://github.com/python/cpython/commit/06e7608207daab9fb82d13ccf2d3664535442f11

[issue34589] Py_Initialize() and Py_Main() should not enable C locale coercion

Change by STINNER Victor : -- pull_requests: +8847 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue34589] Py_Initialize() and Py_Main() should not enable C locale coercion

STINNER Victor added the comment: New changeset 95cc3ee00cfa079751ae2bb9a8d3387053b50489 by Victor Stinner in branch '3.7': Revert "[3.7] bpo-34589: Add -X coerce_c_locale option; C locale coercion off by default (GH-9379)" (GH-9416)

[issue34589] Py_Initialize() and Py_Main() should not enable C locale coercion

STINNER Victor added the comment: The issue is now discussed on python-dev. My latest email: https://mail.python.org/pipermail/python-dev/2018-September/155242.html -- ___ Python tracker

[issue34589] Py_Initialize() and Py_Main() should not enable C locale coercion

Change by STINNER Victor : -- pull_requests: +8836 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue34589] Py_Initialize() and Py_Main() should not enable C locale coercion

Nick Coghlan added the comment: (The other reason this change should be reverted is because it added a new feature to Python 3.7.1, even though in the earlier discussion we had agreed to leave the as-shipped implementation in 3.7.0 alone for the rest of the Python 3.7.x releases, and only

[issue34589] Py_Initialize() and Py_Main() should not enable C locale coercion

Nick Coghlan added the comment: test_c_locale_coercion *did* test the -E and -I options, by running everything in isolated mode. This was only broken by Victor's changes to the test suite (which broke locale coercion in isolated mode, and hence broke the tests). As for why

[issue34589] Py_Initialize() and Py_Main() should not enable C locale coercion

Nick Coghlan added the comment: (Note: I won't have time to work on this myself until this weekend at the earliest) -- ___ Python tracker ___

[issue34589] Py_Initialize() and Py_Main() should not enable C locale coercion

Nick Coghlan added the comment: The only reason this got through my original review was because PEP 540 was implemented when I was going through a personal situation that lead to me quitting my job cold without a new one to go to, so my review of Victors changes to the PEP 538

[issue34589] Py_Initialize() and Py_Main() should not enable C locale coercion

Nick Coghlan added the comment: 3.7.1 should not ship until PEP 538 is once more implemented as documented, without Victor's personal editorialising and feature additions to a maintenance release. -- ___ Python tracker

[issue34589] Py_Initialize() and Py_Main() should not enable C locale coercion

Nick Coghlan added the comment: Victor, no - you've completely broken PEP 538 now. Please just give up, and implement the PEP as written, and stop trying to use your copious amounts of available development time to railroad me. -- ___ Python

[issue34589] Py_Initialize() and Py_Main() should not enable C locale coercion

Change by Ned Deily : -- versions: +Python 3.7 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue34589] Py_Initialize() and Py_Main() should not enable C locale coercion

STINNER Victor added the comment: test_c_locale_coercion didn't test the -E option: I wrote PR 9382 to test it. -- ___ Python tracker ___

[issue34589] Py_Initialize() and Py_Main() should not enable C locale coercion

Change by STINNER Victor : -- pull_requests: +8806 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue34589] Py_Initialize() and Py_Main() should not enable C locale coercion

Ned Deily added the comment: I'm marking this as a 3.7.1 "release blocker" until Nick has had a chance to review the merged PR and he and Victor are in agreement that this is a satisfactory approach for 3.7.1. -- ___ Python tracker

[issue34589] Py_Initialize() and Py_Main() should not enable C locale coercion

Change by Ned Deily : -- priority: normal -> release blocker ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue34589] Py_Initialize() and Py_Main() should not enable C locale coercion

STINNER Victor added the comment: New changeset 144f1e2c6f4a24bd288c045986842c65cc289684 by Victor Stinner in branch '3.7': [3.7] bpo-34589: Add -X coerce_c_locale option; C locale coercion off by default (GH-9379)

[issue34589] Py_Initialize() and Py_Main() should not enable C locale coercion

Change by STINNER Victor : -- pull_requests: +8803 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue34589] Py_Initialize() and Py_Main() should not enable C locale coercion

STINNER Victor added the comment: New changeset dbdee0073cf0b88fe541980ace1f650900f455cc by Victor Stinner in branch 'master': bpo-34589: Add -X coerce_c_locale command line option (GH-9378) https://github.com/python/cpython/commit/dbdee0073cf0b88fe541980ace1f650900f455cc --

[issue34589] Py_Initialize() and Py_Main() should not enable C locale coercion

Change by STINNER Victor : -- pull_requests: +8802 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue34589] Py_Initialize() and Py_Main() should not enable C locale coercion

STINNER Victor added the comment: New changeset 7a0791b6992d420dc52536257f2f093851ed7215 by Victor Stinner in branch 'master': bpo-34589: C locale coercion off by default (GH-9073) https://github.com/python/cpython/commit/7a0791b6992d420dc52536257f2f093851ed7215 --

[issue34589] Py_Initialize() and Py_Main() should not enable C locale coercion

STINNER Victor added the comment: New changeset 188ebfa475a6f6aa2d0ea14ca8e1fbe7865b6d27 by Victor Stinner in branch 'master': bpo-34589: Make _PyCoreConfig.coerce_c_locale private (GH-9371) https://github.com/python/cpython/commit/188ebfa475a6f6aa2d0ea14ca8e1fbe7865b6d27 --

[issue34589] Py_Initialize() and Py_Main() should not enable C locale coercion

Change by STINNER Victor : -- pull_requests: +8794 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue34589] Py_Initialize() and Py_Main() should not enable C locale coercion

STINNER Victor added the comment: > I'd strongly prefer to just go back to the PEP 538 design. It's much simpler > to implement, we don't actually want anyone turning off locale coercion > except for debugging purposes (unlike UTF-8 mode), and the only argument > against doing this the way

[issue34589] Py_Initialize() and Py_Main() should not enable C locale coercion

Nick Coghlan added the comment: I'd strongly prefer to just go back to the PEP 538 design. It's much simpler to implement, we don't actually want anyone turning off locale coercion except for debugging purposes (unlike UTF-8 mode), and the only argument against doing this the way PEP 538

[issue34589] Py_Initialize() and Py_Main() should not enable C locale coercion

STINNER Victor added the comment: > LC_CTYPE=C PYTHONCOERCECLOCALE=0 python3 -E ... The UTF-8 Mode has the same behavior: PYTHONUTF8 env var is ignored when using -E, but enabled by the C locale: you can use -X utf8=0 in that case to ensure that the UTF-8 Mode is disabled. I wanted to

[issue34589] Py_Initialize() and Py_Main() should not enable C locale coercion

Nick Coghlan added the comment: Alternate PR is up at https://github.com/python/cpython/pull/9257 There's one setting that remains in CoreConfig: "warn_on_c_locale", which _Py_UnixMain also uses to decide whether or not to actually emit the passed in locale coercion warning. The coercion

[issue34589] Py_Initialize() and Py_Main() should not enable C locale coercion

Change by Nick Coghlan : -- pull_requests: +8689 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue34589] Py_Initialize() and Py_Main() should not enable C locale coercion

Nick Coghlan added the comment: The actual functional error is that the following will currently give different outputs on Fedora and CentOS 7, whereas in the original PEP 538 implementation it would always print "C", even if locale coercion would otherwise normally work on your current

[issue34589] Py_Initialize() and Py_Main() should not enable C locale coercion

Nick Coghlan added the comment: Reviewing the way things are now, I don't think it's worth rearranging this yet again for 3.7 - instead, I'll just put it back the way I intended it to be (and the way the PEP describes) for 3.8. The issue is that Victor had good reasons for moving the

[issue34589] Py_Initialize() and Py_Main() should not enable C locale coercion

STINNER Victor added the comment: Example of C locale coercion with Python 3.7: $ env -i PYTHONCOERCECLOCALE=0 ./python -X utf8=0 -c 'import sys; print(ascii(sys.argv[1]))' 'hé' 'h\udcc3\udca9' $ env -i PYTHONCOERCECLOCALE=1 ./python -X utf8=0 -c 'import sys; print(ascii(sys.argv[1]))' 'hé'

[issue34589] Py_Initialize() and Py_Main() should not enable C locale coercion

STINNER Victor added the comment: > (...) The documentation could definitely be much clearer on that point > though, as even in the PEP it's only implied by the final paragraph in > https://www.python.org/dev/peps/pep-0538/#legacy-c-locale-coercion-in-the-standalone-python-interpreter-binary

[issue34589] Py_Initialize() and Py_Main() should not enable C locale coercion

STINNER Victor added the comment: > (...) the locale should be coerced before *ANY* calls are made to > Py_DecodeLocale, as the whole point of the architecture of the PEP > implementation was to ensure that *nothing ever gets decoded incorrectly in > the first place*. See my comment:

[issue34589] Py_Initialize() and Py_Main() should not enable C locale coercion

Nick Coghlan added the comment: FWIW, this is also why I don't want PEP 432 to expose any wstr fields in the configuration settings: it means embedding applications have to figure out which encoding Python is expecting to be used for those fields. Everything should be much cleaner if the

[issue34589] Py_Initialize() and Py_Main() should not enable C locale coercion

Nick Coghlan added the comment: (The one exception to "nothing gets decoded incorrectly" is that PYTHONCOERCECLOCALE itself is always interpreted as an ASCII field: the values that it checks for are actually ASCII byte sequences, not Unicode code points. The documentation could definitely

[issue34589] Py_Initialize() and Py_Main() should not enable C locale coercion

Nick Coghlan added the comment: The entire change affecting the PEP 538 implementation in https://github.com/python/cpython/commit/9454060e84a669dde63824d9e2fcaf295e34f687#diff-8c018c3ada66d06c8e101e47a313c2c7 needs to be reverted: the locale should be coerced before *ANY* calls are made to

[issue34589] Py_Initialize() and Py_Main() should not enable C locale coercion

Change by STINNER Victor : -- keywords: +patch pull_requests: +8532 stage: -> patch review ___ Python tracker ___ ___

[issue34589] Py_Initialize() and Py_Main() should not enable C locale coercion

New submission from STINNER Victor : According to Nick Coghlan, author of the PEP 538, the C locale coercion should not be enabled by Py_Initialize() and Py_Main(), only by the python3 program. Attached PR fix this issue. -- components: Interpreter Core messages: 324649 nosy: