[issue32096] Py_DecodeLocale() fails if used before the runtime is initialized.

Nick Coghlan added the comment: Successful test run on the Debian machine that failed above: * http://buildbot.python.org/all/#/builders/27/builds/242 And for the macOS Tiger machine: * http://buildbot.python.org/all/#/builders/30/builds/227 So I think we can call the

[issue32096] Py_DecodeLocale() fails if used before the runtime is initialized.

Nick Coghlan added the comment: Looking more closely at the code, I've realised Victor's right - there's no way for Py_DecodeLocale() to accidentally trigger an attempt to import the "encodings" module. Instead, the error is likely coming from the init_sys_streams step

[issue32096] Py_DecodeLocale() fails if used before the runtime is initialized.

Nick Coghlan added the comment: New changeset 4274609e1856facd80b7ee588b0791fe8963b9e0 by Nick Coghlan in branch 'master': bpo-32096: Ensure new embedding test can find the encodings module (GH-4566)

[issue32096] Py_DecodeLocale() fails if used before the runtime is initialized.

Change by Nick Coghlan : -- pull_requests: +4494 ___ Python tracker ___ ___

[issue32096] Py_DecodeLocale() fails if used before the runtime is initialized.

Nick Coghlan added the comment: Ah, you're right - I forgot about this little hack in the other embedding tests: https://github.com/vstinner/cpython/blob/3fda852ba4d4040657a1b616a1ef60ad437b7845/Programs/_testembed.c#L11 I'll add "./" to the program name in the new test

[issue32096] Py_DecodeLocale() fails if used before the runtime is initialized.

STINNER Victor added the comment: The test calls Py_SetProgramName(). IMHO the bug is that the program name is needed to locate the Python std lib. I don't think that the bug is triggered by Py_DecodeLocale(). -- ___

[issue32096] Py_DecodeLocale() fails if used before the runtime is initialized.

Nick Coghlan added the comment: Huh, those crashes are interesting - I'd guess that it means we have a platform-dependent dependency from Py_DecodeLocale on to Py_SetPythonHome in order to locate the encodings module. If I'm right, that dependency would then mean that

[issue32096] Py_DecodeLocale() fails if used before the runtime is initialized.

STINNER Victor added the comment: The test also failed on x86 Tiger 3.x: http://buildbot.python.org/all/#/builders/30/builds/212 == FAIL: test_pre_initialization_api (test.test_capi.EmbeddingTests)

[issue32096] Py_DecodeLocale() fails if used before the runtime is initialized.

STINNER Victor added the comment: We now check that Py_DecodeLocale() can be called before Py_Initialize(). IMHO we need to document this property in the documentation: I opened bpo-32124 and wrote a PR for that. -- ___

[issue32096] Py_DecodeLocale() fails if used before the runtime is initialized.

STINNER Victor added the comment: > New changeset 9e87e7776f7ace66baaf7247233afdabd00c2b44 by Victor Stinner in > branch 'master': > bpo-32096: Remove obj and mem from _PyRuntime (#4532) The newly added test failed on AMD64 Debian root 3.x:

[issue32096] Py_DecodeLocale() fails if used before the runtime is initialized.

STINNER Victor added the comment: New changeset 9e87e7776f7ace66baaf7247233afdabd00c2b44 by Victor Stinner in branch 'master': bpo-32096: Remove obj and mem from _PyRuntime (#4532) https://github.com/python/cpython/commit/9e87e7776f7ace66baaf7247233afdabd00c2b44

[issue32096] Py_DecodeLocale() fails if used before the runtime is initialized.

Change by STINNER Victor : -- pull_requests: +4467 ___ Python tracker ___ ___

[issue32096] Py_DecodeLocale() fails if used before the runtime is initialized.

Nick Coghlan added the comment: Even the public implementation of PEP 432 is going to bound by the requirement to keep existing embedding logic working, and that's encapsulated in the new test Eric added in his PR: wchar_t *program = Py_DecodeLocale("spam", NULL);

[issue32096] Py_DecodeLocale() fails if used before the runtime is initialized.

STINNER Victor added the comment: "Victor, I think you're fundamentally misunderstanding the goals of PEP 432. The entire point is to let people have a *mostly working Python runtime* during CPython startup. (...)" While the PEP 432 is nice, all changes are

[issue32096] Py_DecodeLocale() fails if used before the runtime is initialized.

Nick Coghlan added the comment: Follow up: this also came up on https://bugs.python.org/issue32030#msg306763, and I think Victor and I are on the same page now :) Since MainInterpreterConfig is currently still a private struct, we can store the existing C level config

[issue32096] Py_DecodeLocale() fails if used before the runtime is initialized.

Nick Coghlan added the comment: Also, the basic rules of thumb I use for deciding whether or not a setting belongs in CoreConfig: * does `PyUnicode_New` need this? (If yes, then include it) * does the importlib bootstrapping need this? (If yes, then include it) Everything

[issue32096] Py_DecodeLocale() fails if used before the runtime is initialized.

Nick Coghlan added the comment: Victor, I think you're fundamentally misunderstanding the goals of PEP 432. The entire point is to let people have a *mostly working Python runtime* during CPython startup. Moving everything that Py_Initialize needs to instead have to

[issue32096] Py_DecodeLocale() fails if used before the runtime is initialized.

STINNER Victor added the comment: I created a new PR adding a a new _PyCoreConfig.pythonpath field: https://github.com/python/cpython/pull/4504 Once it will be merged, I will work on a new PR for PYTHONHOME (add a new _PyCoreConfig.pythonhome field). --

[issue32096] Py_DecodeLocale() fails if used before the runtime is initialized.

STINNER Victor added the comment: > We *don't* currently have anything like that for environment variables, not > even the ones which are "read once at startup, then never read them again". I changed Py_Main() in bpo-32030. Now multiple environment variables are

[issue32096] Py_DecodeLocale() fails if used before the runtime is initialized.

Nick Coghlan added the comment: Speaking of surprises with static initialization of the runtime allocations: both PRs are failing in CI, suggesting that the changes that Py_Initialize makes to the allocator settings aren't being reverted in Py_Finalize, so there's a

[issue32096] Py_DecodeLocale() fails if used before the runtime is initialized.

Nick Coghlan added the comment: Victor, please stop trying to conflate the two questions of "How should we fix the current Py_DecodeLocale regression before 3.7.0a3?" and "What do we want to do long term?". They're far from being the same question, and answering the

[issue32096] Py_DecodeLocale() fails if used before the runtime is initialized.

STINNER Victor added the comment: > As an alternative to that, we could also deprecate using any of those > functions before initializing the runtime. Instead of calling them, you > would set the relevant info on the runtime "config" struct that you pass to > the

[issue32096] Py_DecodeLocale() fails if used before the runtime is initialized.

Eric Snow added the comment: > IMHO for the long term, the best would be to have a structure for > pre-Py_Initialize configuration, maybe _PyCoreConfig, and pass it > to functions that can be called before Py_Initialize(). +1 As an alternative to that, we could

[issue32096] Py_DecodeLocale() fails if used before the runtime is initialized.

Change by Eric Snow : -- pull_requests: +4432 stage: -> patch review ___ Python tracker ___

[issue32096] Py_DecodeLocale() fails if used before the runtime is initialized.

STINNER Victor added the comment: IMHO for the long term, the best would be to have a structure for pre-Py_Initialize configuration, maybe _PyCoreConfig, and pass it to functions that can be called before Py_Initialize(). For example, I'm working on a variant of

[issue32096] Py_DecodeLocale() fails if used before the runtime is initialized.

STINNER Victor added the comment: > 2. statically initialize the "raw" allocator with defaults, enough > to make PyMem_RawMalloc() and PyMem_RawFree() work pre-init (this > is what my PR does) As I explained, the code to initialize PyMem_Raw allocator is complex

[issue32096] Py_DecodeLocale() fails if used before the runtime is initialized.

STINNER Victor added the comment: Nick: "Should we deprecate our implied support for calling Py_DecodeLocale() before calling Py_Initialize()?" Please don't do that. Py_DecodeLocale() is the best available function to decode paths and environment variables to

[issue32096] Py_DecodeLocale() fails if used before the runtime is initialized.

STINNER Victor added the comment: I marked bpo-32096 as a duplicate of this one. I don't want to discuss the same issue in 3 places (2 bpo and python-dev). -- superseder: C API: Clarify which C functions are safe to be called before Py_Initialize() ->

[issue32096] Py_DecodeLocale() fails if used before the runtime is initialized.

STINNER Victor added the comment: "3. use hard-coded defaults in PyMem_RawMalloc() and PyMem_RawFree() if the runtime has not been initialized yet" I dislike this option since it can have a negative impact on performances. The PEP 445 already added a new level of

[issue32096] Py_DecodeLocale() fails if used before the runtime is initialized.

Eric Snow added the comment: I thought issue #32086 was about documentation (which is worth having a separate issue for), not about a fix to the regression. -- ___ Python tracker

[issue32096] Py_DecodeLocale() fails if used before the runtime is initialized.

Eric Snow added the comment: I see at least 3 ways to sort this out: 1. partially revert the _PyRuntime change, sort of temporarily ("revert the change on memory allocators, and retry later to fix it, once other initializations issues are fixed", as

[issue32096] Py_DecodeLocale() fails if used before the runtime is initialized.

Nick Coghlan added the comment: While they're definitely closely related, I don't think this and issue 32086 are actually duplicates: this issue is "fix the current Py_DecodeLocale regression in 3.7 by reverting back to the 3.6 behaviour", while issue 32086 is "Should we

[issue32096] Py_DecodeLocale() fails if used before the runtime is initialized.

STINNER Victor added the comment: Duplicate of bpo-32086. -- resolution: -> duplicate stage: patch review -> resolved status: open -> closed superseder: -> C API: Clarify which C functions are safe to be called before Py_Initialize()

[issue32096] Py_DecodeLocale() fails if used before the runtime is initialized.

Change by Serhiy Storchaka : -- components: +Interpreter Core versions: +Python 3.7 ___ Python tracker ___

[issue32096] Py_DecodeLocale() fails if used before the runtime is initialized.

Decorater added the comment: Interesting, on 3.6.3 on my embedded program it seems to work just fine. Did anything change in it since then? https://github.com/AraHaan/Els_kom_new/blob/master/PC/komextract_new.c -- nosy: +Decorater

[issue32096] Py_DecodeLocale() fails if used before the runtime is initialized.

Change by Eric Snow : -- keywords: +patch pull_requests: +4418 stage: -> patch review ___ Python tracker ___

[issue32096] Py_DecodeLocale() fails if used before the runtime is initialized.

New submission from Eric Snow : (see the python-dev thread [1]) (related: issue #32086) When I consolidated the global runtime state into a single global, _PyRuntime, calls Py_DecodeLocale() started to break if the runtime hadn't been intialized yet. This is