[issue32096] Py_DecodeLocale() fails if used before the runtime is initialized.

[Nick Coghlan]

Nick Coghlan  added the comment:

Successful test run on the Debian machine that failed above:

* http://buildbot.python.org/all/#/builders/27/builds/242

And for the macOS Tiger machine:

* http://buildbot.python.org/all/#/builders/30/builds/227

So I think we can call the regression fixed, which is where we wanted to get to 
before the next alpha release.

--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32096] Py_DecodeLocale() fails if used before the runtime is initialized.

[Nick Coghlan]

Nick Coghlan  added the comment:

Looking more closely at the code, I've realised Victor's right - there's no way 
for Py_DecodeLocale() to accidentally trigger an attempt to import the 
"encodings" module.

Instead, the error is likely coming from the init_sys_streams step towards the 
end of the initialization process. The way the embedded test cases are 
currently being run unfortunately truncated that traceback.

Rather than trying to improve the test case error reporting under the scope of 
this issue, I've instead filed https://bugs.python.org/issue32136 to cover 
factoring the runtime embedding tests out to their own test file.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32096] Py_DecodeLocale() fails if used before the runtime is initialized.

[Nick Coghlan]

Nick Coghlan  added the comment:


New changeset 4274609e1856facd80b7ee588b0791fe8963b9e0 by Nick Coghlan in 
branch 'master':
bpo-32096: Ensure new embedding test can find the encodings module (GH-4566)
https://github.com/python/cpython/commit/4274609e1856facd80b7ee588b0791fe8963b9e0


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32096] Py_DecodeLocale() fails if used before the runtime is initialized.

[Nick Coghlan]

Change by Nick Coghlan :


--
pull_requests: +4494

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32096] Py_DecodeLocale() fails if used before the runtime is initialized.

[Nick Coghlan]

Nick Coghlan  added the comment:

Ah, you're right - I forgot about this little hack in the other embedding 
tests: 
https://github.com/vstinner/cpython/blob/3fda852ba4d4040657a1b616a1ef60ad437b7845/Programs/_testembed.c#L11

I'll add "./" to the program name in the new test as well, and see if that's 
enough to make the failing build bots happy.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32096] Py_DecodeLocale() fails if used before the runtime is initialized.

[STINNER Victor]

STINNER Victor  added the comment:

The test calls Py_SetProgramName(). IMHO the bug is that the program name
is needed to locate the Python std lib.

I don't think that the bug is triggered by Py_DecodeLocale().

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32096] Py_DecodeLocale() fails if used before the runtime is initialized.

[Nick Coghlan]

Nick Coghlan  added the comment:

Huh, those crashes are interesting - I'd guess that it means we have a 
platform-dependent dependency from Py_DecodeLocale on to Py_SetPythonHome in 
order to locate the encodings module. If I'm right, that dependency would then 
mean that embedding applications can only rely on Py_DecodeLocale to do "char 
*" to "wchar_t *" conversions if they can also rely on the locale encoding 
always being a builtin one that bypasses the search for the encodings module.

Perhaps we should be recommending temporarily doing 'setenv("PYTHONHOME", 
home)' (and then reverting that after calling Py_Initialize so it doesn't get 
inherited by subprocesses) as the preferred approach to handling platforms with 
"char *" based native filesystem APIs, and adding such a setting to that 
particular `_testembed` test?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32096] Py_DecodeLocale() fails if used before the runtime is initialized.

[STINNER Victor]

STINNER Victor  added the comment:

The test also failed on x86 Tiger 3.x:

http://buildbot.python.org/all/#/builders/30/builds/212

==
FAIL: test_pre_initialization_api (test.test_capi.EmbeddingTests)
--
Traceback (most recent call last):
  File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/test/test_capi.py", 
line 602, in test_pre_initialization_api
out, err = self.run_embedded_interpreter("pre_initialization_api", env=env)
  File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/test/test_capi.py", 
line 464, in run_embedded_interpreter
(p.returncode, err))
AssertionError: -6 != 0 : bad returncode -6, stderr is "Could not find platform 
independent libraries \nCould not find platform dependent libraries 
\nConsider setting $PYTHONHOME to [:]\nFatal 
Python error: initfsencoding: unable to load the file system 
codec\nModuleNotFoundError: No module named 'encodings'\n\nCurrent thread 
0xa000d000 (most recent call first):\n"

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32096] Py_DecodeLocale() fails if used before the runtime is initialized.

[STINNER Victor]

STINNER Victor  added the comment:

We now check that Py_DecodeLocale() can be called before Py_Initialize(). IMHO 
we need to document this property in the documentation: I opened bpo-32124 and 
wrote a PR for that.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32096] Py_DecodeLocale() fails if used before the runtime is initialized.

[STINNER Victor]

STINNER Victor  added the comment:

> New changeset 9e87e7776f7ace66baaf7247233afdabd00c2b44 by Victor Stinner in 
> branch 'master':
> bpo-32096: Remove obj and mem from _PyRuntime (#4532)

The newly added test failed on AMD64 Debian root 3.x:

http://buildbot.python.org/all/#/builders/27/builds/226

==
FAIL: test_pre_initialization_api (test.test_capi.EmbeddingTests)
--
Traceback (most recent call last):
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/test/test_capi.py", 
line 602, in test_pre_initialization_api
out, err = self.run_embedded_interpreter("pre_initialization_api", env=env)
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/test/test_capi.py", 
line 464, in run_embedded_interpreter
(p.returncode, err))
AssertionError: -6 != 0 : bad returncode -6, stderr is "Could not find platform 
independent libraries \nCould not find platform dependent libraries 
\nConsider setting $PYTHONHOME to [:]\nFatal 
Python error: initfsencoding: Unable to get the locale 
encoding\nModuleNotFoundError: No module named 'encodings'\n\nCurrent thread 
0x7f6456f8c700 (most recent call first):\n"

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32096] Py_DecodeLocale() fails if used before the runtime is initialized.

[STINNER Victor]

STINNER Victor  added the comment:


New changeset 9e87e7776f7ace66baaf7247233afdabd00c2b44 by Victor Stinner in 
branch 'master':
bpo-32096: Remove obj and mem from _PyRuntime (#4532)
https://github.com/python/cpython/commit/9e87e7776f7ace66baaf7247233afdabd00c2b44


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32096] Py_DecodeLocale() fails if used before the runtime is initialized.

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +4467

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32096] Py_DecodeLocale() fails if used before the runtime is initialized.

[Nick Coghlan]

Nick Coghlan  added the comment:

Even the public implementation of PEP 432 is going to bound by the requirement 
to keep existing embedding logic working, and that's encapsulated in the new 
test Eric added in his PR:

wchar_t *program = Py_DecodeLocale("spam", NULL);
Py_SetProgramName(program);
Py_Initialize();
Py_Finalize();
PyMem_RawFree(program);

So even if we were to revert the _PyRuntime.mem change in 3.7, we'd still face 
the same problem in 3.8, because we'd still be exposing the traditional 
configuration API - the new multi-step configuration API would be *optional* 
for folks that wanted to override the default settings more easily, rather than 
a backwards compatibility break with the previously supported way of doing 
things.

As a result, my preferred option is now to make exactly the promises we need to 
ensure that the above code works correctly, and then have Py_Initialize and 
Py_Finalize enforce those constraints:

* the public Py_Initialize API should fail if the memory allocators have 
already been set to something other than the default
* Py_Finalize should revert the memory allocators to their default setting

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32096] Py_DecodeLocale() fails if used before the runtime is initialized.

[STINNER Victor]

STINNER Victor  added the comment:

"Victor, I think you're fundamentally misunderstanding the goals of PEP 432. 
The entire point is to let people have a *mostly working Python runtime* during 
CPython startup. (...)"

While the PEP 432 is nice, all changes are currently done in private APIs, 
symbols starting with _Py. I would prefer that nobody uses these new APIs 
before the conversion is complete. And from what I saw, I can say that the 
conversion just started, there are still a lot of changes that should be done.

While having _PyRuntime.mem is nice to have in the long term, it doesn't add 
any value *right now*, except of making the existing C API harder to use.

I would prefer to do things in this order:

* Revert _PyRutime.mem
* Finish PEP 432 implementation
* Recreate _PyRutime.mem

Maybe we can complete these 3 steps before Python 3.7, but I'm not sure about 
that.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32096] Py_DecodeLocale() fails if used before the runtime is initialized.

[Nick Coghlan]

Nick Coghlan  added the comment:

Follow up: this also came up on https://bugs.python.org/issue32030#msg306763, 
and I think Victor and I are on the same page now :)

Since MainInterpreterConfig is currently still a private struct, we can store 
the existing C level config state directly in there for now, and then look at 
upgrading to Python types on a case by case basis.

Once they're all both consolidated *and* upgraded, then we can consider making 
the new incremental configuration API public.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32096] Py_DecodeLocale() fails if used before the runtime is initialized.

[Nick Coghlan]

Nick Coghlan  added the comment:

Also, the basic rules of thumb I use for deciding whether or not a setting 
belongs in CoreConfig:

* does `PyUnicode_New` need this? (If yes, then include it)
* does the importlib bootstrapping need this? (If yes, then include it)

Everything else goes in MainInterpreterConfig as a real Python object.

We may need other structs internally to help manage the way Py_Main populates 
MainInterpreterConfig, but those should be made a required part of the future 
public initialization API (although we may decide to expose them as "use them 
if you want to better emulate CPython's default behaviour" helper APIs).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32096] Py_DecodeLocale() fails if used before the runtime is initialized.

[Nick Coghlan]

Nick Coghlan  added the comment:

Victor, I think you're fundamentally misunderstanding the goals of PEP 432.

The entire point is to let people have a *mostly working Python runtime* during 
CPython startup. Moving everything that Py_Initialize needs to instead have to 
happen before Py_InitializeRuntime (aka _Py_CoreInitialize) defeats that point.

CoreConfig should thus contain *as little as possible*, with most of the 
environmental querying work moving into Py_ReadMainInterpreterConfig.

So could you please move everything you've added to CoreConfig (that isn't 
genuinely required to from the moment the runtime starts doing anything) out 
again, and either put it into the main interpreter config as Python objects (as 
described in PEP 432), or else into a new intermediate configuration struct?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32096] Py_DecodeLocale() fails if used before the runtime is initialized.

[STINNER Victor]

STINNER Victor  added the comment:

I created a new PR adding a a new _PyCoreConfig.pythonpath field:
https://github.com/python/cpython/pull/4504

Once it will be merged, I will work on a new PR for PYTHONHOME (add a new 
_PyCoreConfig.pythonhome field).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32096] Py_DecodeLocale() fails if used before the runtime is initialized.

[STINNER Victor]

STINNER Victor  added the comment:

>  We *don't* currently have anything like that for environment variables, not 
> even the ones which are "read once at startup, then never read them again".

I changed Py_Main() in bpo-32030. Now multiple environment variables are read 
once at startup and put into _PyCoreConfig:

https://github.com/python/cpython/blob/803ddd8ce22f0de3ab42fb98a225a704c000ef06/Modules/main.c#L1365-L1413

I added new fields to _PyCoreConfig:

https://github.com/python/cpython/blob/803ddd8ce22f0de3ab42fb98a225a704c000ef06/Include/pystate.h#L27-L39

I suggest to continue to add more fields to _PyCoreConfig to move all code to 
configure Python before Py_Initialize(), and later to let users embedding 
Python to configure Python as they want, without losing features. For example, 
to enable the new "development mode" (-X dev), now you "just" have to set 
_PyCoreConfig.dev_mode to 1.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32096] Py_DecodeLocale() fails if used before the runtime is initialized.

[Nick Coghlan]

Nick Coghlan  added the comment:

Speaking of surprises with static initialization of the runtime allocations: 
both PRs are failing in CI, suggesting that the changes that Py_Initialize 
makes to the allocator settings aren't being reverted in Py_Finalize, so 
there's a mismatch between the allocation function and the deallocation 
function.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32096] Py_DecodeLocale() fails if used before the runtime is initialized.

[Nick Coghlan]

Nick Coghlan  added the comment:

Victor, please stop trying to conflate the two questions of "How should we fix 
the current Py_DecodeLocale regression before 3.7.0a3?" and "What do we want to 
do long term?".

They're far from being the same question, and answering the second one properly 
is going to be much harder and more involved than just doing the bare minimum 
needed to make the previously supported embedding logic work again (even if it 
means postponing some of the dynamic allocator changes we'd like to make).

Omitting PYTHONHOME and PYTHONPATH from the core config is deliberate, as the 
interpreter doesn't support external imports yet when just the core has been 
initialized - only builtin and frozen ones.

Anything related to external imports should ultimately end up in the main 
interpreter configuration: 
https://www.python.org/dev/peps/pep-0432/#supported-configuration-settings

Longer term, I also want to rewrite getpath.c in Python (or at least primarily 
using Python lists and strings via the C API instead of relaying C arrays and C 
string manipulation functions).

However, our work on refactoring Py_Main has also shown me that we're going to 
need some additional structs to hold the raw(ish) command line arguments and 
environment variables in order to easily pass them around to other internal 
configuration APIs.

Modules/main.c already defines _Py_CommandLineDetails for the command line 
settings: https://github.com/python/cpython/blob/master/Modules/main.c#L382

We *don't* currently have anything like that for environment variables, not 
even the ones which are "read once at startup, then never read them again".

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32096] Py_DecodeLocale() fails if used before the runtime is initialized.

[STINNER Victor]

STINNER Victor  added the comment:

> As an alternative to that, we could also deprecate using any of those 
> functions before initializing the runtime.   Instead of calling them, you 
> would set the relevant info on the runtime "config" struct that you pass to 
> the replacement for PyInitialize().

Currently, _PyCoreConfig is not complete: you cannot pass PYTHONPATH or 
PYTHONHOME. I'm working on patches to implement that.

Moreover, there is the question how to decode a bytes path (for PYTHONPATH) 
into a wchar_t* string.

Disallow calling Py_DecodeLocale() before Py_Initialize(): ok, but which 
alternative do you propose to decode OS data?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32096] Py_DecodeLocale() fails if used before the runtime is initialized.

[Eric Snow]

Eric Snow  added the comment:

> IMHO for the long term, the best would be to have a structure for
> pre-Py_Initialize configuration, maybe _PyCoreConfig, and pass it
> to functions that can be called before Py_Initialize().

+1

As an alternative to that, we could also deprecate using any of those functions 
before initializing the runtime.   Instead of calling them, you would set the 
relevant info on the runtime "config" struct that you pass to the replacement 
for PyInitialize().  At that point we would not need some of those functions 
any longer and we could remove them (eventually, once backward-compatibility is 
resolved).  Given that the community of CPython embedders is relatively small, 
we're still in a position to iron this out

Regardless, I see where you're coming from.  I'm okay with reverting the 
Object/obmalloc.c parts, but, like I said, I'd rather avoid it if possible.

Solution #2 (that I listed above), AKA PR #4481, is focused and effective.  
Unfortunately, it feels like a bit like a hack to me, though it is a 
well-contained hack.  So I'm not convinced it's the best solution.  However, I 
like it as much as I like reverting the allocators.

Solution #3, AKA PR #4495, is nice and clean, but potentially adds a little 
overhead to all PyMem_RawMalloc() and PyMem_RawFree() calls.  All the other 
PyMem_* functions are unaffected, so perhaps the overall impact is not 
significant enough to worry.


>> 2. statically initialize the "raw" allocator with defaults, enough
>>   to make PyMem_RawMalloc() and PyMem_RawFree() work pre-init (this
>>   is what my PR does)
>
> As I explained, the code to initialize PyMem_Raw allocator is complex
> and I would really prefer to only initialize it "partially" to prevent
> bad surprises.

The surprises would only be pre-initialization, right?  After the runtime is 
initialized, the allocators are in the proper fully-initialized state.  So it 
mostly boils down to what parts of the C-API embedders can use before 
initialization and how those functions interact with the raw memory allocator.  
Those constraints narrow down the scope of potential problems to a manageable 
size (I won't say small, but it feels that way).

Ultimately, I favor solution #3 if we can see that it does not impact 
performance.  If we can't come to an agreement in a timely fashion then I'll go 
along with #1 (revert), so that we don't leave the embedding story broken.  If 
we go that route, do you think we could resolve the initialization issues 
within the 3.7 timeframe?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32096] Py_DecodeLocale() fails if used before the runtime is initialized.

[Eric Snow]

Change by Eric Snow :


--
pull_requests: +4432
stage:  -> patch review

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32096] Py_DecodeLocale() fails if used before the runtime is initialized.

[STINNER Victor]

STINNER Victor  added the comment:

IMHO for the long term, the best would be to have a structure for 
pre-Py_Initialize configuration, maybe _PyCoreConfig, and pass it to functions 
that can be called before Py_Initialize().

For example, I'm working on a variant of Py_GetPath() which accepts a 
_PyCoreConfig to be able to pass the value of the PYTHONPATH environment 
variable.

That's a more complex solution, so I proposed to first revert, to have more 
time to design the "correct API".

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32096] Py_DecodeLocale() fails if used before the runtime is initialized.

[STINNER Victor]

STINNER Victor  added the comment:

> 2. statically initialize the "raw" allocator with defaults, enough
>   to make PyMem_RawMalloc() and PyMem_RawFree() work pre-init (this
>   is what my PR does)

As I explained, the code to initialize PyMem_Raw allocator is complex and I 
would really prefer to only initialize it "partially" to prevent bad surprises.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32096] Py_DecodeLocale() fails if used before the runtime is initialized.

[STINNER Victor]

STINNER Victor  added the comment:

Nick: "Should we deprecate our implied support for calling Py_DecodeLocale() 
before calling Py_Initialize()?"

Please don't do that. Py_DecodeLocale() is the best available function to 
decode paths and environment variables to initialize Python. The implementation 
of Py_DecodeLocale() is very complex, but we need this complexity to decode 
"correctly" OS data.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32096] Py_DecodeLocale() fails if used before the runtime is initialized.

[STINNER Victor]

STINNER Victor  added the comment:

I marked bpo-32096 as a duplicate of this one. I don't want to discuss the same 
issue in 3 places (2 bpo and python-dev).

--
superseder: C API: Clarify which C functions are safe to be called before 
Py_Initialize() -> 

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32096] Py_DecodeLocale() fails if used before the runtime is initialized.

[STINNER Victor]

STINNER Victor  added the comment:

"3. use hard-coded defaults in PyMem_RawMalloc() and PyMem_RawFree() if the 
runtime has not been initialized yet"

I dislike this option since it can have a negative impact on performances. The 
PEP 445 already added a new level of indirection and so made memory allocations 
a little bit slower.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32096] Py_DecodeLocale() fails if used before the runtime is initialized.

[Eric Snow]

Eric Snow  added the comment:

I thought issue #32086 was about documentation (which is worth having a 
separate issue for), not about a fix to the regression.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32096] Py_DecodeLocale() fails if used before the runtime is initialized.

[Eric Snow]

Eric Snow  added the comment:

I see at least 3 ways to sort this out:

 1. partially revert the _PyRuntime change, sort of temporarily
   ("revert the change on memory allocators, and retry later to fix
   it, once other initializations issues are fixed", as suggested by
   Victor in the email thread)
 2. statically initialize the "raw" allocator with defaults, enough
   to make PyMem_RawMalloc() and PyMem_RawFree() work pre-init (this
   is what my PR does)
 3. use hard-coded defaults in PyMem_RawMalloc() and PyMem_RawFree()
   if the runtime has not been initialized yet

I considered implementing #3 instead, but wasn't sure about the performance 
impact.  It would add a pointer comparison to NULL and a branch on each 
PyMem_RawMalloc() and PyMem_RawFree() call.  I'm not convinved it would make 
much of a difference.  Furthermore, I consider #3 to be the simplest solution, 
both to implement and to maintain, so I'll probably try it out.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32096] Py_DecodeLocale() fails if used before the runtime is initialized.

[Nick Coghlan]

Nick Coghlan  added the comment:

While they're definitely closely related, I don't think this and issue 32086 
are actually duplicates: this issue is "fix the current Py_DecodeLocale 
regression in 3.7 by reverting back to the 3.6 behaviour", while issue 32086 is 
"Should we deprecate our implied support for calling Py_DecodeLocale() before 
calling Py_Initialize()?".


The latter is certainly a valid question, but we should restore the 3.6 
behaviour while we're considering it.

--
resolution: duplicate -> 
stage: resolved -> 
status: closed -> open

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32096] Py_DecodeLocale() fails if used before the runtime is initialized.

[STINNER Victor]

STINNER Victor  added the comment:

Duplicate of bpo-32086.

--
resolution:  -> duplicate
stage: patch review -> resolved
status: open -> closed
superseder:  -> C API: Clarify which C functions are safe to be called before 
Py_Initialize()

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32096] Py_DecodeLocale() fails if used before the runtime is initialized.

[Serhiy Storchaka]

Change by Serhiy Storchaka :


--
components: +Interpreter Core
versions: +Python 3.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32096] Py_DecodeLocale() fails if used before the runtime is initialized.

[Decorater]

Decorater  added the comment:

Interesting, on 3.6.3 on my embedded program it seems to work just fine.

Did anything change in it since then?

https://github.com/AraHaan/Els_kom_new/blob/master/PC/komextract_new.c

--
nosy: +Decorater

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32096] Py_DecodeLocale() fails if used before the runtime is initialized.

[Eric Snow]

Change by Eric Snow :


--
keywords: +patch
pull_requests: +4418
stage:  -> patch review

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32096] Py_DecodeLocale() fails if used before the runtime is initialized.

[Eric Snow]

New submission from Eric Snow :

(see the python-dev thread [1])
(related: issue #32086)

When I consolidated the global runtime state into a single global, _PyRuntime, 
calls Py_DecodeLocale() started to break if the runtime hadn't been intialized 
yet.  This is because that function relies on PyMem_RawMalloc() and 
PyMem_RawFree(), which rely on the raw allocator having been initialized as 
part of the runtime (it used to be intialized statically).

The documentation for various "Process-wide parameters" [2] explicitly directs 
users to call Py_DecodeLocale() where necessary.  The docs for 
Py_DecodeLocale(), in turn, explicitly refer to calling PyMem_RawFree().  So 
changing the pre-runtime-init behavior of Py_DecodeLocale() and PyMem_RawFree() 
is a regression that should be fixed.

[1] https://mail.python.org/pipermail/python-dev/2017-November/150605.html
[2] https://docs.python.org/3/c-api/init.html#process-wide-parameters

--
assignee: eric.snow
messages: 306585
nosy: eric.snow, ncoghlan, serhiy.storchaka, vstinner
priority: normal
severity: normal
status: open
title: Py_DecodeLocale() fails if used before the runtime is initialized.
type: crash

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com