[issue38865] Can Py_Finalize() be called if the current interpreter is not the main interpreter?

[STINNER Victor]

STINNER Victor  added the comment:

See also bpo-37776: Test Py_Finalize() from a subinterpreter.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38865] Can Py_Finalize() be called if the current interpreter is not the main interpreter?

[STINNER Victor]

STINNER Victor  added the comment:

I changed my mind. I managed to implement bpo-39984 without needing my PR 
19063, so I closed it.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38865] Can Py_Finalize() be called if the current interpreter is not the main interpreter?

[STINNER Victor]

STINNER Victor  added the comment:

See also bpo-36225: "Lingering subinterpreters should be implicitly cleared on 
shutdown".

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38865] Can Py_Finalize() be called if the current interpreter is not the main interpreter?

[STINNER Victor]

STINNER Victor  added the comment:

This issue became a blocker issue while working on bpo-39984, when I tried to 
move pending calls from _PyRuntimeState to PyInterpreterState.

I wrote PR 19063 to deny calling Py_FinalizeEx() from a subinterpreter.


Eric:
> On the other hand, if the "main" interpreter isn't special after runtime init 
> is done (i.e. a subinterpreter is effectively indistinguishable by nature) 
> then it certainly would not matter which one finalizes (or is active during 
> fork() for that matter).  On top of that, the effort we are making to 
> identify the main interpreter (both in the code and in the docs) becomes 
> unnecessary.

Well. Maybe we will be able to reach this point later, but right now, there are 
concrete implementation issues. The main interpreter remains special. Only the 
main interpreter is allowed to call fork(), spawn daemon threads, handle 
signals, etc.

If we manage to make subinterpreters less special, we can change 
Py_FinalizeEx() again later.


I asked:
> Is Python supposed to magically destroy the 3 interpreters?

Currently, Py_FinalizeEx() fails with a fatal error if there are remaining 
subinterpreters:
---
Fatal Python error: PyInterpreterState_Delete: remaining subinterpreters
Python runtime state: finalizing (tstate=0xfaa1c0)

Abandon (core dumped)
---

IMHO it's a good behavior.

if we want, we can modify Py_FinalizeEx() to magically end subinterpters later. 
The opposite may be more annoying for embedders.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38865] Can Py_Finalize() be called if the current interpreter is not the main interpreter?

[STINNER Victor]

Change by STINNER Victor :


--
keywords: +patch
pull_requests: +18417
stage:  -> patch review
pull_request: https://github.com/python/cpython/pull/19063

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38865] Can Py_Finalize() be called if the current interpreter is not the main interpreter?

[Eric Snow]

Eric Snow  added the comment:

I could swear the topic of destroy-everything-in-PyFinalize has come up before 
but I don't remember anything specific.  Doing so there sure makes sense 
though...

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38865] Can Py_Finalize() be called if the current interpreter is not the main interpreter?

[Eric Snow]

Eric Snow  added the comment:

> * Is Python supposed to magically destroy the 3 interpreters?

Doesn't it?  Gah.  I guess I was thinking of PyOS_AfterFork_Child(), which 
calls _PyInterpreterState_DeleteExceptMain(). :/  In September there was a nice 
comment right above Py_FinalizeEx() saying that we do not clean up other 
interpreters or threads, but I haven't checked when it got removed.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38865] Can Py_Finalize() be called if the current interpreter is not the main interpreter?

[Eric Snow]

Eric Snow  added the comment:

tl;dr Py_Finalize() probably *should* only be allowed under the main 
interpreter.

As you know well, when the runtime starts we do it at first relative to a 
partially initialized main interpreter and the finish initialization with a 
fully operational main interpreter.  My expectation is that some of the global 
runtime state relies somehow on the main interpreter's state.  Consequently, 
finalization would then need to happen under the main interpreter.

The same idea applies to fork-without-exec, where we destroy all interpreters 
but the current one.  We now only allow os.fork() in the main interpreter.  So 
I'd expect the same treatement for finalization.

On the other hand, if the "main" interpreter isn't special after runtime init 
is done (i.e. a subinterpreter is effectively indistinguishable by nature) then 
it certainly would not matter which one finalizes (or is active during fork() 
for that matter).  On top of that, the effort we are making to identify the 
main interpreter (both in the code and in the docs) becomes unnecessary.

If we look at it from that angle, though, we do have cases where the main 
interpreter is still significant (regardless of the relationship between the 
main interpreter and the global runtime state).  At the very least, the main 
interpreter is solely responsible for some global resources, like signals.  So 
the main interpreter cannot go away until nothing else in the runtime relies on 
those resources.  I expect that we won't go down that route, which means 
finalization should happen only under the main interpreter.

Also keep in mind that this only matters relative to the C-API and I expect 
only embedders (not extensions) will be finalizing the runtime.  With PEP 554 
the only case where you would run into problems is when running subinterpreters 
in daemon threads.  And we already know about problems with daemon threads 
during runtime finalization! :)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38865] Can Py_Finalize() be called if the current interpreter is not the main interpreter?

[STINNER Victor]

New submission from STINNER Victor :

Programs/_testembed.c contains the following test used by test_embed:

static int test_audit_subinterpreter(void)
{
Py_IgnoreEnvironmentFlag = 0;
PySys_AddAuditHook(_audit_subinterpreter_hook, NULL);
_testembed_Py_Initialize();

Py_NewInterpreter(); 
Py_NewInterpreter(); 
Py_NewInterpreter(); 
 
Py_Finalize(); 

switch (_audit_subinterpreter_interpreter_count) {
case 3: return 0;
case 0: return -1;
default: return _audit_subinterpreter_interpreter_count;
}
}

When Py_Finalize() is called, the current interpreter is a subinterpreter (the 
3rd interpreter), not the main interpreter.

* Is it correct to call Py_Finalize() in such case?
* Is Python supposed to magically destroy the 3 interpreters?

In bpo-38858, I'm trying to reuse the same code to initialize and finalize the 
"main" interpreter and subinterpreters. I had an issue with 
test_audit_subinterpreter() when working on the PR 17293.

I modified my PR 17293 to not expect that Py_Finalize() can only be called from 
the main interpreter, but actually check if the current interpreter is the main 
interpreter or not. It fix test_audit_subinterpreter() but again, I'm not sure 
what is the correct behavior?

--

Last year, we had a similar discussion about calling Py_Main() *after* 
Py_Initialize(). I hacked the code to make it possible because it was supported 
previously, even if the Py_Main() configuration is only partially applied. But 
I understood that Nick Coghlan would prefer to deprecate supporting to call 
Py_Initialize() before Py_Main().

PEP 587 added Py_RunMain() which provides a different solution to this problem:
https://docs.python.org/dev/c-api/init_config.html#c.Py_RunMain

--
components: Interpreter Core
messages: 357080
nosy: eric.snow, nanjekyejoannah, ncoghlan, vstinner
priority: normal
severity: normal
status: open
title: Can Py_Finalize() be called if the current interpreter is not the main 
interpreter?
versions: Python 3.9

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com