[issue36225] [subinterpreters] Lingering subinterpreters should be implicitly cleared on shutdown

[Chris Roberts]

Change by Chris Roberts :


--
nosy: +nasageek

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36225] [subinterpreters] Lingering subinterpreters should be implicitly cleared on shutdown

[STINNER Victor]

STINNER Victor  added the comment:

If tomorrow Python allows to run two interpreters in parallel (see bpo-40512: 
"Meta issue: per-interpreter GIL"), I don't think that it will be safe to 
execute object finalizers of a subinterpreter from the main interpreter in 
Py_Finalize().

IMO subinterpreters must be finalized manually by the programmer, since it's 
too tricky.

The safest option is to display a warning in Py_Finalize() if there are running 
subinterpreters.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36225] [subinterpreters] Lingering subinterpreters should be implicitly cleared on shutdown

[Eric Snow]

Eric Snow  added the comment:

> I see multiple non trivial problems:
>
> * To finalize the subinterpreter A, Py_Finalize() may switch the current 
> Python thread state from the main interpreter to a Python thread in the 
> subintrepeter A. It can lead to new funny issues.

I'm not sure what you mean with this.  Are you talking about how another thread 
might be running that threadstate already?  That doesn't sounds like a problem 
specific to this issue but rather a problem with Py_EndInterpreter() in general.

> * A subinterpreter can be stuck for whatever reason and refuse to stop. For 
> example, the subinterpreter A may wait for an even from subinterpreter B. If 
> we don't run two interpreters "in parallel" (in threads), it may be stuck 
> forever.

The wait_for_thread_shutdown() call in Py_FinalizeEx() already has a similar 
behavior.  In the case of one interpreter blocking another like that, how would 
that be possible where it isn't already a problem?

> * Python 3.10 still has weird code to allow daemon threads to continue to run 
> after Py_Finalize() complete. Maybe we need to add such hacks for 
> subinterpreter which fail to be finalized? For example, kill them (exit) as 
> soon as they attempt to acquire the GIL. Search for tstate_must_exit() in 
> Python/ceval_gil.h.

I consider the problem of daemon threads to be a separate problem.  The 
solution proposed here for finalization doesn't change any behavior regarding 
daemon threads.

> By the way, currently Py_Finalize() calls PyInterpreterState_Clear() which 
> call object finalizers in the main thread of the main interpreter, whereas 
> these finalizers might expect to be called from the thread which created them.

Do you have any examples?  I'm having trouble imagining such a case.

> I don't know if we must change anything else.
>
> Again, all these problems are very complex :-(

I agree.  However, automatically finalizing all other interpreters at the 
beginning of Py_FinalizeEx() doesn't introduce any new problems.  At the same 
time, it makes sure that any resources in use in other interpreters have a 
chance to be released and that global resources used there don't cause crashes 
during runtime finalization.

> The simple option, which is sadly a backward incompatible change, is to raise 
> a fatal error in Py_Finalize() if a subinterpreter is still running.

As long as we require Py_FinalizeEx() to be called from the main interpreter, I 
don't see how the proposed change (in PR #17575) would be backward incompatible.

> Maybe we can start by logging a warning into stderr for now, before 
> introducing the backward incompatible change. Maybe even only emit it when -X 
> dev is used? https://docs.python.org/dev/library/devmode.html

I'd like to see a ResourceWarning if any other interpreters were still around.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36225] [subinterpreters] Lingering subinterpreters should be implicitly cleared on shutdown

[STINNER Victor]

STINNER Victor  added the comment:

I would not qualify the new Python 3.7 behavior (call Py_FatalError()) as a 
regression, so I remove "3.7regression" keyword.

Also, I don't think that we can easily fix this issue in a stable branch, I 
would prefer to start working on implementation this issue in the master 
branch, and only later consider to *maybe* backport the change. So I changed 
the Python version to "3.10".

I see multiple non trivial problems:

* To finalize the subinterpreter A, Py_Finalize() may switch the current Python 
thread state from the main interpreter to a Python thread in the subintrepeter 
A. It can lead to new funny issues.

* A subinterpreter can be stuck for whatever reason and refuse to stop. For 
example, the subinterpreter A may wait for an even from subinterpreter B. If we 
don't run two interpreters "in parallel" (in threads), it may be stuck forever.

* Python 3.10 still has weird code to allow daemon threads to continue to run 
after Py_Finalize() complete. Maybe we need to add such hacks for 
subinterpreter which fail to be finalized? For example, kill them (exit) as 
soon as they attempt to acquire the GIL. Search for tstate_must_exit() in 
Python/ceval_gil.h.


By the way, currently Py_Finalize() calls PyInterpreterState_Clear() which call 
object finalizers in the main thread of the main interpreter, whereas these 
finalizers might expect to be called from the thread which created them. I 
don't know if we must change anything else.

Again, all these problems are very complex :-(

The simple option, which is sadly a backward incompatible change, is to raise a 
fatal error in Py_Finalize() if a subinterpreter is still running.

Maybe we can start by logging a warning into stderr for now, before introducing 
the backward incompatible change. Maybe even only emit it when -X dev is used? 
https://docs.python.org/dev/library/devmode.html

--
keywords:  -3.7regression
versions: +Python 3.10 -Python 3.7, Python 3.8, Python 3.9

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36225] [subinterpreters] Lingering subinterpreters should be implicitly cleared on shutdown

[Ned Deily]

Ned Deily  added the comment:

I note this is marked as a 3.7regression and still open. Since the cutoff for 
the final 3.7 bugfix mode release is in a few days, I'm assuming this means 
that 3.7 users will have to live with this regression.  If you feel that is a 
problem, speak up now.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36225] [subinterpreters] Lingering subinterpreters should be implicitly cleared on shutdown

[STINNER Victor]

Change by STINNER Victor :


--
components: +Subinterpreters
title: Lingering subinterpreters should be implicitly cleared on shutdown -> 
[subinterpreters] Lingering subinterpreters should be implicitly cleared on 
shutdown

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com