[issue36710] Pass _PyRuntimeState as an argument rather than using the _PyRuntime global variable

[STINNER Victor]

STINNER Victor  added the comment:


New changeset 61b649296110dcb38e67db382a7e8427c5eb75d0 by Victor Stinner in 
branch 'master':
bpo-36710: Pass tstate explicitly in abstract.c (GH-21075)
https://github.com/python/cpython/commit/61b649296110dcb38e67db382a7e8427c5eb75d0


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36710] Pass _PyRuntimeState as an argument rather than using the _PyRuntime global variable

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +20243
pull_request: https://github.com/python/cpython/pull/21075

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36710] Pass _PyRuntimeState as an argument rather than using the _PyRuntime global variable

[Mark Shannon]

Mark Shannon  added the comment:

Even if `_PyRuntime` ends up as just a list of interpreters and doesn't 
disappear completely, it won't be used anything like as much as it is now.

Many of the functions that it getting passed to will no longer need it, so why 
bother passing it now?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36710] Pass _PyRuntimeState as an argument rather than using the _PyRuntime global variable

[STINNER Victor]

STINNER Victor  added the comment:

> Instead of passing `_PyRuntimeState` around everywhere, why not just let it 
> disappear in time.

Passing runtime (_PyRuntimeState) is a temporary move until more and more 
fields are moved from _PyRuntimeState into PyInterpreterState. I just created 
bpo-39984 "Move some ceval fields from _PyRuntime.ceval to 
PyInterpreterState.ceval" yesterday ;-)

Once we will manage to move the GIL into PyInterpreterState, we would still 
have to pass PyInterpreterState or PyThreadState to function which require to 
access the GIL. Passing explicitly runtime is a first step to prepare to 
migration to PyInterpreterState or PyThreadState. My intent is to show that 
many functions rely on "global variables": pass these variables instead.

If you are thinking about getting the current Python thread state using a 
thread local storage, that's a different topic and I'm not aware of an open 
issue to track this idea.


> Currently `_PyRuntimeState` manages "global" state, mainly the GIL and some 
> config. Once the GIL has been migrated to the sub-interpreters, the config 
> part can be factored out and `_PyRuntimeState` can just disappear.

I don't think that we will be able to fully remove _PyRuntimeState. It seems 
like the PEP 554 "Multiple Interpreters in the Stdlib" requires a registry of 
interpreters and it currently lives in _PyRuntimeState.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36710] Pass _PyRuntimeState as an argument rather than using the _PyRuntime global variable

[Steve Dower]

Steve Dower  added the comment:

> Instead of passing `_PyRuntimeState` around everywhere, why not just let it 
> disappear in time.

Agreed. It's valuable to pass the thread state, but the runtime state should 
only be needed to create a new thread state (and arguably not even then).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36710] Pass _PyRuntimeState as an argument rather than using the _PyRuntime global variable

[Mark Shannon]

Mark Shannon  added the comment:

Instead of passing `_PyRuntimeState` around everywhere, why not just let it 
disappear in time.

Currently `_PyRuntimeState` manages "global" state, mainly the GIL and some 
config.
Once the GIL has been migrated to the sub-interpreters, the config part can be 
factored out and `_PyRuntimeState` can just disappear.

--
nosy: +Mark.Shannon

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36710] Pass _PyRuntimeState as an argument rather than using the _PyRuntime global variable

[Eric Snow]

Eric Snow  added the comment:

Thanks, Victor!

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36710] Pass _PyRuntimeState as an argument rather than using the _PyRuntime global variable

[STINNER Victor]

STINNER Victor  added the comment:

I continued this work by passing tstate to internal C functions: bpo-38644.

I also added PyInterpreterState.runtime field, so it's now possible to retrieve 
the runtime from tstate:

runtime = tstate->interp->runtime;

I wrote an article on passing tstate to internal C functions:

https://vstinner.github.io/cpython-pass-tstate.html

I consider that this issue is now done. I close the issue.

--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36710] Pass _PyRuntimeState as an argument rather than using the _PyRuntime global variable

[STINNER Victor]

STINNER Victor  added the comment:


New changeset 01b1cc12e7c6a3d6a3d27ba7c731687d57aae92a by Victor Stinner in 
branch 'master':
bpo-36710: Add PyInterpreterState.runtime field (GH-17270)
https://github.com/python/cpython/commit/01b1cc12e7c6a3d6a3d27ba7c731687d57aae92a


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36710] Pass _PyRuntimeState as an argument rather than using the _PyRuntime global variable

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +16763
pull_request: https://github.com/python/cpython/pull/17270

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36710] Pass _PyRuntimeState as an argument rather than using the _PyRuntime global variable

[STINNER Victor]

STINNER Victor  added the comment:


New changeset 2e96906da764402b4c8062dbf99171ca506f9e12 by Victor Stinner in 
branch 'master':
bpo-36710: Pass tstate parameter to GC collect() (GH-17267)
https://github.com/python/cpython/commit/2e96906da764402b4c8062dbf99171ca506f9e12


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36710] Pass _PyRuntimeState as an argument rather than using the _PyRuntime global variable

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +16759
pull_request: https://github.com/python/cpython/pull/17267

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36710] Pass _PyRuntimeState as an argument rather than using the _PyRuntime global variable

[STINNER Victor]

STINNER Victor  added the comment:

I started a thread on python-dev about this issue:
"Pass the Python thread state to internal C functions"
https://mail.python.org/archives/list/python-...@python.org/thread/PQBGECVGVYFTVDLBYURLCXA3T7IPEHHO/

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36710] Pass _PyRuntimeState as an argument rather than using the _PyRuntime global variable

[Paulo Henrique Silva]

Change by Paulo Henrique Silva :


--
nosy: +phsilva

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36710] Pass _PyRuntimeState as an argument rather than using the _PyRuntime global variable

[STINNER Victor]

STINNER Victor  added the comment:


New changeset b45d259bdda1de2b2d369458a9ad2e4d6f750687 by Victor Stinner in 
branch 'master':
bpo-36710: Use tstate in pylifecycle.c (GH-14249)
https://github.com/python/cpython/commit/b45d259bdda1de2b2d369458a9ad2e4d6f750687


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36710] Pass _PyRuntimeState as an argument rather than using the _PyRuntime global variable

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +14082
pull_request: https://github.com/python/cpython/pull/14250

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36710] Pass _PyRuntimeState as an argument rather than using the _PyRuntime global variable

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +14081
pull_request: https://github.com/python/cpython/pull/14249

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36710] Pass _PyRuntimeState as an argument rather than using the _PyRuntime global variable

[STINNER Victor]

STINNER Victor  added the comment:


New changeset 987a0dcfa1302df6c1ed8cf14762dc18628e3f33 by Victor Stinner in 
branch 'master':
bpo-36710: Remove PyImport_Cleanup() function (GH-14221)
https://github.com/python/cpython/commit/987a0dcfa1302df6c1ed8cf14762dc18628e3f33


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36710] Pass _PyRuntimeState as an argument rather than using the _PyRuntime global variable

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +14059
pull_request: https://github.com/python/cpython/pull/14221

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36710] Pass _PyRuntimeState as an argument rather than using the _PyRuntime global variable

[STINNER Victor]

STINNER Victor  added the comment:


New changeset 0a28f8d379544eee897979da0ce99f0b449b49dd by Victor Stinner in 
branch 'master':
bpo-36710: Add tstate parameter in import.c (GH-14218)
https://github.com/python/cpython/commit/0a28f8d379544eee897979da0ce99f0b449b49dd


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36710] Pass _PyRuntimeState as an argument rather than using the _PyRuntime global variable

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +14056
pull_request: https://github.com/python/cpython/pull/14218

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36710] Pass _PyRuntimeState as an argument rather than using the _PyRuntime global variable

[STINNER Victor]

STINNER Victor  added the comment:

I wrote my notes on this issue there:
https://pythoncapi.readthedocs.io/runtime.html

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36710] Pass _PyRuntimeState as an argument rather than using the _PyRuntime global variable

[STINNER Victor]

STINNER Victor  added the comment:


New changeset 838f26402de82640698c38ea9d2be65c6cf780d6 by Victor Stinner in 
branch 'master':
bpo-36710: Pass explicitly tstate in sysmodule.c (GH-14060)
https://github.com/python/cpython/commit/838f26402de82640698c38ea9d2be65c6cf780d6


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36710] Pass _PyRuntimeState as an argument rather than using the _PyRuntime global variable

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +13922
pull_request: https://github.com/python/cpython/pull/14060

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36710] Pass _PyRuntimeState as an argument rather than using the _PyRuntime global variable

[STINNER Victor]

STINNER Victor  added the comment:


New changeset 438a12dd9d85f463c0bb7bf1505cd87b98b98170 by Victor Stinner in 
branch 'master':
bpo-36710: Add tstate parameter in ceval.c (GH-13547)
https://github.com/python/cpython/commit/438a12dd9d85f463c0bb7bf1505cd87b98b98170


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36710] Pass _PyRuntimeState as an argument rather than using the _PyRuntime global variable

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +13459

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36710] Pass _PyRuntimeState as an argument rather than using the _PyRuntime global variable

[STINNER Victor]

STINNER Victor  added the comment:


New changeset b4bdecd0fc9112b60a81fec171bc78bc13f2f59c by Victor Stinner in 
branch 'master':
bpo-36710: Add tstate parameter in errors.c (GH-13540)
https://github.com/python/cpython/commit/b4bdecd0fc9112b60a81fec171bc78bc13f2f59c


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36710] Pass _PyRuntimeState as an argument rather than using the _PyRuntime global variable

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +13454

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36710] Pass _PyRuntimeState as an argument rather than using the _PyRuntime global variable

[STINNER Victor]

STINNER Victor  added the comment:


New changeset 09532feeece39d5ba68a0d47115ce1967bfbd58e by Victor Stinner in 
branch 'master':
bpo-36710: Add 'ceval' local variable to ceval.c (GH-12934)
https://github.com/python/cpython/commit/09532feeece39d5ba68a0d47115ce1967bfbd58e


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36710] Pass _PyRuntimeState as an argument rather than using the _PyRuntime global variable

[Steve Dower]

Steve Dower  added the comment:

I think Neil is right, though I believe we'll have a clear enough internal 
boundary that we should only rarely have to maintain TSS for the sake of legacy 
callers. The build option should just turn off the entire legacy API, which 
would also make it easier to remove one day.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36710] Pass _PyRuntimeState as an argument rather than using the _PyRuntime global variable

[Neil Schemenauer]

Neil Schemenauer  added the comment:

I think there are two questions to answer.  First, do we want to support 
multiple runtimes per process?  Second, if we do, what is the best way to do 
that?  Some people would argue that multiple runtimes are not needed or are too 
hard to do.  Maybe they are correct, I'm not sure.  We should try to get a 
consensus on that first question.

If we do decide to do it, then we need to answer the second question.  Passing 
a "context" argument around seems the best solution.  That is how the Java JNI 
does it.  It sounds like that's how Javascript VMs do it too.  We don't need to 
get creative.  Look at what other VMs do and copy the best idea.

If we do decide to do it, evolving the codebase and all extension modules is 
going to be a massive task.  I would imagine that we can have a backwards 
compatible API layer that uses TSS.  The layer that passes context explicitly 
would still have to maintain the TSS.  There could be a build option that turns 
that backwards compatibility on or off.  If off, you would gain some 
performance advantage because TSS does not have to be kept up-to-date.

My feeling right now that even though this is a massive job, it is the correct 
thing to do.  CPUs continue to gain cores.  Improving CPython's ability to do 
multi-threading and multi-processing should be a priority for CPython core 
developers.

--
nosy: +nascheme

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36710] Pass _PyRuntimeState as an argument rather than using the _PyRuntime global variable

[Eric Snow]

Eric Snow  added the comment:

FWIW, I don't mean to side-track this issue.  If we want to have any further 
discussion about broader solutions then let's take this to capi-sig.  In fact, 
I've started a thread there.  I'd post the link, but I think it got stuck in 
moderation. :)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36710] Pass _PyRuntimeState as an argument rather than using the _PyRuntime global variable

[Jeroen Demeyer]

Jeroen Demeyer  added the comment:

Changing *every* C API function to include a state parameter looks very 
cumbersome. Another alternative would be to store the interpreter state in 
every Python object (or every class, that would be sufficient). That way, you 
would only need to pass context to C API functions which do not take a Python 
object as argument.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36710] Pass _PyRuntimeState as an argument rather than using the _PyRuntime global variable

[Steve Dower]

Steve Dower  added the comment:

Changing every API to take the context parameter would bring us into alignment 
with the JavaScript VMs.

I'm working on a project that embeds a few of these, as well as Python, and our 
thread management is much worse than their context parameter. Though I'm of 
course very sympathetic to the compatibility argument (but then the shims would 
just load the context from TSS and pass it around, so they're not too bad).

Eric's breakdown of context scopes seems spot on, and it means that we only 
really need the thread state to be passed around. The few places that would be 
satisfied by runtime state now (GIL, GC) should become interpreter state, which 
is most easily found from a thread state anyway.

Runtime state should eventually probably become runtime configuration (those 
settings we need to create interpreters) and a minimum amount of state to track 
live interpreters. I see no reason to pass it around anywhere other than 
interpreter creation, and as a transitional step toward that goal it should be 
accessible through the active interpreter state.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36710] Pass _PyRuntimeState as an argument rather than using the _PyRuntime global variable

[Eric Snow]

Eric Snow  added the comment:

I don't think this change is the right way to go (yet), but something related 
might be.  First, let's be clear on the status quo for CPython.  (This has 
gotten long, but I want to be clear.)


Status Quo


For simplicity sake, let's say nearly all the code operates relative to the 3 
levels of runtime state:

* global  - _PyRuntimeState
* interpreter - PyInterpreterState
* thread  - PyThreadState

Furthermore, there are 3 groups of functions in the C-API:

* context-sensitive   - operate relative to the current Python thread
* runtime-dependent   - operate relative to some part of the runtime state, 
regardless of thread
* runtime-independent - have nothing to do with CPython's runtime state

Most of the C-API is context-sensitive.  A small portion is runtime-dependent.  
A handful of functions are runtime-independent (effectively otherwise stateless 
helper functions that only happen to be part of the C-API).

Each context-sensitive function relies on there being a "runtime context" it 
can use relative to the current OS thread.  That context consists of the 
current (i.e. active) PyThreadState, the corresponding PyInterpreterState, and 
the global _PyRuntimeState.  That context is derived from data in TSS (see 
caveats below).  This group includes most of the C-API.

Each runtime-dependent function operates against one or more runtime state 
target, regardless of the current thread context (or even if there isn't one).  
The target state (e.g. PyInterpreterState) is always passed explicitly.  Again, 
this is only a small portion of the C-API.

Caveats:
* for context-sensitive functions, we get the global runtime state from the 
global C variable (_PyRuntime) rather than via the implicit thread context
* for some of the runtime-dependent functions that target _PyRuntimeState, we 
rely on the global C variable

All of this is the pattern we use currently.  Using TSS to identify the 
implicit runtime context has certain benefits and costs:

benefits:
* sticking with the status quo means no backward incompatibility for existing 
C-extension code
* easier to distinguish the context-sensitive functions from the 
runtime-dependent ones
* (debatable) callers don't have to track, nor pass through, an extra argument

costs:
* extra complexity in keeping TSS correct
* makes the C-API bigger (extra macros, etc.)


Alternative
=

For every context-sensitive function we could add a new first parameter, 
"context", that provides the runtime context to use.  That would be something 
like this:

struct {
PyThreadState *tstate;
...
} PyRuntimeContext;

The interpreter state and global runtime state would still be accessible via 
the same indirection we have now.

Taking this alternative would eliminate the previous costs.  Having a 
consistent "PyRuntimeContext *context" first parameter would maintain the easy 
distinction from runtime-dependent functions.  Asking callers to pass in the 
context explicitly is probably better regardless.  As to backward 
compatibility, we could maintain a shim to bridge between the old way and the 
new.


About the C-global _PyRuntime
==

Currently the global runtime state (_PyRuntimeState) is stored in a static 
global C variable, _PyRuntime.  I added it at the time I consolidated many of 
the existing C globals into a single struct.  Having a C global makes it easy 
to do the wrong thing, so it may be good to do something else.

That would mean allocating a _PyRuntimeState on the heap early in startup and 
pass that around where needed.  I expect that would not have any meaningful 
performance penalty.  It would probably also simplify some of the code we 
currently use to manage _PyRuntime correctly.

As a bonus, this would be important if we decided that 
multiple-runtimes-per-process were a desirable thing.  That's a neat idea, 
though I don't see a need currently.  So on its own it's not really a 
justification for dropping a static _PyRuntime. :)  However, I think the other 
reasons are enough.


Conclusions


This issue has a specific objective that I think is premature.  We have an 
existing pattern and we should stick with that until we decide to change to a 
new pattern.  That said, a few things should get corrected and we should 
investigate alternative patterns for the context-sensitive C-API.

As to getting rid of the _PyRuntime global variable in favor of putting it on 
the heap, I'm not opposed.  However, doing so should probably be handled in a 
separate issue.

Here are my thoughts on actionable items:

1. look for a better pattern for the context-sensitive C-API
2. clearly document which of the 3 groups each C-API function belongs to
3. add a "runtime" field to the PyInterpreterState pointing to the parent 
_PyRuntimeState
4. (maybe) add a _PyRuntimeState_GET() macro, a la PyThreadState_GET()
5. for context-sensitive C-API that uses the global runtime

[issue36710] Pass _PyRuntimeState as an argument rather than using the _PyRuntime global variable

[Eric Snow]

Eric Snow  added the comment:

FWIW, PEP 554 is part of a larger project that I've been working on (slowly) 
for several years now. [1]  The concrete objective is to leverage 
subinterpreters as the mechanism by which we can achieve multi-core parallelism 
in Python code.  Moving the GIL (and some other parts of _PyRuntimeState, as 
Victor indicated) down to per-interpreter state is essential to that.

However, I don't thing making _PyRuntime a per-interpreter thing is right.  The 
runtime holds the set of interpreters, as well as any state state shared by the 
interpreters.

Also, to be clear, the status quo is not a problem for me, so make sure I'm not 
used as the justification for the change (thoughtful as that is of Victor). :)


[1] https://github.com/ericsnowcurrently/multi-core-python

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36710] Pass _PyRuntimeState as an argument rather than using the _PyRuntime global variable

[Jeroen Demeyer]

Jeroen Demeyer  added the comment:

> It's wrong to share a single gc state between two interpreters

And what's your solution for that? I'm not asking for a complete 
ready-to-implement answer, but at least a basic idea. Otherwise it's impossible 
for me to judge whether your PR 12934 helps with that or not.

Basically I would like to see something like PEP 579 but for this problem.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36710] Pass _PyRuntimeState as an argument rather than using the _PyRuntime global variable

[Jeroen Demeyer]

Jeroen Demeyer  added the comment:

So what's the relation between _PyRuntime and PyInterpreterState? If the latter 
is a structure per interpreter, what's the point of also making the former per 
interpreter? It would be better to move data from _PyRuntime to 
PyInterpreterState.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36710] Pass _PyRuntimeState as an argument rather than using the _PyRuntime global variable

[STINNER Victor]

STINNER Victor  added the comment:

Jeroen Demeyer:
> Sorry, but I don't see the relation between this issue and PEP 554.

The long term plan for PEP 554 is to support having one GIL per interpreter for 
best performances. The GIL lives in _PyRuntime.

It's not just about the GIL. Currently, the gc module stores its state into 
_PyRuntime. It's wrong to share a single gc state between two interpreters: 
each interpreter should have its own "namespace" completely isolated from the 
other namespaces. For example, _PyRuntime.gc.garbage is a Python list: each 
interpreter should have its own list.

My PR 12934 is only a first step to prepare ceval.c for that.

Said differently, if I understood correctly, each interpreter must have its own 
_PyRuntime instance.

Maybe tomorrow, we will keep a single _PyRuntime instance, *but* my work is 
needed to identify the relationship between the current implementation of 
Python and _PyRuntime.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36710] Pass _PyRuntimeState as an argument rather than using the _PyRuntime global variable

[Jeroen Demeyer]

Jeroen Demeyer  added the comment:

> The long term goal is to support multiple interpreter instances per process:
> Eric Snow's PEP 554 "Multiple Interpreters in the Stdlib"
> https://www.python.org/dev/peps/pep-0554/

Sorry, but I don't see the relation between this issue and PEP 554. It seems to 
me that the PEP is about making subinterpreters available from pure Python 
(instead of only at the C level). It doesn't say anything about the 
*implementation* of subinterpreters, which is what this issue is about.

So I'm still missing the bigger picture where this issue fits in.

> The current implementation of CPython leaks dozens of *Python* objects at 
> exit.

That may be an issue to be fixed, but again I don't see the relation with this 
issue.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36710] Pass _PyRuntimeState as an argument rather than using the _PyRuntime global variable

[STINNER Victor]

STINNER Victor  added the comment:


New changeset 99e69d44f499625786a2e6461a954adcd0037d69 by Victor Stinner in 
branch 'master':
bpo-36710: Fix compiler warning on PyThreadState_Delete() (GH-12962)
https://github.com/python/cpython/commit/99e69d44f499625786a2e6461a954adcd0037d69


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36710] Pass _PyRuntimeState as an argument rather than using the _PyRuntime global variable

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +12888

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36710] Pass _PyRuntimeState as an argument rather than using the _PyRuntime global variable

[STINNER Victor]

STINNER Victor  added the comment:


New changeset 9db0324712f6982d89620b420f507a6aa5da152f by Victor Stinner in 
branch 'master':
bpo-36710: Add runtime parameter in gcmodule.c (GH-12958)
https://github.com/python/cpython/commit/9db0324712f6982d89620b420f507a6aa5da152f


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36710] Pass _PyRuntimeState as an argument rather than using the _PyRuntime global variable

[STINNER Victor]

STINNER Victor  added the comment:


New changeset 10c8e6af910e3a26e59f913a3c1e4830ca71b1af by Victor Stinner in 
branch 'master':
bpo-36710: Add runtime variable in pystate.c (GH-12956)
https://github.com/python/cpython/commit/10c8e6af910e3a26e59f913a3c1e4830ca71b1af


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36710] Pass _PyRuntimeState as an argument rather than using the _PyRuntime global variable

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +12884

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36710] Pass _PyRuntimeState as an argument rather than using the _PyRuntime global variable

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +12882

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36710] Pass _PyRuntimeState as an argument rather than using the _PyRuntime global variable

[STINNER Victor]

STINNER Victor  added the comment:

I created bpo-36724: Clear _PyRuntime at exit.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36710] Pass _PyRuntimeState as an argument rather than using the _PyRuntime global variable

[STINNER Victor]

STINNER Victor  added the comment:

> I don't really understand the rationale for these changes. What's wrong with 
> the global variable _PyRuntime?
> What's the long-term goal for _PyRuntime? If you can't get rid of all 
> occurrences of _PyRuntime, how does it help to get rid of *some* occurences?

The long term goal is to support multiple interpreter instances per process:

Eric Snow's PEP 554 "Multiple Interpreters in the Stdlib"
https://www.python.org/dev/peps/pep-0554/

Right now, there is a single instance of _PyRuntimeState: _PyRuntime. I would 
be interested to have one _PyRuntimeState per interpreter.

Maybe _PyRuntimeState should be reworked in the meanwhile. Right now, it's a 
strange beast: it contains things which are set before Python initialization 
and things which are set after. It contains C types and Python objects. Maybe 
some parts should be moved into PyInterpreterState or even PyThreadState. I 
don't know at this point. It takes time to look at each individual structure 
field...

Anyway, more generally, IMHO it's a bad practice to rely on a global variable. 
Python runtime should be "stateless".

The current implementation of CPython leaks dozens of *Python* objects at exit. 
For example, I started to work on this issue while working on 
https://bugzilla.redhat.com/show_bug.cgi?id=1696322 : Python doesn't clear 2 
warnings variables at exit. When I looked into this issue, I also noticed that 
_PyRuntime.gc.garbage remains *alive* after Py_Finalize().

That's plain wrong: *all* Python objects must be cleared by Py_Finalize(). Two 
interpreters must *not* share any Python object.

Well, the PEP should better explain the rationale than me :-)

--

When I wrote PR 12934, I noticed that even getting the current thread state 
rely on _PyRuntime:

#define _PyThreadState_GET() \

((PyThreadState*)_Py_atomic_load_relaxed(&_PyRuntime.gilstate.tstate_current))

That's wrong in the case of ceval.c: it should be possible to run 
_PyEval_EvalFrameDefault() twice at the same time in two threads using two 
"isolated" interpreters.

Well, PR 12934 doesn't fix all issues. It's just one small step towards 
"stateless" runtime and the even more long term of having one GIL per 
*interpeter*, rather than a single GIL per *process*.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36710] Pass _PyRuntimeState as an argument rather than using the _PyRuntime global variable

[Jeroen Demeyer]

Jeroen Demeyer  added the comment:

I don't really understand the rationale for these changes. What's wrong with 
the global variable _PyRuntime?

What's the long-term goal for _PyRuntime? If you can't get rid of all 
occurrences of _PyRuntime, how does it help to get rid of *some* occurences?

--
nosy: +jdemeyer

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36710] Pass _PyRuntimeState as an argument rather than using the _PyRuntime global variable

[Eric Snow]

Change by Eric Snow :


--
nosy: +eric.snow

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36710] Pass _PyRuntimeState as an argument rather than using the _PyRuntime global variable

[STINNER Victor]

STINNER Victor  added the comment:


New changeset 43125224d6da5febb34101ebfd36536d791d68cd by Victor Stinner in 
branch 'master':
bpo-36710: Add runtime variable to Py_InitializeEx() (GH-12939)
https://github.com/python/cpython/commit/43125224d6da5febb34101ebfd36536d791d68cd


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36710] Pass _PyRuntimeState as an argument rather than using the _PyRuntime global variable

[STINNER Victor]

STINNER Victor  added the comment:


New changeset 8e91c246e468515b877690e090c73f496552541d by Victor Stinner in 
branch 'master':
bpo-36710: Add runtime variable to Py_FinalizeEx() (GH-12937)
https://github.com/python/cpython/commit/8e91c246e468515b877690e090c73f496552541d


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36710] Pass _PyRuntimeState as an argument rather than using the _PyRuntime global variable

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +12863

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36710] Pass _PyRuntimeState as an argument rather than using the _PyRuntime global variable

[STINNER Victor]

STINNER Victor  added the comment:


New changeset b930a2d2b1247bdba560db341ba90a9cbb538eb3 by Victor Stinner in 
branch 'master':
bpo-36710: PyOS_AfterFork_Child() pass runtime parameter (GH-12936)
https://github.com/python/cpython/commit/b930a2d2b1247bdba560db341ba90a9cbb538eb3


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36710] Pass _PyRuntimeState as an argument rather than using the _PyRuntime global variable

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +12861

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36710] Pass _PyRuntimeState as an argument rather than using the _PyRuntime global variable

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +12860

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36710] Pass _PyRuntimeState as an argument rather than using the _PyRuntime global variable

[STINNER Victor]

STINNER Victor  added the comment:


New changeset 8bb3230149538c25c1bacced5e64a3c071475f73 by Victor Stinner in 
branch 'master':
bpo-36710: Add runtime parameter to _PyThreadState_Init() (GH-12935)
https://github.com/python/cpython/commit/8bb3230149538c25c1bacced5e64a3c071475f73


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36710] Pass _PyRuntimeState as an argument rather than using the _PyRuntime global variable

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +12859

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36710] Pass _PyRuntimeState as an argument rather than using the _PyRuntime global variable

[STINNER Victor]

Change by STINNER Victor :


--
keywords: +patch
pull_requests: +12858
stage:  -> patch review

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36710] Pass _PyRuntimeState as an argument rather than using the _PyRuntime global variable

[STINNER Victor]

New submission from STINNER Victor :

Eric Snow moved global variables into a _PyRuntimeState structure which is made 
of sub-structures. There is a single instance of _PyRuntimeState: the 
_PyRuntime global variable.

I would like to add "_PyRuntimeState *" parameters to functions to avoid 
relying directly on _PyRuntime global variable. The long term goal is to have 
"stateless" code: don't rely on global variables, only on input parameters. In 
practice, we will continue to use thread local storage (TLS) to get the 
"current context" like the current interpreter and the current Python thread 
state.

--
components: Interpreter Core
messages: 340772
nosy: vstinner
priority: normal
severity: normal
status: open
title: Pass _PyRuntimeState as an argument rather than using the _PyRuntime 
global variable
versions: Python 3.8

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com