Eric Snow <[email protected]> added the comment:
I don't think this change is the right way to go (yet), but something related
might be. First, let's be clear on the status quo for CPython. (This has
gotten long, but I want to be clear.)
Status Quo
============
For simplicity sake, let's say nearly all the code operates relative to the 3
levels of runtime state:
* global - _PyRuntimeState
* interpreter - PyInterpreterState
* thread - PyThreadState
Furthermore, there are 3 groups of functions in the C-API:
* context-sensitive - operate relative to the current Python thread
* runtime-dependent - operate relative to some part of the runtime state,
regardless of thread
* runtime-independent - have nothing to do with CPython's runtime state
Most of the C-API is context-sensitive. A small portion is runtime-dependent.
A handful of functions are runtime-independent (effectively otherwise stateless
helper functions that only happen to be part of the C-API).
Each context-sensitive function relies on there being a "runtime context" it
can use relative to the current OS thread. That context consists of the
current (i.e. active) PyThreadState, the corresponding PyInterpreterState, and
the global _PyRuntimeState. That context is derived from data in TSS (see
caveats below). This group includes most of the C-API.
Each runtime-dependent function operates against one or more runtime state
target, regardless of the current thread context (or even if there isn't one).
The target state (e.g. PyInterpreterState) is always passed explicitly. Again,
this is only a small portion of the C-API.
Caveats:
* for context-sensitive functions, we get the global runtime state from the
global C variable (_PyRuntime) rather than via the implicit thread context
* for some of the runtime-dependent functions that target _PyRuntimeState, we
rely on the global C variable
All of this is the pattern we use currently. Using TSS to identify the
implicit runtime context has certain benefits and costs:
benefits:
* sticking with the status quo means no backward incompatibility for existing
C-extension code
* easier to distinguish the context-sensitive functions from the
runtime-dependent ones
* (debatable) callers don't have to track, nor pass through, an extra argument
costs:
* extra complexity in keeping TSS correct
* makes the C-API bigger (extra macros, etc.)
Alternative
=============
For every context-sensitive function we could add a new first parameter,
"context", that provides the runtime context to use. That would be something
like this:
struct {
PyThreadState *tstate;
...
} PyRuntimeContext;
The interpreter state and global runtime state would still be accessible via
the same indirection we have now.
Taking this alternative would eliminate the previous costs. Having a
consistent "PyRuntimeContext *context" first parameter would maintain the easy
distinction from runtime-dependent functions. Asking callers to pass in the
context explicitly is probably better regardless. As to backward
compatibility, we could maintain a shim to bridge between the old way and the
new.
About the C-global _PyRuntime
==============================
Currently the global runtime state (_PyRuntimeState) is stored in a static
global C variable, _PyRuntime. I added it at the time I consolidated many of
the existing C globals into a single struct. Having a C global makes it easy
to do the wrong thing, so it may be good to do something else.
That would mean allocating a _PyRuntimeState on the heap early in startup and
pass that around where needed. I expect that would not have any meaningful
performance penalty. It would probably also simplify some of the code we
currently use to manage _PyRuntime correctly.
As a bonus, this would be important if we decided that
multiple-runtimes-per-process were a desirable thing. That's a neat idea,
though I don't see a need currently. So on its own it's not really a
justification for dropping a static _PyRuntime. :) However, I think the other
reasons are enough.
Conclusions
====================
This issue has a specific objective that I think is premature. We have an
existing pattern and we should stick with that until we decide to change to a
new pattern. That said, a few things should get corrected and we should
investigate alternative patterns for the context-sensitive C-API.
As to getting rid of the _PyRuntime global variable in favor of putting it on
the heap, I'm not opposed. However, doing so should probably be handled in a
separate issue.
Here are my thoughts on actionable items:
1. look for a better pattern for the context-sensitive C-API
2. clearly document which of the 3 groups each C-API function belongs to
3. add a "runtime" field to the PyInterpreterState pointing to the parent
_PyRuntimeState
4. (maybe) add a _PyRuntimeState_GET() macro, a la PyThreadState_GET()
5. for context-sensitive C-API that uses the global runtime state, get it from
the current PyInterpreterState
6. for runtime-dependent C-API that targets the global runtime state, ensure
the _PyRuntimeState is always an explicit parameter
7. (maybe) drop _PyRuntime and create a _PyRuntimeState on the heap during
startup to pass around
----------
nosy: +ncoghlan, steve.dower
_______________________________________
Python tracker <[email protected]>
<https://bugs.python.org/issue36710>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
