On Tue, 14 Dec 2021 10:38:25 -0700
Eric Snow <[email protected]> wrote:
>
> So we (the core devs) would effectively be requiring those extensions
> to support subinterpreters, regardless of letting them opt out. This
> situation has been weighing heavily on my mind since Nathaniel brought
> this up. Here are some ideas I've had or heard of about what we could
> do to help:
>
> * add a page to the C-API documentation about how to support subinterpreters
> * identify the extensions most likely to be impacted and offer to help
> * add more helpers to the C-API to make adding subinterpreter support
> less painful
> * fall back to loading the extension in its own namespace (e.g. use
> ldm_open())
> * fall back to copying the extension's file and loading from the copied file
> * ...
As a data point, in PyArrow, we have a bunch of C++ code that interacts
with Python but doesn't belong in a particular Python module. That C++
code can of course have global state, including perhaps Python objects.
What might be nice would be a C API to allow creating interpreter-local
opaque structs, for example:
void* Py_GetInterpreterLocal(const char* unique_name);
void* Py_SetInterpreterLocal(const char* unique_name,
void* ptr, void(*)() destructor);
Then in extension code you'd be able to write, e.g.:
typedef struct {
PyObject* cached_decimal;
std::vector<std::string> some_other_data;
} MyLocalState ;
static void destroy_state(MyLocalState* state) {
Py_XDECREF(state->cached_decimal);
delete state;
}
static MyLocalState* get_state() {
MyLocalState* state = NULL;
state = Py_GetInterpreterLocal("pyarrow._lib.internal");
if (state == NULL) {
state = new MyLocalState;
Py_SetInterpreterLocal("pyarrow._lib.internal", state,
destroy_state);
}
return state;
}
PyObject* get_decimal_module() {
MyLocalState* state = get_state();
if (state->cached_decimal == NULL) {
state->cached_decimal = PyImport_ImportModule("decimal");
}
return state->cached_decimal;
}
>
> I'd appreciate your thoughts on what we can do to help. Thanks!
>
> -eric
_______________________________________________
Python-Dev mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at
https://mail.python.org/archives/list/[email protected]/message/TTVWSOOVMY2AKYARI5VSQ3RF4U7DHLX6/
Code of Conduct: http://python.org/psf/codeofconduct/
