On Tue, 14 Dec 2021 10:38:25 -0700 Eric Snow <ericsnowcurren...@gmail.com> wrote: > > So we (the core devs) would effectively be requiring those extensions > to support subinterpreters, regardless of letting them opt out. This > situation has been weighing heavily on my mind since Nathaniel brought > this up. Here are some ideas I've had or heard of about what we could > do to help: > > * add a page to the C-API documentation about how to support subinterpreters > * identify the extensions most likely to be impacted and offer to help > * add more helpers to the C-API to make adding subinterpreter support > less painful > * fall back to loading the extension in its own namespace (e.g. use > ldm_open()) > * fall back to copying the extension's file and loading from the copied file > * ...
As a data point, in PyArrow, we have a bunch of C++ code that interacts with Python but doesn't belong in a particular Python module. That C++ code can of course have global state, including perhaps Python objects. What might be nice would be a C API to allow creating interpreter-local opaque structs, for example: void* Py_GetInterpreterLocal(const char* unique_name); void* Py_SetInterpreterLocal(const char* unique_name, void* ptr, void(*)() destructor); Then in extension code you'd be able to write, e.g.: typedef struct { PyObject* cached_decimal; std::vector<std::string> some_other_data; } MyLocalState ; static void destroy_state(MyLocalState* state) { Py_XDECREF(state->cached_decimal); delete state; } static MyLocalState* get_state() { MyLocalState* state = NULL; state = Py_GetInterpreterLocal("pyarrow._lib.internal"); if (state == NULL) { state = new MyLocalState; Py_SetInterpreterLocal("pyarrow._lib.internal", state, destroy_state); } return state; } PyObject* get_decimal_module() { MyLocalState* state = get_state(); if (state->cached_decimal == NULL) { state->cached_decimal = PyImport_ImportModule("decimal"); } return state->cached_decimal; } > > I'd appreciate your thoughts on what we can do to help. Thanks! > > -eric _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/TTVWSOOVMY2AKYARI5VSQ3RF4U7DHLX6/ Code of Conduct: http://python.org/psf/codeofconduct/