On Tue, 14 Dec 2021 10:38:25 -0700
Eric Snow <ericsnowcurren...@gmail.com> wrote:
> 
> So we (the core devs) would effectively be requiring those extensions
> to support subinterpreters, regardless of letting them opt out.  This
> situation has been weighing heavily on my mind since Nathaniel brought
> this up.  Here are some ideas I've had or heard of about what we could
> do to help:
> 
> * add a page to the C-API documentation about how to support subinterpreters
> * identify the extensions most likely to be impacted and offer to help
> * add more helpers to the C-API to make adding subinterpreter support
> less painful
> * fall back to loading the extension in its own namespace (e.g. use 
> ldm_open())
> * fall back to copying the extension's file and loading from the copied file
> * ...

As a data point, in PyArrow, we have a bunch of C++ code that interacts
with Python but doesn't belong in a particular Python module.  That C++
code can of course have global state, including perhaps Python objects.

What might be nice would be a C API to allow creating interpreter-local
opaque structs, for example:

void* Py_GetInterpreterLocal(const char* unique_name);
void* Py_SetInterpreterLocal(const char* unique_name,
                             void* ptr, void(*)() destructor);


Then in extension code you'd be able to write, e.g.:


typedef struct {
  PyObject* cached_decimal;
  std::vector<std::string> some_other_data;
} MyLocalState ;

static void destroy_state(MyLocalState* state) {
  Py_XDECREF(state->cached_decimal);
  delete state;
}

static MyLocalState* get_state() {
  MyLocalState* state = NULL;
  state = Py_GetInterpreterLocal("pyarrow._lib.internal");
  if (state == NULL) {
    state = new MyLocalState;
    Py_SetInterpreterLocal("pyarrow._lib.internal", state,
                           destroy_state);
  }
  return state;
}


PyObject* get_decimal_module() {
  MyLocalState* state = get_state();
  if (state->cached_decimal == NULL) {
    state->cached_decimal = PyImport_ImportModule("decimal");
  }
  return state->cached_decimal;
}



> 
>  I'd appreciate your thoughts on what we can do to help.  Thanks!
> 
> -eric



_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/TTVWSOOVMY2AKYARI5VSQ3RF4U7DHLX6/
Code of Conduct: http://python.org/psf/codeofconduct/

Reply via email to