Hi all, CPython has supported multiple interpreters (in the same process) for a long time, but only through the C-API. I'm working on exposing that functionality to Python code (see PEP 554), aiming for 3.12. I expect that users will find the feature useful (particularly with a per-interpreter GIL--see PEP 684) and that it will be used a lot more over the coming years. This has the potential to impact extension module projects, especially large ones like numpy, which is why I'm reaching out to you.
Use of multiple interpreters depends on isolation between them. When an extension module is imported in multiple interpreters, it is loaded separately into a new module object in each. Extensions often store module data/state in C globals, which means the the multiple instances end up sharing data. This causes problems, more so once we have one GIL per interpreter. Over the years we have added machinery to help extensions get the necessary isolation, moving away from global variables. This includes PEPs 384, 3121, and 489. This has culminated in the guide you can find in PEP 630. Note that nothing should change when only a single interpreter is in use (basically the status quo). With PEP 684, importing an incompatible extension outside the main (initial) interpreter will now be an ImportError. (Currently the behavior is undefined and too often results in hard-to-debug failures and crashes.) Thus extension module maintainers do have the option to *not* support multiple interpreters. Unfortunately, that doesn't mean their users won't pester them about adding support. We all recognize how that dynamic can be draining on a project. The potential burden on maintainers is a serious factor for these upcoming changes. numpy is likely to be affected more than any other project. That's why I'm starting this thread. PEP 684 discusses all of the above. What I'm after with this thread is: * to make sure the numpy maintainers are clear on what interpreter isolation requires of the project * a clear picture of what changes numpy would need (and how much work that would be) * feedback on what the CPython team can do to minimize that work (incl. adding new C APIs) I'm fine with having the discussion here, but I will probably create a new category on discuss.python.org for a variety of similar threads related to multiple interpreters and supporting them. Having our discussion there may lead to more participation from more CPython core devs than just me. Do you have any preference for or against any particular venue? Thanks! -eric _______________________________________________ NumPy-Discussion mailing list -- numpy-discussion@python.org To unsubscribe send an email to numpy-discussion-le...@python.org https://mail.python.org/mailman3/lists/numpy-discussion.python.org/ Member address: arch...@mail-archive.com
