Hi Nathaniel, Thanks for your very interesting analysis :-)
Le ven. 17 avr. 2020 à 23:12, Nathaniel Smith <n...@pobox.com> a écrit : > - The asyncio designers (esp. Guido) did a very extensive analysis of > these libraries' design choices, spoke to the maintainers about what > they'd learned from hard experience, etc. asyncio and Twisted could perfectly live outside CPython code base. asyncio even started as a 3rd party project named Tulip for Python 3.3! Subinterpreters are different: the implementation rely a lot on CPython internals. Developing it outside CPython would be way harder if not impossible. For example, Eric had to add PyThreadState.id and PyInterpreterState.id members. He made many changes in Py_NewInterpreter() and Py_EndInterpreter() which are part of Python internals. Moreover, there are many large refactoring changes which are required to properly implement subinterpreters, especially to better isolate each others: _PyRuntimeState structure to better identify shared globals, PyConfig API (PEP 587), move more structures per-interpreter (GC state, warnings state, pending calls, etc.). > - Even today, the limitations imposed by the stdlib release cycle > still add substantial difficulty to maintaining asyncio From what I understood, asyncio has a very large API since it has tons of features (subprocesses, transports and protocols, streams, futures and tasks, etc.) and requires a lot of glue code to provide the same API on all platforms which is really hard. I understood that the subinterpreters module is very light: just a few functions to provide "channels" and that's it (in short). $ ./python Python 3.9.0a5+ (heads/master:6a9e80a931, Apr 21 2020, 02:28:15) >>> import _xxsubinterpreters >>> len(dir(_xxsubinterpreters)) 30 >>> import asyncio >>> len(dir(asyncio)) 122 >>> dir(_xxsubinterpreters) ['ChannelClosedError', 'ChannelEmptyError', 'ChannelError', 'ChannelID', 'ChannelNotEmptyError', 'ChannelNotFoundError', 'InterpreterID', 'RunFailedError', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__spec__', '_channel_id', 'channel_close', 'channel_create', 'channel_destroy', 'channel_list_all', 'channel_recv', 'channel_release', 'channel_send', 'create', 'destroy', 'get_current', 'get_main', 'is_running', 'is_shareable', 'list_all', 'run_string'] > OTOH, AFAICT the new concurrency model in PEP 554 has never actually > been used, and it isn't even clear whether it's useful at all. Antoine (BDFL-delegate of the PEP) wants to mark the PEP as provisional: it means that we *can* still remove the whole module later if we decide that it's not worth it. The same was done with asyncio. By the way, Antoine was also the BDFL-delegate of the heavy PEP 3156 (asyncio) ;-) > Designing useful concurrency models is *stupidly* hard. And on top of > that, it requires major reworks of the interpreter internals + > disrupts the existing C extension module ecosystem -- which is very > different from asyncio, where folks who didn't use it could just > ignore it. A lot of work done to isolate subinterpreters also helps to "fix" CPython internals. For example, converting C extension modules to multiphase initialization (PEP 489) and add a module state helps to destroy all module objects at exit and also helps to break reference cycles in the garbage collector. It helps to fix this issue created in 2007: "Py_Finalize() doesn't clear all Python objects at exit" https://bugs.python.org/issue1635741 It also enhances the implementation of the "embed Python" use case: "leak" less objects at exit, better cleanup things, etc. That's why I'm supporter of the overall project: even if subinterpreters never take off, the implementation will make CPython better ;-) FYI it even helps indirectly my project to clean the C API ;-) PEP 554 implementation looks quite small: 2 800 lines of C code (Modules/_xxsubinterpretersmodule.c and Objects/interpreteridobject.c) and 2 100 lines of tests (Lib/test/test__xxsubinterpreters.py). Compare it to asyncio: 13 000 lines (Lib/asyncio/*.py) of Python and 20 800 lines of tests (Lib/test/test_asyncio/*.py). Victor -- Night gathers, and now my watch begins. It shall not end until my death. _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/LUV4AHNVK4LX2FPUTVQUEU7GYZ5HPDVY/ Code of Conduct: http://python.org/psf/codeofconduct/
