[issue10915] Make the PyGILState API compatible with multiple interpreters

Change by Maciej Szulik : -- nosy: +maciej.szulik ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue10915] Make the PyGILState API compatible with multiple interpreters

STINNER Victor added the comment: See also bpo-1021318: "PyThreadState_Next not thread safe". -- ___ Python tracker ___ ___

[issue10915] Make the PyGILState API compatible with multiple interpreters

Change by Paulo Henrique Silva : -- nosy: +phsilva ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue10915] Make the PyGILState API compatible with multiple interpreters

Change by Eric Snow : -- versions: +Python 3.9 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue10915] Make the PyGILState API compatible with multiple interpreters

Change by Ronald Oussoren : -- nosy: +ronaldoussoren ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue10915] Make the PyGILState API compatible with multiple interpreters

Nick Coghlan added the comment: A more recent discussion of this on python-dev: https://mail.python.org/pipermail/python-dev/2019-January/156095.html The situation there appears to be a case of "Hand off an OS level thread from the creating interpreter to a different subinterpreter. As far

[issue10915] Make the PyGILState API compatible with multiple interpreters

Antoine Pitrou added the comment: Patch probably doesn't apply anymore. -- stage: patch review -> needs patch versions: +Python 3.8 -Python 3.5, Python 3.6 ___ Python tracker

[issue10915] Make the PyGILState API compatible with multiple interpreters

Change by Petr Viktorin : -- nosy: +petr.viktorin ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue10915] Make the PyGILState API compatible with multiple interpreters

Roundup Robot added the comment: New changeset e590c632c9fa by Victor Stinner in branch 'default': Add more checks on the GIL https://hg.python.org/cpython/rev/e590c632c9fa -- ___ Python tracker

[issue10915] Make the PyGILState API compatible with multiple interpreters

Changes by Eric Snow ericsnowcurren...@gmail.com: -- nosy: +eric.snow versions: +Python 3.5, Python 3.6 -Python 3.3 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10915 ___

[issue10915] Make the PyGILState API compatible with multiple interpreters

Roundup Robot added the comment: New changeset 5d078b0bae75 by Victor Stinner in branch 'default': Issue #19787: PyThread_set_key_value() now always set the value http://hg.python.org/cpython/rev/5d078b0bae75 -- nosy: +python-dev ___ Python tracker

[issue10915] Make the PyGILState API compatible with multiple interpreters

STINNER Victor added the comment: See also issue #15751. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10915 ___ ___ Python-bugs-list mailing

[issue10915] Make the PyGILState API compatible with multiple interpreters

Changes by STINNER Victor victor.stin...@gmail.com: -- nosy: +haypo ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10915 ___ ___ Python-bugs-list

[issue10915] Make the PyGILState API compatible with multiple interpreters

Nick Coghlan ncogh...@gmail.com added the comment: Graham - the cases you describe are the things I was saying don't currently work in my post and wouldn't be helped by Antoine's patch. Your thoughts on how we could possibly make it work actually parallel mine (although there may be fun and

[issue10915] Make the PyGILState API compatible with multiple interpreters

Antoine Pitrou pit...@free.fr added the comment: Graham - the cases you describe are the things I was saying don't currently work in my post and wouldn't be helped by Antoine's patch. Your thoughts on how we could possibly make it work actually parallel mine (although there may be fun and

[issue10915] Make the PyGILState API compatible with multiple interpreters

Nick Coghlan ncogh...@gmail.com added the comment: Good point - consider that comment revised to refer to the GIL acquisition counter in the thread state struct. It may just be a matter of having ThreadState_Swap complain loudly if the gilstate_counter isn't set to a value it knows how to

[issue10915] Make the PyGILState API compatible with multiple interpreters

Antoine Pitrou pit...@free.fr added the comment: The bulk of use cases is going to be simple callbacks via the same thread that called out of Python in the first place. [...] This is what SWIG effectively does in its generated wrappers for callbacks. Thanks for clarifying. I agree the TLS

[issue10915] Make the PyGILState API compatible with multiple interpreters

Nick Coghlan ncogh...@gmail.com added the comment: There's no point improving the multiple interpreter support if it doesn't help applications that are currently broken because of that lack of support. I believe the patch as currently proposed actually makes things *worse*. With autoTLSkey as

[issue10915] Make the PyGILState API compatible with multiple interpreters

Nick Coghlan ncogh...@gmail.com added the comment: Added Mark Hammond to the nosy list, as the original author and implementor of PEP 311 (which added the GILState APIs). Mark, since your PEP deliberately punted on multiple interpreter support, feel free to take yourself off the list again.

[issue10915] Make the PyGILState API compatible with multiple interpreters

Graham Dumpleton graham.dumple...@gmail.com added the comment: Nick, I think you are making the wrong assumption that an external threads will only ever call into the same interpreter. This is not the case. In mod_wsgi and mod_python there is a pool of external threads that for distinct HTTP

[issue10915] Make the PyGILState API compatible with multiple interpreters

New submission from Antoine Pitrou pit...@free.fr: It should be relatively easy to devise a new PyGILState API with support for multiple interpreters. We just need two new functions (similar to the two existing ones) taking a PyInterpreterState* parameter; a TLS key can be added to the

[issue10915] Make the PyGILState API compatible with multiple interpreters

Changes by Antoine Pitrou pit...@free.fr: -- nosy: +ncoghlan ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10915 ___ ___ Python-bugs-list mailing

[issue10915] Make the PyGILState API compatible with multiple interpreters

Antoine Pitrou pit...@free.fr added the comment: Here is a sketch, including conversion of ctypes to the new API. Converting sqlite would require a bit more work. -- keywords: +patch Added file: http://bugs.python.org/file20416/gilstateinterp.patch

[issue10915] Make the PyGILState API compatible with multiple interpreters

Changes by Antoine Pitrou pit...@free.fr: Removed file: http://bugs.python.org/file20416/gilstateinterp.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10915 ___

[issue10915] Make the PyGILState API compatible with multiple interpreters

Changes by Antoine Pitrou pit...@free.fr: -- stage: needs patch - patch review Added file: http://bugs.python.org/file20417/gilstateinterp.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10915

[issue10915] Make the PyGILState API compatible with multiple interpreters

Graham Dumpleton graham.dumple...@gmail.com added the comment: Can you please provide an example of what user would do and what changes existing extension modules would need to make? When I looked at this exact problem some time back, I worked out that you probably only need a single new

[issue10915] Make the PyGILState API compatible with multiple interpreters

Antoine Pitrou pit...@free.fr added the comment: Can you please provide an example of what user would do and what changes existing extension modules would need to make? The patch contains such a change for ctypes. It's quite simple actually. By default stuff would work on the main

[issue10915] Make the PyGILState API compatible with multiple interpreters

Nick Coghlan ncogh...@gmail.com added the comment: A TLS based approach would presumably allow an embedding application like mod_wsgi to tinker with the state of threads created by naive modules that are unaware of the existence of subinterpreters. That said, I don't see anything that

[issue10915] Make the PyGILState API compatible with multiple interpreters

Antoine Pitrou pit...@free.fr added the comment: A TLS based approach would presumably allow an embedding application like mod_wsgi to tinker with the state of threads created by naive modules that are unaware of the existence of subinterpreters. The question is how mod_wsgi could know about

[issue10915] Make the PyGILState API compatible with multiple interpreters

Graham Dumpleton graham.dumple...@gmail.com added the comment: The bulk of use cases is going to be simple callbacks via the same thread that called out of Python in the first place. Thus ultimately all it is doing is: Py_BEGIN_ALLOW_THREADS Call into some foreign C library. C library wants

[issue10915] Make the PyGILState API compatible with multiple interpreters

Graham Dumpleton graham.dumple...@gmail.com added the comment: As to the comment: IMO we should really promote clean APIs which allow solving the whole problem, rather than devise an internal hack to try to improve things slightly. The reality is that if you force a change on every single