Re: [Python-ideas] Exposing CPython's subinterpreter C-API in the stdlib.

2017-05-27 Thread Stephan Houben
Hi Nick, > I guess I'll have to scale back my hopes on that front to be closer to > what Stephan described - even a deep copy equivalent is often going to > be cheaper than a full serialise/transmit/deserialise cycle or some > other form of inter-process communication. I would like to add that in

Re: [Python-ideas] Exposing CPython's subinterpreter C-API in the stdlib.

2017-05-27 Thread Nick Coghlan
On 27 May 2017 at 03:30, Guido van Rossum wrote: > On Fri, May 26, 2017 at 8:28 AM, Nick Coghlan wrote: >> >> [...] assuming the rest of idea works out >> well, we'd eventually like to move to a tiered model where the GIL >> becomes a read/write lock. Most code execution in subinterpreters >> wou

Re: [Python-ideas] Exposing CPython's subinterpreter C-API in the stdlib.

2017-05-26 Thread Guido van Rossum
On Fri, May 26, 2017 at 8:28 AM, Nick Coghlan wrote: > [...] assuming the rest of idea works out > well, we'd eventually like to move to a tiered model where the GIL > becomes a read/write lock. Most code execution in subinterpreters > would then only need a read lock on the GIL, and hence could

Re: [Python-ideas] Exposing CPython's subinterpreter C-API in the stdlib.

2017-05-26 Thread Nick Coghlan
On 26 May 2017 at 23:49, Stephan Houben wrote: > Hi Nick, > > As far as I understand, the (to me) essential difference between your > approach and my proposal is that: > > Approach 1 (PEP-489): >* Single (global) GIL. >* PyObject's may be shared across interpreters (zero-copy transfer) > >

Re: [Python-ideas] Exposing CPython's subinterpreter C-API in the stdlib.

2017-05-26 Thread Stephan Houben
Hi Nick, As far as I understand, the (to me) essential difference between your approach and my proposal is that: Approach 1 (PEP-489): * Single (global) GIL. * PyObject's may be shared across interpreters (zero-copy transfer) Approach 2 (mine) * Per-interpreter GIL. * PyObject's must

Re: [Python-ideas] Exposing CPython's subinterpreter C-API in the stdlib.

2017-05-26 Thread Nick Coghlan
On 26 May 2017 at 22:08, Stephan Houben wrote: > Hi all, > > Personally I feel that the current subinterpreter support falls short > in the sense that it still requires > a single GIL across interpreters. > > If interpreters would have their own individual GIL, > we could have true shared-nothing

Re: [Python-ideas] Exposing CPython's subinterpreter C-API in the stdlib.

2017-05-26 Thread Stephan Houben
Hi all, Personally I feel that the current subinterpreter support falls short in the sense that it still requires a single GIL across interpreters. If interpreters would have their own individual GIL, we could have true shared-nothing multi-threaded support similar to Javascript's "Web Workers".

Re: [Python-ideas] Exposing CPython's subinterpreter C-API in the stdlib.

2017-05-26 Thread Petr Viktorin
On 05/25/2017 09:01 PM, Eric Snow wrote: On Thu, May 25, 2017 at 11:19 AM, Nathaniel Smith wrote: My impression is that the code to support them inside CPython is fine, but they're broken and not very useful in the sense that lots of C extensions don't really support them, so in practice you ca

Re: [Python-ideas] Exposing CPython's subinterpreter C-API in the stdlib.

2017-05-26 Thread Ronald Oussoren
> On 25 May 2017, at 19:03, Eric Snow wrote: > > On Wed, May 24, 2017 at 8:30 PM, Guido van Rossum wrote: >> Hm... Curiously, I've heard a few people at PyCon > > I'd love to get in touch with them and discuss the situation. I've > spoken with Graham Dumpleton on several occasions about > sub

Re: [Python-ideas] Exposing CPython's subinterpreter C-API in the stdlib.

2017-05-26 Thread Nathaniel Smith
On Thu, May 25, 2017 at 12:01 PM, Eric Snow wrote: > More significantly, I genuinely believe that isolated > interpreters in the same process is a tool that many people will find > extremely useful and will help the Python community. Consequently, > exposing subinterpreters in the stdlib would re

Re: [Python-ideas] Exposing CPython's subinterpreter C-API in the stdlib.

2017-05-25 Thread Paul Moore
On 25 May 2017 at 20:01, Eric Snow wrote: > More significantly, I genuinely believe that isolated > interpreters in the same process is a tool that many people will find > extremely useful and will help the Python community. Consequently, > exposing subinterpreters in the stdlib would result in a

Re: [Python-ideas] Exposing CPython's subinterpreter C-API in the stdlib.

2017-05-25 Thread Eric Snow
On Thu, May 25, 2017 at 11:55 AM, Brett Cannon wrote: > I'm +1 on Nick's idea of the low-level, private API existing first to > facilitate testing, but putting off any public API until we're sure we can > make it function in a way we're happy with to more generally expose. Same here. I hadn't ex

Re: [Python-ideas] Exposing CPython's subinterpreter C-API in the stdlib.

2017-05-25 Thread Eric Snow
On Thu, May 25, 2017 at 11:19 AM, Nathaniel Smith wrote: > My impression is that the code to support them inside CPython is fine, but > they're broken and not very useful in the sense that lots of C extensions > don't really support them, so in practice you can't reliably use them to run > arbitra

Re: [Python-ideas] Exposing CPython's subinterpreter C-API in the stdlib.

2017-05-25 Thread Brett Cannon
On Thu, 25 May 2017 at 08:06 Nick Coghlan wrote: > On 25 May 2017 at 13:30, Guido van Rossum wrote: > > Hm... Curiously, I've heard a few people at PyCon mention they thought > > subinterpreters were broken and not useful (and they share the GIL > anyways) > > and should be taken out. > > Taking

Re: [Python-ideas] Exposing CPython's subinterpreter C-API in the stdlib.

2017-05-25 Thread Nathaniel Smith
On May 24, 2017 20:31, "Guido van Rossum" wrote: Hm... Curiously, I've heard a few people at PyCon mention they thought subinterpreters were broken and not useful (and they share the GIL anyways) and should be taken out. So we should at least have clarity on which direction we want to take... M

Re: [Python-ideas] Exposing CPython's subinterpreter C-API in the stdlib.

2017-05-25 Thread Jim Baker
Eric, Something like these subinterpreters in CPython are used from Jython's Java API. Like nearly all of Jython* this can be directly imported into using Python code, as seen in tests using this feature: https://github.com/ jythontools/jython/blob/master/Lib/test/test_pythoninterpreter_jy.py Mo

Re: [Python-ideas] Exposing CPython's subinterpreter C-API in the stdlib.

2017-05-25 Thread Eric Snow
On Wed, May 24, 2017 at 8:30 PM, Guido van Rossum wrote: > Hm... Curiously, I've heard a few people at PyCon I'd love to get in touch with them and discuss the situation. I've spoken with Graham Dumpleton on several occasions about subinterpreters and what needs to be fixed. > mention they thou

Re: [Python-ideas] Exposing CPython's subinterpreter C-API in the stdlib.

2017-05-25 Thread Nick Coghlan
On 25 May 2017 at 13:30, Guido van Rossum wrote: > Hm... Curiously, I've heard a few people at PyCon mention they thought > subinterpreters were broken and not useful (and they share the GIL anyways) > and should be taken out. Taking them out entirely would break mod_wsgi (and hence the use of Ap

Re: [Python-ideas] Exposing CPython's subinterpreter C-API in the stdlib.

2017-05-24 Thread Guido van Rossum
Hm... Curiously, I've heard a few people at PyCon mention they thought subinterpreters were broken and not useful (and they share the GIL anyways) and should be taken out. So we should at least have clarity on which direction we want to take... On Wed, May 24, 2017 at 6:01 PM, Eric Snow wrote: >

Re: [Python-ideas] Exposing CPython's subinterpreter C-API in the stdlib.

2017-05-24 Thread Nathaniel Smith
CC'ing PyPy-dev... On Wed, May 24, 2017 at 6:01 PM, Eric Snow wrote: > Although I haven't been able to achieve the pace that I originally > wanted, I have been able to work on my multi-core Python idea > little-by-little. Most notably, some of the blockers have been > resolved at the recent PyCo

[Python-ideas] Exposing CPython's subinterpreter C-API in the stdlib.

2017-05-24 Thread Eric Snow
Although I haven't been able to achieve the pace that I originally wanted, I have been able to work on my multi-core Python idea little-by-little. Most notably, some of the blockers have been resolved at the recent PyCon sprints and I'm ready to move onto the next step: exposing multiple interpret