Re: [Python-ideas] Exposing CPython's subinterpreter C-API in the stdlib.

2017-05-27 Thread Stephan Houben
Hi Nick, > I guess I'll have to scale back my hopes on that front to be closer to > what Stephan described - even a deep copy equivalent is often going to > be cheaper than a full serialise/transmit/deserialise cycle or some > other form of inter-process communication. I would like to add that

Re: [Python-ideas] Exposing CPython's subinterpreter C-API in the stdlib.

2017-05-27 Thread Nick Coghlan
On 27 May 2017 at 03:30, Guido van Rossum wrote: > On Fri, May 26, 2017 at 8:28 AM, Nick Coghlan wrote: >> >> [...] assuming the rest of idea works out >> well, we'd eventually like to move to a tiered model where the GIL >> becomes a read/write lock. Most

Re: [Python-ideas] Exposing CPython's subinterpreter C-API in the stdlib.

2017-05-26 Thread Nick Coghlan
On 26 May 2017 at 22:08, Stephan Houben wrote: > Hi all, > > Personally I feel that the current subinterpreter support falls short > in the sense that it still requires > a single GIL across interpreters. > > If interpreters would have their own individual GIL, > we could

Re: [Python-ideas] Exposing CPython's subinterpreter C-API in the stdlib.

2017-05-26 Thread Stephan Houben
Hi all, Personally I feel that the current subinterpreter support falls short in the sense that it still requires a single GIL across interpreters. If interpreters would have their own individual GIL, we could have true shared-nothing multi-threaded support similar to Javascript's "Web Workers".

Re: [Python-ideas] Exposing CPython's subinterpreter C-API in the stdlib.

2017-05-26 Thread Petr Viktorin
On 05/25/2017 09:01 PM, Eric Snow wrote: On Thu, May 25, 2017 at 11:19 AM, Nathaniel Smith wrote: My impression is that the code to support them inside CPython is fine, but they're broken and not very useful in the sense that lots of C extensions don't really support them, so

Re: [Python-ideas] Exposing CPython's subinterpreter C-API in the stdlib.

2017-05-26 Thread Ronald Oussoren
> On 25 May 2017, at 19:03, Eric Snow wrote: > > On Wed, May 24, 2017 at 8:30 PM, Guido van Rossum wrote: >> Hm... Curiously, I've heard a few people at PyCon > > I'd love to get in touch with them and discuss the situation. I've > spoken with

Re: [Python-ideas] Exposing CPython's subinterpreter C-API in the stdlib.

2017-05-26 Thread Nathaniel Smith
On Thu, May 25, 2017 at 12:01 PM, Eric Snow wrote: > More significantly, I genuinely believe that isolated > interpreters in the same process is a tool that many people will find > extremely useful and will help the Python community. Consequently, > exposing

Re: [Python-ideas] Exposing CPython's subinterpreter C-API in the stdlib.

2017-05-25 Thread Eric Snow
On Thu, May 25, 2017 at 11:55 AM, Brett Cannon wrote: > I'm +1 on Nick's idea of the low-level, private API existing first to > facilitate testing, but putting off any public API until we're sure we can > make it function in a way we're happy with to more generally expose. Same

Re: [Python-ideas] Exposing CPython's subinterpreter C-API in the stdlib.

2017-05-25 Thread Eric Snow
On Thu, May 25, 2017 at 11:19 AM, Nathaniel Smith wrote: > My impression is that the code to support them inside CPython is fine, but > they're broken and not very useful in the sense that lots of C extensions > don't really support them, so in practice you can't reliably use them

Re: [Python-ideas] Exposing CPython's subinterpreter C-API in the stdlib.

2017-05-25 Thread Brett Cannon
On Thu, 25 May 2017 at 08:06 Nick Coghlan wrote: > On 25 May 2017 at 13:30, Guido van Rossum wrote: > > Hm... Curiously, I've heard a few people at PyCon mention they thought > > subinterpreters were broken and not useful (and they share the GIL > anyways)

Re: [Python-ideas] Exposing CPython's subinterpreter C-API in the stdlib.

2017-05-25 Thread Nathaniel Smith
On May 24, 2017 20:31, "Guido van Rossum" wrote: Hm... Curiously, I've heard a few people at PyCon mention they thought subinterpreters were broken and not useful (and they share the GIL anyways) and should be taken out. So we should at least have clarity on which direction we

Re: [Python-ideas] Exposing CPython's subinterpreter C-API in the stdlib.

2017-05-25 Thread Nick Coghlan
On 25 May 2017 at 13:30, Guido van Rossum wrote: > Hm... Curiously, I've heard a few people at PyCon mention they thought > subinterpreters were broken and not useful (and they share the GIL anyways) > and should be taken out. Taking them out entirely would break mod_wsgi (and

Re: [Python-ideas] Exposing CPython's subinterpreter C-API in the stdlib.

2017-05-24 Thread Guido van Rossum
Hm... Curiously, I've heard a few people at PyCon mention they thought subinterpreters were broken and not useful (and they share the GIL anyways) and should be taken out. So we should at least have clarity on which direction we want to take... On Wed, May 24, 2017 at 6:01 PM, Eric Snow

Re: [Python-ideas] Exposing CPython's subinterpreter C-API in the stdlib.

2017-05-24 Thread Nathaniel Smith
CC'ing PyPy-dev... On Wed, May 24, 2017 at 6:01 PM, Eric Snow wrote: > Although I haven't been able to achieve the pace that I originally > wanted, I have been able to work on my multi-core Python idea > little-by-little. Most notably, some of the blockers have been

[Python-ideas] Exposing CPython's subinterpreter C-API in the stdlib.

2017-05-24 Thread Eric Snow
Although I haven't been able to achieve the pace that I originally wanted, I have been able to work on my multi-core Python idea little-by-little. Most notably, some of the blockers have been resolved at the recent PyCon sprints and I'm ready to move onto the next step: exposing multiple