Re: [Python-ideas] PEP 554: Stdlib Module to Support Multiple Interpreters in Python Code

On 14 September 2017 at 08:25, Nick Coghlan wrote: > On 13 September 2017 at 20:45, Koos Zevenhoven wrote: >> It's still just *an* interpreter that happens to run __main__. And who says >> it even needs to be the only one? > > Koos, I've asked multiple

Re: [Python-ideas] PEP 554: Stdlib Module to Support Multiple Interpreters in Python Code

On 13 September 2017 at 14:10, Nathaniel Smith wrote: > Subinterpreters are basically an attempt to reimplement the OS's > process isolation in user-space, right? Not really, they're more an attempt to make something resembling Rust's memory model available to Python programs -

Re: [Python-ideas] PEP 554: Stdlib Module to Support Multiple Interpreters in Python Code

On 13 September 2017 at 20:45, Koos Zevenhoven wrote: > On Wed, Sep 13, 2017 at 6:14 AM, Nick Coghlan wrote: >> >> On 13 September 2017 at 00:35, Koos Zevenhoven wrote:> >> >> > I don't see how the situation benefits from calling

Re: [Python-ideas] PEP 554: Stdlib Module to Support Multiple Interpreters in Python Code

On Wed, Sep 13, 2017 at 6:14 AM, Nick Coghlan wrote: > On 13 September 2017 at 00:35, Koos Zevenhoven wrote:> > > I don't see how the situation benefits from calling something the "main > > interpreter". Subinterpreters can be a way to take something >

Re: [Python-ideas] PEP 554: Stdlib Module to Support Multiple Interpreters in Python Code

On Tue, Sep 12, 2017 at 1:46 PM, Eric Snow wrote: > On Thu, Sep 7, 2017 at 11:19 PM, Nathaniel Smith wrote: >> On Thu, Sep 7, 2017 at 8:11 PM, Eric Snow >> wrote: >>> My concern is that this is a chicken-and-egg problem.

Re: [Python-ideas] PEP 554: Stdlib Module to Support Multiple Interpreters in Python Code

On 13 September 2017 at 00:35, Koos Zevenhoven wrote: > On Tue, Sep 12, 2017 at 1:40 PM, Nick Coghlan wrote: >> >> On 11 September 2017 at 18:02, Koos Zevenhoven wrote: >> > On Mon, Sep 11, 2017 at 8:32 AM, Nick Coghlan

Re: [Python-ideas] PEP 554: Stdlib Module to Support Multiple Interpreters in Python Code

On Sun, Sep 10, 2017 at 12:14 PM, Antoine Pitrou wrote: > What could improve performance significantly would be to share objects > without any form of marshalling; but it's not obvious it's possible in > the subinterpreters model *if* it also tries to remove the GIL. Yep.

Re: [Python-ideas] PEP 554: Stdlib Module to Support Multiple Interpreters in Python Code

On Sun, Sep 10, 2017 at 7:52 AM, Koos Zevenhoven wrote: > I assume the concept of a main interpreter is inherited from the previous > levels of support in the C API, but what exactly is the significance of > being "the main interpreter"? Instead, could they just all be >

Re: [Python-ideas] PEP 554: Stdlib Module to Support Multiple Interpreters in Python Code

Yep. See http://bugs.python.org/issue10915 and http://bugs.python.org/issue15751. The issue of C-extension support for subinterpreters is, of course, a critical one here. At the very least, incompatible modules should be able to opt out of subinterpreter support. I've updated the PEP to

Re: [Python-ideas] PEP 554: Stdlib Module to Support Multiple Interpreters in Python Code

On Thu, Sep 7, 2017 at 11:19 PM, Nathaniel Smith wrote: > On Thu, Sep 7, 2017 at 8:11 PM, Eric Snow wrote: >> My concern is that this is a chicken-and-egg problem. The situation >> won't improve until subinterpreters are more readily available. > >

Re: [Python-ideas] PEP 554: Stdlib Module to Support Multiple Interpreters in Python Code

On Tue, Sep 12, 2017 at 6:30 PM, Koos Zevenhoven wrote: > On Tue, Sep 12, 2017 at 5:53 PM, Stefan Krah wrote: > >> On Tue, Sep 12, 2017 at 05:35:34PM +0300, Koos Zevenhoven wrote: >> > I don't see how the situation benefits from calling something the

Re: [Python-ideas] PEP 554: Stdlib Module to Support Multiple Interpreters in Python Code

On 11 September 2017 at 18:02, Koos Zevenhoven wrote: > On Mon, Sep 11, 2017 at 8:32 AM, Nick Coghlan wrote: >> The line between it and the "CPython Runtime" is fuzzy for both >> practical and historical reasons, but the regular Python CLI will >> always

Re: [Python-ideas] PEP 554: Stdlib Module to Support Multiple Interpreters in Python Code

On Mon, Sep 11, 2017 at 8:32 AM, Nick Coghlan wrote: > On 11 September 2017 at 00:52, Koos Zevenhoven wrote: > > On Thu, Sep 7, 2017 at 9:26 PM, Eric Snow > > wrote: > > [...] > > > >> > >> get_main(): > >> > >>Return the

Re: [Python-ideas] PEP 554: Stdlib Module to Support Multiple Interpreters in Python Code

On 11 September 2017 at 00:52, Koos Zevenhoven wrote: > On Thu, Sep 7, 2017 at 9:26 PM, Eric Snow > wrote: > [...] > >> >> get_main(): >> >>Return the main interpreter. >> > > I assume the concept of a main interpreter is inherited from the

Re: [Python-ideas] PEP 554: Stdlib Module to Support Multiple Interpreters in Python Code

On Thu, 7 Sep 2017 21:08:48 -0700 Nathaniel Smith wrote: > > Awesome, thanks for bringing numbers into my wooly-headed theorizing :-). > > On my laptop I actually get a worse result from your benchmark: 531 ms > for 100 MB == ~200 MB/s round-trip, or 400 MB/s one-way. So yeah, >

Re: [Python-ideas] PEP 554: Stdlib Module to Support Multiple Interpreters in Python Code

On Thu, Sep 7, 2017 at 9:26 PM, Eric Snow wrote: ​[...]​ > get_main(): > >Return the main interpreter. > > ​I assume the concept of a main interpreter is inherited from the previous levels of support in the C API, but what exactly is the significance of being

Re: [Python-ideas] PEP 554: Stdlib Module to Support Multiple Interpreters in Python Code

> On 8 Sep 2017, at 05:11, Eric Snow wrote: > On Thu, Sep 7, 2017 at 3:48 PM, Nathaniel Smith wrote: > >> Numpy is the one I'm >> most familiar with: when we get subinterpreter bugs we close them >> wontfix, because supporting subinterpreters

Re: [Python-ideas] PEP 554: Stdlib Module to Support Multiple Interpreters in Python Code

On Thu, Sep 7, 2017 at 8:11 PM, Eric Snow wrote: > First of all, thanks for the feedback and encouragement! Responses > in-line below. I hope it's helpful! More responses in-line as well. > On Thu, Sep 7, 2017 at 3:48 PM, Nathaniel Smith wrote: >>

Re: [Python-ideas] PEP 554: Stdlib Module to Support Multiple Interpreters in Python Code

On Thu, Sep 7, 2017 at 6:14 PM, Matthew Rocklin wrote: > Those numbers were for common use in Python tools and reflected my anecdotal > experience at the time with normal Python tools. I'm sure that there are > mechanisms to achieve faster speeds than what I experienced.

Re: [Python-ideas] PEP 554: Stdlib Module to Support Multiple Interpreters in Python Code

On Thu, Sep 7, 2017 at 12:44 PM, Paul Moore wrote: > On 7 September 2017 at 20:14, Eric Snow wrote: >> I didn't include such a queue in this proposal because I wanted to >> keep it as focused as possible. I'll add a note to the PEP about >>

Re: [Python-ideas] PEP 554: Stdlib Module to Support Multiple Interpreters in Python Code

First of all, thanks for the feedback and encouragement! Responses in-line below. -eric On Thu, Sep 7, 2017 at 3:48 PM, Nathaniel Smith wrote: > My concern about this is the same as it was last time -- the work > looks neat, but right now, almost no-one uses subinterpreters >

Re: [Python-ideas] PEP 554: Stdlib Module to Support Multiple Interpreters in Python Code

Those numbers were for common use in Python tools and reflected my anecdotal experience at the time with normal Python tools. I'm sure that there are mechanisms to achieve faster speeds than what I experienced. That being said, here is a small example. In [1]: import multiprocessing In [2]:

Re: [Python-ideas] PEP 554: Stdlib Module to Support Multiple Interpreters in Python Code

On Thu, Sep 7, 2017 at 5:15 PM Nathaniel Smith wrote: > On Thu, Sep 7, 2017 at 4:23 PM, Nick Coghlan wrote: > > The gist of the idea is that with subinterpreters, your starting point > > is multiprocessing-style isolation (i.e. you have to use pickle to > >

Re: [Python-ideas] PEP 554: Stdlib Module to Support Multiple Interpreters in Python Code

On Thu, Sep 7, 2017 at 4:23 PM, Nick Coghlan wrote: > On 7 September 2017 at 15:48, Nathaniel Smith wrote: >> I've actually argued with the PyPy devs to try to convince them to add >> subinterpreter support as part of their experiments with GIL-removal, >>

Re: [Python-ideas] PEP 554: Stdlib Module to Support Multiple Interpreters in Python Code

On 7 September 2017 at 15:48, Nathaniel Smith wrote: > I've actually argued with the PyPy devs to try to convince them to add > subinterpreter support as part of their experiments with GIL-removal, > because I think the semantics would genuinely be nicer to work with > than raw

Re: [Python-ideas] PEP 554: Stdlib Module to Support Multiple Interpreters in Python Code

On Thu, Sep 7, 2017 at 11:26 AM, Eric Snow wrote: > Hi all, > > As part of the multi-core work I'm proposing the addition of the > "interpreters" module to the stdlib. This will expose the existing > subinterpreters C-API to Python code. I've purposefully kept the

Re: [Python-ideas] PEP 554: Stdlib Module to Support Multiple Interpreters in Python Code

On Thu, Sep 7, 2017 at 1:14 PM, Sebastian Krause wrote: > How is the GIL situation with subinterpreters these days, is the > long-term goal still "solving multi-core Python", i.e. using > multiple CPU cores from within the same process? Or is it mainly > used for

Re: [Python-ideas] PEP 554: Stdlib Module to Support Multiple Interpreters in Python Code

On Thu, Sep 7, 2017 at 12:44 PM, Paul Moore wrote: > Ah, OK. so if I create a new interpreter, none of the classes, > functions, or objects defined in my calling code will exist within the > target interpreter? That makes sense, but I'd missed that nuance from > the

Re: [Python-ideas] PEP 554: Stdlib Module to Support Multiple Interpreters in Python Code

Eric Snow wrote: > 1. add a basic queue class for passing objects between interpreters > * only support strings at first (though Nick pointed out we could > fall back to pickle or marshal for unsupported objects) > 2. implement CSP on top of subinterpreters > 3.

Re: [Python-ideas] PEP 554: Stdlib Module to Support Multiple Interpreters in Python Code

On 7 September 2017 at 20:14, Eric Snow wrote: > On Thu, Sep 7, 2017 at 11:52 AM, Paul Moore wrote: >> Is there any reason why passing a callable and args is unsafe, and/or >> difficult? Naively, I'd assume that >> >> interp.call('f(a)') >>

Re: [Python-ideas] PEP 554: Stdlib Module to Support Multiple Interpreters in Python Code

On Thu, Sep 7, 2017 at 11:52 AM, Paul Moore wrote: > The only quibble I have is that I'd prefer it if we had a > run(callable, *args, **kwargs) method. Either instead of, or as well > as, the run(string) one here. > > Is there any reason why passing a callable and args is

Re: [Python-ideas] PEP 554: Stdlib Module to Support Multiple Interpreters in Python Code

On 7 September 2017 at 19:26, Eric Snow wrote: > As part of the multi-core work I'm proposing the addition of the > "interpreters" module to the stdlib. This will expose the existing > subinterpreters C-API to Python code. I've purposefully kept the API > simple.

[Python-ideas] PEP 554: Stdlib Module to Support Multiple Interpreters in Python Code

Hi all, As part of the multi-core work I'm proposing the addition of the "interpreters" module to the stdlib. This will expose the existing subinterpreters C-API to Python code. I've purposefully kept the API simple. Please let me know what you think. -eric