On 14 September 2017 at 08:25, Nick Coghlan wrote:
> On 13 September 2017 at 20:45, Koos Zevenhoven wrote:
>> It's still just *an* interpreter that happens to run __main__. And who says
>> it even needs to be the only one?
>
> Koos, I've asked multiple times now for you to describe the practical
On 14 September 2017 at 08:46, Nick Coghlan wrote:
> On 13 September 2017 at 14:10, Nathaniel Smith wrote:
>> Subinterpreters are basically an attempt to reimplement the OS's
>> process isolation in user-space, right?
>
> Not really, they're more an attempt to make something resembling
> Rust's m
On 13 September 2017 at 14:10, Nathaniel Smith wrote:
> Subinterpreters are basically an attempt to reimplement the OS's
> process isolation in user-space, right?
Not really, they're more an attempt to make something resembling
Rust's memory model available to Python programs - having the default
On 13 September 2017 at 20:45, Koos Zevenhoven wrote:
> On Wed, Sep 13, 2017 at 6:14 AM, Nick Coghlan wrote:
>>
>> On 13 September 2017 at 00:35, Koos Zevenhoven wrote:>
>>
>> > I don't see how the situation benefits from calling something the "main
>> > interpreter". Subinterpreters can be a wa
On Wed, Sep 13, 2017 at 6:14 AM, Nick Coghlan wrote:
> On 13 September 2017 at 00:35, Koos Zevenhoven wrote:>
>
> I don't see how the situation benefits from calling something the "main
> > interpreter". Subinterpreters can be a way to take something
> non-thread-safe
> > and make it thread-safe
On Tue, Sep 12, 2017 at 1:46 PM, Eric Snow wrote:
> On Thu, Sep 7, 2017 at 11:19 PM, Nathaniel Smith wrote:
>> On Thu, Sep 7, 2017 at 8:11 PM, Eric Snow
>> wrote:
>>> My concern is that this is a chicken-and-egg problem. The situation
>>> won't improve until subinterpreters are more readily av
On 13 September 2017 at 00:35, Koos Zevenhoven wrote:
> On Tue, Sep 12, 2017 at 1:40 PM, Nick Coghlan wrote:
>>
>> On 11 September 2017 at 18:02, Koos Zevenhoven wrote:
>> > On Mon, Sep 11, 2017 at 8:32 AM, Nick Coghlan
>> > wrote:
>> >> The line between it and the "CPython Runtime" is fuzzy fo
On Sun, Sep 10, 2017 at 12:14 PM, Antoine Pitrou wrote:
> What could improve performance significantly would be to share objects
> without any form of marshalling; but it's not obvious it's possible in
> the subinterpreters model *if* it also tries to remove the GIL.
Yep. This is one of the main
On Sun, Sep 10, 2017 at 7:52 AM, Koos Zevenhoven wrote:
> I assume the concept of a main interpreter is inherited from the previous
> levels of support in the C API, but what exactly is the significance of
> being "the main interpreter"? Instead, could they just all be
> subinterpreters of the sam
Yep. See http://bugs.python.org/issue10915 and
http://bugs.python.org/issue15751. The issue of C-extension support
for subinterpreters is, of course, a critical one here. At the very
least, incompatible modules should be able to opt out of
subinterpreter support. I've updated the PEP to discuss
On Thu, Sep 7, 2017 at 11:19 PM, Nathaniel Smith wrote:
> On Thu, Sep 7, 2017 at 8:11 PM, Eric Snow wrote:
>> My concern is that this is a chicken-and-egg problem. The situation
>> won't improve until subinterpreters are more readily available.
>
> Okay, but you're assuming that "more libraries
On Tue, Sep 12, 2017 at 6:30 PM, Koos Zevenhoven wrote:
> On Tue, Sep 12, 2017 at 5:53 PM, Stefan Krah wrote:
>
>> On Tue, Sep 12, 2017 at 05:35:34PM +0300, Koos Zevenhoven wrote:
>> > I don't see how the situation benefits from calling something the "main
>> > interpreter". Subinterpreters can
On Tue, Sep 12, 2017 at 5:53 PM, Stefan Krah wrote:
> On Tue, Sep 12, 2017 at 05:35:34PM +0300, Koos Zevenhoven wrote:
> > I don't see how the situation benefits from calling something the "main
> > interpreter". Subinterpreters can be a way to take something
> > non-thread-safe and make it threa
On Tue, Sep 12, 2017 at 05:35:34PM +0300, Koos Zevenhoven wrote:
> I don't see how the situation benefits from calling something the "main
> interpreter". Subinterpreters can be a way to take something
> non-thread-safe and make it thread-safe, because in an
> interpreter-per-thread scheme, most of
On Tue, Sep 12, 2017 at 1:40 PM, Nick Coghlan wrote:
> On 11 September 2017 at 18:02, Koos Zevenhoven wrote:
> > On Mon, Sep 11, 2017 at 8:32 AM, Nick Coghlan
> wrote:
> >> The line between it and the "CPython Runtime" is fuzzy for both
> >> practical and historical reasons, but the regular Pyt
On 11 September 2017 at 18:02, Koos Zevenhoven wrote:
> On Mon, Sep 11, 2017 at 8:32 AM, Nick Coghlan wrote:
>> The line between it and the "CPython Runtime" is fuzzy for both
>> practical and historical reasons, but the regular Python CLI will
>> always have a "first created, last destroyed" mai
On Mon, Sep 11, 2017 at 8:32 AM, Nick Coghlan wrote:
> On 11 September 2017 at 00:52, Koos Zevenhoven wrote:
> > On Thu, Sep 7, 2017 at 9:26 PM, Eric Snow
> > wrote:
> > [...]
> >
> >>
> >> get_main():
> >>
> >>Return the main interpreter.
> >>
> >
> > I assume the concept of a main interpr
On 11 September 2017 at 00:52, Koos Zevenhoven wrote:
> On Thu, Sep 7, 2017 at 9:26 PM, Eric Snow
> wrote:
> [...]
>
>>
>> get_main():
>>
>>Return the main interpreter.
>>
>
> I assume the concept of a main interpreter is inherited from the previous
> levels of support in the C API, but what
On Thu, 7 Sep 2017 21:08:48 -0700
Nathaniel Smith wrote:
>
> Awesome, thanks for bringing numbers into my wooly-headed theorizing :-).
>
> On my laptop I actually get a worse result from your benchmark: 531 ms
> for 100 MB == ~200 MB/s round-trip, or 400 MB/s one-way. So yeah,
> transferring dat
On Thu, Sep 7, 2017 at 9:26 PM, Eric Snow
wrote:
[...]
> get_main():
>
>Return the main interpreter.
>
>
I assume the concept of a main interpreter is inherited from the previous
levels of support in the C API, but what exactly is the significance of
being "the main interpreter"? Instead,
> On 8 Sep 2017, at 05:11, Eric Snow wrote:
> On Thu, Sep 7, 2017 at 3:48 PM, Nathaniel Smith wrote:
>
>> Numpy is the one I'm
>> most familiar with: when we get subinterpreter bugs we close them
>> wontfix, because supporting subinterpreters properly would require
>> non-trivial auditing, add
On Thu, Sep 7, 2017 at 8:11 PM, Eric Snow wrote:
> First of all, thanks for the feedback and encouragement! Responses
> in-line below.
I hope it's helpful! More responses in-line as well.
> On Thu, Sep 7, 2017 at 3:48 PM, Nathaniel Smith wrote:
>> My concern about this is the same as it was la
On Thu, Sep 7, 2017 at 6:14 PM, Matthew Rocklin wrote:
> Those numbers were for common use in Python tools and reflected my anecdotal
> experience at the time with normal Python tools. I'm sure that there are
> mechanisms to achieve faster speeds than what I experienced. That being
> said, here
On Thu, Sep 7, 2017 at 12:44 PM, Paul Moore wrote:
> On 7 September 2017 at 20:14, Eric Snow wrote:
>> I didn't include such a queue in this proposal because I wanted to
>> keep it as focused as possible. I'll add a note to the PEP about
>> this.
>
> This all sounds very reasonable. Thanks for t
First of all, thanks for the feedback and encouragement! Responses
in-line below.
-eric
On Thu, Sep 7, 2017 at 3:48 PM, Nathaniel Smith wrote:
> My concern about this is the same as it was last time -- the work
> looks neat, but right now, almost no-one uses subinterpreters
> (basically it's J
Those numbers were for common use in Python tools and reflected my
anecdotal experience at the time with normal Python tools. I'm sure that
there are mechanisms to achieve faster speeds than what I experienced.
That being said, here is a small example.
In [1]: import multiprocessing
In [2]: data
On Thu, Sep 7, 2017 at 5:15 PM Nathaniel Smith wrote:
> On Thu, Sep 7, 2017 at 4:23 PM, Nick Coghlan wrote:
> > The gist of the idea is that with subinterpreters, your starting point
> > is multiprocessing-style isolation (i.e. you have to use pickle to
> > transfer data between subinterpreters)
On Thu, Sep 7, 2017 at 4:23 PM, Nick Coghlan wrote:
> On 7 September 2017 at 15:48, Nathaniel Smith wrote:
>> I've actually argued with the PyPy devs to try to convince them to add
>> subinterpreter support as part of their experiments with GIL-removal,
>> because I think the semantics would genu
On 7 September 2017 at 15:48, Nathaniel Smith wrote:
> I've actually argued with the PyPy devs to try to convince them to add
> subinterpreter support as part of their experiments with GIL-removal,
> because I think the semantics would genuinely be nicer to work with
> than raw threads, but they'r
On Thu, Sep 7, 2017 at 11:26 AM, Eric Snow wrote:
> Hi all,
>
> As part of the multi-core work I'm proposing the addition of the
> "interpreters" module to the stdlib. This will expose the existing
> subinterpreters C-API to Python code. I've purposefully kept the API
> simple. Please let me kn
On Thu, Sep 7, 2017 at 1:14 PM, Sebastian Krause wrote:
> How is the GIL situation with subinterpreters these days, is the
> long-term goal still "solving multi-core Python", i.e. using
> multiple CPU cores from within the same process? Or is it mainly
> used for isolation?
The GIL is still proce
On Thu, Sep 7, 2017 at 12:44 PM, Paul Moore wrote:
> Ah, OK. so if I create a new interpreter, none of the classes,
> functions, or objects defined in my calling code will exist within the
> target interpreter? That makes sense, but I'd missed that nuance from
> the description. Again, this is pro
Eric Snow wrote:
> 1. add a basic queue class for passing objects between interpreters
> * only support strings at first (though Nick pointed out we could
> fall back to pickle or marshal for unsupported objects)
> 2. implement CSP on top of subinterpreters
> 3. expand the queue's supported ty
On 7 September 2017 at 20:14, Eric Snow wrote:
> On Thu, Sep 7, 2017 at 11:52 AM, Paul Moore wrote:
>> Is there any reason why passing a callable and args is unsafe, and/or
>> difficult? Naively, I'd assume that
>>
>> interp.call('f(a)')
>>
>> would be precisely as safe as
>>
>> interp.ca
On Thu, Sep 7, 2017 at 11:52 AM, Paul Moore wrote:
> The only quibble I have is that I'd prefer it if we had a
> run(callable, *args, **kwargs) method. Either instead of, or as well
> as, the run(string) one here.
>
> Is there any reason why passing a callable and args is unsafe, and/or
> difficul
On 7 September 2017 at 19:26, Eric Snow wrote:
> As part of the multi-core work I'm proposing the addition of the
> "interpreters" module to the stdlib. This will expose the existing
> subinterpreters C-API to Python code. I've purposefully kept the API
> simple. Please let me know what you thi
Hi all,
As part of the multi-core work I'm proposing the addition of the
"interpreters" module to the stdlib. This will expose the existing
subinterpreters C-API to Python code. I've purposefully kept the API
simple. Please let me know what you think.
-eric
https://www.python.org/dev/peps/pep
37 matches
Mail list logo
