Re: [Python-ideas] The future of Python parallelism. The GIL. Subinterpreters. Actors.

Eric, > On 17 Jul 2018, at 20:35, Eric Snow wrote: > > With this in mind, here's how I'm approaching the problem: > > 1. interp A "shares" an object with interp B (e.g. through a channel) >* the object is incref'ed under A before it is sent to B > 2. the object is wrapped in a proxy owned

Re: [Python-ideas] The future of Python parallelism. The GIL. Subinterpreters. Actors.

On 18 July 2018 at 05:35, Eric Snow wrote: > In order to make all this work the missing piece is a mechanism by > which the decref (#3) happens under the original interpreter. At the > moment Emily Morehouse and I are pursuing an approach that extends the > existing ceval "pending call"

Re: [Python-ideas] The future of Python parallelism. The GIL. Subinterpreters. Actors.

Hi Nathaniel, 2018-07-19 1:33 GMT+02:00 Nathaniel Smith : > Note that this everything you said here also exactly describes the > programming model for the existing 'multiprocessing' module: > "structured clone" is equivalent to how multiprocessing uses pickle to > transfer arbitrary objects, or

Re: [Python-ideas] The future of Python parallelism. The GIL. Subinterpreters. Actors.

Hi, > > Note that this everything you said here also exactly describes the > programming model for the existing 'multiprocessing' module: > "structured clone" is equivalent to how multiprocessing uses pickle to > transfer arbitrary objects, or you can use multiprocessing.Array to > get a shared

Re: [Python-ideas] The future of Python parallelism. The GIL. Subinterpreters. Actors.

On Wed, Jul 18, 2018 at 11:49 AM, Stephan Houben wrote: > Basically, what I am suggesting is a direct translation of Javascript's > Web Worker API > (https://developer.mozilla.org/en-US/docs/Web/API/Web_Workers_API) > to Python. > > The Web Worker API is generally considered a "share-nothing"

Re: [Python-ideas] The future of Python parallelism. The GIL. Subinterpreters. Actors.

On Wed, Jul 18, 2018 at 2:38 PM MRAB wrote: > What if an object is not going to be shared, but instead "moved" from > one subinterpreter to another? The first subinterpreter would no longer > have a reference to the object. > > If the object's refcount is 1 and the object doesn't refer to any

Re: [Python-ideas] The future of Python parallelism. The GIL. Subinterpreters. Actors.

On 2018-07-18 20:35, Eric Snow wrote: On Wed, Jul 18, 2018 at 12:49 PM Stephan Houben wrote: Antoine said that what I proposed earlier was very similar to what Eric is trying to do, but from the direction the discussion has taken so far that appears not to be the case. It looks like we are

Re: [Python-ideas] The future of Python parallelism. The GIL. Subinterpreters. Actors.

On Wed, Jul 18, 2018 at 12:49 PM Stephan Houben wrote: > Antoine said that what I proposed earlier was very similar to what Eric > is trying to do, but from the direction the discussion has taken so far > that appears not to be the case. It looks like we are after the same thing actually. :)

Re: [Python-ideas] The future of Python parallelism. The GIL. Subinterpreters. Actors.

Hi Python in the age of the multi-core processor is an important question. And garbage collection is one of the many issues involved. I've been thinking about the garbage collection problem, and lurking on this list, for a while. I think it's now about time I showed myself, and shared my

Re: [Python-ideas] The future of Python parallelism. The GIL. Subinterpreters. Actors.

On Wed, Jul 18, 2018 at 1:37 AM Barry Scott wrote: > Let me try a longer answer. The inc+test and dec+test do not require a > lock if coded correctly. All OS and run times have solved this to provide > locks. All processors provide the instructions that are the building blocks > for lock

Re: [Python-ideas] The future of Python parallelism. The GIL. Subinterpreters. Actors.

On Wed, Jul 18, 2018 at 3:31 AM Antoine Pitrou wrote: > Please read in context: we are not talking about making all refcounts > atomic, only a couple refcounts on shared objects (which probably > won't be Python objects, actually). I have no plans to use refcounts for shared data (outside of

Re: [Python-ideas] The future of Python parallelism. The GIL. Subinterpreters. Actors.

On Wed, 18 Jul 2018 08:21:31 +0100 Ronald Oussoren via Python-ideas wrote: > Some past attempts at getting rid of the GIL used atomic inc/dec, and that > resulted in bad performance because these instructions aren’t cheap. Please read in context: we are not talking about making all refcounts

Re: [Python-ideas] The future of Python parallelism. The GIL. Subinterpreters. Actors.

> On 18 Jul 2018, at 08:21, Ronald Oussoren wrote: > > Op 18 jul. 2018 om 08:02 heeft Barry > het volgende geschreven: > >> >> On 17 Jul 2018, at 21:00, Eric Snow wrote: On Tue, Jul 17, 2018 at 1:44 PM Barry wrote: The decrement itself

Re: [Python-ideas] The future of Python parallelism. The GIL. Subinterpreters. Actors.

> On 17 Jul 2018, at 21:00, Eric Snow wrote: > > On Tue, Jul 17, 2018 at 1:44 PM Barry wrote: >> The decrement itself is not the problem, that can be made thread safe. > > Yeah, by using the GIL. Otherwise, please elaborate. My > understanding is that if the decrement itself were not the

Re: [Python-ideas] The future of Python parallelism. The GIL. Subinterpreters. Actors.

Op 18 jul. 2018 om 08:02 heeft Barry het volgende geschreven: > > >>> On 17 Jul 2018, at 21:00, Eric Snow wrote: >>> >>> On Tue, Jul 17, 2018 at 1:44 PM Barry wrote: >>> The decrement itself is not the problem, that can be made thread safe. >> >> Yeah, by using the GIL. Otherwise,

Re: [Python-ideas] The future of Python parallelism. The GIL. Subinterpreters. Actors.

>> On 17 Jul 2018, at 21:00, Eric Snow wrote: >> >> On Tue, Jul 17, 2018 at 1:44 PM Barry wrote: >> The decrement itself is not the problem, that can be made thread safe. > > Yeah, by using the GIL. Otherwise, please elaborate. My > understanding is that if the decrement itself were not

Re: [Python-ideas] The future of Python parallelism. The GIL. Subinterpreters. Actors.

(Apologies for the slow reply, I'm in the middle of a relocation at the moment so e-mail access isn't consistent, and will be a lot worse over the next few weeks.) On Tue, Jul 10, 2018 at 07:31:49AM -0700, David Foster wrote: > I was not aware of PyParallel. The PyParellel

Re: [Python-ideas] The future of Python parallelism. The GIL. Subinterpreters. Actors.

MRAB wrote: The shared object's refcount would be incremented and the sharing function would return a proxy to the shared object. Refcounting in the thread/process would be done on the proxy. When the proxy is closed or garbage-collected, the shared object's refcount would be decremented.

Re: [Python-ideas] The future of Python parallelism. The GIL. Subinterpreters. Actors.

> On 17 Jul 2018, at 20:35, Eric Snow wrote: > >> On Mon, Jul 16, 2018 at 11:08 AM Antoine Pitrou wrote: >> On Mon, 16 Jul 2018 18:00:37 +0100 >> MRAB wrote: >>> Could you explicitly share an object in a similar way to how you >>> explicitly open a file? >>> >>> The shared object's

Re: [Python-ideas] The future of Python parallelism. The GIL. Subinterpreters. Actors.

On Tue, Jul 17, 2018 at 1:44 PM Barry wrote: > The decrement itself is not the problem, that can be made thread safe. Yeah, by using the GIL. Otherwise, please elaborate. My understanding is that if the decrement itself were not the problem then we'd have gotten rid of the GIL already. > Do

Re: [Python-ideas] The future of Python parallelism. The GIL. Subinterpreters. Actors.

On Mon, Jul 16, 2018 at 11:08 AM Antoine Pitrou wrote: > On Mon, 16 Jul 2018 18:00:37 +0100 > MRAB wrote: > > Could you explicitly share an object in a similar way to how you > > explicitly open a file? > > > > The shared object's refcount would be incremented and the sharing > > function would

Re: [Python-ideas] The future of Python parallelism. The GIL. Subinterpreters. Actors.

On Mon, 16 Jul 2018 18:00:37 +0100 MRAB wrote: > Could you explicitly share an object in a similar way to how you > explicitly open a file? > > The shared object's refcount would be incremented and the sharing > function would return a proxy to the shared object. > > Refcounting in the

Re: [Python-ideas] The future of Python parallelism. The GIL. Subinterpreters. Actors.

On 2018-07-16 05:24, Chris Angelico wrote: On Mon, Jul 16, 2018 at 1:21 PM, Nathaniel Smith wrote: On Sun, Jul 15, 2018 at 6:00 PM, Chris Angelico wrote: On Mon, Jul 16, 2018 at 10:31 AM, Nathaniel Smith wrote: On Sun, Jul 8, 2018 at 11:27 AM, David Foster wrote: * The Actor model can be

Re: [Python-ideas] The future of Python parallelism. The GIL. Subinterpreters. Actors.

On Mon, 16 Jul 2018 07:00:34 +0200 Stephan Houben wrote: > What about the following model: you have N Python interpreters, each with > their own GIL. Each *Python* object belongs to precisely one interpreter. This is roughly what Eric's subinterpreters approach tries to do. Regards Antoine.

Re: [Python-ideas] The future of Python parallelism. The GIL. Subinterpreters. Actors.

On Sun, 15 Jul 2018 20:21:56 -0700 Nathaniel Smith wrote: > > If you need shared-memory threads, on multiple cores, for CPU-bound > logic, where the logic is implemented in Python, then yeah, you > basically need a free-threaded implementation of Python. Jython is > such an implementation. PyPy

Re: [Python-ideas] The future of Python parallelism. The GIL. Subinterpreters. Actors.

Nick Coghlan wrote: > It was never extended beyond Windows, and a Windows-only solution > doesn't meet the needs of a lot of folks interested in more efficient > exploitation of multiple local CPU cores. On the other hand Windows has a higher need for a better multi-core story. A reasonable

Re: [Python-ideas] The future of Python parallelism. The GIL. Subinterpreters. Actors.

On Mon, Jul 16, 2018 at 3:00 PM, Stephan Houben wrote: > What about the following model: you have N Python interpreters, each with > their own GIL. Each *Python* object belongs to precisely one interpreter. > > However, the interpreters share some common data storage: perhaps a shared > Numpy

Re: [Python-ideas] The future of Python parallelism. The GIL. Subinterpreters. Actors.

What about the following model: you have N Python interpreters, each with their own GIL. Each *Python* object belongs to precisely one interpreter. However, the interpreters share some common data storage: perhaps a shared Numpy array, or a shared Sqlite in-memory db. Or some key-value store

Re: [Python-ideas] The future of Python parallelism. The GIL. Subinterpreters. Actors.

On Mon, Jul 16, 2018 at 1:21 PM, Nathaniel Smith wrote: > On Sun, Jul 15, 2018 at 6:00 PM, Chris Angelico wrote: >> On Mon, Jul 16, 2018 at 10:31 AM, Nathaniel Smith wrote: >>> On Sun, Jul 8, 2018 at 11:27 AM, David Foster wrote: * The Actor model can be used with some effort via the

Re: [Python-ideas] The future of Python parallelism. The GIL. Subinterpreters. Actors.

On Sun, Jul 15, 2018 at 6:00 PM, Chris Angelico wrote: > On Mon, Jul 16, 2018 at 10:31 AM, Nathaniel Smith wrote: >> On Sun, Jul 8, 2018 at 11:27 AM, David Foster wrote: >>> * The Actor model can be used with some effort via the “multiprocessing” >>> module, but it doesn’t seem that streamlined

Re: [Python-ideas] The future of Python parallelism. The GIL. Subinterpreters. Actors.

On Mon, Jul 16, 2018 at 10:31 AM, Nathaniel Smith wrote: > On Sun, Jul 8, 2018 at 11:27 AM, David Foster wrote: >> * The Actor model can be used with some effort via the “multiprocessing” >> module, but it doesn’t seem that streamlined and forces there to be a >> separate OS process per line of

Re: [Python-ideas] The future of Python parallelism. The GIL. Subinterpreters. Actors.

On Sun, Jul 8, 2018 at 11:27 AM, David Foster wrote: > * The Actor model can be used with some effort via the “multiprocessing” > module, but it doesn’t seem that streamlined and forces there to be a > separate OS process per line of execution, which is relatively expensive. What do you mean by

Re: [Python-ideas] The future of Python parallelism. The GIL. Subinterpreters. Actors.

No one talked about this, but modern cpus with multiple numa nodes are atrocious to any shared memory (maybe threadripper is better, but multiple socket xeon is slow) and more and more all cpus will move to it on single chips, so a share nothing aproach can really make python a good contender on

Re: [Python-ideas] The future of Python parallelism. The GIL. Subinterpreters. Actors.

On 11 July 2018 at 00:31, David Foster wrote: > I was not aware of PyParallel. The PyParellel "parallel thread" > line-of-execution implementation is pretty interesting. Trent, big kudos to > you on that effort. > > Since you're speaking in the past tense and said "but we're not doing it > like

Re: [Python-ideas] The future of Python parallelism. The GIL. Subinterpreters. Actors.

On 7/14/2018 5:40 AM, Antoine Pitrou wrote: On Fri, 13 Jul 2018 18:22:24 -0600 Eric Snow wrote: 2. Give up on making things work inside the same OS process and rather focus on implementing better abstractions on top of the existing multiprocessing API so that the actor model is easier to

Re: [Python-ideas] The future of Python parallelism. The GIL. Subinterpreters. Actors.

On Sat, Jul 14, 2018 at 3:41 AM Antoine Pitrou wrote: > Davin has been mostly inactive. I'm the de facto maintainer for > multiprocessing. Ah, that's great to know. Sorry about the confusion. -eric ___ Python-ideas mailing list

Re: [Python-ideas] The future of Python parallelism. The GIL. Subinterpreters. Actors.

On Fri, 13 Jul 2018 18:22:24 -0600 Eric Snow wrote: > > 2. Give up on making things work inside the same OS process and rather > > focus on implementing better abstractions on top of the existing > > multiprocessing API so that the actor model is easier to program > > against. For example,

Re: [Python-ideas] The future of Python parallelism. The GIL. Subinterpreters. Actors.

On Tue, Jul 10, 2018 at 8:32 AM David Foster wrote: > > I was not aware of PyParallel. The PyParellel "parallel thread" > line-of-execution implementation is pretty interesting. Trent, big kudos > to you on that effort. +1 It's a neat project. Trent's pretty smart. :) > Since you're speaking

Re: [Python-ideas] The future of Python parallelism. The GIL. Subinterpreters. Actors.

On Sun, Jul 8, 2018 at 12:30 PM David Foster wrote: > In the past I have personally viewed Python as difficult to use for > parallel applications, which need to do multiple things simultaneously > for increased performance: > > * The old Threads, Locks, & Shared State model is inefficient in

Re: [Python-ideas] The future of Python parallelism. The GIL. Subinterpreters. Actors.

On 7/10/2018 10:31 AM, David Foster wrote: Since you're speaking in the past tense and said "but we're not doing it like that", I infer that the notion of a parallel thread was turned down for integration into CPython, as that appears to have been the original goal. A far as I remember,

Re: [Python-ideas] The future of Python parallelism. The GIL. Subinterpreters. Actors.

I was not aware of PyParallel. The PyParellel "parallel thread" line-of-execution implementation is pretty interesting. Trent, big kudos to you on that effort. Since you're speaking in the past tense and said "but we're not doing it like that", I infer that the notion of a parallel thread was

Re: [Python-ideas] The future of Python parallelism. The GIL. Subinterpreters. Actors.

On Sun, Jul 08, 2018 at 11:27:08AM -0700, David Foster wrote: > I'd like to solicit some feedback on what might be the most > efficient way to make forward progress on efficient parallelization > in Python inside the same OS process. The most promising areas > appear to be: You might find

Re: [Python-ideas] The future of Python parallelism. The GIL. Subinterpreters. Actors.

On 9 July 2018 at 04:27, David Foster wrote: > I'd like to solicit some feedback on what might be the most efficient way to > make forward progress on efficient parallelization in Python inside the same > OS process. The most promising areas appear to be: > > 1. Make the current subinterpreter