On Sun, Jul 8, 2018 at 11:27 AM, David Foster <davidf...@gmail.com> wrote: > * The Actor model can be used with some effort via the “multiprocessing” > module, but it doesn’t seem that streamlined and forces there to be a > separate OS process per line of execution, which is relatively expensive.
What do you mean by "the Actor model"? Just shared-nothing concurrency? (My understanding is that in academia it means shared-nothing + every thread/process/whatever gets an associated queue + queues are globally addressable + queues have unbounded buffering + every thread/process/whatever is implemented as a loop that reads messages from its queue and responds to them, with no internal concurrency. I don't know why this particular bundle of features is considered special. Lots of people seem to use it in looser sense though.) > I'd like to solicit some feedback on what might be the most efficient way to > make forward progress on efficient parallelization in Python inside the same > OS process. The most promising areas appear to be: > > 1. Make the current subinterpreter implementation in Python have more > complete isolation, sharing almost no state between subinterpreters. In > particular not sharing the GIL. The "Interpreter Isolation" section of PEP > 554 enumerates areas that are currently shared, some of which probably > shouldn't be. > > 2. Give up on making things work inside the same OS process and rather focus > on implementing better abstractions on top of the existing multiprocessing > API so that the actor model is easier to program against. For example, > providing some notion of Channels to communicate between lines of execution, > a way to monitor the number of Messages waiting in each channel for > throughput profiling and diagnostics, Supervision, etc. In particular I > could do this by using an existing library like Pykka or Thespian and > extending it where necessary. I guess I would distinguish though between "multiple processes" and "the multiprocessing module". The module might be at the point in its lifecycle where starting over is at least worth considering, and one thing I'm hoping to do with Trio is experiment with making worker process patterns easier to work with. But the nice thing about these two options is that subinterpreters are basically a way to emulate multiple Python processes within a single OS process, which means they're largely interchangeable. There are trade-offs in terms of compatibility, how much work needs to be done, probably speed, but if you come up with a great API based around one model then you should be able to switch out the backend later without affecting users. So if you want to start experimenting now, I'd use multiple processes and plan to switch to subinterpreters later if it turns out to make sense. -n -- Nathaniel J. Smith -- https://vorpus.org _______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
