Re: [Python-Dev] bpo-34837: Multiprocessing.Pool API Extension - Pass Data to Workers w/o Globals

On Mon, Oct 22, 2018 at 2:01 PM Michael Selik wrote: > This thread seems more appropriate for python-ideas than python-dev. > > > On Mon, Oct 22, 2018 at 5:28 AM Sean Harrington > wrote: > >> Michael - the initializer/globals pattern still might be necessary if you >

Re: [Python-Dev] bpo-34837: Multiprocessing.Pool API Extension - Pass Data to Workers w/o Globals

executing at once, when `func` has a non-trivial memory footprint. On Fri, Oct 19, 2018 at 4:06 PM Michael Selik wrote: > On Fri, Oct 19, 2018 at 5:01 AM Sean Harrington > wrote: > >> I like the idea to extend the Pool class [to optimize the case when only >> one fu

Re: [Python-Dev] bpo-34837: Multiprocessing.Pool API Extension - Pass Data to Workers w/o Globals

On Fri, Oct 19, 2018 at 7:32 AM Joni Orponen wrote: > On Fri, Oct 19, 2018 at 9:09 AM Thomas Moreau < > thomas.moreau.2...@gmail.com> wrote: > >> Hello, >> >> I have been working on the concurent.futures module lately and I think >> this optimization should be avoided in the context of python

Re: [Python-Dev] bpo-34837: Multiprocessing.Pool API Extension - Pass Data to Workers w/o Globals

ain it won't have edge > case negative effects. > > > On Tue, Oct 16, 2018 at 2:53 PM Sean Harrington > wrote: > >> Is your concern something like the following? >> >> with Pool(8) as p: >> gen = p.imap_unordered(func, ls) >> first_elem

Re: [Python-Dev] bpo-34837: Multiprocessing.Pool API Extension - Pass Data to Workers w/o Globals

od behavior in some way if the user, > for whatever reason, mixed techniques? > > imap_unordered will only block when nexting the generator. If the user > mingles nexting that generator with, say, apply_async, could the change > you're proposing have some side-effect? > >

Re: [Python-Dev] bpo-34837: Multiprocessing.Pool API Extension - Pass Data to Workers w/o Globals

> >> On Fri, 12 Oct 2018 08:33:32 -0400 >> Sean Harrington wrote: >> > Hi Nathaniel - this if this solution can be made performant, than I >> would >> > be more than satisfied. >> > >> > I think this would require removing "func&qu

Re: [Python-Dev] bpo-34837: Multiprocessing.Pool API Extension - Pass Data to Workers w/o Globals

Yes - “func” (and “self” which func is bound to) would be copied to each child worker process, where they are stored and applied to each element of the iterable being mapped over. On Fri, Oct 12, 2018 at 10:41 AM Antoine Pitrou wrote: > On Fri, 12 Oct 2018 09:42:50 -0400 > Sean Harrington

Re: [Python-Dev] bpo-34837: Multiprocessing.Pool API Extension - Pass Data to Workers w/o Globals

o account". Do you see a difficulty in accomplishing the second behavior? On Fri, Oct 12, 2018 at 9:25 AM Antoine Pitrou wrote: > > Le 12/10/2018 à 15:17, Sean Harrington a écrit : > > The implementation details need to be flushed out, but agnostic of > > these

Re: [Python-Dev] bpo-34837: Multiprocessing.Pool API Extension - Pass Data to Workers w/o Globals

Fri, Oct 12, 2018 at 9:07 AM Antoine Pitrou wrote: > On Fri, 12 Oct 2018 08:33:32 -0400 > Sean Harrington wrote: > > Hi Nathaniel - this if this solution can be made performant, than I would > > be more than satisfied. > > > > I think this would require removing &quo

Re: [Python-Dev] bpo-34837: Multiprocessing.Pool API Extension - Pass Data to Workers w/o Globals

gs... On Thu, Oct 4, 2018 at 6:15 AM Nathaniel Smith wrote: > On Wed, Oct 3, 2018 at 6:30 PM, Sean Harrington > wrote: > > with Pool(func_kwargs={"big_cache": big_cache}) as pool: > > pool.map(func, ls) > > I feel like it would be nicer to spell

Re: [Python-Dev] bpo-34837: Multiprocessing.Pool API Extension - Pass Data to Workers w/o Globals

You don't like using Pool.starmap and itertools.repeat or a comprehension > that repeats an object? > > > > On Wed, Oct 3, 2018, 6:30 PM Sean Harrington wrote: > >> Hi guys - >> >> The solution to "lazily initialize" an expensive object in the worker >

Re: [Python-Dev] bpo-34837: Multiprocessing.Pool API Extension - Pass Data to Workers w/o Globals

a different module, testing w/o globals, etc...).. This would essentially be an efficient implementation of Pool.starmap(), where kwargs are static, and passed to each application of "func" over our iterable. Thoughts? On Sat, Sep 29, 2018 at 3:00 PM Michael Selik wrote: > On Sat,

Re: [Python-Dev] bpo-34837: Multiprocessing.Pool API Extension - Pass Data to Workers w/o Globals

On Sat, Sep 29, 2018 at 8:18 AM Antoine Pitrou wrote: > On Sat, 29 Sep 2018 08:13:19 -0400 > Sean Harrington wrote: > > > > > > Hmm... We might have a disagreement on the target audience of the > > > multiprocessing module. multiprocessing isn't very

Re: [Python-Dev] bpo-34837: Multiprocessing.Pool API Extension - Pass Data to Workers w/o Globals

On Fri, Sep 28, 2018 at 9:27 PM Michael Selik wrote: > On Fri, Sep 28, 2018 at 2:11 PM Sean Harrington > wrote: > > kwarg on Pool.__init__ called `expect_initret`, that defaults to False. > When set to True: > > Capture the return value of the initializer kwarg of Poo

Re: [Python-Dev] bpo-34837: Multiprocessing.Pool API Extension - Pass Data to Workers w/o Globals

On Sat, Sep 29, 2018 at 6:24 AM Antoine Pitrou wrote: > > Hi Sean, > > On Fri, 28 Sep 2018 19:23:06 -0400 > Sean Harrington wrote: > > My simple argument is that the > > developer should not be constrained to make the objects passed globally > > available

Re: [Python-Dev] bpo-34837: Multiprocessing.Pool API Extension - Pass Data to Workers w/o Globals

Hi Antoine - see inline below for my response...thanks for your time! On Fri, Sep 28, 2018 at 6:45 PM Antoine Pitrou wrote: > > Hi, > > On Fri, 28 Sep 2018 17:07:33 -0400 > Sean Harrington wrote: > > > > In *short*, the implementation of the feature works as fol

[Python-Dev] bpo-34837: Multiprocessing.Pool API Extension - Pass Data to Workers w/o Globals

I am proposing an extension to the multiprocessing.Pool API that allows for an alternative way to pass data to Pool worker processes, *without* using globals. A PR has been opened , extensive test coverage is also included, with all tests & CI passing