One idea would be for the Pool method to generate a uuid and slap it on the function as an attribute. If a function being passed in doesn't have one, generate one. If it already has one, just pass that instead of pickling. The child process will keep a cache mapping uuids to functions.
I'm still worried about unintended consequences. On Thu, Oct 18, 2018 at 9:00 AM Michael Selik <michael.se...@gmail.com> wrote: > On Thu, Oct 18, 2018 at 8:35 AM Sean Harrington <seanhar...@gmail.com> > wrote: > >> The most common use case comes up when passing instance methods (of >> really big objects!) to Pool.map(). >> > > This reminds me of that old joke: "A patient says to the doctor, 'Doctor, > it hurts when I ...!' The doctor replies, 'Well, don't do that.'" > > Further, let me pivot on my idea of __qualname__...we can use the `id` of >> `func` as the cache key to address your concern, and store this `id` on the >> `task` tuple (i.e. an integer in-lieu of the `func` previously stored >> there). >> > > Possible. Does the Pool keep a reference to the passed function in the > main process? If not, couldn't the garbage collector free that memory > location and a new function could replace it? Then it could have the same > qualname and id in CPython. Edge case, for sure. Worse, it'd be hard to > reproduce as it'd be dependent on the vagaries of memory allocation. > > >
_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
