Jesse Noller <jnoller <at> gmail.com> writes:
> We already have an implementation that spawns a
> subprocess and then pushes the required state to the child. The
> fundamental need for things to be pickleable *all the time* kinda
> makes it annoying to work with.
>
just a lurker here... but this topic hits home with me so thought I'd chime
in. I'm a windows user and I would *love* to use multiprocessing a lot more
because *in theory* it solves a lot of the problems I deal with very nicely
(lot sof financial data number crunching). However, the pickling requirement
makes it very very difficult to actually get any reasonably complex code to
work properly with it.
A lot of the time the functions I want to call in the spawned processes are
actually fairly self contained and don't need most of the environment of the
parent process shoved into it, so it's annoying that it fails because some data
I don't even need in the child process can't be pickled.
What about having an option to skip all the parent environment data pickling
and require the user to manually invoke any imports that are needed in the
target functions as the first step inside their target function?
for example...
def target_function(object_from_module_xyz):
import xyz
return object_from_module_xyz.do_something()
and if I forgot to import all the stuff necessary for the arguments being
passed into my function to work, then it's my own problem.
Although maybe there is some obvious problem with this that I am not seeing.
Anyway, just food for thought.
- Matt
_______________________________________________
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
