Matt Knox a écrit :
Jesse Noller <jnoller <at> gmail.com> writes:
We already have an implementation that spawns a
subprocess and then pushes the required state to the child. The
fundamental need for things to be pickleable *all the time* kinda
makes it annoying to work with.
just a lurker here... but this topic hits home with me so thought I'd chime
in. I'm a windows user and I would *love* to use multiprocessing a lot more
because *in theory* it solves a lot of the problems I deal with very nicely
(lot sof financial data number crunching). However, the pickling requirement
makes it very very difficult to actually get any reasonably complex code to
work properly with it.
A lot of the time the functions I want to call in the spawned processes are
actually fairly self contained and don't need most of the environment of the
parent process shoved into it, so it's annoying that it fails because some data
I don't even need in the child process can't be pickled.
What about having an option to skip all the parent environment data pickling
and require the user to manually invoke any imports that are needed in the
target functions as the first step inside their target function?
for example...
def target_function(object_from_module_xyz):
import xyz
return object_from_module_xyz.do_something()
and if I forgot to import all the stuff necessary for the arguments being
passed into my function to work, then it's my own problem.
Although maybe there is some obvious problem with this that I am not seeing.
Anyway, just food for thought.
- Matt
Hello
I don't really get it there... it seems to me that multiprocessing only
requires picklability for the objects it needs to transfer, i.e those
given as arguments to the called function, and thsoe put into
multiprocessing queues/pipes. Global program data needn't be picklable -
on windows it gets wholly recreated by the child process, from python
bytecode.
So if you're having pickle errors, it must be because the
"object_from_module_xyz" itself is *not* picklable, maybe because it
contains references to unpicklable objets. In such case, properly
implementing pickle magic methods inside the object should do it,
shouldn't it ?
Regards,
Pascal
_______________________________________________
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
