Pascal Chambon writes: > I don't really get it there... it seems to me that multiprocessing only > requires picklability for the objects it needs to transfer, i.e those > given as arguments to the called function, and thsoe put into > multiprocessing queues/pipes. Global program data needn't be picklable - > on windows it gets wholly recreated by the child process, from python > bytecode. > > So if you're having pickle errors, it must be because the > "object_from_module_xyz" itself is *not* picklable, maybe because it > contains references to unpicklable objets. In such case, properly > implementing pickle magic methods inside the object should do it, > shouldn't it ?
I'm also a long time lurker (and in financial software, coincidentally). Pascal is correct here. We use a number of C++ libraries wrapped via Boost.Python to do various calculations. The typical function calls return wrapped C++ types. Boost.Python types are not, unfortunately, pickleable. For a number of technical reasons, and also unfortunately, we often have to load these libraries in their own process, but we want to hide this from our users. We accomplish this by pickling the instance data, but importing the types fresh when we unpickle, all implemented in the magic pickle methods. We would lose any information that was dynamically added to the type in the remote process, but we simply don't do that. We very often have many unpickleable objects imported somewhere when we spin off our processes using the multiprocess library, and this does not cause any problems. Jesse Noller <jnoller <at> gmail.com> writes: > We already have an implementation that spawns a > subprocess and then pushes the required state to the child. The > fundamental need for things to be pickleable *all the time* kinda > makes it annoying to work with. This requirement puts a fairly large additional strain on working with unwieldy, wrapped C++ libraries in a multiprocessing environment. I'm not very knowledgeable on the internals of the system, but would it be possible to have some kind of fallback system whereby if an object fails to pickle we instead send information about how to import it? This has all kinds of limitations - it only works for importable things (i.e. not instances), it can potentially lose information dynamically added to the object, etc., but I thought I would throw the idea out there so someone knowledgeable can decide if it has any merit. Ben _______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
