Charles-François Natali added the comment: > Also, does pickle currently handle byte strings larger than 4GB?
The 2.7 failure is indeed a pickle limitation, which should now be fixed by issue #13555. > On a machine with 256GB of RAM, it makes more sense to send arrays > of this size than say on a laptop... Richard was saying that you shouldn't serialize such a large array, that's just a huge performance bottleneck. The right way would be to use a shared memory. > multiprocessing currently only allows sharing of such shared arrays > using inheritance. You mean through fork() COW? > Perhaps we need a picklable mmap type which can be sent over pipes > and queues. (On Unix this would probably require fd passing.) If you use POSIX semaphores, you could pass the semaphore path and use sem_open in the other process (but that would mean you can't unlink it right after open). ---------- nosy: +neologix _______________________________________ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue17560> _______________________________________ _______________________________________________ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
