On Tue, May 5, 2020 at 3:47 PM Guido van Rossum <gu...@python.org> wrote: > > This sounds like a significant milestone! > > Is there some kind of optimized communication possible yet between > subinterpreters? (Otherwise I still worry that it's no better than > subprocesses -- and it could be worse because when one subinterpreter > experiences a hard crash or runs out of memory, all others have to die with > it.)
As far as I understand it, the subinterpreter folks have given up on optimized passing of objects, and are only hoping to do optimized (zero-copy) passing of raw memory buffers. On my laptop, some rough measurements [1] suggest that simply piping bytes between processes goes at ~2.8 gigabytes/second, and that pickle/unpickle is ~10x slower than that. So that would suggest that once subinterpreters are fully optimized, they might provide a maximum ~10% speedup vs multiprocessing, for a program that's doing nothing except passing pickled objects back and forth. Of course, any real program that's spawning parallel workers will presumably be designed so its workers spend most of their time doing work on that data, not just passing it back and forth. That makes a 10% speedup highly unrealistic; in real-world programs it will be much smaller. So IIUC, subinterpreter communication is currently about the same speed as multiprocessing communication, and the plan is to keep it that way. -n [1] Of course there are a lot of assumptions in my quick back-of-the-envelope calculation: pickle speed depends on the details of the objects being pickled, there are other serialization formats, there are other IPC methods that might be faster but are more complicated (shared memory), the stdlib 'multiprocessing' library might not be as good as it could be (the above measurements are for an ideal multiprocessing library, I haven't tested the one we currently have in the stdlib), etc. So maybe there's some situation where subinterpreters look better. But I've been pointing out this issue to Eric et al for years and they haven't disputed it, so I guess they haven't found one yet. -- Nathaniel J. Smith -- https://vorpus.org _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/GZEJOIT6SZDUVPND64VKFFKFX6AJWZ7W/ Code of Conduct: http://python.org/psf/codeofconduct/
