Hi Nathaniel, 2018-07-19 1:33 GMT+02:00 Nathaniel Smith <n...@pobox.com>:
> Note that this everything you said here also exactly describes the > programming model for the existing 'multiprocessing' module: > "structured clone" is equivalent to how multiprocessing uses pickle to > transfer arbitrary objects, or you can use multiprocessing.Array to > get a shared view on raw "C"-style data. > This is true. In fact, I am a big fan of multiprocessing and I think it is often overlooked/underrated. Experience with multiprocessing is also what has me convinced that share-nothing or share-explicit approach to concurrency is a useful programming model. The main limitation of multiprocessing comes when you need to go outside Python, and you need to interact with C/C++ libraries or operating services from multiple processes. The support for this generally varies from "extremely weak" to "none at all". For example, things I would like to in parallel with a main thread/process: * Upload data to the GPU using OpenGL or OpenCL * Generate a picture in pyqt QImage, then hand over zero-copy to main thread * interact with a complex scenegraph in C++ (shared with main thread) This is impossible right now but would be possible if the interpreters were all in-process. In addition, there are things which are now hard with "multiprocessing" but could be fixed. For example, sharing a Numpy array is possible but very inconvenient. You need to first allocate the raw data segment, communicate that, then create in each process an array which uses this data segment. Ideally, this would rather work like this: ar = numpy.zeros((30, 30), shared=True) and then "ar" would automatically be shared. This is fixable but given the other limitations above the question is if it is worthwhile to fix it now. It would be a lot simpler to fix if we had the in-process model. But yeah, I am actually also very open to ideas on how multiprocessing could be made more convenient and powerful. Perhaps there are ways, and I am just not seeing them. Stephan
_______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
