Allan Haldane <allanhald...@gmail.com> wrote: > You probably already know this, but I just wanted to note that the > mpi4py module has worked around pickle too. They discuss how they > efficiently transfer numpy arrays in mpi messages here: > http://pythonhosted.org/mpi4py/usrman/overview.html#communicating-python-objects-and-array-data
Unless I am mistaken, they use the PEP 3118 buffer interface to support NumPy as well as a number of other Python objects. However, this protocol makes buffer aquisition an expensive operation. You can see this in Cython if you use typed memory views. Assigning a NumPy array to a typed memoryview (i,e, buffer acqisition) is slow. They are correct that avoiding pickle means we save some memory. It also avoids creating and destroying temporary Python objects, and associated reference counting. However, because of the expensive buffer acquisition, I am not sure how much faster their apporach will be. I prefer to use the NumPy C API, and bypass any unneccesary overhead. The idea is to make IPC of NumPy arrays fast, and then we cannot have an expensive buffer acquisition in there. Sturla _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org https://mail.scipy.org/mailman/listinfo/numpy-discussion
