Hi Clive, I'd like to understand the high level design for a system where a Go > process can communicate an Arrow data structure to a python process on the > same CPU > I see two options - Different processes with hared memory, probably using plasma - Same process. The either Go uses Python shared library or Python using Go compiled to shared library (-build-mode=c-shared)
> - and for the python process to zero-copy gain access to that data, change > it and inform the Go process. This is low latency so I don't want to save > to file. > IIRC arrow is not built for mutation. You build an Array/Table once and then use it. Would this need the use of Plasma as a zero-copy store for the data between > the two processes or do I need to use IPC? But with IPC you are > transferring the data which is not needed in this case as I understand it. > Any pointers to examples would be appreciated. > See above about options. Note that currently the Go arrow implementation doesn't support IPC or plasma (though it's in the works). Yoni & I are working on another option which is using the C++ arrow library from Go. It does support plasma and since it uses the same underlying C++ library that Python does you'll be able to pass a pointer around without copying data. It's at very alpha-ish state but you're more than welcomed to give it a try - https://github.com/353solutions/carrow Happy hacking, Miki
