My bad, IPC in Go seems to be implemented - https://issues.apache.org/jira/browse/ARROW-3679
On Mon, Jul 8, 2019 at 10:18 AM Sebastien Binet <[email protected]> wrote: > As far as i know, Go does support IPC (as in the arrow IPC format) > > Another option which has been discussed at some point was to have a shared > memory allocator so the arrow arrays could be shared between processes. > > I haven't looked in details what implementing plasma support for Go would > need on the Go side... > > -s > > > sent from my droid > > On Mon, Jul 8, 2019, 08:29 Miki Tebeka <[email protected]> wrote: > >> Hi Clive, >> >> I'd like to understand the high level design for a system where a Go >>> process can communicate an Arrow data structure to a python process on the >>> same CPU >>> >> I see two options >> - Different processes with hared memory, probably using plasma >> - Same process. The either Go uses Python shared library or Python using >> Go compiled to shared library (-build-mode=c-shared) >> >> >>> - and for the python process to zero-copy gain access to that data, >>> change it and inform the Go process. This is low latency so I don't want >>> to save to file. >>> >> IIRC arrow is not built for mutation. You build an Array/Table once and >> then use it. >> >> Would this need the use of Plasma as a zero-copy store for the data >>> between the two processes or do I need to use IPC? But with IPC you are >>> transferring the data which is not needed in this case as I understand it. >>> Any pointers to examples would be appreciated. >>> >> See above about options. Note that currently the Go arrow implementation >> doesn't support IPC or plasma (though it's in the works). >> >> Yoni & I are working on another option which is using the C++ arrow >> library from Go. It does support plasma and since it uses the same >> underlying C++ library that Python does you'll be able to pass a pointer >> around without copying data. It's at very alpha-ish state but you're more >> than welcomed to give it a try - https://github.com/353solutions/carrow >> >> Happy hacking, >> Miki >> >
