I've started down the path of building a very simple file format for transferring Arrow data between nodes in my project.
Somebody on Reddit quite reasonably pointed out that I should look at Feather (which I didn't actually know about until now) and also mentioned that has been deprecated now in favor of some new format in the Arrow project itself? I'm also aware that the IPC mechanism might potentially be suitable but I haven't had time to read the specs yet. I'm waiting on Google Flatbuffers for Rust though before starting to contribute IPC support and I need something usable in the meantime (and I'm happy to donate whatever I build if it is useful). I'd appreciate hearing opinions on where Arrow is going in terms of defining file formats. Thanks, Andy.
