Hey All, I used my Strata talk today as a forcing function to make additional progress on a GRPC-based Arrow RPC protocol [1]. I’m calling it “Apache Arrow Flight”. You can take a look at the work here [2]. I’ll work to clean up my work and explain my thoughts about the protocol in the coming days. High-level: use protobuf as a encapsulation format so that any client that is supported in GRPC will work. However, we can optimize the read/write path for targeted languages and hand control the serialization/deserialization and memory handling. (I did that in this Java patch [3][4][5].) I also looked at starting to use GRPC generated bindings within Python but it looks like some glue code may be needed in the C++ layer since Python delegates down frequently. I also am still trying to understand GRPC back-pressure patterns and whether the protocol realistically needs to change to cover real-world high performance use cases.
I’ll send out some slides about the ideas and update README, etc. soon. Thanks, Jacques [1] https://github.com/jacques-n/arrow/blob/flight/java/flight/src/main/protobuf/flight.proto [2] http://github.com/jacques-n/arrow/ [3] https://github.com/jacques-n/arrow/tree/flight/java/flight/src/main/java/org/apache/arrow/flight/grpc [4] https://github.com/jacques-n/arrow/blob/flight/java/flight/src/main/java/org/apache/arrow/flight/ArrowMessage.java#L253 <https://github.com/jacques-n/arrow/blob/flight/java/flight/src/main/java/org/apache/arrow/flight/ArrowMessage.java#L253> [5] https://github.com/jacques-n/arrow/blob/flight/java/flight/src/main/java/org/apache/arrow/flight/ArrowMessage.java#L185
