toddtomkinson commented on issue #37937:
URL: https://github.com/apache/arrow/issues/37937#issuecomment-2290011767
Hello! I'm working on implementing a few of the Flight RPC endpoints into
our existing gRPC server (where we use the new async/callback api exclusively)
and came across this ticket. I have everything working, but would love to be
able to utilize the the gRPC zero-copy serialization in my server (right now
I'm returning populated FlightData objects, which requires a copy of the data
in the IpcPayload objects).
I could reuse the existing functions if, instead of hiding all of the gRPC
server internals, arrow exposed the functions in
flight/transport/grpc/serialization_internal.{h,cc}--specifically the
FlightDataSerialize function. My plan was to build a
ServerWriteReactor<ByteBuffer> implementation that my application could add
RecordBatches to, that would in turn translate each RecordBatch into a series
of FlightPayload objects (similar to what is done in the RecordBatchStream)
and in turn to a series of ByteBuffers (using FlightDataSerialize). The reactor
would send the ByteBuffers as soon as they were available and the previous send
was complete.
If there isn't already work in process to implement this ticket I'd be happy
to take a stab at exposing the serialization internals and then exposing a
reactor implementation as described above. With those in place it would be much
easier for projects to implement Flight RPC interfaces into existing gRPC
servers, while still taking advantage of the zero-copy serialization.
Thoughts?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
