Hi,
Have been hacking on this for a while, but wanted to make sure I'm on
the right track.
Is it possible to supply a pre-serialized IPC stream of data from a
Flight server's DoGet function? It looks like a table *object* (or
schema + record batches) can be supplied to the FlightDataStream
parameter (using a RecordBatchStream) but not plain bytes.
I've had a look at implementing a FlightDataStream for plain bytes. I
can see the byte stream needs to be split up into FlightPayloads, but
it's not clear what goes where in each one.
Given the following defs for FlightPayloads...
struct ARROW_FLIGHT_EXPORT FlightPayload {
std::shared_ptr<Buffer> descriptor;
std::shared_ptr<Buffer> app_metadata;
ipc::IpcPayload ipc_message;
};
struct IpcPayload {
MessageType type = MessageType::NONE;
std::shared_ptr<Buffer> metadata;
std::vector<std::shared_ptr<Buffer>> body_buffers;
int64_t body_length = 0;
};
AFAICT it looks like:
"descriptor" is ignored for DoGet
"app_metadata" can be ignored
"type" is set to whatever message it is (e.g. schema, batch, etc)
"metadata" buffer should contain a schema IPC bytes?
"body_buffers" should contain IPC bytes for each batch? (Why are there
multiple buffers? Is there any reason not to just use the first slot of
the buffers vector?)
Any advice appreciated.
Thanks, Matt
