kou commented on PR #28: URL: https://github.com/apache/arrow-flight-sql-postgresql/pull/28#issuecomment-1464448737
> it seems like we should be able to allow read/write in parallel at some level (though that will be tricky). Yes. I think so too. > we may consider formalizing a version of C Data Interface for shared memory. Interesting. But it's one of "non-goals"...: https://arrow.apache.org/docs/format/CDataInterface.html#non-goals > (I think the current version doesn't work because it relies on absolute pointers.) Right. > Then we could even avoid the IPC serialization/deserialization steps (though: I think in this case that's "only" a copy-and-concatenate). We may be able to reduce IPC serialization/deserialization steps by another approach. Flight server needs to serialize Apache Arrow data to send it to client. The current implementation uses `arrow::flight::RecordBatchStream` for it but it requires `arrow::RecordBatchReader`. The current implementation deserializes data received from an executor for the `arrow::RecordBatchReader`. We may be able to avoid the deserilization step by implementing a custom `arrow::flight::FlightDataStream` that accepts `arrow::io::InputStream`. The `arrow::io::InputStream` produces Apache Arrow IPC streaming format data. So the custom `arrow::flight::FilghtDataStream` just passes produced data as-is without additional deserialization/serialization steps. > (If it's a serious enough bottleneck, we could probably come up with all sorts of schemes as needed.) Right. We need to profile as the first step for optimization. It may not be a bottleneck. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
