vtk9 opened a new issue, #207:
URL: https://github.com/apache/arrow-go/issues/207
### Describe the enhancement requested
**Use case**: there is an arrow table in shared memory. It was written using
Python's Arrow `RecordBatchStreamWriter` (although even if it was written using
`RecordBatchStreamWriter`, I believe the same problem would exist due to the
current state of the Go Arrow implementation). We would like to load it lazily
and stream it directly to an arrow flight client. We would do so by opening up
the file in process memory (lazily), then we would first aggregate the list of
records (lazily) and then read the contents of the records for the first time
when streaming the data to the client.
via
```
func (s *MyServer) DoGet(
ticket *flight.Ticket,
server flight.FlightService_DoGetServer,
) error {
...
w := flight.NewRecordWriter(server, writerOpts...)
for _, r := range s.recordsInTable {
if err = w.Write(r); err != nil {
return err
}
}
```
Ideally, up until the records sent, no data would be actually copied/ready.
However, this doesn't appear to be possible at the moment since there is a copy
done when processing the records
https://github.com/apache/arrow-go/blob/main/arrow/ipc/message.go#L217 instead
or referencing the already allocated/used shared memory. Instead, there might
need to be a new `MessageReader` implementation where this copy is avoided.
Thank you!
### Component(s)
Other
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
