Hi,
I have a simple Flight server in Go. I am able to receive data to a Go server
from a Python client:
import pyarrow.flight as flight
import polars as pl
df = pl.read_csv("test.csv")
data_table = df.to_arrow()
client = pa.flight.connect("grpc://0.0.0.0:58474")
upload_descriptor = pa.flight.FlightDescriptor.for_path("uploaded.parquet")
writer, _ = client.do_put(upload_descriptor, data_table.schema)
writer.write_table(data_table)
writer.close()
This works perfectly.
But I am getting very stuck with the methods to format the DoPut to include the
descriptor and schema for the Go client.
My approximate code is:
client, err := flight.NewClientWithMiddleware(address, nil,
nil, grpc.WithTransportCredentials(insecure.NewCredentials()))
stream, err := client.DoPut(context.Background())
if err != nil {
panic(err)
}
wr := flight.NewRecordWriter(stream)
descriptor := &flight.FlightDescriptor{
Type: flight.DescriptorPATH,
Path: []string{*output},
}
wr.SetFlightDescriptor(descriptor)
schema := flight.SerializeSchema(rr.Schema(), memory.DefaultAllocator)
defer rr.Release()
for rr.Next() {
rec := rr.Record()
print("rec", rec.NumRows())
err = wr.Write(rec)
}
My question is how to correctly insert the flight.FlightDescriptor and
flight.SerializeSchema pieces.
The comments for these methods suggest it can be a bit intricate.
I can't find an example from the docs, the Flight SQL example in the
repository, or docs such as
https://voltrondata.com/resources/data-transfer-with-apache-arrow-and-golang or
https://blog.djnavarro.net/posts/2022-10-18_arrow-flight/
All the examples I could find (blogs and tests) only show DoGet, where it
appears the NewRecordWriter does most of the heavy lifting
Thanks
Thanks
