Hey Micah, There hasn't really been formal discussions of Flight "backends", but there has been some talk about supporting protocols besides gRPC (which is why the implementation tries to abstract away from gRPC). So it might be interesting to treat this as another "protocol" in Flight clients that can only read from BigQuery.
If the current gRPC stub definitions are reasonably stable (in your opinion), I might try implementing support. That might get reasonable performance still, especially in Python (where I've found that a lot of performance is lost copying messages into/out of CPython to work with Protobuf & gRPC - presumably grpc-c++ wouldn't do that and/or we could do the read optimizations ourselves). Best, David On 7/27/19, Micah Kornfield <emkornfi...@gmail.com> wrote: > Hi David, > >> I see the original thread mentioned Flight support, do you think it'd >> be possible to support Flight natively? Or conversely, maybe this >> could be a candidate for a new Flight "backend" as has been discussed. > > Right now our main priority is addressing the caveats I mentioned above. > After that I would like to see if we can get some of the client side > optimizations Flight uses either into gRPC directly or perhaps specialized > clients. Given my current backlog, I don't think these will happen anytime > soon. > > In my opinion (this is really above my pay-grade) native flight support > probably hinges on two things: > 1. Customer demand for it. > 2. The Flight APIs would likely need to conform to the Cloud API design > guidelines [1]. I haven't looked closely enough to see if there are any > incompatibilities between the two. > > I don't recall seeing the discussion on "new flight backends", could you > provide a pointer? This might be a shorter path for support. I'd also > like to make an adapter for the C++ Datasets API, but again, given my > current backlog it will take a while for this to happen. > > Thanks, > Micah > > [1] https://cloud.google.com/apis/design/ > > On Sat, Jul 27, 2019 at 6:17 AM David Li <li.david...@gmail.com> wrote: > >> This is super awesome, thanks for sharing! >> >> I see the original thread mentioned Flight support, do you think it'd >> be possible to support Flight natively? Or conversely, maybe this >> could be a candidate for a new Flight "backend" as has been discussed. >> >> Best, >> David >> >> On 7/26/19, Micah Kornfield <emkornfi...@gmail.com> wrote: >> > Hi Arrow Dev, >> > As a follow-up to an old thread [1] on working with BigQuery and Arrow. >> > I >> > just wanted to share some work that Brian Hulette and I helped out >> > with. >> > >> > I'm happy to announce there is now preliminary support for reading >> > Arrow >> > data in the BigQuery Storage API [1]. Python library support is >> available >> > in the latest release of google-cloud-bigquery-storage [2][3]. >> > >> > Caveats: >> > - Small cached tables are not supported (same with Avro) >> > - Row filters aren't supported yet. >> > >> > Cheers, >> > Micah >> > >> > [1] >> > >> https://lists.apache.org/thread.html/6d374dc6c948d3e84b1f0feda1d48eddf905a99c0ef569d46af7f7af@%3Cdev.arrow.apache.org%3E >> > [2] https://cloud.google.com/bigquery/docs/reference/storage/ >> > [3] https://pypi.org/project/google-cloud-bigquery-storage/ >> > [4] >> > >> https://googleapis.github.io/google-cloud-python/latest/bigquery_storage/gapic/v1beta1/reader.html#google.cloud.bigquery_storage_v1beta1.reader.ReadRowsIterable.to_arrow >> > >> >
