if someone were to register arrow flight protos on buf bsr [1], it looks like you would be able to use buf curl [2] without having a checkout of the arrow repo. Here's an example for what it looks like for substrait [3]. [1]: https://buf.build/explore[2]: https://buf.build/blog/buf-curl[3]: https://buf.build/substrait/substrait Sent from Proton Mail for iOS On Thu, Mar 20, 2025 at 17:43, Bryce Mecum <bryceme...@gmail.com> wrote: I had a checkout of the arrow monorepo [1] and the file is at
./format/Flight.proto [2]. [1] https://github.com/apache/arrow/ [2] https://github.com/apache/arrow/blob/main/format/Flight.proto On Thu, Mar 20, 2025 at 5:40 PM Z A <z11...@outlook.com> wrote: > > Hi Bryce, > Thanks for the reply. Where can i get the path for the "-import-path" and also where can i get the file Flight.proto? > ________________________________ > From: Bryce Mecum <bryceme...@gmail.com> > Sent: Tuesday, March 18, 2025 12:15 PM > To: user@arrow.apache.org <user@arrow.apache.org> > Subject: Re: api gateway with arrow flight grpc > > Yes, you can use a tool like grpcurl to call RPCs on an Arrow Flight > server by doing something like this, > > grpcurl -plaintext \ > -import-path /path/to/clone/of/arrow/format/ \ > -proto Flight.proto \ > localhost:8815 \ > arrow.flight.protocol.FlightService/DoGet > > On Tue, Mar 18, 2025 at 7:26 AM Z A <z11...@outlook.com> wrote: > > > > I am thinking to use it as a grpc proxy, so not HTTP <-> Arrow Flight grpc, but I am still looking into it. Thanks for explaining those patterns. > > Can i use grpcurl to call Arrow Flight? > > > > ________________________________ > > From: kekronbekron <kekronbek...@protonmail.com> > > Sent: Sunday, March 16, 2025 11:06 PM > > To: user@arrow.apache.org <user@arrow.apache.org> > > Subject: Re: api gateway with arrow flight grpc > > > > Yes, an API gateway in front surely helps, especially if you "front" this with an HTTP server rather than gRPC. > > 3 different patterns - > > > > 1: > > API gateway / reverse proxy --> HTTP server --> Flight Server > > GET bla.com/pipe --> connects to a duckdb file via Flight and sends stuff over > > > > 2: > > API gateway / reverse proxy --> HTTP server --> Flight SQL > > GET bla.com/pipe --> connects to a duckdb file and sends SQL over > > Advantage is that you can now connect to the Flight SQL server with JDBC/ADBC even outside of this app, essentially adding concurrency to the backing (Duck)DBs... on top of concurrent access to the DB(s) _in_ this program > > > > 3: > > API gateway / reverse proxy --> HTTP server --> ADBC > > https://arrow.apache.org/blog/2025/03/10/fast-streaming-inserts-in-duckdb-with-adbc/ > > This removes the need for Flight altogether and appends directly into DBs with the ADBC driver manager. > > With HTTP in the front, you can route requests to whichever DB you want. > > However, managing replication (or dual write) b/w clones/copies of DBs needs some thinking. > > In DuckDB's case, it's challenging because remote attach over https/s3 is readonly. > > So even if you stream out IPC in your HTTP server's function (for the /pipe route), you'd need a second HTTP server to receive IPC and write it in... which is fine and still works. > > But do note that this means accessing the DBs entirely via HTTP, so most of the securty etc will be at the API gateway / HTTP layer. > > > > > > > > On Monday, March 17th, 2025 at 06:41, Z A <z11...@outlook.com> wrote: > > > > Thanks all for the reply and insights! I am going to investigate the grpc passthru feature which is supported by the API gateway product. > > Btw, I wrote a simple Arrow Flight Server in python, but I couldn't figure out how to call it from grpcurl (all my attempts failed), is that even possible? > > ________________________________ > > From: Hélder Gregório <helder.grego...@dremio.com> > > Sent: Saturday, March 15, 2025 8:28 AM > > To: user@arrow.apache.org <user@arrow.apache.org> > > Subject: Re: api gateway with arrow flight grpc > > > > Hi, > > > > I'm not sure about integration with the services you mentioned but assuming some responsibilities of the API Gateway are handling authentication and load balancing, you can implement a flight proxy by creating a service that implements FlightProducer interface. > > The service can access headers for validation if needed and redirect vectors it receives to the designated flight server using another flight client. > > > > Hope it helps > > > > On Sat, Mar 15, 2025 at 3:03 AM kekronbekron <kekronbek...@protonmail.com> wrote: > > > > Sure - > > > > https://www.definite.app/blog/duck-takes-flight > > https://blobs.duckdb.org/events/duckcon6/rusty-conover-airport-for-duckdb-letting-duckdb-take-apache-arrow-flights.pdf > > > > > > > > > > On Saturday, March 15th, 2025 at 01:25, Bryce Mecum <bryceme...@gmail.com> wrote: > > > > > Hi kekronbekron, can you share any pointers to the pattern you mention > > > and where people are talking about it? It sounds like something I > > > might be interested in tracking. > > > > > > On Thu, Mar 13, 2025 at 7:27 PM kekronbekron > > > kekronbek...@protonmail.com wrote: > > > > > > > I'm embarking on exactly this. > > > > Amusing how this pattern has become "vocal" in the past few weeks, though Flight has existed for a few years now. > > > > > > > > On Friday, March 14th, 2025 at 03:56, Paul Whalen pgwha...@gmail.com wrote: > > > > > > > > I haven’t used anything you’ve mentioned, but I have deployed an arrow flight server in an istio/envoy service mesh, and seen all the benefits of gRPC in that context without any trouble. > > > > > > > > I’ve often though it would be fun to write an envoy filter for the Flight or Flight SQL protocol, but it isn’t clear to me that there’s much demand for it (for comparison, there’s one for Postgres: https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/other_protocols/postgres). > > > > > > > > Paul > > > > > > > > On Mar 13, 2025, at 5:05 PM, Z A z11...@outlook.com wrote: > > > > > > > > Thanks Felipe! > > > > ________________________________ > > > > From: Felipe Oliveira Carvalho felipe...@gmail.com > > > > Sent: Thursday, March 13, 2025 12:29 AM > > > > To: user@arrow.apache.org user@arrow.apache.org > > > > Subject: Re: api gateway with arrow flight grpc > > > > > > > > No, but if these are gRPC proxies they should work. > > > > > > > > On Wed, 12 Mar 2025 at 18:13 Z A z11...@outlook.com wrote: > > > > > > > > Hi, > > > > I just subscribed to this mailing list, and apologize if this is a silly question. > > > > Has anyone ever done any integration of API Gateway (i.e. Kong, Tyk, KrakenD, etc.) with your own Arrow Flight Server? > > > > Thanks! > > > >
signature.asc
Description: OpenPGP digital signature
