avantgardnerio opened a new issue, #2581: URL: https://github.com/apache/arrow-rs/issues/2581
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.** The Flight protocol has the same shortcomings of many protocols such as FTP and zookeeper: it expects that the interweb is a happy place where everyone knows their public IP address and everything is routable the same way from every where. Unfortunately this is not the case IRL, so I would argue that it is best practice to stream everything over a single connection. This has manifested itself in weeks of work recently in two facets: 1. The ArrowFlightSQL JDBC driver doesn't support alternate endpoints, and PRs to fix this are currently on hold 2. When trying to deploy to Kubernetes (I would posit a common situation), it is very hard to get kubernetes to behave in such a way as that it can horizontally scale, and yet have each executor be reachable by unique address to the outside world. **Describe the solution you'd like** Rather than hack on kubernetes routing _and_ build a JDBC fork, it would be much more convenient if we can simply proxy all Flights through the existing connection to the scheduler. It was a relatively minor amount of work to get this working (PR soon incoming), and due to coroutines, performance impacts should be minor, as each stream is just another open file handle. Unfortunately, that means that the scheduler needs to be able to implement both the Flight and FlightSQL protocols, since there will be `do_get` requests coming in for SQL operations as well as Flights. The change I would propose is to create a new method with a default implementation that gets called if the `do_get` doesn't match any existing FlightSQL request, and allow the implementer to override and handle that if they so choose. The default behavior remains the same. **Describe alternatives you've considered** Solve problems in the networking layer instead of code. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
