phillipleblanc commented on issue #2298: URL: https://github.com/apache/datafusion-ballista/issues/2298#issuecomment-5310343513
Unfortunately the FlightSQL protocol as is doesn't match well with the async query nature of Ballista. This was something we were thinking about how to enable in Spice, and we eventually scoped FlightSQL out from being able to start Ballista jobs - it didn't match our HA requirements and the timeout issues that you mentioned. I do think its useful to have a FlightSQL frontend for Ballista to make it easier for testing though. > Catalog / planning model. SQL text has to be planned against something. > embedder-supplied catalog via SessionBuilder Spice supplies the catalog via SessionBuilder, so that matches what we already do. I'm not sure who the users of a standalone Ballista cluster are and what their requirements would be. If its just for testing, then maybe a simple bootstrap set of DDL commands by the scheduler would suffice? > Long-running queries. Blocking GetFlightInfo until the job completes (old behaviour) will hit client and proxy timeouts for TPC-H-scale queries. I don't think that FlightSQL as designed works for this. We may want to think about either implementing our own Ballista extension to FlightSQL that has first class support for submitting a job via SQL, being able to poll for its status, retrieve results, etc. Could potentially do this without any protobuf changes if we used custom Flight Actions and/or `PollFlightInfo` - but I think its worth at least prototyping what a custom FlightSQL extension looks like that would allow clients the ability to manage these long-running jobs. If its useful enough, then we could even think about trying to update the FlightSQL spec itself to have this. But I think that would be a separate feature, getting regular FlightSQL working again is still a good goal. > Metadata surface. Agree that we should implement enough for clients to be usable. You are free to reference the spiceai implementation for FlightSQL here which does implement enough for real clients: https://github.com/spiceai/spiceai/blob/trunk/crates/runtime/src/flight/mod.rs > Prepared statements. Agree we should have a TTL/eviction policy for this. spiceai has a session store that uses `moka` for this: https://github.com/spiceai/spiceai/blob/trunk/crates/runtime/src/flight/session.rs#L80 > Auth Flight already has an authentication method that Ballista can use, anything additional can be implemented using tonic middleware so I don't think Ballista needs anything special for this. > Lifecycle and cleanup Yes, and if we go down the custom protobuf extension route then we could potentially tie into that as well. > Write path. I think deferring for now makes sense. > Config naming My understanding is that this is for the executors? Reusing it for the scheduler frontend could be confusing. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
