+1 On Thu, Aug 10, 2023 at 10:11 PM Sutou Kouhei <k...@clear-code.com> wrote:
> +1 > > In <20230811.105721.969618469307878987....@clear-code.com> > "[VOTE][Format][Flight] Long-running queries support" on Fri, 11 Aug > 2023 10:57:21 +0900 (JST), > Sutou Kouhei <k...@clear-code.com> wrote: > > > Hi, > > > > I would like to propose long-running queries support for > > Flight RPC. > > > > See the following pull request and discussion for details: > > > > * GH-36155: [C++][Go][Java][FlightRPC] Add support for long-running > queries > > https://github.com/apache/arrow/pull/36946 > > > > * [DISCUSS][Format][Flight] Long-running queries support > > https://lists.apache.org/thread/qcjpcw6m3p15wqxp6n6rqzlx01v1fl3v > > > > This is based on one of the following proposals: > > > > [DISCUSS] Flight RPC/Flight SQL/ADBC enhancements > > https://lists.apache.org/thread/247z3t06mf132nocngc1jkp3oqglz7jp > > > > Google Docs: (Arrow ML) Arrow Flight RPC/Flight SQL Proposals > > > https://docs.google.com/document/d/1jhPyPZSOo2iy0LqIJVUs9KWPyFULVFJXTILDfkadx2g/edit#heading=h.anpr1q5slm1v > > > > Summary: > > > > * Background: Queries generally don't complete instantly (as > > much as we would like them to). So where can we put the > > 'query evaluation time'? > > > > * In GetFlightInfo: block and wait for the query to complete. > > * Con: this is a long-running blocking call, which may > > fail or time out. Then when the client retries, the > > server has to redo all the work. > > * Con: parts of the result may be ready before others, but > > the client can't do anything until everything is ready. > > > > * In DoGet: return a fixed number of partitions > > * Con: this makes handling worker failures hard. Systems > > like Trino support fault-tolerant execution by replacing > > workers at runtime. But GetFlightInfo has already > > passed, so we can't notify the client of new workers. > > * Con: we have to know or fix the partitioning up front. > > > > Neither solution is optimal. > > > > * Proposal: Add PollFlightInfo as a pollable version of > > GetFlightInfo. Clients can poll the current query status > > and start reading the currently available results so far > > before the query is completed. > > > > * Changes: > > > > * Add PollFlightInfo and PollInfo > > > > Flight.proto: > > > https://github.com/apache/arrow/pull/36946/files#diff-53b6c132dcc789483c879f667a1c675792b77aae9a056b257d6b20287bb09dba > > Documentation: > > > http://crossbow.voltrondata.com/pr_docs/36946/format/Flight.html#downloading-data-by-running-a-heavy-query > > > > * The pull request includes reference implementations for > > C++, Go and Java. > > > > > > The vote will be open for at least 72 hours. > > > > [ ] +1 Accept this proposal > > [ ] +0 > > [ ] -1 Do not accept this proposal because... > > > > > > Thanks, > > -- > > kou >
