kou opened a new pull request, #36946:
URL: https://github.com/apache/arrow/pull/36946
### Rationale for this change
In Flight RPC, FlightInfo includes addresses of workers alongside result
partition info. This lets clients fetch data directly from workers, in parallel
or even distributed across multiple machines. But this also comes with
tradeoffs.
Queries generally don't complete instantly (as much as we would like them
to). So where can we put the 'query evaluation time'?
* In `GetFlightInfo`: block and wait for the query to complete.
* Con: this is a long-running blocking call, which may fail or time out.
Then when the client retries, the server has to redo all the work.
* Con: parts of the result may be ready before others, but the client
can't do anything until everything is ready.
* In `DoGet`: return a fixed number of partitions
* Con: this makes handling worker failures hard. Systems like Trino
support fault-tolerant execution by replacing workers at runtime. But
GetFlightInfo has already passed, so we can't notify the client of new workers.
* Con: we have to know or fix the partitioning up front.
Neither solution is optimal.
### What changes are included in this PR?
We can address this by adding a retryable version of `GetFlightInfo`:
`PollFlightInfo(FlightDescriptor)`
`PollFlightInfo` returns `RetryInfo`:
```proto
message RetryInfo {
// The currently available results so far.
FlightInfo info = 1;
// The descriptor the client should use on the next try.
// If unset, the query is complete.
FlightDescriptor flight_descriptor = 2;
// Query progress. Must be in [0.0, 1.0] but need not be
// monotonic or nondecreasing. If unknown, do not set.
optional double progress = 3;
// Expiration time for this request. After this passes, the server
// might not accept the retry descriptor anymore (and the query may
// be cancelled). This may be updated on a call to PollFlightInfo.
google.protobuf.Timestamp expiration_time = 4;
}
```
See the documentation changes for details of them.
### Are these changes tested?
Yes.
This has C++ and Go implementations and an integration test with them.
### Are there any user-facing changes?
Yes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
