lidavidm commented on code in PR #36946:
URL: https://github.com/apache/arrow/pull/36946#discussion_r1279217548
##########
format/Flight.proto:
##########
@@ -65,6 +65,33 @@ service FlightService {
*/
rpc GetFlightInfo(FlightDescriptor) returns (FlightInfo) {}
+ /*
+ * For a given FlightDescriptor, start a query and get information
+ * to poll the updated statuses. This is a useful interface if the
+ * query may be a long-running query. The first PollFlightInfo call
+ * should be returned as quickly as possible. (GetFlightInfo isn't
+ * returned until a query isn't completed.)
+ *
+ * A client can consume the currently available results so far
+ * before the query isn't completed yet. See RertyInfo.info for
+ * details.
+ *
+ * A client can poll the updated query status by calling
+ * PollFlightInfo() with RetryInfo.flight_descriptor. A server
+ * should not respond until the result would be different from last
+ * time. That way, the client can "long poll" for updates
+ * without constantly making requests. Clients can set a short timeout
+ * to avoid blocking calls if desired.
+ *
+ * A client can't use RetryInfo.flight_descriptor after
+ * RetryInfo.timestamp passes. A server might not accept the retry
+ * descriptor anymore and the query may be cancelled.
+ *
+ * A client may be able to use the CancelFLightInfo action with
+ * RetryInfo.info to cancel the running query.
Review Comment:
```suggestion
* A client may use the CancelFlightInfo action with
* RetryInfo.info to cancel the running query.
```
##########
format/Flight.proto:
##########
@@ -65,6 +65,33 @@ service FlightService {
*/
rpc GetFlightInfo(FlightDescriptor) returns (FlightInfo) {}
+ /*
+ * For a given FlightDescriptor, start a query and get information
+ * to poll the updated statuses. This is a useful interface if the
+ * query may be a long-running query. The first PollFlightInfo call
+ * should be returned as quickly as possible. (GetFlightInfo isn't
+ * returned until a query isn't completed.)
Review Comment:
```suggestion
* should return as quickly as possible. (GetFlightInfo doesn't
* return until the query is complete.)
```
##########
cpp/src/arrow/flight/middleware.h:
##########
@@ -57,6 +57,7 @@ enum class FlightMethod : char {
DoAction = 7,
ListActions = 8,
DoExchange = 9,
+ PollFlightInfo = 10,
Review Comment:
We'll have to update this in Python (I can put up a PR for that later)
##########
cpp/src/arrow/flight/types.h:
##########
@@ -650,6 +650,71 @@ class ARROW_FLIGHT_EXPORT FlightInfo {
mutable bool reconstructed_schema_;
};
+/// \brief The information to process a long-running query.
+class ARROW_FLIGHT_EXPORT RetryInfo {
+ public:
+ /// The currently available results so far.
+ std::unique_ptr<FlightInfo> info = NULLPTR;
+ /// The descriptor the client should use on the next try. If unset,
+ /// the query is complete.
+ std::optional<FlightDescriptor> descriptor = std::nullopt;
+ /// Query progress. Must be in [0.0, 1.0] but need not be
+ /// monotonic or nondecreasing. If unknown, do not set.
+ std::optional<double> progress = std::nullopt;
+ /// Expiration time for this request. After this passes, the server
+ /// might not accept the retry descriptor anymore (and the query may
+ /// be cancelled). This may be updated on a call to PollFlightInfo.
+ std::optional<Timestamp> expiration_time = std::nullopt;
+
+ RetryInfo()
+ : info(NULLPTR),
+ descriptor(std::nullopt),
+ progress(std::nullopt),
+ expiration_time(std::nullopt) {}
+
+ explicit RetryInfo(std::unique_ptr<FlightInfo> info,
+ std::optional<FlightDescriptor> descriptor,
+ std::optional<double> progress,
+ std::optional<Timestamp> expiration_time)
+ : info(std::move(info)),
+ descriptor(descriptor),
Review Comment:
```suggestion
descriptor(std::move(descriptor)),
```
##########
format/Flight.proto:
##########
@@ -65,6 +65,33 @@ service FlightService {
*/
rpc GetFlightInfo(FlightDescriptor) returns (FlightInfo) {}
+ /*
+ * For a given FlightDescriptor, start a query and get information
+ * to poll the updated statuses. This is a useful interface if the
+ * query may be a long-running query. The first PollFlightInfo call
+ * should be returned as quickly as possible. (GetFlightInfo isn't
+ * returned until a query isn't completed.)
+ *
+ * A client can consume the currently available results so far
+ * before the query isn't completed yet. See RertyInfo.info for
+ * details.
+ *
+ * A client can poll the updated query status by calling
+ * PollFlightInfo() with RetryInfo.flight_descriptor. A server
+ * should not respond until the result would be different from last
+ * time. That way, the client can "long poll" for updates
+ * without constantly making requests. Clients can set a short timeout
+ * to avoid blocking calls if desired.
+ *
+ * A client can't use RetryInfo.flight_descriptor after
+ * RetryInfo.timestamp passes. A server might not accept the retry
+ * descriptor anymore and the query may be cancelled.
Review Comment:
```suggestion
* A client can't use RetryInfo.flight_descriptor after
* RetryInfo.expiration_time passes. A server might not accept the
* retry descriptor anymore and the query may be cancelled.
```
##########
format/Flight.proto:
##########
@@ -65,6 +65,33 @@ service FlightService {
*/
rpc GetFlightInfo(FlightDescriptor) returns (FlightInfo) {}
+ /*
+ * For a given FlightDescriptor, start a query and get information
+ * to poll the updated statuses. This is a useful interface if the
+ * query may be a long-running query. The first PollFlightInfo call
+ * should be returned as quickly as possible. (GetFlightInfo isn't
+ * returned until a query isn't completed.)
+ *
+ * A client can consume the currently available results so far
+ * before the query isn't completed yet. See RertyInfo.info for
+ * details.
Review Comment:
```suggestion
* A client can consume the current results so far before
* the query is completed. See RetryInfo.info for details.
```
##########
docs/source/format/Flight.rst:
##########
@@ -153,6 +153,70 @@ A client that wishes to download the data would:
.. _google.protobuf.Timestamp:
https://protobuf.dev/reference/protobuf/google.protobuf/#timestamp
+Downloading Data by Running a Heavy Query
+-----------------------------------------
+
+A client may need to request a heavy query to download
+data. ``GetFlightInfo`` isn't returned until the requested query is
+completed. It means that the client is blocked. The client can use
+``PollFlightInfo`` instead of ``GetFlightInfo``:
+
+.. figure:: ./Flight/PollFlightInfo.mmd.svg
+
+ Polling a long-running query by ``PollFlightInfo``.
+
+#. Construct or acquire a ``FlightDescriptor``, as before.
+#. Call ``PollFlightInfo(FlightDescriptor)`` to get a ``RetryInfo``
+ message.
+
+ A server should respond as quickly as possible on the first
+ call. So the client shouldn't wait for the first ``RetryInfo``
+ response.
+
+ If the query isn't finished, ``RetryInfo.flight_descriptor`` has a
+ ``FlightDescriptor``. The client should use the descriptor (not the
+ original ``FlightDescriptor``) to call the next
+ ``PollFlightInfo()``. A server should recognize a
+ ``RetryInfo.flight_descriptor`` that is not necessarily the latest
+ in case the client misses an update in between.
+
+ If the query is finished, ``RetryInfo.flight_descriptor`` is
+ unset.
+
+ ``RetryInfo.info`` is the currently available results so far. It's
+ a complete ``FlightInfo`` each time not just the delta between the
+ previous and current ``FlightInfo``. A server should only append to
+ the endpoints in ``RetryInfo.info`` each time. So the client can
+ run ``DoGet(Ticket)`` with the ``Ticket`` in the ``RetryInfo.info``
+ even when the query isn't finished yet. ``FlightInfo.ordered`` is
+ also valid.
+
+ A server should not respond until the result would be different
+ from last time. That way, the client can "long poll" for updates
+ without constantly making requests. Clients can set a short timeout
+ to avoid blocking calls if desired.
+
+ ``RetryInfo.progress`` may be set. It represents progress of the
+ query. If it's set, the value must be in ``[0.0, 1.0]``. The value
+ must not be monotonic or nondecreasing. A server may respond by
+ only updating the ``RetryInfo.progress`` value though it shouldn't
+ spam the client with updates.
Review Comment:
```suggestion
``RetryInfo.progress`` may be set. It represents progress of the
query. If it's set, the value must be in ``[0.0, 1.0]``. The value
is not necessarily monotonic or nondecreasing. A server may respond by
only updating the ``RetryInfo.progress`` value, though it shouldn't
spam the client with updates.
```
##########
docs/source/format/Flight.rst:
##########
@@ -153,6 +153,70 @@ A client that wishes to download the data would:
.. _google.protobuf.Timestamp:
https://protobuf.dev/reference/protobuf/google.protobuf/#timestamp
+Downloading Data by Running a Heavy Query
+-----------------------------------------
+
+A client may need to request a heavy query to download
+data. ``GetFlightInfo`` isn't returned until the requested query is
+completed. It means that the client is blocked. The client can use
+``PollFlightInfo`` instead of ``GetFlightInfo``:
Review Comment:
```suggestion
A client may need to request a heavy query to download
data. However, ``GetFlightInfo`` doesn't return until the query
completes, so the client is blocked. In this situation, the client
can use ``PollFlightInfo`` instead of ``GetFlightInfo``:
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
