Hi, I'm currently working on adding the 'DoPartialGet' functionality in my flight client and server. The flight client is written in Rust, while the flight server is written in Rust and C++.
The purpose of 'DoPartialGet' is to enable a client to request a specific part of a flight endpoint without fully consuming it. As the official flight protocol does not have this feature, I am implementing it using existing methods and messages. The workflow of the client utilizing 'DoPartialGet' involes the following steps: 1. The client initiates a query with FlightSQL on the server. 2. The server responds with the resulting endpoints. 3. The client treats the endpoints as an array of record arrays. 4. The client displays the virtually concatenated records to the user page by page, without downloading the entire content of the endpoints. 5. To achieve this, the client invokes 'DoPartialGet' for the endpoint(s) containing records for a particular page. For the 5th step, the client needs to identify which endpoint(s) hold the records for the page. To address this, each endpoint should report the number of records it contains to the client in the 2nd step. Unfortunately, the current FlightEndpoint message lacks a 'record_count' field (It would be nice if added). As a workaround, I decided to use the 'app_metadata' field of FlightEndpoint message. Now, here lies the issue: neither FlightEndpoint nor the FlightInfo message in the arrow.flight.protocol.rs file of arrow-rs includes an 'app_metadata' field (It seems outdated). Consequently, I am currently facing a roadblock. Is there a plan to update the flight implementation of arrow-rs? On a related note, I'm planning to use DoExchange to implement DoPartialGet since DoExchange allows custom information, such as offset and count, to be sent to the server, while also allowing the server to send FlightData. Thank you.
