ianmcook commented on issue #40597: URL: https://github.com/apache/arrow/issues/40597#issuecomment-2510256233
Another thought re the comment above: If the record batch offsets are not available, another way to achieve this would be as follows: When the user calls for the first record batch: 1. A thread in the client app sends a GET request for first file and begins to enqueue the response body into a FIFO queue. 2. A second thread in the client app waits until the queue contains a full record batch, then tells the first thread to interrupt the download and gives the user/application the first record batch. When the user calls for a subsequent record batch: 1. The first thread in the client app sets variable `N` to the number of bytes in the queue, then dequeues the bytes for the first record batch from the buffer, and sends a GET request for `Range: bytes=N-` and again begins to enqueue the response body into the queue. 2. The second thread again waits until the queue contains a full record batch, then tells the first thread to interrupt the download and gives the user/application the next record batch. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
