Dandandan commented on issue #26682: URL: https://github.com/apache/datafusion/issues/26682#issuecomment-6575456791
> object storage the first rows of a large row group arrive only after the whole row group has been downloaded This is correct currently, but it could be improved. It is a limitation of `[arrow-rs-object-store](https://github.com/apache/arrow-rs-object-store)` that most API methods only return after downloading the full body, but the object storage backend will just stream the `GET` response. Streaming decode while downloading would be possible with some changes/additions in API, which would reduce the "latency-to-first-batch" on large row groups and could reduce memory requirements / increase throughput as well. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
