alamb commented on PR #6837: URL: https://github.com/apache/arrow-rs/pull/6837#issuecomment-3487838013
> Hello, I have found object_store being quite slow downloading large files in Azure. I believe this is caused by the lack of parallelism in the download. I believe this feature would improve things significantly. Are you planning to merge this (or a version of this PR ) at some point? TLDR is probably not There is a larger discussion about what the appropriate building block for a parallel downloader would be: - https://github.com/apache/arrow-rs-object-store/issues/14 - https://github.com/apache/arrow-rs-object-store/issues/274 It is possible to implement parallel downloading using an ObjectStore wrapper (wrap another implementation of ObjectStore and then implement the parallel download). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
