crepererum commented on issue #274: URL: https://github.com/apache/arrow-rs-object-store/issues/274#issuecomment-2758392289
> Also FWIW one could just fetch the first chunk instead of doing a separate head request, so long as some of the range requested is satisfiable it should be ok. Can you request a range that is potentially larger than the file? > Edit: actually going back to the original description the request is for files on the order of 100s of MB, an approach that returns in order runs the risk of pulling the entire file into memory... This feels problematic... I think it depends on your concrete deployment / integration. Enabling chunking also increases cost, so I wouldn't do that by default anyways. Also you can chunk even for very large files if you use a [buffered-stream](https://docs.rs/futures/latest/futures/stream/trait.StreamExt.html#method.buffered) like approach. That enables concurrency while -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
