midnattsol opened a new pull request, #6837:
URL: https://github.com/apache/arrow-rs/pull/6837
# Which issue does this PR close?
Closes #5277
# Rationale for this change
The current `get` method in `ObjectStore` requires users to handle multipart
downloads and large file transfers manually, which can be complex and
error-prone. This PR proposes a new `download` method to streamline this
process by:
- Automatically managing concurrency for large file downloads.
- Supporting chunk buffering to optimize memory usage.
- Allowing configurable retry mechanisms to handle transient errors.
This change simplifies the API for users and aligns with the feature request
outlined in issue #5277.
# What changes are included in this PR?
- Added a new `download` method in the `object_store::local` module.
- Introduced the `TransferConfig` struct for user-configurable transfer
options, such as concurrency, buffer size, and retry limits.
- Implemented helper functions:
- `download_chunk_stream`: Downloads individual chunks of data.
- `write_multi_chunks`: Writes downloaded chunks to a local file.
- Added cancellation support for graceful handling of aborted downloads.
- Updated documentation to reflect the new API.
# Are there any user-facing changes?
Yes:
- Users can now utilize the `download` method to simplify file downloads
from the object store.
- Transfer behavior can be configured using the new `TransferConfig` struct.
- Documentation has been updated to include the usage and configuration
details for the `download` method.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
