alamb commented on code in PR #4971:
URL: https://github.com/apache/arrow-rs/pull/4971#discussion_r1370203481
##########
object_store/src/multipart.rs:
##########
@@ -263,3 +267,42 @@ impl<T: PutPart> std::fmt::Debug for WriteMultiPart<T> {
.finish()
}
}
+
+/// A low-level interface for interacting with multipart upload APIs
+///
+/// Most use-cases should prefer [`ObjectStore::put_multipart`] as this is
supported by more
+/// backends, including [`LocalFileSystem`], and automatically handles
uploading fixed
+/// size parts of sufficient size in parallel
+///
+/// [`ObjectStore::put_multipart`]: crate::ObjectStore::put_multipart
+/// [`LocalFileSystem`]: crate::local::LocalFileSystem
+#[async_trait]
+pub trait MultiPartStore: Send + Sync + 'static {
+ /// Creates a new multipart upload, returning the [`MultipartId`]
+ async fn create_multipart(&self, path: &Path) -> Result<MultipartId>;
+
+ /// Uploads a new part with index `part_idx`
+ ///
+ /// Most stores require that all parts apart from the final are at least 5
MiB. Additionally
+ /// some stores require all parts excluding the last to be the same size,
e.g. [R2]
+ ///
+ /// [R2]:
https://developers.cloudflare.com/r2/objects/multipart-objects/#limitations
+ async fn put_part(
+ &self,
+ path: &Path,
+ id: &MultipartId,
+ part_idx: usize,
+ data: Bytes,
+ ) -> Result<PartId>;
+
+ /// Completes a multipart upload
Review Comment:
Can we also document here what the expectations are for a successful
completion? For example, add the note about "parts is the order you want the
parts to appear in the final output"
Also can we say anything about what happens to `PartId` that are not used
(as in put_part is called and returns a `PartId` but then that PartId is not
supplied here)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
