crepererum commented on code in PR #4971:
URL: https://github.com/apache/arrow-rs/pull/4971#discussion_r1369995595
##########
object_store/src/multipart.rs:
##########
@@ -263,3 +267,42 @@ impl<T: PutPart> std::fmt::Debug for WriteMultiPart<T> {
.finish()
}
}
+
+/// A low-level interface for interacting with multipart upload APIs
+///
+/// Most use-cases should prefer [`ObjectStore::put_multipart`] as this is
supported by more
+/// backends, including [`LocalFileSystem`], and automatically handles
uploading fixed
+/// size parts of sufficient size in parallel
+///
+/// [`ObjectStore::put_multipart`]: crate::ObjectStore::put_multipart
+/// [`LocalFileSystem`]: crate::local::LocalFileSystem
+#[async_trait]
+pub trait MultiPartStore: Send + Sync + 'static {
Review Comment:
Depends. I think on Linux you can make this work w/ `fallocate` which let's
you insert data into the middle of a file. Since IO to a single file is likely
serialized anyways (due to the fseek logic), you could totally insert one part
after another (given you know the order, see my question above).
If FS doesn't implement this, I think having a half-baked interface on the
default trait is useless: eventually many user will likely converge to the
well-defined trait. And that nice new version cannot be tested w/ local FS (if
we don't implement it), so there's little value in leaving the half-baked
version on the default trait.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
