Xuanwo commented on code in PR #2143:
URL:
https://github.com/apache/incubator-opendal/pull/2143#discussion_r1186714261
##########
core/src/services/gcs/backend.rs:
##########
@@ -236,6 +239,23 @@ impl GcsBuilder {
};
self
}
+
+ /// The buffer size should be a multiple of 256 KiB (256 x 1024 bytes),
unless it's the last chunk that completes the upload.
+ /// Larger chunk sizes typically make uploads faster, but note that
there's a tradeoff between speed and memory usage.
+ /// It's recommended that you use at least 8 MiB for the chunk size.
+ /// Reference: [Perform resumable
uploads](https://cloud.google.com/storage/docs/performing-resumable-uploads)
+ pub fn write_fixed_size(&mut self, fixed_buffer_size: usize) ->
Result<&mut Self> {
+ if fixed_buffer_size.checked_rem_euclid(256 * 1024).is_none() {
+ self.write_fixed_size = Some(fixed_buffer_size);
+ } else {
+ return Err(
+ Error::new(ErrorKind::ConfigInvalid, "The buffer size is
misconfigured")
Review Comment:
> Will there be somehow `write_max_size` or something else in each backend?
Yes, s3 similiar services will have `write_max_size`. And we will add
`write_buffer_size` (the name is not decided) for gcs or s3 services so that
they can buffer the data first before they start using resumable upload.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
