OliLay commented on code in PR #41564:
URL: https://github.com/apache/arrow/pull/41564#discussion_r1599998938
##########
cpp/src/arrow/filesystem/s3fs.cc:
##########
@@ -1545,6 +1559,13 @@ class ObjectInputFile final : public
io::RandomAccessFile {
// (for rational, see: https://github.com/apache/arrow/issues/34363)
static constexpr int64_t kPartUploadSize = 10 * 1024 * 1024;
+// Above this threshold, use a multi-part upload instead of a single request
upload.
+static constexpr int64_t kMultiPartUploadThresholdSize = 5 * 1024 * 1024;
Review Comment:
Would also be a valid approach and would simplify the code a bit. AWS S3
docs themselves state that you could/should use `PutObject` request for data up
to 100MB (see
[here](https://docs.aws.amazon.com/AmazonS3/latest/userguide/mpuoverview.html)),
so orienting at the part size which is 10MB would also be fine. What do other
think?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
