spatel11 commented on a change in pull request #16987:
URL: https://github.com/apache/beam/pull/16987#discussion_r818825768
##########
File path:
sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/s3/S3FileSystem.java
##########
@@ -548,12 +554,15 @@ CompleteMultipartUploadResponse multipartCopy(
Math.min(objectSize - 1, bytePosition +
uploadBufferSizeBytes - 1)))
Review comment:
Mmm. Yes I think you're right. This link here also seems to back up
this idea:
https://docs.aws.amazon.com/whitepapers/latest/s3-optimizing-performance-best-practices/use-byte-range-fetches.html
> If objects are PUT using a multipart upload, it’s a good practice to GET
them in the same part sizes (or at least aligned to part boundaries) for best
performance.
Since the threshold for multipartCopy is 5GB, we could use that in place of
the `uploadBufferSizeBytes` (in both aws1 and aws2 filesystems).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
