SinghAsDev commented on a change in pull request #3813:
URL: https://github.com/apache/iceberg/pull/3813#discussion_r789055918
##########
File path: aws/src/main/java/org/apache/iceberg/aws/s3/S3OutputStream.java
##########
@@ -74,13 +78,16 @@
private final AwsProperties awsProperties;
private CountingOutputStream stream;
- private final List<File> stagingFiles = Lists.newArrayList();
+ private final List<FileAndDigest> stagingFiles = Lists.newArrayList();
private final File stagingDirectory;
private File currentStagingFile;
private String multipartUploadId;
private final Map<File, CompletableFuture<CompletedPart>> multiPartMap =
Maps.newHashMap();
private final int multiPartSize;
private final int multiPartThresholdSize;
+ private final boolean isEtagCheckEnabled;
+ private final MessageDigest completeMessageDigest;
+ private MessageDigest currentPartMessageDigest;
Review comment:
Thanks @danielcweeks for the review. We don't do full checksum if we are
doing multi-part upload, we only do full checksum if it is a single put
request. If I am understanding correctly wrapping the stream with
`DigestOutputStream` will make us handle counts explicitly, which will beat the
purpose of this suggestion to simplify the changes. Please correct me if I am
missing something here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
