SinghAsDev commented on a change in pull request #3813:
URL: https://github.com/apache/iceberg/pull/3813#discussion_r789195309
##########
File path: aws/src/main/java/org/apache/iceberg/aws/s3/S3OutputStream.java
##########
@@ -172,15 +187,23 @@ public void write(byte[] b, int off, int len) throws
IOException {
private void newStream() throws IOException {
if (stream != null) {
- stream.close();
+ closeStream();
}
createStagingDirectoryIfNotExists();
currentStagingFile = File.createTempFile("s3fileio-", ".tmp",
stagingDirectory);
currentStagingFile.deleteOnExit();
- stagingFiles.add(currentStagingFile);
- stream = new CountingOutputStream(new BufferedOutputStream(new
FileOutputStream(currentStagingFile)));
+ stagingFiles.add(new FileAndDigest(currentStagingFile));
+
+ if (isEtagCheckEnabled) {
+ currentPartMessageDigest.reset();
Review comment:
@danielcweeks @rdblue either we can create a new MessageDigest for each
file and use MessageDigest in `FileAndDigest` or we can use bytes. Former is
likely more readable and later is likely more memory efficient. I feel memory
efficiency here is not the biggest concern, so I am happy to go back to what I
had originally. However, let me know what you folks think.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
