scwhittle commented on a change in pull request #15301:
URL: https://github.com/apache/beam/pull/15301#discussion_r686656470
##########
File path:
sdks/java/core/src/main/java/org/apache/beam/sdk/io/FileBasedSink.java
##########
@@ -770,12 +770,10 @@ final void moveToOutputFiles(
}
// During a failure case, files may have been deleted in an earlier
step. Thus
// we ignore missing files here.
- FileSystems.rename(
- srcFiles,
- dstFiles,
- StandardMoveOptions.IGNORE_MISSING_FILES,
- StandardMoveOptions.SKIP_IF_DESTINATION_EXISTS);
- removeTemporaryFiles(srcFiles);
Review comment:
I modified FileSystems.rename to delete srcs that existed but were
filtered due to dest existing.
I kept the existing methods in FileBasedSink because it appears they are
designed to be called by subclasses. I changed to pass an empty set for known
files after the rename to avoid the unnecessary delete.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
