TheR1sing3un commented on code in PR #17472:
URL: https://github.com/apache/hudi/pull/17472#discussion_r2587598605
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/timeline/versioning/v2/LSMTimelineWriter.java:
##########
@@ -208,16 +209,19 @@ private void createManifestFile(HoodieLSMTimelineManifest
manifest, int currentV
int newVersion = currentVersion < 0 ? 1 : currentVersion + 1;
// create manifest file
final StoragePath manifestFilePath =
LSMTimeline.getManifestFilePath(newVersion, archivePath);
- metaClient.getStorage().createImmutableFileInPath(manifestFilePath,
Option.of(HoodieInstantWriter.convertByteArrayToWriter(content)));
+ // create the manifest file with overwrite semantics, to handle the case
like that:
+ // writer_1 creates the manifest file successfully, but fails before
updating the version file,
+ // so we need to allow writer_2 to overwrite the manifest file wth the
same version number.
+ FileIOUtils.createFileInPath(metaClient.getStorage(), manifestFilePath,
Option.of(HoodieInstantWriter.convertByteArrayToWriter(content)));
Review Comment:
> The method `createImmutableFileInPath` should keep the atomicity
I agree that.
But what I want to express is that the entire archive process should
maintain atomicity, that is, creating archived data file, manifest and
modifying version should maintain atomicity.
However, it is possible for us to fail at any step now.
For instance, if the failure occurs between creating the manifest and
modifying the version, an invalid manifest file will remain on the file system,
which will cause our retry to fail.
What I want to solve is the atomicity problem in this process
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
