danny0405 commented on code in PR #17472:
URL: https://github.com/apache/hudi/pull/17472#discussion_r2587545610
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/timeline/versioning/v2/LSMTimelineWriter.java:
##########
@@ -208,16 +209,19 @@ private void createManifestFile(HoodieLSMTimelineManifest
manifest, int currentV
int newVersion = currentVersion < 0 ? 1 : currentVersion + 1;
// create manifest file
final StoragePath manifestFilePath =
LSMTimeline.getManifestFilePath(newVersion, archivePath);
- metaClient.getStorage().createImmutableFileInPath(manifestFilePath,
Option.of(HoodieInstantWriter.convertByteArrayToWriter(content)));
+ // create the manifest file with overwrite semantics, to handle the case
like that:
+ // writer_1 creates the manifest file successfully, but fails before
updating the version file,
+ // so we need to allow writer_2 to overwrite the manifest file wth the
same version number.
+ FileIOUtils.createFileInPath(metaClient.getStorage(), manifestFilePath,
Option.of(HoodieInstantWriter.convertByteArrayToWriter(content)));
Review Comment:
The method `createImmutableFileInPath` should keep the atomicity, that means
the file is either created with full info or does not exist. Especially, on
HDFS, we use rename to achieve this, while on OSS, we just use create, because
most of the OSS support file creation with atomicity.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
