jonvex commented on code in PR #8043:
URL: https://github.com/apache/hudi/pull/8043#discussion_r1118885381
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/deltastreamer/DeltaSync.java:
##########
@@ -698,8 +717,13 @@ private Pair<Option<String>, JavaRDD<WriteStatus>>
writeToSink(JavaRDD<HoodieRec
boolean hasErrors = totalErrorRecords > 0;
if (!hasErrors || cfg.commitOnErrors) {
HashMap<String, String> checkpointCommitMetadata = new HashMap<>();
+ Option<BiConsumer<HoodieTableMetaClient, HoodieCommitMetadata>>
extraPreCommitFunc = Option.empty();
if (checkpointStr != null) {
- checkpointCommitMetadata.put(CHECKPOINT_KEY, checkpointStr);
+ if (identifier.isPresent()) {
+ extraPreCommitFunc = Option.of(new
HoodieDeltaStreamerMultiwriterCheckpoint(this, checkpointStr,
latestCheckpointWritten));
Review Comment:
That does happen inside. latestCheckpointWritten is used to detect if
multiple deltastreamers are using the same id. We keep track of the last
checkpoint that we wrote so that if it doesn't match the latest commit, then we
know that another deltastreamer used the same id
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
