nsivabalan commented on a change in pull request #3426:
URL: https://github.com/apache/hudi/pull/3426#discussion_r703014796
##########
File path:
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataUtil.java
##########
@@ -223,11 +174,17 @@
return convertFilesToRecords(partitionToDeletedFiles,
partitionToAppendedFiles, instantTime, "Restore");
}
- public static List<HoodieRecord>
convertMetadataToRecords(HoodieRollbackMetadata rollbackMetadata, String
instantTime, Option<String> lastSyncTs) {
+ public static List<HoodieRecord>
convertMetadataToRecords(HoodieRollbackMetadata rollbackMetadata, String
instantTime,
+ Option<String> lastSyncTs, boolean wasSynced) {
Map<String, Map<String, Long>> partitionToAppendedFiles = new HashMap<>();
Map<String, List<String>> partitionToDeletedFiles = new HashMap<>();
processRollbackMetadata(rollbackMetadata, partitionToDeletedFiles,
partitionToAppendedFiles, lastSyncTs);
+ if (!wasSynced) {
Review comment:
nts on processRollbackMetadata(...).
with multi-writer, lastsyncTs could be mis-leading. So, just calling it out
for understanding purpose.
lets say,
dataset timeline:
C1.complete, C2.complete, C3.inflight, C4.complete
metadata timeline:
C1.complete, C2.complete, C3.inflight, C4.complete.
rollback is triggered for C3.
Case1: C3 is committed to metadata.
Case2: C3 is not committed to metadata yet.
Lets see what happens when we processRollbackMetadata() R5 for rollback of
C3.
As per code, 2 cases will be ignored while trying to process rollback
metadata.
Case A: if instant to rollback > lastSyncTs. // metadata table is not yet
caught up. skip processing.
Case B: If instant to rollback was never committed to metadata and is part
of active timeline. refers to failed commit. skip processing.
For any other case, we will process the rollback metadata and add/delete
appropriate files.
Case1: C3 is committed to metadata.
As per logic above,
Case A is false.
Case B true. since C3 is part of active timeline, we can't skip processing.
And so we will process the rollback since C3 was committed to metadata
table.
Case2: C3 not committed to metadata.
As per logic above,
Case A is false.
Case B false. since C3 is committed to metadata, we can skip processing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
