prashantwason commented on a change in pull request #3210:
URL: https://github.com/apache/hudi/pull/3210#discussion_r687072435
##########
File path:
hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/client/functional/TestHoodieBackedMetadata.java
##########
@@ -480,6 +482,131 @@ public void testRollbackUnsyncedCommit(HoodieTableType
tableType) throws Excepti
client.syncTableMetadata();
validateMetadata(client);
}
+
+ // If an unsynced commit is automatically rolled back during next commit,
the rollback commit gets a timestamp
+ // greater than than the new commit which is started. Ensure that in this
case the rollback is not processed
+ // as the earlier failed commit would not have been committed.
+ //
+ // Dataset: C1 C2 C3.inflight[failed] C4 R5[rolls
back C3]
+ // Metadata: C1.delta C2.delta
+ //
+ // When R5 completes, C3.xxx will be deleted. When C4 completes, C4 and R5
will be committed to Metadata Table in
Review comment:
Having a config is acceptable as there can be various use cases for HUDI.
We saw a production issue where the metadata table did not have the latest
files. If there was no exception:
1. Only an error would have been logged
2. Metadata table would have returned an older file listing
3. Data would have got written to older version of files
The above is a classic data lass scenario which would be very difficult to
catch once the logs have rolled over. Hence, I want to err on the side of
data-consistency.
To contrast it to another component in HUDI wherein we do something similar
- If the RemoteFileSystemView is unable to get the listing from TimelineServer,
it logs an error and then falls back to file listing. There are no data
consistency issues in this fallback so this makes sense to prevent user
interventions. But wrong results from metadata table can lead to data
consistency issues.
BTW, there may be a better way to sync the rollback. Also much of this
complexity will go away with the synchronous design.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
