alexeykudinkin opened a new pull request #4716: URL: https://github.com/apache/hudi/pull/4716
## *Tips* - *Thank you very much for contributing to Apache Hudi.* - *Please review https://hudi.apache.org/contribute/how-to-contribute before opening a pull request.* ## What is the purpose of the pull request This change is addressing issues in regards to Metadata Table observing ingesting duplicated records leading to it persisting incorrect file-sizes for the files referred to in those records. There are multiple issues that were leading to that: 1. [HUDI-3322] Incorrect Rollback Plan generation: Rollback Plan generated for MOR tables was overly expansively listing all log-files with the latest base-instant as the ones that have been affected by the rollback, leading to invalid MT records being ingested referring to those. 2. [HUDI-3343] Metadata Table including Uncommitted Log Files during Bootstrap: Since MT is bootstrapped at the end of the commit operation execution (after FS activity, but before committing to the timeline), it was actually incorrectly ingesting some files that were part of the intermediate state of the operation being committed. This change will unblock Stack of PRs based off #4556 ## Brief change log - Cleaned up Rollback request generation flow - Cleaned up superfluous `writtenLogFilesSize` payload - Fixed RollbackUtils to always leverage appropriate instant when fetching metadata - Propagate `HoodieWriteStat` w/in `ListingBasedRollbackRequest` - Fixed RollbackPlan generation to properly refeference files bearing blocks - Fixed superfluous `RollbackPlan` generation - Fixing MT to be bootstapped _before_ carrying out any FS operations, to make sure it's not ingesting intermediate step upon bootstrapping - Fix MarkerBasedRollbackStrategy to only ref the latest log-file as the one blocks have been appended to ## Verify this pull request This pull request is already covered by existing tests, such as *(please describe tests)*. This PR fixing following tests that started to fail after changes in #4556: `TestHoodieSparkMergeOnReadTableRollback#testMORTableRestore` `TestHoodieSparkMergeOnReadTableRollback#testRollbackWithDeltaAndCompactionCommit` ## Committer checklist - [ ] Has a corresponding JIRA in PR title & commit - [ ] Commit message is descriptive of the change - [ ] CI is green - [ ] Necessary doc changes done or have another open PR - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
