Davis-Zhang-Onehouse opened a new pull request, #20704: URL: https://github.com/apache/hudi/pull/20704
### Describe the issue this Pull Request addresses Part 3 of 4 of the metadata table validator v2 series. **Stacked on #20703, which is stacked on #20702; please review those first. The diff here includes both.** The commit this PR adds is the last one on the branch, "add the FS-MDT diff dimension with timeline adjudication for metadata table validator v2". #20702 explains the series and how v2 relates to `HoodieMetadataTableValidator`. v1's latest-file-slice check throws at the first difference between the metadata table (MDT) and storage, and cannot say whether the timeline explains it. A file written by a commit still in flight, a file a pending clean is about to delete, and a file nothing will ever reference all look the same to it. This PR adds v2's second dimension: it compiles every difference, then decides each one against the state of its writing instant before and after the run. ### Summary and Changelog Library code under `org.apache.hudi.utilities.util`, with unit tests. Nothing calls it until the last PR of the series. - `FileSliceMismatchAnalyzer`: phase 1. Compiles every disagreement between a partition's latest file slices according to the MDT and according to a storage listing. Each file's writing instant comes from its name, or, for a log file, from its first block header (`LogFileInstantReader`, `LogFileReader`). A file whose instant cannot be attributed is dropped and counted; it never gets a placeholder instant. For a file group the MDT does not know at all, every slice on storage is reported. - `FileDiffAdjudicator`, `Verdict`, `DiffPattern`, `FileDiffRecord`, `DiffTally`: phase 2. Decides each difference from its writing instant's state at the two timeline reads that bracket the run. A failed read or an unresolvable state ends at `NOT_SETTLED`, never at a fatal verdict (`DANGLING`, `MDT_CORRUPTION`, `OUTBOUND_DELETION`, `CODE_BUG`). - `InstantMembershipResolver`: whether a completed commit, active or archived (through #20702's batch resolver), names a file. A storage-only file under a completed commit that does not name it is `DANGLING`. This is what catches a retried log write. - `PendingServiceState`, `ServiceActivityDelta`, `PendingServiceReconciler`: whether a clean or rollback, pending before the run or active during it, explains a file the MDT no longer lists. A plan that cannot be read is `UNRESOLVED`, never "nothing explains this file". - `HoodieMetadataValidationUtils`: the file-slice comparison helpers v1 and v2 share (base-file equality, in-flight cleaning tolerance, metadata table availability). These are v1's existing helpers; the last PR switches `HoodieMetadataTableValidator` over to them and removes its private copies. - `FsMdtDiffDimension`: runs both phases on the executors against a remote view of the driver-hosted MDT file system view. It enforces that phase 1 finishes before the second timeline read. ### Impact None at runtime until the last PR wires the dimension behind `--use-validator-v2` (default off). No public API, storage format or config change. ### Risk Level none. New library code with no callers. Unit tests cover 193 cases across 13 classes. The end-to-end suites for this dimension arrive with the entry point in the last PR. ### Documentation Update none in this PR. `--validate-fs-mdt-diff` and `--validation-mode` arrive in the last PR, with their CLI descriptions. ### Contributor's checklist - [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [x] Enough context is provided in the sections above - [x] Adequate tests were added if applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
