Davis-Zhang-Onehouse opened a new pull request, #20704:
URL: https://github.com/apache/hudi/pull/20704

   ### Describe the issue this Pull Request addresses
   
   Part 3 of 4 of the metadata table validator v2 series. **Stacked on #20703, 
which is stacked on #20702; please review those first. The diff here includes 
both.** The commit this PR adds is the last one on the branch, "add the FS-MDT 
diff dimension with timeline adjudication for metadata table validator v2".
   
   #20702 explains the series and how v2 relates to 
`HoodieMetadataTableValidator`.
   
   v1's latest-file-slice check throws at the first difference between the 
metadata table (MDT) and storage, and cannot say whether the timeline explains 
it. A file written by a commit still in flight, a file a pending clean is about 
to delete, and a file nothing will ever reference all look the same to it. This 
PR adds v2's second dimension: it compiles every difference, then decides each 
one against the state of its writing instant before and after the run.
   
   ### Summary and Changelog
   
   Library code under `org.apache.hudi.utilities.util`, with unit tests. 
Nothing calls it until the last PR of the series.
   
   - `FileSliceMismatchAnalyzer`: phase 1. Compiles every disagreement between 
a partition's latest file slices according to the MDT and according to a 
storage listing. Each file's writing instant comes from its name, or, for a log 
file, from its first block header (`LogFileInstantReader`, `LogFileReader`). A 
file whose instant cannot be attributed is dropped and counted; it never gets a 
placeholder instant. For a file group the MDT does not know at all, every slice 
on storage is reported.
   - `FileDiffAdjudicator`, `Verdict`, `DiffPattern`, `FileDiffRecord`, 
`DiffTally`: phase 2. Decides each difference from its writing instant's state 
at the two timeline reads that bracket the run. A failed read or an 
unresolvable state ends at `NOT_SETTLED`, never at a fatal verdict (`DANGLING`, 
`MDT_CORRUPTION`, `OUTBOUND_DELETION`, `CODE_BUG`).
   - `InstantMembershipResolver`: whether a completed commit, active or 
archived (through #20702's batch resolver), names a file. A storage-only file 
under a completed commit that does not name it is `DANGLING`. This is what 
catches a retried log write.
   - `PendingServiceState`, `ServiceActivityDelta`, `PendingServiceReconciler`: 
whether a clean or rollback, pending before the run or active during it, 
explains a file the MDT no longer lists. A plan that cannot be read is 
`UNRESOLVED`, never "nothing explains this file".
   - `HoodieMetadataValidationUtils`: the file-slice comparison helpers v1 and 
v2 share (base-file equality, in-flight cleaning tolerance, metadata table 
availability). These are v1's existing helpers; the last PR switches 
`HoodieMetadataTableValidator` over to them and removes its private copies.
   - `FsMdtDiffDimension`: runs both phases on the executors against a remote 
view of the driver-hosted MDT file system view. It enforces that phase 1 
finishes before the second timeline read.
   
   ### Impact
   
   None at runtime until the last PR wires the dimension behind 
`--use-validator-v2` (default off). No public API, storage format or config 
change.
   
   ### Risk Level
   
   none. New library code with no callers. Unit tests cover 193 cases across 13 
classes. The end-to-end suites for this dimension arrive with the entry point 
in the last PR.
   
   ### Documentation Update
   
   none in this PR. `--validate-fs-mdt-diff` and `--validation-mode` arrive in 
the last PR, with their CLI descriptions.
   
   ### Contributor's checklist
   
   - [x] Read through [contributor's 
guide](https://hudi.apache.org/contribute/how-to-contribute)
   - [x] Enough context is provided in the sections above
   - [x] Adequate tests were added if applicable
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

  • [PR] feat(utilities): add th... Davis-Zhang-Onehouse (via GitHub) via commits

Reply via email to