Davis-Zhang-Onehouse opened a new pull request, #20702:
URL: https://github.com/apache/hudi/pull/20702

   ### Describe the issue this Pull Request addresses
   
   `HoodieMetadataTableValidator` compares the metadata table (MDT) with a 
storage listing, but on large tables its latest-file-slice check does not scale 
and cannot tell a real inconsistency from a transient one. It builds both file 
system views per partition on the executors, resolves the archived timeline 
once per file group, and reports any difference without saying whether the 
timeline explains it. This series adds **`HoodieMetadataTableValidatorV2`**, a 
separate, opt-in validator for the latest file slices. It is reached from the 
existing tool with `--use-validator-v2` (off by default), so nothing changes 
for current callers.
   
   How v2 relates to the existing validator:
   
   - v1 stays as it is and remains the default. Every v1 validation (column 
stats, bloom filters, record index, partitions, and so on) stays v1-only.
   - v2 covers latest file slice validation only, as two independent dimensions:
     - **duplicate base files**: a file slice holding more than one base file;
     - **FS-MDT diff**: files that storage and the MDT disagree on.
   - v2 adjudicates every difference against the timeline (active and archived, 
plus pending clean, rollback and compaction state). A run fails only on 
positive proof. A difference it cannot settle is reported as not settled, and 
never as clean.
   - v2 hosts the MDT file system view once on the driver, behind an embedded 
timeline server, so executors no longer rebuild it per partition.
   
   The series is stacked; each PR compiles and passes its own tests at its 
position:
   
   1. **This PR**: the archived timeline batch resolver and the shared 
primitives (library code only, nothing calls it yet).
   2. The duplicate base file dimension.
   3. The FS-MDT diff dimension with timeline adjudication.
   4. The `HoodieMetadataTableValidatorV2` entry point, run reports, the 
`--use-validator-v2` flag on `HoodieMetadataTableValidator`, two small 
resource-leak fixes it depends on 
(`SpillableMapBasedFileSystemView.closeResources`, `EmbeddedTimelineService` 
shutdown), and the end-to-end suites.
   
   PRs 2–4 are opened as drafts stacked on this one. Please review this one 
first.
   
   ### Summary and Changelog
   
   Adds library code under `org.apache.hudi.utilities.util` (and 
`util.archive`), with unit tests. No existing class is modified.
   
   - `util.archive.*`: resolves a **batch** of instants against the archived 
timeline in one scan per instant range, for both the V1 (log-format commits 
archive) and V2 (LSM timeline) layouts. The validator previously read the 
archive once per file group. `ArchiveInstantSearcher` locates the archive files 
that can hold a range, `HoodieArchiveFileScanner` reads them and reports 
damaged or partially merged files as such, and `ArchivedInstantWorkingSet` 
holds what was found.
   - `InstantState`, `InstantStateIndex`, `InstantBuckets`, 
`InstantRangePartitioner`, `InstantRangeBalancer`: a serializable index of 
every active instant's state, read once on the driver so executors cannot 
disagree about which instants completed, and a total range partitioner (a 
binary search over sorted boundaries) that gives each Spark task a contiguous 
instant range without splitting an instant.
   - `CommitWriteStats`, `CommitPathProjector`, `ArchivedCommitPaths`, 
`PartitionPaths`: what a commit recorded for a file group. "The commit named 
nothing", "named several" and "could not be read" stay three different answers.
   - `MismatchRecord`, `MismatchPattern`, `MismatchSide`: the finding model 
shared by both dimensions.
   - `Inconclusive`, `InconclusiveCause`, `ValidationFailures`, 
`ValidationMode`: the outcome vocabulary. A run that could not decide says why, 
with a stable marker per cause, and is never reported as a pass.
   - `Branch`, `BranchBeacon`: an enumerated decision-branch surface, logged at 
debug and used by tests to assert which branch decided a verdict.
   - `ArchivalProbe`, `RecomputeProbe`, `ExecutorMetaClientFactory`, 
`CloseableExecutor`, `JavaSerializedPayload`: helpers for detecting that the 
archiver overlapped a run, rebuilding meta clients on executors, and 
broadcasting driver state independently of the configured Spark serializer.
   
   No code was copied from outside the project.
   
   ### Impact
   
   None at runtime: no existing code path reaches the new classes until PR 4 
wires them behind `--use-validator-v2`. There are no public API, storage format 
or config changes.
   
   ### Risk Level
   
   none. This is new library code with no callers in this PR. Its unit tests 
(155 cases across 20 classes) run in `hudi-utilities`.
   
   ### Documentation Update
   
   none in this PR. The `--use-validator-v2` flag and the v2-specific options 
arrive in PR 4, with their CLI descriptions.
   
   ### Contributor's checklist
   
   - [x] Read through [contributor's 
guide](https://hudi.apache.org/contribute/how-to-contribute)
   - [x] Enough context is provided in the sections above
   - [x] Adequate tests were added if applicable
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

  • [PR] feat(utilities): add th... Davis-Zhang-Onehouse (via GitHub) via commits

Reply via email to