yihua opened a new pull request, #19626: URL: https://github.com/apache/hudi/pull/19626
### Describe the issue this Pull Request addresses Closes #19625 ### Summary and Changelog The metadata table validator consulted the archived timeline to classify file slices found only on storage, but loaded the whole archive and then filtered the range in memory. That could OOM an executor while the error message was being built, which discarded the mismatch verdict the validator had already reached, and the half-open `findInstantsInRange` always misclassified an archived instant equal to the range lower bound as a missing commit. Changes: - Add `TimelineFactory.createArchivedTimeline(metaClient, startTs, endTs)` and a matching `HoodieTableMetaClient.getArchivedTimeline(startTs, endTs)`, returning the completed instants in the closed range. The V1 factory delegates to the existing `ArchivedTimelineV1(metaClient, startTs, endTs)`; a new `ArchivedTimelineV2` constructor uses `ClosedClosedTimeRangeFilter`, so both layouts agree on boundary semantics. The accessor deliberately bypasses `archivedTimelineMap`, which is keyed by start instant alone and would otherwise hand a range-bounded timeline to a caller asking for an open-ended one. - Switch `HoodieMetadataTableValidator.computeDiffSummary` to that accessor and drop the in-memory `findInstantsInRange`, which fixes both the unbounded load and the off-by-one. - Wrap the summary in `computeDiffSummarySafely`, so a failure while computing a diagnostic degrades the message instead of propagating and masking the mismatch it describes. ### Impact `TimelineFactory` gains an abstract method, so any out-of-tree subclass needs to implement it. Both in-tree implementations and the test format are updated. No storage format or config change. Validator behavior changes in two user-visible ways: an archived instant equal to the range lower bound is now correctly reported as archived rather than as a missing commit, which also stops a spurious `missingCommits` from triggering a scan that reads every rollback plan from storage; and the archive read for the diagnostic is now proportional to the instant range rather than to the whole archive. One intentional deviation worth flagging for review: the new accessor returns completed instants only, matching its `createArchivedTimeline(metaClient, startTs)` sibling. The V1 archive also holds requested and inflight entries, and the previous no-arg path surfaced them, so on a table-version-6 table an instant archived without ever completing now classifies as a missing commit rather than as archived. That is the intended reading, since such an instant is a failed write, but it is a behavior change rather than a pure bug fix. ### Risk Level low The change is confined to a new API plus one validator call site. Verified with red-then-green: reverting the validator change makes `testValidateFileSlicesReportsMismatchWhenDiffSummaryFails` fail with `expected: <HoodieValidationException> but was: <HoodieIOException>`, reproducing the lost-verdict failure mode, and makes `testValidateFileSlicesTreatsArchivedBoundaryInstantAsCommitted` fail because the range accessor is never reached. Substituting the half-open filter for `ClosedClosedTimeRangeFilter` in the V2 constructor makes the new range test fail on the lower bound, confirming it pins inclusivity rather than passing vacuously. `TestArchivedTimelineV1`, `TestArchivedTimelineV2`, `TestHoodieMetadataTableValidator` and the pluggable-table-format suite that exercises the new abstract method all pass, and checkstyle is clean on the touched modules. ### Documentation Update none ### Contributor's checklist - [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [x] Enough context is provided in the sections above - [x] Adequate tests were added if applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
