yihua opened a new issue, #19625:
URL: https://github.com/apache/hudi/issues/19625
### Describe the problem you faced
`HoodieMetadataTableValidator.computeDiffSummary` builds the diagnostic
message for a detected file-slice mismatch. To decide whether the file slices
it found only on storage belong to commits that were archived (so the metadata
table is genuinely missing a committed file) or to commits that exist nowhere
(so they are orphans of a failed or rolled-back write), it consults the
archived timeline:
```java
Set<String> archivedInstants = metaClient.getArchivedTimeline()
.findInstantsInRange(minInstant, maxInstant)
.getInstantsAsStream()
.map(HoodieInstant::requestedTime)
.collect(Collectors.toSet());
```
Two problems follow from filtering in memory rather than pushing the range
into the load.
**1. The whole archive is loaded.** `getArchivedTimeline()` is the no-arg
accessor, so every archived instant is materialized with its commit metadata,
and the result is cached in `archivedTimelineMap` for the rest of the task. On
a large table this is enough to OOM an executor. Because the load happens while
the error message is being built, the `HoodieValidationException` carrying the
mismatch verdict is never thrown: the validator reports an unexplained failure
and the real finding, a genuine file-slice count mismatch, is lost. "No
mismatch reported for this table" is therefore not evidence the table is clean.
Note that a range filter alone would not fix the memory use.
`ArchivedTimelineLoaderV1` applies the filter only after deserializing each
record, and the reverse-chronological early `break` that skips older archive
files is disabled entirely when the filter is null, which is exactly today's
path.
**2. An off-by-one on the lower bound.** `findInstantsInRange` is half-open:
`InstantComparison.isInRange` is `GREATER_THAN startTs` and
`LESSER_THAN_OR_EQUALS endTs`. `minInstant` is itself a member of the set being
classified, so it can never appear in `archivedInstants` and always survives
into `missingCommits`. When the set has a single element, `minInstant` equals
`maxInstant`, the range is empty, and that instant is always reported missing.
Beyond the wrong message, a spurious `missingCommits` triggers a scan that
reads every `.rollback.requested` plan from storage.
### To Reproduce
Run the metadata table validator against a table with an archived timeline
and at least one file slice on storage whose base instant is not in the active
timeline. The instant equal to the lower bound of the computed range is always
classified as a missing commit even when it is archived.
### Expected behavior
The archive lookup should load only the instants in the range it cares
about, and should treat both bounds as inclusive so an archived instant equal
to the lower bound is recognized as archived. Separately, a failure while
computing a diagnostic summary should degrade the message rather than discard
the mismatch verdict the validator had already reached.
### Environment Description
* Hudi version : 1.1.0 and later, including master
* Storage (HDFS/S3/GCS..) : S3
* Running on Docker? (yes/no) : no
### Additional context
`TimelineFactory` exposes only `createArchivedTimeline(metaClient)` and
`createArchivedTimeline(metaClient, startTs)`, and `HoodieTableMetaClient`
correspondingly has no closed-range accessor, which is why the range ended up
being applied in memory. `ArchivedTimelineV1` already has a public constructor
taking `(metaClient, startTs, endTs)` backed by `ClosedClosedTimeRangeFilter`;
it is simply unreachable through the sanctioned API.
`getArchivedTimeline(minInstant)` is not a workaround: `StartTsFilter` is
open-ended upward, so it still reads nearly every archive file when
`minInstant` is old, and it loads instant details.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
