kbuci opened a new pull request, #18215:
URL: https://github.com/apache/hudi/pull/18215
### Summary and Changelog
**Summary:** Metadata table (MDT) archival should be able to archive
instants that are before the earliest commit to retain (in latest clean) on the
data table, provided that there are no savepoints. This is since
- the metadata table anyway cannot be safely restored to a point before the
earliest commit to retain (ECTR) on the data table
- For datasets with infrequent ingestion writes and duration based
compaction scheduling on MDT, there can be 1.5-2x the number of instant files
in MDT active timeline compared to data table timeline. This is since each
write to the data table will not only create a deltacommit on MDT, but most
likely compaction commit as well.
**Changelog:**
- **TimelineUtils**
- `getEarliestInstantForMetadataArchival` now takes a third parameter:
`Option<String> earliestUncleanedDataTableInstantTimeOption` (ECTR from the
latest clean).
- Logic uses “earliest possible restore commit” (ECTR if present and on
timeline, else first commit), “smallest” of that and inflight, and when not
archiving beyond savepoint, the minimum of first savepointed write, earliest
inflight, and earliest possible restore commit.
- Added `getEarliestRetainedCommitFromLastClean(HoodieTableMetaClient)` to
read ECTR from the latest completed clean; throws `HoodieIOException` on read
failure.
- Added `findSmallestInstant(Option<HoodieInstant>,
Option<HoodieInstant>)` and
`getEarliestPossibleRestoreCommit(HoodieActiveTimeline, Option<String>)`.
- **HoodieTimeline / BaseHoodieTimeline**
- New method `findFirstSavepointedWrite()`: returns the first write commit
that has been savepointed (used when blocking MDT archival on savepoints).
- **TimelineArchiverV1 and TimelineArchiverV2**
- When computing the earliest instant to retain for the metadata table,
both archivers now pass the data table ECTR:
`TimelineUtils.getEarliestRetainedCommitFromLastClean(dataMetaClient)` is
supplied as the third argument to `getEarliestInstantForMetadataArchival`.
- **Tests**
- **TestTimelineUtils:** All `getEarliestInstantForMetadataArchival` call
sites updated to the 3-arg signature; added cases for ECTR present, ECTR not on
timeline, and savepoints + ECTR (earlier than savepoints and between
savepoints).
- **HoodieTestTable:** Added `addIncrementalClean(String instantTime,
String earliestCommitToRetain)` for tests that need a clean with a given ECTR.
- **TestHoodieTimelineArchiver:** Added
`testArchivalInMetadataTableCanProceedUntilECTR` to assert MDT archival
proceeds up to the data table ECTR.
### Impact
- **User-facing / API:** No new public APIs. MDT archival may retain fewer
instants (archive more)
- **Performance:** Slightly less MDT timeline to scan when ECTR is present,
due to more aggressive archival.
### Risk Level
**Low.** It should be safe if instants on the data table before the ECTR
don't have a corresponding deltacommit on MDT. This is since those instants
must already have been committed, and cannot anyway be rolled back.
### Documentation Update
None.
### Contributor's checklist
- [ ] Read through [contributor's
guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Enough context is provided in the sections above
- [ ] Adequate tests were added if applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]