fhan688 opened a new pull request, #19041:
URL: https://github.com/apache/hudi/pull/19041
### Describe the issue this Pull Request addresses
KEEP_LATEST_BY_HOURS previously derived the earliest commit to retain
(ECTR) from completed instants by requested time. Under timeline v2 /
completion-time semantics, this can make the cleaner treat an instant as old
even when it completed within the configured retention window.
This PR updates KEEP_LATEST_BY_HOURS to derive ECTR from completed
instants by completion time, without clipping the by-hours boundary on pending
instants. Pending-instant archival safety is handled by the archiver's existing
pending-instant protection.
The PR also makes archival optionally depend on the ECTR recorded by the
latest completed clean, so commit metadata is archived only after cleaner has
completed past that boundary.
### Summary and Changelog
Changes:
- Make KEEP_LATEST_BY_HOURS select ECTR using completion time instead of
requested time.
- Keep the returned ECTR as a requested-time boundary, because clean
metadata and timeline retention still identify instants by requested time.
- Fall back to requested time when completion time is unavailable for
compatibility with legacy timeline metadata.
- Remove pending-instant clipping from the by-hours ECTR calculation.
- Keep pending-instant archival safety in the archiver path.
- Add a shared archival utility to derive the earliest instant to retain
from the latest completed clean metadata.
- Apply clean ECTR archive blocking to both TimelineArchiverV1 and
TimelineArchiverV2 when `hoodie.archive.block.on.latest.clean.ectr` is enabled.
- Expose Flink options for:
- `hoodie.clean.max.commits`
- `hoodie.clean.empty.commit.interval.hours`
- `hoodie.archive.block.on.latest.clean.ectr`
- Add tests for completion-time based by-hours ECTR selection,
pending-instant handling, V2 archival behavior, and Flink config propagation.
No code was copied from another project.
### Impact
User-facing behavior changes:
- Tables using KEEP_LATEST_BY_HOURS now evaluate the retention window
using completion time.
- Pending instants no longer shrink the cleaner's by-hours ECTR directly;
archival safety for pending instants remains handled by the archiver.
- When `hoodie.archive.block.on.latest.clean.ectr` is enabled, timeline
archiving avoids archiving commit metadata at or after the ECTR recorded by the
latest completed clean.
- Flink writers can configure the existing clean/archive controls through
Flink options.
This may retain more active timeline instants when archive blocking on
clean ECTR is enabled, which is expected for correctness.
### Risk Level
medium
The change affects cleaner retention boundary calculation for
KEEP_LATEST_BY_HOURS and archival retention decisions when clean ECTR archive
blocking is enabled.
The implementation is conservative:
- by-hours ECTR is based on completed instants only
- missing completion time falls back to requested time
- ECTR remains a requested-time boundary
- clean ECTR archive blocking is gated by the existing config
- pending-instant archival protection remains in the archiver
Verification:
- `mvn -pl hudi-common -am -DskipITs -Dcheckstyle.skip=true
-DfailIfNoTests=false -Dsurefire.failIfNoSpecifiedTests=false
-Dtest=TestCleanerUtils test`
- `mvn -pl hudi-client/hudi-spark-client -am -DskipTests -DskipITs
-Dcheckstyle.skip=true test-compile`
- `git diff --check`
### Documentation Update
The Hudi website/config documentation should be updated for the newly
exposed Flink options:
- `hoodie.clean.max.commits`
- `hoodie.clean.empty.commit.interval.hours`
- `hoodie.archive.block.on.latest.clean.ectr`
### Contributor's checklist
- [x] Read through [contributor's
guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Enough context is provided in the sections above
- [x] Adequate tests were added if applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
