ravjot28 opened a new pull request, #20592:
URL: https://github.com/apache/hudi/pull/20592
### Describe the issue this Pull Request addresses
Closes #20582
With `KEEP_LATEST_BY_HOURS` the earliest commit to retain is the first
completed commit at or after `now - hours`.
When every completed commit is older there is none, and
`CleanPlanner#getPartitionPathsToClean` treats that like
`KEEP_LATEST_COMMITS` with too few commits: it scans no partition and
nothing is cleaned. A table written less often
than its window, or any table with a window of 0 hours, keeps every file
slice.
### Summary and Changelog
When the hours window holds no completed commit while the table has some,
the clean retains from the latest completed
commit, so the file slices older than the window are removed.
- `CleanerUtils#getEarliestCommitToRetain`: for `KEEP_LATEST_BY_HOURS`, when
no completed commit is inside the window,
the earliest commit to retain is the latest completed commit. If a commit
is pending, it is the latest completed
commit before the earliest pending one, as `KEEP_LATEST_COMMITS` does, so
an incremental clean later finds the
partitions of that commit. It never goes back before the commit the
previous clean recorded: a clean inside the
window does not stop at a pending commit, so without this the recorded
commit could move back once a pending commit
holds the fallback, and time travel would then accept timestamps whose
slices were already cleaned.
`CleanPlanner#getEarliestCommitToRetain` reads the previous clean's commit
for the hours policy too (until now only
when `hoodie.clean.max.commits.to.clean` is set). A table without
completed commits still has no commit to retain.
- Nothing else changes in the planner: the clean runs the existing selection
of the hours policy with that commit. Each
file group keeps its latest slice and its latest slice before that commit;
savepointed slices, slices needed by a
pending compaction or log compaction and replaced file groups follow the
existing rules.
- The clean plan and the clean metadata record that commit, so everything
that reads the earliest commit to retain
works as for any other clean of this policy: incremental partition
selection, `hoodie.clean.max.commits.to.clean`,
empty cleans, time travel and savepoint validation
(`TimelineUtils#validateTimestampAsOf`,
`SavepointActionExecutor#getLastCommitRetained`) and the archival guard
for replaced file groups kept by a
savepoint (`ClusteringUtils#getEarliestReplacedSavepointInClean`).
Archival itself calls
`getEarliestCommitToRetain` with the latest commit's time as "now", which
always has a commit in the window, so its
result does not change.
- New tests:
- `TestCleanerUtils`: every commit older than the window returns the
latest one; `hoodie.clean.max.commits.to.clean`
caps it from the previous clean's commit; a pending commit holds it
back, but not before the previous clean's
commit.
-
`TestCleanPlanExecutor#testKeepXHoursCommitToRetainDoesNotMoveBackWhileACommitIsPending`:
a clean inside the
window past a pending commit, then a window of 0 hours; the planner
keeps the recorded commit.
-
`TestCleanPlanExecutor#testKeepXHoursCleansOlderSlicesWhenEveryCommitIsOlderThanTheWindow`
(windows of 0 and 2
hours, three commits 5, 4 and 3 hours old): older slices are removed,
the clean metadata records the latest
commit, time travel and a savepoint to a cleaned commit are refused, and
the next clean selects its partitions
from the recorded commit.
- `TestCleanPlanExecutor#testKeepLatestCommitsMOR` also runs with
`KEEP_LATEST_BY_HOURS` on a merge-on-read table
with log files.
-
`TestCleaner#testKeepLatestByHoursWithoutCommitInWindowKeepsTheReplacedSavepointGuard`:
a savepoint, a replace
commit and commits all older than the window; savepointed slices stay
and the archival guard returns the savepoint.
### Impact
No API or configuration change. Tables with `KEEP_LATEST_BY_HOURS` and no
commit inside the window are cleaned again.
The metadata table derives the hours policy from the data table and is
cleaned the same way. File groups written by
the latest commit keep one older slice until a later commit, as at the edge
of the window.
**Backport:** please include this fix in the next 1.2.x patch release. The
commit cherry-picks cleanly onto
`release-1.2.1`.
### Risk Level
low. The change only fills the earliest commit to retain that was empty for
`KEEP_LATEST_BY_HOURS`; the cases where a
commit is inside the window are unchanged, and the clean that follows is an
ordinary clean of the hours policy.
For this policy the planner reads the previous clean's metadata when it
computes the commit to retain; with
incremental cleaning it is the read the planner already made.
Tests, run on master with JDK 11 and passing checkstyle checks for main and
test sources (no Scala sources changed;
scalastyle did not execute in this module selection), with
`mvn test -pl hudi-client/hudi-client-common,hudi-client/hudi-spark-client
-am -Dtest=...` and the classes listed
below:
- Without the main code change: `TestCleanerUtils` 3 of 13 (the three new
hours cases get no commit to retain),
`TestCleanPlanExecutor` 4 of 26 (both window cases clean no partition; the
merge-on-read hours case runs no clean;
the no-move-back case gets no commit to retain) and `TestCleaner` 1 of 29
(the savepoint guard) fail.
- With the change: `TestCleanerUtils` 13/13, `TestCleanPlanner` 44/44,
`TestCleanPlanExecutor` 26/26, `TestCleaner` 29/29,
`TestHoodieTimelineArchiver` 2/2 and 57/57 (client and Spark client) pass.
### Documentation Update
none (the behaviour now matches the documentation of
`hoodie.clean.hours.retained`)
### Contributor's checklist
- [x] Read through [contributor's
guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Enough context is provided in the sections above
- [x] Adequate tests were added if applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]