ravjot28 opened a new pull request, #20592:
URL: https://github.com/apache/hudi/pull/20592

   ### Describe the issue this Pull Request addresses
   
   Closes #20582
   
   With `KEEP_LATEST_BY_HOURS` the earliest commit to retain is the first 
completed commit at or after `now - hours`.
   When every completed commit is older there is none, and 
`CleanPlanner#getPartitionPathsToClean` treats that like
   `KEEP_LATEST_COMMITS` with too few commits: it scans no partition and 
nothing is cleaned. A table written less often
   than its window, or any table with a window of 0 hours, keeps every file 
slice.
   
   ### Summary and Changelog
   
   When the hours window holds no completed commit while the table has some, 
the clean retains from the latest completed
   commit, so the file slices older than the window are removed.
   
   - `CleanerUtils#getEarliestCommitToRetain`: for `KEEP_LATEST_BY_HOURS`, when 
no completed commit is inside the window,
     the earliest commit to retain is the latest completed commit. If a commit 
is pending, it is the latest completed
     commit before the earliest pending one, as `KEEP_LATEST_COMMITS` does, so 
an incremental clean later finds the
     partitions of that commit. It never goes back before the commit the 
previous clean recorded: a clean inside the
     window does not stop at a pending commit, so without this the recorded 
commit could move back once a pending commit
     holds the fallback, and time travel would then accept timestamps whose 
slices were already cleaned.
     `CleanPlanner#getEarliestCommitToRetain` reads the previous clean's commit 
for the hours policy too (until now only
     when `hoodie.clean.max.commits.to.clean` is set). A table without 
completed commits still has no commit to retain.
   - Nothing else changes in the planner: the clean runs the existing selection 
of the hours policy with that commit. Each
     file group keeps its latest slice and its latest slice before that commit; 
savepointed slices, slices needed by a
     pending compaction or log compaction and replaced file groups follow the 
existing rules.
   - The clean plan and the clean metadata record that commit, so everything 
that reads the earliest commit to retain
     works as for any other clean of this policy: incremental partition 
selection, `hoodie.clean.max.commits.to.clean`,
     empty cleans, time travel and savepoint validation 
(`TimelineUtils#validateTimestampAsOf`,
     `SavepointActionExecutor#getLastCommitRetained`) and the archival guard 
for replaced file groups kept by a
     savepoint (`ClusteringUtils#getEarliestReplacedSavepointInClean`). 
Archival itself calls
     `getEarliestCommitToRetain` with the latest commit's time as "now", which 
always has a commit in the window, so its
     result does not change.
   - New tests:
     - `TestCleanerUtils`: every commit older than the window returns the 
latest one; `hoodie.clean.max.commits.to.clean`
       caps it from the previous clean's commit; a pending commit holds it 
back, but not before the previous clean's
       commit.
     - 
`TestCleanPlanExecutor#testKeepXHoursCommitToRetainDoesNotMoveBackWhileACommitIsPending`:
 a clean inside the
       window past a pending commit, then a window of 0 hours; the planner 
keeps the recorded commit.
     - 
`TestCleanPlanExecutor#testKeepXHoursCleansOlderSlicesWhenEveryCommitIsOlderThanTheWindow`
 (windows of 0 and 2
       hours, three commits 5, 4 and 3 hours old): older slices are removed, 
the clean metadata records the latest
       commit, time travel and a savepoint to a cleaned commit are refused, and 
the next clean selects its partitions
       from the recorded commit.
     - `TestCleanPlanExecutor#testKeepLatestCommitsMOR` also runs with 
`KEEP_LATEST_BY_HOURS` on a merge-on-read table
       with log files.
     - 
`TestCleaner#testKeepLatestByHoursWithoutCommitInWindowKeepsTheReplacedSavepointGuard`:
 a savepoint, a replace
       commit and commits all older than the window; savepointed slices stay 
and the archival guard returns the savepoint.
   
   ### Impact
   
   No API or configuration change. Tables with `KEEP_LATEST_BY_HOURS` and no 
commit inside the window are cleaned again.
   The metadata table derives the hours policy from the data table and is 
cleaned the same way. File groups written by
   the latest commit keep one older slice until a later commit, as at the edge 
of the window.
   
   **Backport:** please include this fix in the next 1.2.x patch release. The 
commit cherry-picks cleanly onto
   `release-1.2.1`.
   
   ### Risk Level
   
   low. The change only fills the earliest commit to retain that was empty for 
`KEEP_LATEST_BY_HOURS`; the cases where a
   commit is inside the window are unchanged, and the clean that follows is an 
ordinary clean of the hours policy.
   For this policy the planner reads the previous clean's metadata when it 
computes the commit to retain; with
   incremental cleaning it is the read the planner already made.
   
   Tests, run on master with JDK 11 and passing checkstyle checks for main and 
test sources (no Scala sources changed;
   scalastyle did not execute in this module selection), with
   `mvn test -pl hudi-client/hudi-client-common,hudi-client/hudi-spark-client 
-am -Dtest=...` and the classes listed
   below:
   - Without the main code change: `TestCleanerUtils` 3 of 13 (the three new 
hours cases get no commit to retain),
     `TestCleanPlanExecutor` 4 of 26 (both window cases clean no partition; the 
merge-on-read hours case runs no clean;
     the no-move-back case gets no commit to retain) and `TestCleaner` 1 of 29 
(the savepoint guard) fail.
   - With the change: `TestCleanerUtils` 13/13, `TestCleanPlanner` 44/44, 
`TestCleanPlanExecutor` 26/26, `TestCleaner` 29/29,
     `TestHoodieTimelineArchiver` 2/2 and 57/57 (client and Spark client) pass.
   
   ### Documentation Update
   
   none (the behaviour now matches the documentation of 
`hoodie.clean.hours.retained`)
   
   ### Contributor's checklist
   
   - [x] Read through [contributor's 
guide](https://hudi.apache.org/contribute/how-to-contribute)
   - [x] Enough context is provided in the sections above
   - [x] Adequate tests were added if applicable
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to