ravjot28 opened a new issue, #20582:
URL: https://github.com/apache/hudi/issues/20582

   ### Bug Description
   
   **What happened:**
   
   With `hoodie.clean.policy=KEEP_LATEST_BY_HOURS`, no clean is planned when 
every completed commit of the table is older
   than `hoodie.clean.hours.retained`. Every older file slice stays until a 
commit lands inside the window again: a table
   written less often than its window never cleans, and a window of 0 hours 
never cleans at all.
   
   **What you expected:**
   
   The file slices older than the window are removed and the latest file slice 
of each file group is kept, as
   `CleanPlanner#getFilesToCleanKeepingLatestHours` documents ("all the files 
with commit time earlier than N hours will
   be removed. Also the latest file for any file group is retained."). Every 
commit being outside the window is the case
   in which the documented behaviour removes the most. A file group written by 
the latest commit may keep the slice before it, as at the
   edge of the window.
   
   **Steps to reproduce:**
   
   1. Create a copy-on-write table with 
`hoodie.clean.policy=KEEP_LATEST_BY_HOURS` and `hoodie.clean.hours.retained=2`.
   2. Write three commits that update the same file group, all more than 2 
hours ago (or set the window to 0 hours, so
      every commit is older than the window).
   3. Run a clean: nothing is planned and all three file slices remain.
   
   **Root cause (master d5230b69e22):**
   
   - 
`hudi-common/src/main/java/org/apache/hudi/common/util/CleanerUtils.java:151-155`:
 for `KEEP_LATEST_BY_HOURS`,
     `getEarliestCommitToRetain` returns the first completed commit at or after 
`now - hoursRetained`; when every
     completed commit is older there is none and it returns empty.
   - 
`hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/clean/CleanPlanner.java:157-160`
 sends
     `KEEP_LATEST_BY_HOURS` to `getPartitionPathsForCleanByCommits`, which 
returns no partition when there is no commit to
     retain (`CleanPlanner.java:200-204`, "No earliest commit to retain. No 
need to scan partitions !!").
   - For `KEEP_LATEST_COMMITS` an empty result correctly means "fewer commits 
than retained, nothing to clean". For the
     hours policy it means "every commit is outside the window", the case in 
which the documented behaviour cleans the
     most.
   
   Related, not the same defect: #16014 (which time the hours window compares), 
#8209, #7600.
   
   ### Environment
   
   **Hudi version:** master (1.3.0-SNAPSHOT at d5230b69e22), 1.2.1
   **Query engine:** any (the clean planner is engine independent); seen with 
Spark 3.5. The metadata table derives
   the hours policy from the data table and is affected the same way.
   **Relevant configs:** `hoodie.clean.policy=KEEP_LATEST_BY_HOURS`, 
`hoodie.clean.hours.retained`
   
   ### Logs and Stack Trace
   
   ```
   INFO CleanPlanner: No earliest commit to retain. No need to scan partitions 
!!
   ```
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to