rangareddy commented on issue #7390: URL: https://github.com/apache/hudi/issues/7390#issuecomment-4989041263
Hi @Gale100 — day-based partitioning is a good fit here, but the design hinges on separating three things that are easy to conflate. Once separated, you don't need a 365×2×24 cleaner setting. **1. Data retention ≠ cleaning. The cleaner will not delete your old day-partitions.** The cleaner only removes `superseded, older file versions` inside file groups that received `updates`. A partition like `2015/...` that never gets a new write keeps its latest file version forever — regardless of the cleaner. So "old folders still exist with the latest data" is exactly the intended behavior; that's your retained data, not something the cleaner reclaims. Your 2 years of history is safe by default. **2. Don't set hoodie.cleaner.commits.retained = 365*2*24.** That config counts commits, not hours, and it controls how many stale file-slice versions are kept per updated file group (i.e., your incremental-query / time-travel lookback). Cranking it to ~17,520 makes Hudi retain every superseded version for ~2 years across all updated file groups → storage bloat, huge file/version counts, and slower cleaning, metadata, and queries. Keep it near the default (10) unless you genuinely need long incremental-read lookback. If you want time-based version retention, use the purpose-built policy instead of converting time to commits: hoodie.clean.policy = KEEP_LATEST_BY_HOURS hoodie.clean.hours.retained = <hours> # default 24 (default policy is `KEEP_LATEST_COMMITS` with `hoodie.clean.commits.retained=10`.) **3. For guaranteed 2-year time-travel / recovery points, use savepoints — not a giant cleaner window.** A savepoint pins the files for a chosen instant so the cleaner can't remove them, and lets you `restore` to it. Take savepoints at the granularity you actually need (e.g., daily or weekly) to get 2 years of recoverable points cheaply. For arbitrary time-travel reads (`as.of.instant`) to keep working, the referenced file versions must still exist (savepoints protect them) and the instant should remain on the timeline — bound timeline growth with archival: hoodie.keep.min.commits / hoodie.keep.max.commits # must be > cleaner commits retained **4. To actually expire data after 2 years, drop whole partitions.** Since you're date-partitioned, use the partition-delete operation rather than record deletes or the cleaner: hoodie.datasource.write.operation = delete_partition This is your retention/expiry mechanism and is independent of cleaning. **5. Scale is fine.** ~730 daily partitions over 2 years is small for Hudi. Keep the metadata table enabled (default on in recent versions) for fast file listing. The "too many files" worry is really about small files — manage that with target file sizing / clustering, not the cleaner. TL;DR: Day partitioning ✅. Leave the cleaner near default — it won't touch un-updated old partitions. Use savepoints (+ archival tuning) for 2-year time-travel/restore, and `delete_partition` to expire data after 2 years. Reserve `KEEP_LATEST_BY_HOURS` for time-based version retention if you need it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
