koochiswathiTR commented on issue #7600:
URL: https://github.com/apache/hudi/issues/7600#issuecomment-1411949976
Hi @umehrot2,
Below are the cleanup config changes.
We process the batch in 5 mints interval.
5 minute ingestion – which is 12 delta commits per hour and 288(12*24)
delta commits per day
Compaction runs every hour, In a day 24 commits.
In a day total number of commits = (Delta commits + compaction commits
) = 312 commits
We configured to retain 3 days of commits 312 *3 = 936 commits
Minimum commits retained is set to 937 ( 936 +1 )
Maximum commits retained is 960 (936 + 24)
HoodieCompactionConfig.CLEANER_POLICY.key() ->
HoodieCleaningPolicy.KEEP_LATEST_COMMITS.name(),
HoodieCompactionConfig.CLEANER_COMMITS_RETAINED.key() -> "936",
HoodieCompactionConfig.MIN_COMMITS_TO_KEEP.key() -> "937", //
CLEANER_COMMITS_RETAINED + 1
HoodieCompactionConfig.MAX_COMMITS_TO_KEEP.key() -> "960", //
CLEANER_COMMITS_RETAINED + 24
Please let us know your thoughts on this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
