deepakpanda93 commented on code in PR #19764: URL: https://github.com/apache/hudi/pull/19764#discussion_r3869058582
########## website/docs/cleaning.md: ########## @@ -221,6 +221,127 @@ cleans run --sparkMaster local --hoodieConfigs hoodie.clean.policy=KEEP_LATEST_C You can find more details and the relevant code for these commands in [`org.apache.hudi.cli.commands.CleansCommand`](https://github.com/apache/hudi/blob/master/hudi-cli/src/main/java/org/apache/hudi/cli/commands/CleansCommand.java) class. +## Partition TTL + +Cleaning bounds how many *versions* of a file are kept, but it never removes a partition: an old partition whose files +have all been cleaned down to a single version still sits in the table forever. Partition TTL (time to live) is the +complementary service. It works at partition granularity, and when a partition is judged expired it deletes the whole +partition rather than trimming file versions inside it. + +Because it removes data outright, TTL is off by default and stays off until you set a retention period. + +### How a partition is judged expired + +TTL asks a strategy which partitions have expired. Two strategies ship with Hudi, selected through +`hoodie.partition.ttl.management.strategy.type`: + +| Strategy | Ages a partition against | +|---|---| +| `KEEP_BY_TIME` (default) | The partition's last commit time, taken from the newest base instant among its latest file slices. A partition that is still being written to therefore stays. | +| `KEEP_BY_CREATION_TIME` | The commit time the partition was created at, read from its partition metadata. Writing to a partition does not extend its life. | + +Both compare that timestamp against `hoodie.partition.ttl.strategy.days.retain`. A custom strategy can be supplied +instead with `hoodie.partition.ttl.strategy.class`, pointing at a subclass of `PartitionTTLStrategy`; when both configs +are present the class takes precedence over the type. + +:::caution +`hoodie.partition.ttl.strategy.days.retain` defaults to `-1`, and the built-in strategies treat any value of `0` or less +as "nothing expires". **TTL does nothing at all until you set a positive retention, even with TTL enabled.** This is +deliberate, so that turning the service on cannot delete data by itself, but it does mean a misconfigured job looks like +a working one: it runs, reports no expired partitions, and deletes nothing. +::: + +Two other conditions make TTL a silent no-op regardless of retention: a table with no completed commit yet, and an +unpartitioned table. + +### Ways to run partition TTL + +**Inline.** Setting `hoodie.partition.ttl.inline=true` runs TTL immediately after each commit, alongside the other inline +table services. + +**As a standalone Spark job.** `org.apache.hudi.utilities.HoodieTTLJob`, in the utilities bundle, runs TTL against an +existing table without enabling it on the writer: + +``` +spark-submit --master local \ Review Comment: You're right, and it was contradictory in exactly the way you describe. Fixed in 55c8b484. The command now takes one path only, the self-contained utilities bundle as the application jar: ``` spark-submit --master local \ --class org.apache.hudi.utilities.HoodieTTLJob \ hudi-utilities-bundle_2.12-1.2.0.jar \ --base-path file:///tmp/events_table \ --hoodie-conf hoodie.partition.ttl.strategy.days.retain=30 ``` followed by a line saying the bundle is self-contained so no `--packages` is needed, and where to get it: Maven Central, or `packaging/hudi-utilities-bundle/target/hudi-utilities-bundle_2.12-*.jar` from a local build. I confirmed `hudi-utilities-bundle_2.12` is published on Maven Central before pointing readers at it. One thing I should own up to about how the redundancy got there. I did not write that invocation from scratch: I copied the convention from the `HoodieCleaner` examples further up this same page, on the reasoning that matching the page's existing style was safer than inventing my own. Those examples do the identical thing, `--packages` for the slim bundle plus a locally built slim bundle jar via `ls`, at lines 157 and 181 of `cleaning.md`. So the flaw was inherited rather than introduced, which is a decent argument for not treating "consistent with the surrounding page" as the same thing as "correct". I have deliberately **not** changed those `HoodieCleaner` examples. They carry the same redundancy and the same local-build assumption, and fixing them would be worth doing, but it is unrelated to partition TTL and would widen a docs PR about one feature into an edit of the page's long-standing cleaner examples. Happy to do it as a follow-up if a committer agrees the full-bundle form is the one the project wants; if instead the slim-bundle-plus-`--packages` form is deliberate for some reason I am not seeing, then this new command should probably be reverted to match it rather than the other way round. That is the one open question here. Build passes with the warning block still byte-identical to a baseline at the same base commit, and I checked the rendered page to confirm the TTL command region contains neither `--packages` nor `slim-bundle`, while the 12 remaining `slim-bundle` mentions are all in the untouched cleaner examples. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
