hudi-agent commented on code in PR #19764: URL: https://github.com/apache/hudi/pull/19764#discussion_r3868978802
########## website/docs/cleaning.md: ########## @@ -221,6 +221,127 @@ cleans run --sparkMaster local --hoodieConfigs hoodie.clean.policy=KEEP_LATEST_C You can find more details and the relevant code for these commands in [`org.apache.hudi.cli.commands.CleansCommand`](https://github.com/apache/hudi/blob/master/hudi-cli/src/main/java/org/apache/hudi/cli/commands/CleansCommand.java) class. +## Partition TTL + +Cleaning bounds how many *versions* of a file are kept, but it never removes a partition: an old partition whose files +have all been cleaned down to a single version still sits in the table forever. Partition TTL (time to live) is the +complementary service. It works at partition granularity, and when a partition is judged expired it deletes the whole +partition rather than trimming file versions inside it. + +Because it removes data outright, TTL is off by default and stays off until you set a retention period. + +### How a partition is judged expired + +TTL asks a strategy which partitions have expired. Two strategies ship with Hudi, selected through +`hoodie.partition.ttl.management.strategy.type`: + +| Strategy | Ages a partition against | +|---|---| +| `KEEP_BY_TIME` (default) | The partition's last commit time, taken from the newest base instant among its latest file slices. A partition that is still being written to therefore stays. | +| `KEEP_BY_CREATION_TIME` | The commit time the partition was created at, read from its partition metadata. Writing to a partition does not extend its life. | + +Both compare that timestamp against `hoodie.partition.ttl.strategy.days.retain`. A custom strategy can be supplied +instead with `hoodie.partition.ttl.strategy.class`, pointing at a subclass of `PartitionTTLStrategy`; when both configs +are present the class takes precedence over the type. + +:::caution +`hoodie.partition.ttl.strategy.days.retain` defaults to `-1`, and the built-in strategies treat any value of `0` or less +as "nothing expires". **TTL does nothing at all until you set a positive retention, even with TTL enabled.** This is +deliberate, so that turning the service on cannot delete data by itself, but it does mean a misconfigured job looks like +a working one: it runs, reports no expired partitions, and deletes nothing. +::: + +Two other conditions make TTL a silent no-op regardless of retention: a table with no completed commit yet, and an +unpartitioned table. + +### Ways to run partition TTL + +**Inline.** Setting `hoodie.partition.ttl.inline=true` runs TTL immediately after each commit, alongside the other inline +table services. + +**As a standalone Spark job.** `org.apache.hudi.utilities.HoodieTTLJob`, in the utilities bundle, runs TTL against an +existing table without enabling it on the writer: + +``` +spark-submit --master local \ + --packages org.apache.hudi:hudi-utilities-slim-bundle_2.12:1.0.2,org.apache.hudi:hudi-spark3.5-bundle_2.12:1.0.2 \ + --class org.apache.hudi.utilities.HoodieTTLJob `ls packaging/hudi-utilities-slim-bundle/target/hudi-utilities-slim-bundle-*.jar` \ + --base-path file:///tmp/events_table \ + --hoodie-conf hoodie.partition.ttl.strategy.days.retain=30 +``` + +**From Spark SQL**, with the [`run_ttl`](procedures.md#run_ttl) procedure, which is the easiest way to try TTL on a table +before committing to running it on every write: + +```sql +call run_ttl(table => 'events_table', retain_days => 30); +``` + +However it is triggered, TTL writes a replace commit that drops the expired partitions, the same commit type used by the +`delete_partition` operation. + +### Keeping a first run under control + +The first TTL run on an existing table is the risky one, because every historical partition becomes a candidate at once. +Three configs bound it. + +`hoodie.partition.ttl.strategy.max.delete.partitions` caps how many partitions a single run may delete, defaulting to +`1000`. The limit exists to keep one replace commit from growing unmanageably large; partitions over the cap are simply +left for the next run, so a backlog drains across several runs rather than in one commit. + +`hoodie.partition.ttl.strategy.partition.selected` takes a comma-separated list of partition paths and restricts TTL to +exactly those. When it is unset, TTL considers every partition in the table. Setting it is the safest way to try a +retention policy on one partition before applying it everywhere. + +`hoodie.partition.ttl.strategy.stats.max.parallelism` bounds the parallelism used to collect each candidate partition's Review Comment: 🤖 This config, `hoodie.partition.ttl.strategy.stats.max.parallelism`, is the one TTL key in this section that does not appear in the generated config reference (`configurations.md`) — the other six all do, with matching `Config Param` names. Could you confirm the exact key name and its `200` default against `HoodieTTLConfig` in the source? If it exists but isn't surfaced in the reference, it may be worth noting; if the key is slightly off, a user setting it would silently get no effect. The accompanying claim that "the effective value is the smaller of that and the candidate count" would also be good to verify against the strategy implementation. @yihua could you sanity-check this config key/default? <sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag quality.</i></sub> ########## website/docs/cleaning.md: ########## @@ -221,6 +221,127 @@ cleans run --sparkMaster local --hoodieConfigs hoodie.clean.policy=KEEP_LATEST_C You can find more details and the relevant code for these commands in [`org.apache.hudi.cli.commands.CleansCommand`](https://github.com/apache/hudi/blob/master/hudi-cli/src/main/java/org/apache/hudi/cli/commands/CleansCommand.java) class. +## Partition TTL + +Cleaning bounds how many *versions* of a file are kept, but it never removes a partition: an old partition whose files +have all been cleaned down to a single version still sits in the table forever. Partition TTL (time to live) is the +complementary service. It works at partition granularity, and when a partition is judged expired it deletes the whole +partition rather than trimming file versions inside it. + +Because it removes data outright, TTL is off by default and stays off until you set a retention period. + +### How a partition is judged expired + +TTL asks a strategy which partitions have expired. Two strategies ship with Hudi, selected through +`hoodie.partition.ttl.management.strategy.type`: + +| Strategy | Ages a partition against | +|---|---| +| `KEEP_BY_TIME` (default) | The partition's last commit time, taken from the newest base instant among its latest file slices. A partition that is still being written to therefore stays. | +| `KEEP_BY_CREATION_TIME` | The commit time the partition was created at, read from its partition metadata. Writing to a partition does not extend its life. | + +Both compare that timestamp against `hoodie.partition.ttl.strategy.days.retain`. A custom strategy can be supplied +instead with `hoodie.partition.ttl.strategy.class`, pointing at a subclass of `PartitionTTLStrategy`; when both configs +are present the class takes precedence over the type. + +:::caution +`hoodie.partition.ttl.strategy.days.retain` defaults to `-1`, and the built-in strategies treat any value of `0` or less +as "nothing expires". **TTL does nothing at all until you set a positive retention, even with TTL enabled.** This is +deliberate, so that turning the service on cannot delete data by itself, but it does mean a misconfigured job looks like +a working one: it runs, reports no expired partitions, and deletes nothing. +::: + +Two other conditions make TTL a silent no-op regardless of retention: a table with no completed commit yet, and an +unpartitioned table. + +### Ways to run partition TTL + +**Inline.** Setting `hoodie.partition.ttl.inline=true` runs TTL immediately after each commit, alongside the other inline +table services. + +**As a standalone Spark job.** `org.apache.hudi.utilities.HoodieTTLJob`, in the utilities bundle, runs TTL against an +existing table without enabling it on the writer: + +``` +spark-submit --master local \ Review Comment: 🤖 The standalone-job example both pulls the utilities slim bundle via `--packages` and passes a locally built `hudi-utilities-slim-bundle-*.jar` (via the backtick `ls`) as the application jar. That's a bit contradictory — a reader with a local build wouldn't also need `--packages` for the same bundle, and a reader without one won't have the jar the `ls` expects. It might help to pick one path (either the Maven coordinates or the local jar) so the command is copy-paste runnable. <sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag quality.</i></sub> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
