deepakpanda93 commented on code in PR #19764: URL: https://github.com/apache/hudi/pull/19764#discussion_r3869056549
########## website/docs/cleaning.md: ########## @@ -221,6 +221,127 @@ cleans run --sparkMaster local --hoodieConfigs hoodie.clean.policy=KEEP_LATEST_C You can find more details and the relevant code for these commands in [`org.apache.hudi.cli.commands.CleansCommand`](https://github.com/apache/hudi/blob/master/hudi-cli/src/main/java/org/apache/hudi/cli/commands/CleansCommand.java) class. +## Partition TTL + +Cleaning bounds how many *versions* of a file are kept, but it never removes a partition: an old partition whose files +have all been cleaned down to a single version still sits in the table forever. Partition TTL (time to live) is the +complementary service. It works at partition granularity, and when a partition is judged expired it deletes the whole +partition rather than trimming file versions inside it. + +Because it removes data outright, TTL is off by default and stays off until you set a retention period. + +### How a partition is judged expired + +TTL asks a strategy which partitions have expired. Two strategies ship with Hudi, selected through +`hoodie.partition.ttl.management.strategy.type`: + +| Strategy | Ages a partition against | +|---|---| +| `KEEP_BY_TIME` (default) | The partition's last commit time, taken from the newest base instant among its latest file slices. A partition that is still being written to therefore stays. | +| `KEEP_BY_CREATION_TIME` | The commit time the partition was created at, read from its partition metadata. Writing to a partition does not extend its life. | + +Both compare that timestamp against `hoodie.partition.ttl.strategy.days.retain`. A custom strategy can be supplied +instead with `hoodie.partition.ttl.strategy.class`, pointing at a subclass of `PartitionTTLStrategy`; when both configs +are present the class takes precedence over the type. + +:::caution +`hoodie.partition.ttl.strategy.days.retain` defaults to `-1`, and the built-in strategies treat any value of `0` or less +as "nothing expires". **TTL does nothing at all until you set a positive retention, even with TTL enabled.** This is +deliberate, so that turning the service on cannot delete data by itself, but it does mean a misconfigured job looks like +a working one: it runs, reports no expired partitions, and deletes nothing. +::: + +Two other conditions make TTL a silent no-op regardless of retention: a table with no completed commit yet, and an +unpartitioned table. + +### Ways to run partition TTL + +**Inline.** Setting `hoodie.partition.ttl.inline=true` runs TTL immediately after each commit, alongside the other inline +table services. + +**As a standalone Spark job.** `org.apache.hudi.utilities.HoodieTTLJob`, in the utilities bundle, runs TTL against an +existing table without enabling it on the writer: + +``` +spark-submit --master local \ + --packages org.apache.hudi:hudi-utilities-slim-bundle_2.12:1.0.2,org.apache.hudi:hudi-spark3.5-bundle_2.12:1.0.2 \ + --class org.apache.hudi.utilities.HoodieTTLJob `ls packaging/hudi-utilities-slim-bundle/target/hudi-utilities-slim-bundle-*.jar` \ + --base-path file:///tmp/events_table \ + --hoodie-conf hoodie.partition.ttl.strategy.days.retain=30 +``` + +**From Spark SQL**, with the [`run_ttl`](procedures.md#run_ttl) procedure, which is the easiest way to try TTL on a table +before committing to running it on every write: + +```sql +call run_ttl(table => 'events_table', retain_days => 30); +``` + +However it is triggered, TTL writes a replace commit that drops the expired partitions, the same commit type used by the +`delete_partition` operation. + +### Keeping a first run under control + +The first TTL run on an existing table is the risky one, because every historical partition becomes a candidate at once. +Three configs bound it. + +`hoodie.partition.ttl.strategy.max.delete.partitions` caps how many partitions a single run may delete, defaulting to +`1000`. The limit exists to keep one replace commit from growing unmanageably large; partitions over the cap are simply +left for the next run, so a backlog drains across several runs rather than in one commit. + +`hoodie.partition.ttl.strategy.partition.selected` takes a comma-separated list of partition paths and restricts TTL to +exactly those. When it is unset, TTL considers every partition in the table. Setting it is the safest way to try a +retention policy on one partition before applying it everywhere. + +`hoodie.partition.ttl.strategy.stats.max.parallelism` bounds the parallelism used to collect each candidate partition's Review Comment: Good catch on the discrepancy, and thanks for asking rather than assuming the key was wrong. Addressed in 55c8b484. **The key and default are correct.** From `HoodieTTLConfig` on master: ```java public static final String PARTITION_TTL_STRATEGY_PARAM_PREFIX = "hoodie.partition.ttl.strategy."; public static final ConfigProperty<Integer> STATS_MAX_PARALLELISM = ConfigProperty .key(PARTITION_TTL_STRATEGY_PARAM_PREFIX + "stats.max.parallelism") .defaultValue(200) .sinceVersion("1.3.0") ``` so the full key is `hoodie.partition.ttl.strategy.stats.max.parallelism`, default `200`. **The `min` claim also holds.** `KeepByTimeStrategy#getLastCommitTimeForPartitions`: ```java int statsParallelism = Math.min(partitionPaths.size(), writeConfig.getPartitionTTLStatsMaxParallelism()); ``` **You found a real gap, though, just not the one you suspected.** The reason it is missing from `configurations.md` is `sinceVersion("1.3.0")`: the config does not exist in `release-1.2.0` (I checked the tag directly, zero occurrences), and the generated reference reflects a released version. So a reader on 1.2.0 who copied that line would set a key that silently does nothing, which is exactly the failure mode you were worried about, arriving by a different route. Rather than drop the config, the section now says so explicitly. The prose adds: > This config is new in 1.3.0, so it has no effect on earlier releases and does not yet appear in the generated configuration reference; the other six configs above do. and the table row is marked `Since 1.3.0`. This is also why the PR is scoped to `website/docs` only, which the description covers: the versioned copies would each need this row removed, since 1.2.0 and earlier genuinely lack the config. Worth noting for completeness that your count is right on the other side too. I verified each of the seven keys against `docs/configurations.md`: six match with one occurrence each, and this one is the sole absentee. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
