This is an automated email from the ASF dual-hosted git repository.
voonhous pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/hudi.git
The following commit(s) were added to refs/heads/asf-site by this push:
new 67e61754674b docs(cleaning): document partition TTL (#19764)
67e61754674b is described below
commit 67e61754674b7356bb8aa05b85fbf24dcf730693
Author: deepakpanda93 <[email protected]>
AuthorDate: Mon Aug 31 18:21:41 2026 +0530
docs(cleaning): document partition TTL (#19764)
* docs(cleaning): document partition TTL
Closes #19141. Partition TTL has existed since 1.0.0 but appears nowhere in
prose
on the site; the only trace is the generated config reference. This adds a
Partition TTL section to the cleaning page, which is the natural home since
that
page already covers retention, and TTL is the partition-level counterpart
to the
cleaner's file-version retention.
All configs and behaviour read from HoodieTTLConfig, KeepByTimeStrategy,
KeepByCreationTimeStrategy and PartitionTTLStrategy on master.
The most important thing the config reference cannot convey is that
hoodie.partition.ttl.strategy.days.retain defaults to -1, and
KeepByTimeStrategy#getExpiredPartitionPaths returns an empty list whenever
the
resulting retention is zero or negative. TTL therefore does nothing at all
until
a positive retention is set, even when enabled. That gets a caution, along
with
the two other silent no-ops: a table with no completed commit, and an
unpartitioned table.
Documents both built-in strategies rather than only the default, because
they age
a partition against different timestamps: KEEP_BY_TIME uses the partition's
last
commit time, so an actively written partition survives, while
KEEP_BY_CREATION_TIME uses the partition's created commit time, so writing
to it
does not extend its life. Also notes that
hoodie.partition.ttl.strategy.class takes precedence over the strategy type
when
both are set.
Covers all three ways to run TTL, not just the inline config: inline after
each
commit, the standalone HoodieTTLJob utility, and the run_ttl Spark SQL
procedure,
which is cross-referenced to its existing entry on the procedures page
rather
than duplicated. However it is triggered, TTL lands as a replace commit,
the same
commit type delete_partition uses.
Groups the three bounding configs around the first run, which is where they
matter, since every historical partition becomes a candidate at once:
max.delete.partitions caps a single run at 1000 partitions so one replace
commit
cannot grow unmanageably large and a backlog drains over several runs;
partition.selected restricts the candidate set and is the safe way to trial
a
policy; and stats.max.parallelism bounds the parallelism for collecting
candidate
commit times.
Note the key names carry the strategy prefix, so the cap is
hoodie.partition.ttl.strategy.max.delete.partitions rather than the shorter
form
the ticket discussion used.
Scoped to website/docs only. stats.max.parallelism is new in 1.3.0 and
absent
from release-1.2.0, so versioned copies would need it removed; the next
docs are
the only place every config described here exists.
The worked example and the standalone-job invocation use a local file:///tmp
base path with tableName and basePath vals, matching the convention the
quick start guide already uses, rather than an s3a bucket path a reader
cannot
run as written. All three examples in the section refer to the same table
name,
events_table, so a reader can follow them in sequence.
npm run build passes with the warning set byte-identical to a baseline
built at
the same base commit, which also confirms the new procedures.md#run_ttl
anchor
resolves.
* docs(cleaning): flag stats.max.parallelism as 1.3.0 and simplify the TTL
job command
Review feedback on apache/hudi#19764.
The reviewer noticed that
hoodie.partition.ttl.strategy.stats.max.parallelism is
the one config in the section that does not appear in the generated
configurations.md, and asked whether the key or default might be wrong.
They are
not: HoodieTTLConfig on master defines it as
PARTITION_TTL_STRATEGY_PARAM_PREFIX + "stats.max.parallelism" with
defaultValue
200, and KeepByTimeStrategy#getLastCommitTimeForPartitions computes
Math.min(partitionPaths.size(),
writeConfig.getPartitionTTLStatsMaxParallelism()),
which is the "smaller of that and the candidate count" claim.
The reason it is missing from the reference is that it carries
sinceVersion("1.3.0") and is absent from release-1.2.0, while the generated
reference reflects a released version. That is worth telling the reader
rather
than leaving them to wonder, since a user on 1.2.0 could set the key and
see no
effect. The prose now says it is new in 1.3.0 and not yet in the generated
reference, and the config table row is marked "Since 1.3.0".
The standalone job command pulled the utilities slim bundle through
--packages
while also passing a locally built slim bundle jar as the application jar,
which
supplies the same artifact twice and is only runnable by someone who has
built
Hudi. It now passes the self-contained utilities bundle as the application
jar
with no --packages at all, and says where to get it. This diverges from the
HoodieCleaner examples further up the page, which have the same redundancy;
those
are left alone as out of scope for this PR.
Build passes with the warning set still byte-identical to a baseline at the
same
base commit.
* docs(cleaning): make the TTL example's partition paths match its layout
Review feedback on apache/hudi#19764, and a real defect rather than a
wording
nit: the example would have deleted nothing.
It set partition.selected to event_date=2026-01-01, a Hive-style partition
path,
without enabling Hive-style partitioning. KeyGeneratorOptions
HIVE_STYLE_PARTITIONING_ENABLE defaults to "false", and its own
documentation
says "By default false (the names of partition folders are only partition
values)", so the folder on storage would have been 2026-01-01.
That mismatch matters because the value is not normalised anywhere.
PartitionTTLStrategy#getPartitionPathsForTTL does
Arrays.asList(partitionSelected.split(",")) and uses the result as the
partition
list directly, so an entry that does not correspond to a real folder simply
selects a partition that is not there. TTL then completes and deletes
nothing,
which is the same silent no-op the section already warns about for
days.retain.
The example now sets hoodie.datasource.write.hive_style_partitioning to
true, so
the folder layout matches the selected value, and the partition.selected
paragraph explains that the list is used verbatim, that the layout depends
on
that config, and what each of the two forms looks like.
Build passes with the warning set byte-identical to a baseline built at the
same
base commit.
---
website/docs/cleaning.md | 134 +++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 134 insertions(+)
diff --git a/website/docs/cleaning.md b/website/docs/cleaning.md
index 936944347668..2cf925e7774e 100644
--- a/website/docs/cleaning.md
+++ b/website/docs/cleaning.md
@@ -221,6 +221,140 @@ cleans run --sparkMaster local --hoodieConfigs
hoodie.clean.policy=KEEP_LATEST_C
You can find more details and the relevant code for these commands in
[`org.apache.hudi.cli.commands.CleansCommand`](https://github.com/apache/hudi/blob/master/hudi-cli/src/main/java/org/apache/hudi/cli/commands/CleansCommand.java)
class.
+## Partition TTL
+
+Cleaning bounds how many *versions* of a file are kept, but it never removes a
partition: an old partition whose files
+have all been cleaned down to a single version still sits in the table
forever. Partition TTL (time to live) is the
+complementary service. It works at partition granularity, and when a partition
is judged expired it deletes the whole
+partition rather than trimming file versions inside it.
+
+Because it removes data outright, TTL is off by default and stays off until
you set a retention period.
+
+### How a partition is judged expired
+
+TTL asks a strategy which partitions have expired. Two strategies ship with
Hudi, selected through
+`hoodie.partition.ttl.management.strategy.type`:
+
+| Strategy | Ages a partition against |
+|---|---|
+| `KEEP_BY_TIME` (default) | The partition's last commit time, taken from the
newest base instant among its latest file slices. A partition that is still
being written to therefore stays. |
+| `KEEP_BY_CREATION_TIME` | The commit time the partition was created at, read
from its partition metadata. Writing to a partition does not extend its life. |
+
+Both compare that timestamp against
`hoodie.partition.ttl.strategy.days.retain`. A custom strategy can be supplied
+instead with `hoodie.partition.ttl.strategy.class`, pointing at a subclass of
`PartitionTTLStrategy`; when both configs
+are present the class takes precedence over the type.
+
+:::caution
+`hoodie.partition.ttl.strategy.days.retain` defaults to `-1`, and the built-in
strategies treat any value of `0` or less
+as "nothing expires". **TTL does nothing at all until you set a positive
retention, even with TTL enabled.** This is
+deliberate, so that turning the service on cannot delete data by itself, but
it does mean a misconfigured job looks like
+a working one: it runs, reports no expired partitions, and deletes nothing.
+:::
+
+Two other conditions make TTL a silent no-op regardless of retention: a table
with no completed commit yet, and an
+unpartitioned table.
+
+### Ways to run partition TTL
+
+**Inline.** Setting `hoodie.partition.ttl.inline=true` runs TTL immediately
after each commit, alongside the other inline
+table services.
+
+**As a standalone Spark job.** `org.apache.hudi.utilities.HoodieTTLJob`, in
the utilities bundle, runs TTL against an
+existing table without enabling it on the writer:
+
+```
+spark-submit --master local \
+ --class org.apache.hudi.utilities.HoodieTTLJob \
+ hudi-utilities-bundle_2.12-1.2.0.jar \
+ --base-path file:///tmp/events_table \
+ --hoodie-conf hoodie.partition.ttl.strategy.days.retain=30
+```
+
+The utilities bundle is self-contained, so it is passed as the application jar
and no `--packages` is needed. Download it
+from Maven Central, or build it locally and point at
+`packaging/hudi-utilities-bundle/target/hudi-utilities-bundle_2.12-*.jar`.
+
+**From Spark SQL**, with the [`run_ttl`](procedures.md#run_ttl) procedure,
which is the easiest way to try TTL on a table
+before committing to running it on every write:
+
+```sql
+call run_ttl(table => 'events_table', retain_days => 30);
+```
+
+However it is triggered, TTL writes a replace commit that drops the expired
partitions, the same commit type used by the
+`delete_partition` operation.
+
+### Keeping a first run under control
+
+The first TTL run on an existing table is the risky one, because every
historical partition becomes a candidate at once.
+Three configs bound it.
+
+`hoodie.partition.ttl.strategy.max.delete.partitions` caps how many partitions
a single run may delete, defaulting to
+`1000`. The limit exists to keep one replace commit from growing unmanageably
large; partitions over the cap are simply
+left for the next run, so a backlog drains across several runs rather than in
one commit.
+
+`hoodie.partition.ttl.strategy.partition.selected` takes a comma-separated
list of partition paths and restricts TTL to
+exactly those. When it is unset, TTL considers every partition in the table.
Setting it is the safest way to try a
+retention policy on one partition before applying it everywhere.
+
+The list is used verbatim, so each entry has to match the partition path as it
exists on storage. That layout depends on
+`hoodie.datasource.write.hive_style_partitioning`: with the default of `false`
a partition folder is named for the value
+alone, such as `2026-01-01`, while with it enabled the folder is
`event_date=2026-01-01`. Passing the wrong form selects a
+partition that does not exist, and TTL then deletes nothing while still
reporting success.
+
+`hoodie.partition.ttl.strategy.stats.max.parallelism` bounds the parallelism
used to collect each candidate partition's
+last commit time, defaulting to `200`; the effective value is the smaller of
that and the candidate count. It matters
+mainly on that first run, where a table with many historical partitions may
want a higher value. This config is new in
+1.3.0, so it has no effect on earlier releases and does not yet appear in the
generated
+[configuration reference](https://hudi.apache.org/docs/next/configurations/);
the other six configs above do.
+
+### Partition TTL configs
+
+| Config | Default | Description |
+|---|---|---|
+| `hoodie.partition.ttl.inline` | `false` | Run TTL immediately after each
commit |
+| `hoodie.partition.ttl.management.strategy.type` | `KEEP_BY_TIME` |
`KEEP_BY_TIME` or `KEEP_BY_CREATION_TIME` |
+| `hoodie.partition.ttl.strategy.class` | none | A `PartitionTTLStrategy`
subclass; takes precedence over the type above |
+| `hoodie.partition.ttl.strategy.days.retain` | `-1` | Days to retain. Nothing
expires while this is `0` or less |
+| `hoodie.partition.ttl.strategy.partition.selected` | none | Comma-separated
partition paths to restrict TTL to |
+| `hoodie.partition.ttl.strategy.max.delete.partitions` | `1000` | Maximum
partitions deleted in one run |
+| `hoodie.partition.ttl.strategy.stats.max.parallelism` | `200` | Parallelism
for collecting candidate partition commit times. Since 1.3.0 |
+
+### A worked example
+
+Retaining 30 days on a date-partitioned event table, run inline, restricted on
the first pass to a single partition so
+the effect can be checked before it is applied to the whole table:
+
+```scala
+val tableName = "events_table"
+val basePath = "file:///tmp/events_table"
+
+df.write.format("hudi")
+ .option("hoodie.table.name", tableName)
+ .option("hoodie.datasource.write.recordkey.field", "event_id")
+ .option("hoodie.datasource.write.partitionpath.field", "event_date")
+ // partition folders are named event_date=<value>, which partition.selected
below must match
+ .option("hoodie.datasource.write.hive_style_partitioning", "true")
+ // enable TTL and give it a retention, without which it does nothing
+ .option("hoodie.partition.ttl.inline", "true")
+ .option("hoodie.partition.ttl.management.strategy.type", "KEEP_BY_TIME")
+ .option("hoodie.partition.ttl.strategy.days.retain", "30")
+ // first pass: one partition only
+ .option("hoodie.partition.ttl.strategy.partition.selected",
"event_date=2026-01-01")
+ .mode("append")
+ .save(basePath)
+```
+
+Once the deleted partitions look right, drop the `partition.selected` line to
let TTL consider the whole table. On a
+table with a long history, expect the backlog to drain over several commits
because of the
+`max.delete.partitions` cap.
+
+:::caution
+Partition TTL deletes data. A partition removed by TTL is gone from the table
as of that replace commit, recoverable only
+for as long as the cleaner and archival have not yet removed the file versions
and timeline entries a time travel query
+would need. Validate a retention policy with `run_ttl`, or with
`partition.selected`, before enabling it inline.
+:::
+
## Related Resources
<h3>Blogs</h3>
