deepakpanda93 opened a new pull request, #19764:
URL: https://github.com/apache/hudi/pull/19764
### Describe the issue this Pull Request addresses
Closes #19141.
Partition TTL has existed since 1.0.0, but it appears **nowhere** in prose
on the site. The only trace is the generated
config reference, which lists the keys and defaults without saying what they
do together. Confirmed the gap rather than
assuming it: `partition.ttl` matches only `configurations.md` across
`website/docs`, `website/versioned_docs` and
`website/learn`.
### Summary and Changelog
One file, `website/docs/cleaning.md` (+121). A **Partition TTL** section is
added before Related Resources.
`cleaning.md` is the natural home: that page is already about retention, and
TTL is the partition-level counterpart to
the cleaner's file-version retention. The section opens on exactly that
contrast, since it is the thing most likely to
confuse someone who has already read the cleaner docs:
> Cleaning bounds how many *versions* of a file are kept, but it never
removes a partition: an old partition whose files
> have all been cleaned down to a single version still sits in the table
forever.
<!-- SCREENSHOT 1 HERE: Partition TTL overview + How a partition is judged
expired -->
### The thing a config table cannot tell you
`hoodie.partition.ttl.strategy.days.retain` defaults to `-1`, and
`KeepByTimeStrategy#getExpiredPartitionPaths` returns
an empty list whenever the resulting retention is zero or less:
```java
if (!lastCompletedInstant.isPresent() || ttlInMilis <= 0
||
!hoodieTable.getMetaClient().getTableConfig().getPartitionFields().isPresent())
{
return Collections.emptyList();
}
```
So **TTL does nothing at all until a positive retention is set, even when
enabled.** That gets a `:::caution`, phrased
around the failure mode rather than the default value: a misconfigured job
looks like a working one, because it runs,
reports no expired partitions, and deletes nothing.
That same guard shows two further silent no-ops which the section also
states: a table with no completed commit, and an
**unpartitioned** table.
### What the section covers beyond the configs
**Both built-in strategies, not just the default.** They age a partition
against different timestamps, which is the
practically important difference:
| Strategy | Ages against |
|---|---|
| `KEEP_BY_TIME` (default) | The partition's last commit time, so an
actively written partition survives |
| `KEEP_BY_CREATION_TIME` | The partition's created commit time, so writing
to it does not extend its life |
Also that `hoodie.partition.ttl.strategy.class` takes precedence over the
strategy type when both are set
(`PartitionTTLStrategyType#getPartitionTTLStrategyClassName`).
**All three ways to run TTL, not only the inline config.** Inline after each
commit, the standalone
`org.apache.hudi.utilities.HoodieTTLJob`, and the `run_ttl` Spark SQL
procedure. The procedure is cross-referenced to its
existing entry on the procedures page rather than duplicated. Whichever path
is used, TTL lands as a replace commit, the
same commit type `delete_partition` uses (`startDeletePartitionCommit` then
`managePartitionTTL`, committed with
`REPLACE_COMMIT_ACTION`).
**The three bounding configs framed around the first run**, which is where
they matter, since every historical partition
becomes a candidate at once. `max.delete.partitions` caps a run at 1000 so
one replace commit cannot grow unmanageably
large, and the code comment says exactly that ("Avoid a single replace
commit too large") — a backlog therefore drains
over several runs. `partition.selected` restricts the candidate set and is
the safe way to trial a policy.
`stats.max.parallelism` bounds the parallelism for collecting candidate
commit times.
<!-- SCREENSHOT 2 HERE: Ways to run partition TTL + Keeping a first run
under control -->
**A config table and a worked example**, the example following the quick
start guide's own `val tableName` /
`val basePath = "file:///tmp/..."` convention so it is runnable as written,
and restricting the first pass to one
partition so the effect can be checked before it applies to the whole table.
A closing caution states plainly that TTL
deletes data.
<!-- SCREENSHOT 3 HERE: Partition TTL configs + A worked example -->
### One correction worth flagging
The max-partitions config is
**`hoodie.partition.ttl.strategy.max.delete.partitions`**, not the shorter
`hoodie.partition.ttl.max.partitions.to.delete` form that circulates in
discussion of this feature. Everything defined
through `PARTITION_TTL_STRATEGY_PARAM_PREFIX` carries the `strategy.`
prefix, so `days.retain`, `partition.selected`,
`max.delete.partitions` and `stats.max.parallelism` all sit under it. Note
also the asymmetry in Hudi's own naming, which
the section reproduces faithfully rather than tidying: the strategy **type**
is
`hoodie.partition.ttl.management.strategy.type` while the strategy **class**
is `hoodie.partition.ttl.strategy.class`.
### Version scope
`website/docs` only, deliberately.
`hoodie.partition.ttl.strategy.stats.max.parallelism` is new in 1.3.0 (#19137)
and is
**absent from release-1.2.0**, verified against the tag. The next docs are
therefore the only place where every config
described here exists; versioned copies would each need that row removed,
which I would rather do as a follow-up if
reviewers want it than ship as six near-duplicate sections.
### Verification
Everything read from **master**, not from the ticket: `HoodieTTLConfig` for
the keys, defaults and since-versions,
`KeepByTimeStrategy` and `KeepByCreationTimeStrategy` for expiry semantics
and the no-op guard, `PartitionTTLStrategy`
for candidate selection, `BaseHoodieTableServiceClient` for the inline
trigger, `HoodieTTLJob` for the CLI parameters,
and `RunTTLProcedure` for the procedure name and its config mapping.
`npm run build` passes with the warning block **byte-identical** to a
baseline built from the same base commit
(`5971a1ac3ba3`), 13,265 lines each. That parity also confirms the new
`procedures.md#run_ttl` cross-reference resolves,
since a dangling anchor would have added a warning. Rendering checked under
`npm run serve`: all six headings resolve
with unique anchors and 12 TOC entries, both cautions render, and the
heading is `Partition TTL configs` rather than a
second `Configs`, which would have collided with the existing cleaning `###
Configs` and produced a `#configs-1` anchor.
One limitation stated plainly: this is verified by reading the code, not by
running TTL against a table. The claim I would
most like a committer to confirm is the practical consequence of the
`days.retain` default, since that is the line
readers will act on.
### Impact
Documentation only. No code, config, or behaviour change.
### Risk Level
none
### Documentation Update
This PR is the documentation update — the cleaning page,
`/docs/next/cleaning#partition-ttl`.
### Contributor's checklist
- [x] Read through [contributor's
guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Enough context is provided in the sections above
- [x] Adequate tests were added if applicable
cc @wangxianghu (who filed the issue), @yihua
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
