This is an automated email from the ASF dual-hosted git repository.
voonhous pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/hudi.git
The following commit(s) were added to refs/heads/asf-site by this push:
new e313ed231e84 docs(cleaning): document that clean metadata records
instant (start) times (#19552)
e313ed231e84 is described below
commit e313ed231e84d98f93052c0b5d90cb96f33add4c
Author: deepakpanda93 <[email protected]>
AuthorDate: Sun Aug 9 13:34:20 2026 +0530
docs(cleaning): document that clean metadata records instant (start) times
(#19552)
* docs(cleaning): document that clean metadata records instant (start) times
Hudi 1.x stamps every action with both a requested instant time and a
completion time and orders the timeline by completion time, but nothing in
the docs said which of the two the cleaner records. A user reading clean
metadata to debug cannot tell whether earliestCommitToRetain is a start time
or a completion time.
It is a start time, on every field, verified at release-1.2.0 and master:
- HoodieCleanerPlan.earliestInstantToRetain is built from
hoodieInstant.requestedTime() (CleanPlanActionExecutor:111 and :177), and
HoodieCleanMetadata.earliestCommitToRetain is copied from the plan
(CleanActionExecutor:172-179 and :265).
- lastCompletedCommitTimestamp is
getCommitTimeline().lastInstant().requestedTime() (CleanPlanner:655-657),
so despite the name it is a start time; the source carries the same note
at CleanPlanActionExecutor:178.
- startCleanTime is inflightInstant.requestedTime() on both the normal and
the empty-clean path (CleanActionExecutor:228-232 and :256).
- Incremental clean planning likewise ranges over instant.requestedTime()
(CleanPlanner:241-245).
Add a section to the Cleaning page tabulating these four fields, and a
matching field table to the Cleaning section of the 1.0 tech spec.
Note this documents behavior as it stands. The parent ticket HUDI-8077,
which would have moved clean metadata to completion time, is resolved
Won't Do and its PR apache/hudi#11972 was closed unmerged.
Applied to next and to version-1.2.0, the current released docs. The tech
spec is served unversioned from the learn plugin, so it has a single copy.
Closes #17274.
* docs(cleaning): name both endpoints of the incremental clean range
Review feedback on apache/hudi#19552: "between the previous clean's
earliestCommitToRetain and the current one" left the upper endpoint to be
inferred.
Name it, and while here state the bounds exactly. CleanPlanner:241-245
filters
completed commits with
requestedTime >= cleanMetadata.getEarliestCommitToRetain()
requestedTime < newInstantToRetain.requestedTime()
so the range is half-open: inclusive of the previous clean's
earliestCommitToRetain, exclusive of this clean's. newInstantToRetain is the
instant returned by CleanPlanner#getEarliestCommitToRetain
(CleanPlanActionExecutor:129), the same one written to the plan as
earliestInstantToRetain (:177), so "this clean's earliestCommitToRetain"
names
it accurately.
Applied to both cleaning.md copies and the tech spec, which carried the same
sentence.
* docs(cleaning): note the two clean-metadata fields that read wrong
Two values in the new clean-metadata table are easy to misread, and neither
is visible in a single-writer reproduction.
lastCompletedCommitTimestamp mixes the two orderings. CleanPlanner:655-656
is getCommitTimeline().lastInstant().map(HoodieInstant::requestedTime), and
getCommitTimeline() (CleanPlanner:121-126) is getCompletedCommitsTimeline(),
which 1.x orders by completion time. So the instant is picked by completion
order and then recorded as a start time. Given commits A (requested t1,
completed t4) and B (requested t2, completed t3) with t1 < t2 < t3 < t4, the
field holds t1 even though the completed commit B started at t2. Saying "the
last completed write" invited reading it as the largest requested time.
earliestCommitToRetain can be an empty string. CleanerUtils:126-156 has no
branch for KEEP_LATEST_FILE_VERSIONS, so getEarliestCommitToRetain() is
empty
under that policy, and CleanPlanActionExecutor:177 stores null in the plan.
That policy still cleans - getPartitionPathsToClean falls through to
getPartitionPathsForFullCleaning (CleanPlanner:161-165) - so the run
produces
clean stats, CleanActionExecutor:184 substitutes "" for the missing instant,
and CleanerUtils.convertCleanMetadata:93 copies it straight into
HoodieCleanMetadata. A reader debugging retention on that policy sees an
empty field and no explanation.
Applied to both cleaning.md copies as a short list under the table, and
folded into the corresponding tech spec rows.
---------
Co-authored-by: voon <[email protected]>
---
website/docs/cleaning.md | 29 +++++++++++++++++++++++-
website/learn/tech-specs.md | 14 +++++++++++-
website/versioned_docs/version-1.2.0/cleaning.md | 29 +++++++++++++++++++++++-
3 files changed, 69 insertions(+), 3 deletions(-)
diff --git a/website/docs/cleaning.md b/website/docs/cleaning.md
index fa92192f04eb..936944347668 100644
--- a/website/docs/cleaning.md
+++ b/website/docs/cleaning.md
@@ -3,7 +3,7 @@ title: Cleaning
toc: true
toc_min_heading_level: 2
toc_max_heading_level: 4
-last_modified_at: 2026-05-27T00:00:00-00:00
+last_modified_at: 2026-08-06T15:17:36+05:30
---
## Background
Cleaning is a table service employed by Hudi to reclaim space occupied by
older versions of data and keep storage costs
@@ -86,6 +86,33 @@ takes precedence over the regex.
| `hoodie.clean.partition.filter.regex` | (none) | Java regex pattern; only
partitions whose path matches are cleaned. |
| `hoodie.clean.partition.filter.selected` | (none) | Comma-separated list of
partition paths to clean; takes precedence over the regex when both are set. |
+### Instant Times in Clean Metadata
+
+Hudi 1.x stamps every action with both a requested instant time and a
completion time, and orders actions on the
+timeline by completion time — see [timeline](timeline.md). The cleaner's own
plan and metadata, however, record
+**instant (start) times** throughout. Keep this in mind when reading them for
debugging.
+
+| Field | Written to | Value |
+|---|---|---|
+| `earliestInstantToRetain.timestamp` | `HoodieCleanerPlan` (the
`clean.requested` instant) | Instant time of the oldest commit this clean run
retains. |
+| `earliestCommitToRetain` | `HoodieCleanMetadata` (the completed `clean`
instant) | Copied from the plan, so also an instant time. |
+| `lastCompletedCommitTimestamp` | both | Instant time of the write that
completed most recently before the clean was planned. Despite the name, this is
a start time, not a completion time. |
+| `startCleanTime` | `HoodieCleanMetadata` | Instant time of the clean action
itself. |
+
+Two details are easy to trip over when reading these values back:
+
+- `lastCompletedCommitTimestamp` mixes the two orderings. The instant is taken
from the end of the completed-commits
+ timeline, which is ordered by completion time, but what gets recorded is
that instant's start time. Concurrent writers
+ can complete in a different order than they started in, so this is not
always the largest instant time among the
+ completed commits.
+- `earliestCommitToRetain` is an empty string under the
`KEEP_LATEST_FILE_VERSIONS` policy. That policy retains a fixed
+ number of file versions per file group rather than a range of the timeline,
so the plan carries no
+ `earliestInstantToRetain` for the metadata to copy.
+
+Incremental clean planning follows the same convention: it selects the commits
whose **requested** instant time is at or
+after the previous clean's `earliestCommitToRetain` and before this clean's,
then scans only the partitions those
+commits touched.
+
### Configs
For details about all possible configurations and their default values see the
[configuration
docs](https://hudi.apache.org/docs/next/configurations/#Clean-Configs).
For Flink related configs refer
[here](https://hudi.apache.org/docs/next/configurations/#FLINK_SQL).
diff --git a/website/learn/tech-specs.md b/website/learn/tech-specs.md
index 0f8d3acd2acc..f542c4c5b651 100644
--- a/website/learn/tech-specs.md
+++ b/website/learn/tech-specs.md
@@ -2,7 +2,7 @@
| **Syntax** | **Description** |
| ---|-----------------|
-| Last Updated | Jul 2026 |
+| Last Updated | Aug 2026 |
| [Table
Version](https://github.com/apache/hudi/blob/master/hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableVersion.java)
| 9 |
| Reflects release | Hudi 1.2.0 (May 2026) |
@@ -698,6 +698,18 @@ For e.g, there are a couple of retention policies
supported in Apache Hudi platf
Apache Hudi provides snapshot isolation between writers and readers by
managing multiple files with MVCC concurrency. These file versions provide
history
and enable time travel and rollbacks, but it is important to manage how much
history you keep to balance your storage costs.
+Cleaning is planned and tracked in terms of **instant (start) times**, even
though actions on the timeline are ordered
+by completion time. The fields below all hold the requested instant time of
the action they refer to.
+
+| Field | Description |
+|---|---|
+| earliestInstantToRetain | Instant time, action and state of the oldest
commit the clean run retains. Held in `HoodieCleanerPlan`, and copied to
`HoodieCleanMetadata` as `earliestCommitToRetain`. Absent under
`KEEP_LATEST_FILE_VERSIONS`, which retains a fixed number of file versions per
file group rather than a range of the timeline; `earliestCommitToRetain` is
then an empty string |
+| lastCompletedCommitTimestamp | Instant time of the write that completed most
recently before the clean was planned. Despite the name, this is a start time.
The instant is chosen by completion order, so with concurrent writers it is not
always the largest instant time among completed commits |
+| startCleanTime | Instant time of the clean action itself, in
`HoodieCleanMetadata` |
+
+Incremental clean planning selects commits whose requested instant time is at
or after the previous clean's
+`earliestCommitToRetain` and before this clean's, and scans only the
partitions those commits touched.
+
### Indexing
diff --git a/website/versioned_docs/version-1.2.0/cleaning.md
b/website/versioned_docs/version-1.2.0/cleaning.md
index 5cc6a8726469..4e1707ed03e3 100644
--- a/website/versioned_docs/version-1.2.0/cleaning.md
+++ b/website/versioned_docs/version-1.2.0/cleaning.md
@@ -3,7 +3,7 @@ title: Cleaning
toc: true
toc_min_heading_level: 2
toc_max_heading_level: 4
-last_modified_at: 2026-05-27T00:00:00-00:00
+last_modified_at: 2026-08-06T15:17:36+05:30
---
## Background
Cleaning is a table service employed by Hudi to reclaim space occupied by
older versions of data and keep storage costs
@@ -86,6 +86,33 @@ takes precedence over the regex.
| `hoodie.clean.partition.filter.regex` | (none) | Java regex pattern; only
partitions whose path matches are cleaned. |
| `hoodie.clean.partition.filter.selected` | (none) | Comma-separated list of
partition paths to clean; takes precedence over the regex when both are set. |
+### Instant Times in Clean Metadata
+
+Hudi 1.x stamps every action with both a requested instant time and a
completion time, and orders actions on the
+timeline by completion time — see [timeline](timeline.md). The cleaner's own
plan and metadata, however, record
+**instant (start) times** throughout. Keep this in mind when reading them for
debugging.
+
+| Field | Written to | Value |
+|---|---|---|
+| `earliestInstantToRetain.timestamp` | `HoodieCleanerPlan` (the
`clean.requested` instant) | Instant time of the oldest commit this clean run
retains. |
+| `earliestCommitToRetain` | `HoodieCleanMetadata` (the completed `clean`
instant) | Copied from the plan, so also an instant time. |
+| `lastCompletedCommitTimestamp` | both | Instant time of the write that
completed most recently before the clean was planned. Despite the name, this is
a start time, not a completion time. |
+| `startCleanTime` | `HoodieCleanMetadata` | Instant time of the clean action
itself. |
+
+Two details are easy to trip over when reading these values back:
+
+- `lastCompletedCommitTimestamp` mixes the two orderings. The instant is taken
from the end of the completed-commits
+ timeline, which is ordered by completion time, but what gets recorded is
that instant's start time. Concurrent writers
+ can complete in a different order than they started in, so this is not
always the largest instant time among the
+ completed commits.
+- `earliestCommitToRetain` is an empty string under the
`KEEP_LATEST_FILE_VERSIONS` policy. That policy retains a fixed
+ number of file versions per file group rather than a range of the timeline,
so the plan carries no
+ `earliestInstantToRetain` for the metadata to copy.
+
+Incremental clean planning follows the same convention: it selects the commits
whose **requested** instant time is at or
+after the previous clean's `earliestCommitToRetain` and before this clean's,
then scans only the partitions those
+commits touched.
+
### Configs
For details about all possible configurations and their default values see the
[configuration
docs](https://hudi.apache.org/docs/next/configurations/#Clean-Configs).
For Flink related configs refer
[here](https://hudi.apache.org/docs/next/configurations/#FLINK_SQL).
