This is an automated email from the ASF dual-hosted git repository.
voonhous pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/hudi.git
The following commit(s) were added to refs/heads/asf-site by this push:
new 1950c2a00205 docs(streamer): document checkpoint v2 reset for the Hudi
incremental source (#19573)
1950c2a00205 is described below
commit 1950c2a00205920f11befe32f5a84f619251fe7c
Author: deepakpanda93 <[email protected]>
AuthorDate: Wed Aug 19 14:31:47 2026 +0530
docs(streamer): document checkpoint v2 reset for the Hudi incremental
source (#19573)
* docs(streamer): document checkpoint v2 reset for the Hudi incremental
source
apache/hudi#12718 changed how --checkpoint is interpreted for the Hudi
incremental source once the target table is at table version 8 or higher,
but
the Checkpointing section still only described the Kafka and DFS formats. A
user resetting a checkpoint on such a table hits an IllegalArgumentException
with no documentation to consult.
Verified end to end on Spark 3.5.7 with Hudi 1.2.0, running HoodieStreamer
with HoodieIncrSource against a table-version-9 source table whose four
commits have distinct requested and completion times:
- No override ingests everything and stores streamer.checkpoint.key.v2 =
20260810122008215, the completion time of the last source commit. So the
checkpoint really is completion time based, not request time based.
- --checkpoint 20260810122006588, a bare timestamp, fails with
"Illegal checkpoint key override ... Valid format is either
resumeFromInstantRequestTime:<checkpoint value> or
resumeFromInstantCompletionTime:<checkpoint value>."
- --checkpoint resumeFromInstantRequestTime:20260810122006588 and
--checkpoint resumeFromInstantCompletionTime:20260810122007406 both
succeed and both ingest exactly ids 31,32,41,42, confirming the two forms
name the same boundary and that the request time form is translated to the
matching completion time.
- Both record the value passed under streamer.checkpoint.reset.key.v2
verbatim.
- S3EventsHoodieIncrSource with a bare --checkpoint is not rejected by the
parser at all, confirming the DATASOURCES_NOT_SUPPORTED_WITH_CKPT_V2
exclusion in CheckpointUtils; it fails later on unrelated S3 event schema.
The upgrade guard is documented from
StreamerCheckpointUtils.assertNoCheckpointOverrideDuringUpgradeForHoodieIncSource,
which throws when --checkpoint or --ignore-checkpoint is combined with a
Hudi incremental source on a run that would upgrade or downgrade the table.
Applied to next, 1.2.0, 1.1.1, 1.0.2 and 1.0.1. Not 1.0.0:
HoodieIncrSourceCheckpointValUtils does not exist in that release, so the
prefixed format would be wrong there.
Closes #16813.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
* docs(streamer): name the checkpoint metadata keys correctly, note the
1.0.1 floor
Two corrections after running the end-to-end test across every 1.x release.
Checkpoint metadata key. The section claimed checkpoints are stored as
`streamer.checkpoint.key`, which is not a key Hudi writes. The real ones are
deltastreamer.checkpoint.key / deltastreamer.checkpoint.reset_key
(StreamerCheckpointV1:29-30, written at :55 and :60) and
streamer.checkpoint.key.v2 / streamer.checkpoint.reset.key.v2
(StreamerCheckpointV2:32-33, written at :61 and :66), selected by
CheckpointUtils.shouldTargetCheckpointV2 on the write table version.
Confirmed
by running the streamer both ways against the same source: the default
version-9 target commit carries streamer.checkpoint.key.v2, and a target
forced to hoodie.write.table.version=6 carries
`deltastreamer.checkpoint.key" : "20260810134321000`.
StreamerCheckpointV2 also has addV1Props(), which would add the v1 keys
alongside the v2 ones, so I checked before wording this as an either/or. It
has no callers anywhere in main or test at release-1.2.0, and no commit
produced during testing carried both sets, so the mapping is stated plainly
without hedging.
Version floor. The prefixed --checkpoint form was tested on 1.0.0, 1.0.1,
1.0.2, 1.1.0, 1.1.1 and 1.2.0. It works from 1.0.1 onward, on table version
8
(1.0.1, 1.0.2) and table version 9 (1.1.0, 1.1.1, 1.2.0) alike, and both
prefixes resolve to the same resume point on the earliest and latest of
those.
On 1.0.0 it fails outright with
"DateTimeParseException: Text 'resumeFromInstant' could not be parsed at
index 0", while a bare value is accepted and read as a completion time.
Since
readers on the current docs may be running 1.0.0, the section now states the
1.0.1 floor rather than leaving them to hit that error.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
* docs(streamer): name the checkpoint reset key correctly too
Review feedback on apache/hudi#19573: the existing bullet said --checkpoint
sets `streamer.checkpoint.reset_key`, which does not reconcile with the
`streamer.checkpoint.reset.key.v2` named in the new subsection.
It does not reconcile because that key does not exist either. It has the
same
missing-prefix defect as the checkpoint key line already corrected in
d92a88e38e9c: the real reset keys are deltastreamer.checkpoint.reset_key
(StreamerCheckpointV1:30, written at :60) and
streamer.checkpoint.reset.key.v2 (StreamerCheckpointV2:33, written at :66),
chosen by the same write-table-version test.
Reworded the bullet to name both, mirroring the phrasing used for the
checkpoint key a few lines above.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
* docs(streamer): stop tying the checkpoint key to table version alone
Review feedback on apache/hudi#19573 challenged the two lines I edited in
the
Kafka/DFS paragraph, which said the .v2 keys apply at table version 8 and
above and the deltastreamer.* keys below that. Testing shows that rule is
wrong, though not for the reason given in the review.
The reviewer's mechanism does not match release-1.2.0: StreamSync has no
createCheckpoint() method, and extractCheckpointMetadata
(StreamSync:911-927)
uses whatever Checkpoint object the source returned for the next batch,
falling back to buildCheckpointFromGeneralSource only when the source
supplied none.
What actually decides the key is the source's checkpoint type.
InputBatch(batch, String) wraps the value in StreamerCheckpointV2
unconditionally (InputBatch:43-44), so sources that hand back a plain
string,
such as Kafka and DFS, always produce a V2 checkpoint. Confirmed by running
ParquetDFSSource twice against the same input: the v2 key is written at
table
version 9 and equally at table version 6.
HoodieIncrSource is different because it returns a Checkpoint object whose
type is chosen by table version, which is why the earlier version 6 run of
that source produced deltastreamer.checkpoint.key.
So the split is by source, and only within the Hudi incremental source by
table version. Both lines now name the two keys and say what the choice
depends on, without asserting a version rule that does not hold for Kafka or
DFS. The version specific detail stays in the incremental source subsection
below, where it is backed by tests.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
---------
Co-authored-by: Claude Opus 5 (1M context) <[email protected]>
---
website/docs/hoodie_streaming_ingestion.md | 45 +++++++++++++++++++++-
.../version-1.0.1/hoodie_streaming_ingestion.md | 45 +++++++++++++++++++++-
.../version-1.0.2/hoodie_streaming_ingestion.md | 45 +++++++++++++++++++++-
.../version-1.1.1/hoodie_streaming_ingestion.md | 45 +++++++++++++++++++++-
.../version-1.2.0/hoodie_streaming_ingestion.md | 45 +++++++++++++++++++++-
5 files changed, 215 insertions(+), 10 deletions(-)
diff --git a/website/docs/hoodie_streaming_ingestion.md
b/website/docs/hoodie_streaming_ingestion.md
index 1fb282a5d718..6937ed01d22b 100644
--- a/website/docs/hoodie_streaming_ingestion.md
+++ b/website/docs/hoodie_streaming_ingestion.md
@@ -312,14 +312,55 @@ Read more in depth about concurrency control in the
[concurrency control concept
Hudi Streamer uses checkpoints to keep track of what data has been read
already so it can resume without needing to reprocess all data.
When using a Kafka source, the checkpoint is the [Kafka
Offset](https://cwiki.apache.org/confluence/display/KAFKA/Offset+Management)
When using a DFS source, the checkpoint is the 'last modified' timestamp of
the latest file read.
-Checkpoints are saved in the .hoodie commit file as `streamer.checkpoint.key`.
+Checkpoints are saved in the .hoodie commit file. Completion time checkpoints
are stored under
+`streamer.checkpoint.key.v2` and request time checkpoints under
`deltastreamer.checkpoint.key`. Which one applies
+depends on the source, and for the Hudi incremental source also on the table
version — see below.
If you need to change the checkpoints for reprocessing or replaying data you
can use the following options:
-- `--checkpoint` will set `streamer.checkpoint.reset_key` in the commit file
to overwrite the current checkpoint. Format of checkpoint depends on
[KAFKA_CHECKPOINT_TYPE](configurations.md#hoodiestreamersourcekafkacheckpointtype).
By default (for type `string`), checkpoint should be provided as:
`topicName,0:offset0,1:offset1,2:offset2`. For type `timestamp`, checkpoint
should be provided as long value of desired timestamp. For type
`single_offset`, we assume that topic consists of a sing [...]
+- `--checkpoint` will set the matching reset key in the commit file to
overwrite the current checkpoint, either `streamer.checkpoint.reset.key.v2` or
`deltastreamer.checkpoint.reset_key`. Format of checkpoint depends on
[KAFKA_CHECKPOINT_TYPE](configurations.md#hoodiestreamersourcekafkacheckpointtype).
By default (for type `string`), checkpoint should be provided as:
`topicName,0:offset0,1:offset1,2:offset2`. For type `timestamp`, checkpoint
should be provided as long value of desired ti [...]
- `--source-limit` will set a maximum amount of data to read from the source.
For DFS sources, this is max # of bytes read.
For Kafka, this is the max # of events to read.
+#### Resetting the checkpoint for the Hudi incremental source
+
+When Hudi Streamer writes a target table at table version 8 or higher using
`HoodieIncrSource`, it tracks progress by
+**completion time** instead of requested instant time, and records it in the
commit metadata under
+`streamer.checkpoint.key.v2`. A bare timestamp would be ambiguous between the
two, so `--checkpoint` has to state which
+one it is. This form is required from Hudi 1.0.1 onward; on 1.0.0 the prefixes
are not recognised and `--checkpoint`
+takes a bare completion time.
+
+```shell
+# resume after the instant with this requested instant time
+--checkpoint resumeFromInstantRequestTime:20250110120000000
+
+# resume after the instant with this completion time
+--checkpoint resumeFromInstantCompletionTime:20250110120005000
+```
+
+Both forms resume from the same position. The request time form is translated
internally to the completion time of that
+same instant, and ingestion proceeds in completion time order either way.
Whichever value you pass is recorded verbatim
+under `streamer.checkpoint.reset.key.v2`.
+
+Passing a bare timestamp is rejected:
+
+```plain
+Illegal checkpoint key override `20250110120000000`. Valid format is either
+`resumeFromInstantRequestTime:<checkpoint value>` or
`resumeFromInstantCompletionTime:<checkpoint value>`.
+```
+
+:::note
+`S3EventsHoodieIncrSource` and `GcsEventsHoodieIncrSource` are excluded from
completion time checkpoints. They continue
+to take a plain `--checkpoint` value regardless of the table version.
+:::
+
+:::caution
+Do not pass `--checkpoint` or `--ignore-checkpoint` on the run that upgrades
or downgrades the table version while using
+a Hudi incremental source. Rather than risk applying the override under the
wrong checkpoint semantics, Hudi fails the
+job with a message asking you to drop those options. Let the upgrade finish
first, then reset the checkpoint on a later
+run.
+:::
+
### Transformers
Hudi Streamer supports custom transformation on records before writing to
storage. This is done by supplying
diff --git a/website/versioned_docs/version-1.0.1/hoodie_streaming_ingestion.md
b/website/versioned_docs/version-1.0.1/hoodie_streaming_ingestion.md
index 5f559242e8b9..949e45194cf9 100644
--- a/website/versioned_docs/version-1.0.1/hoodie_streaming_ingestion.md
+++ b/website/versioned_docs/version-1.0.1/hoodie_streaming_ingestion.md
@@ -310,14 +310,55 @@ Read more in depth about concurrency control in the
[concurrency control concept
`HoodieStreamer` uses checkpoints to keep track of what data has been read
already so it can resume without needing to reprocess all data.
When using a Kafka source, the checkpoint is the [Kafka
Offset](https://cwiki.apache.org/confluence/display/KAFKA/Offset+Management)
When using a DFS source, the checkpoint is the 'last modified' timestamp of
the latest file read.
-Checkpoints are saved in the .hoodie commit file as `streamer.checkpoint.key`.
+Checkpoints are saved in the .hoodie commit file. Completion time checkpoints
are stored under
+`streamer.checkpoint.key.v2` and request time checkpoints under
`deltastreamer.checkpoint.key`. Which one applies
+depends on the source, and for the Hudi incremental source also on the table
version — see below.
If you need to change the checkpoints for reprocessing or replaying data you
can use the following options:
-- `--checkpoint` will set `streamer.checkpoint.reset_key` in the commit file
to overwrite the current checkpoint. Format of checkpoint depends on
[KAFKA_CHECKPOINT_TYPE](configurations.md#hoodiestreamersourcekafkacheckpointtype).
By default (for type `string`), checkpoint should be provided as:
`topicName,0:offset0,1:offset1,2:offset2`. For type `timestamp`, checkpoint
should be provided as long value of desired timestamp. For type
`single_offset`, we assume that topic consists of a sing [...]
+- `--checkpoint` will set the matching reset key in the commit file to
overwrite the current checkpoint, either `streamer.checkpoint.reset.key.v2` or
`deltastreamer.checkpoint.reset_key`. Format of checkpoint depends on
[KAFKA_CHECKPOINT_TYPE](configurations.md#hoodiestreamersourcekafkacheckpointtype).
By default (for type `string`), checkpoint should be provided as:
`topicName,0:offset0,1:offset1,2:offset2`. For type `timestamp`, checkpoint
should be provided as long value of desired ti [...]
- `--source-limit` will set a maximum amount of data to read from the source.
For DFS sources, this is max # of bytes read.
For Kafka, this is the max # of events to read.
+#### Resetting the checkpoint for the Hudi incremental source
+
+When Hudi Streamer writes a target table at table version 8 or higher using
`HoodieIncrSource`, it tracks progress by
+**completion time** instead of requested instant time, and records it in the
commit metadata under
+`streamer.checkpoint.key.v2`. A bare timestamp would be ambiguous between the
two, so `--checkpoint` has to state which
+one it is. This form is required from Hudi 1.0.1 onward; on 1.0.0 the prefixes
are not recognised and `--checkpoint`
+takes a bare completion time.
+
+```shell
+# resume after the instant with this requested instant time
+--checkpoint resumeFromInstantRequestTime:20250110120000000
+
+# resume after the instant with this completion time
+--checkpoint resumeFromInstantCompletionTime:20250110120005000
+```
+
+Both forms resume from the same position. The request time form is translated
internally to the completion time of that
+same instant, and ingestion proceeds in completion time order either way.
Whichever value you pass is recorded verbatim
+under `streamer.checkpoint.reset.key.v2`.
+
+Passing a bare timestamp is rejected:
+
+```plain
+Illegal checkpoint key override `20250110120000000`. Valid format is either
+`resumeFromInstantRequestTime:<checkpoint value>` or
`resumeFromInstantCompletionTime:<checkpoint value>`.
+```
+
+:::note
+`S3EventsHoodieIncrSource` and `GcsEventsHoodieIncrSource` are excluded from
completion time checkpoints. They continue
+to take a plain `--checkpoint` value regardless of the table version.
+:::
+
+:::caution
+Do not pass `--checkpoint` or `--ignore-checkpoint` on the run that upgrades
or downgrades the table version while using
+a Hudi incremental source. Rather than risk applying the override under the
wrong checkpoint semantics, Hudi fails the
+job with a message asking you to drop those options. Let the upgrade finish
first, then reset the checkpoint on a later
+run.
+:::
+
### Transformers
`HoodieStreamer` supports custom transformation on records before writing to
storage. This is done by supplying
diff --git a/website/versioned_docs/version-1.0.2/hoodie_streaming_ingestion.md
b/website/versioned_docs/version-1.0.2/hoodie_streaming_ingestion.md
index c922b4831d07..edc61eed2a61 100644
--- a/website/versioned_docs/version-1.0.2/hoodie_streaming_ingestion.md
+++ b/website/versioned_docs/version-1.0.2/hoodie_streaming_ingestion.md
@@ -310,14 +310,55 @@ Read more in depth about concurrency control in the
[concurrency control concept
`HoodieStreamer` uses checkpoints to keep track of what data has been read
already so it can resume without needing to reprocess all data.
When using a Kafka source, the checkpoint is the [Kafka
Offset](https://cwiki.apache.org/confluence/display/KAFKA/Offset+Management)
When using a DFS source, the checkpoint is the 'last modified' timestamp of
the latest file read.
-Checkpoints are saved in the .hoodie commit file as `streamer.checkpoint.key`.
+Checkpoints are saved in the .hoodie commit file. Completion time checkpoints
are stored under
+`streamer.checkpoint.key.v2` and request time checkpoints under
`deltastreamer.checkpoint.key`. Which one applies
+depends on the source, and for the Hudi incremental source also on the table
version — see below.
If you need to change the checkpoints for reprocessing or replaying data you
can use the following options:
-- `--checkpoint` will set `streamer.checkpoint.reset_key` in the commit file
to overwrite the current checkpoint. Format of checkpoint depends on
[KAFKA_CHECKPOINT_TYPE](configurations.md#hoodiestreamersourcekafkacheckpointtype).
By default (for type `string`), checkpoint should be provided as:
`topicName,0:offset0,1:offset1,2:offset2`. For type `timestamp`, checkpoint
should be provided as long value of desired timestamp. For type
`single_offset`, we assume that topic consists of a sing [...]
+- `--checkpoint` will set the matching reset key in the commit file to
overwrite the current checkpoint, either `streamer.checkpoint.reset.key.v2` or
`deltastreamer.checkpoint.reset_key`. Format of checkpoint depends on
[KAFKA_CHECKPOINT_TYPE](configurations.md#hoodiestreamersourcekafkacheckpointtype).
By default (for type `string`), checkpoint should be provided as:
`topicName,0:offset0,1:offset1,2:offset2`. For type `timestamp`, checkpoint
should be provided as long value of desired ti [...]
- `--source-limit` will set a maximum amount of data to read from the source.
For DFS sources, this is max # of bytes read.
For Kafka, this is the max # of events to read.
+#### Resetting the checkpoint for the Hudi incremental source
+
+When Hudi Streamer writes a target table at table version 8 or higher using
`HoodieIncrSource`, it tracks progress by
+**completion time** instead of requested instant time, and records it in the
commit metadata under
+`streamer.checkpoint.key.v2`. A bare timestamp would be ambiguous between the
two, so `--checkpoint` has to state which
+one it is. This form is required from Hudi 1.0.1 onward; on 1.0.0 the prefixes
are not recognised and `--checkpoint`
+takes a bare completion time.
+
+```shell
+# resume after the instant with this requested instant time
+--checkpoint resumeFromInstantRequestTime:20250110120000000
+
+# resume after the instant with this completion time
+--checkpoint resumeFromInstantCompletionTime:20250110120005000
+```
+
+Both forms resume from the same position. The request time form is translated
internally to the completion time of that
+same instant, and ingestion proceeds in completion time order either way.
Whichever value you pass is recorded verbatim
+under `streamer.checkpoint.reset.key.v2`.
+
+Passing a bare timestamp is rejected:
+
+```plain
+Illegal checkpoint key override `20250110120000000`. Valid format is either
+`resumeFromInstantRequestTime:<checkpoint value>` or
`resumeFromInstantCompletionTime:<checkpoint value>`.
+```
+
+:::note
+`S3EventsHoodieIncrSource` and `GcsEventsHoodieIncrSource` are excluded from
completion time checkpoints. They continue
+to take a plain `--checkpoint` value regardless of the table version.
+:::
+
+:::caution
+Do not pass `--checkpoint` or `--ignore-checkpoint` on the run that upgrades
or downgrades the table version while using
+a Hudi incremental source. Rather than risk applying the override under the
wrong checkpoint semantics, Hudi fails the
+job with a message asking you to drop those options. Let the upgrade finish
first, then reset the checkpoint on a later
+run.
+:::
+
### Transformers
`HoodieStreamer` supports custom transformation on records before writing to
storage. This is done by supplying
diff --git a/website/versioned_docs/version-1.1.1/hoodie_streaming_ingestion.md
b/website/versioned_docs/version-1.1.1/hoodie_streaming_ingestion.md
index 286d5765c751..a52dc442bae3 100644
--- a/website/versioned_docs/version-1.1.1/hoodie_streaming_ingestion.md
+++ b/website/versioned_docs/version-1.1.1/hoodie_streaming_ingestion.md
@@ -310,14 +310,55 @@ Read more in depth about concurrency control in the
[concurrency control concept
Hudi Streamer uses checkpoints to keep track of what data has been read
already so it can resume without needing to reprocess all data.
When using a Kafka source, the checkpoint is the [Kafka
Offset](https://cwiki.apache.org/confluence/display/KAFKA/Offset+Management)
When using a DFS source, the checkpoint is the 'last modified' timestamp of
the latest file read.
-Checkpoints are saved in the .hoodie commit file as `streamer.checkpoint.key`.
+Checkpoints are saved in the .hoodie commit file. Completion time checkpoints
are stored under
+`streamer.checkpoint.key.v2` and request time checkpoints under
`deltastreamer.checkpoint.key`. Which one applies
+depends on the source, and for the Hudi incremental source also on the table
version — see below.
If you need to change the checkpoints for reprocessing or replaying data you
can use the following options:
-- `--checkpoint` will set `streamer.checkpoint.reset_key` in the commit file
to overwrite the current checkpoint. Format of checkpoint depends on
[KAFKA_CHECKPOINT_TYPE](configurations.md#hoodiestreamersourcekafkacheckpointtype).
By default (for type `string`), checkpoint should be provided as:
`topicName,0:offset0,1:offset1,2:offset2`. For type `timestamp`, checkpoint
should be provided as long value of desired timestamp. For type
`single_offset`, we assume that topic consists of a sing [...]
+- `--checkpoint` will set the matching reset key in the commit file to
overwrite the current checkpoint, either `streamer.checkpoint.reset.key.v2` or
`deltastreamer.checkpoint.reset_key`. Format of checkpoint depends on
[KAFKA_CHECKPOINT_TYPE](configurations.md#hoodiestreamersourcekafkacheckpointtype).
By default (for type `string`), checkpoint should be provided as:
`topicName,0:offset0,1:offset1,2:offset2`. For type `timestamp`, checkpoint
should be provided as long value of desired ti [...]
- `--source-limit` will set a maximum amount of data to read from the source.
For DFS sources, this is max # of bytes read.
For Kafka, this is the max # of events to read.
+#### Resetting the checkpoint for the Hudi incremental source
+
+When Hudi Streamer writes a target table at table version 8 or higher using
`HoodieIncrSource`, it tracks progress by
+**completion time** instead of requested instant time, and records it in the
commit metadata under
+`streamer.checkpoint.key.v2`. A bare timestamp would be ambiguous between the
two, so `--checkpoint` has to state which
+one it is. This form is required from Hudi 1.0.1 onward; on 1.0.0 the prefixes
are not recognised and `--checkpoint`
+takes a bare completion time.
+
+```shell
+# resume after the instant with this requested instant time
+--checkpoint resumeFromInstantRequestTime:20250110120000000
+
+# resume after the instant with this completion time
+--checkpoint resumeFromInstantCompletionTime:20250110120005000
+```
+
+Both forms resume from the same position. The request time form is translated
internally to the completion time of that
+same instant, and ingestion proceeds in completion time order either way.
Whichever value you pass is recorded verbatim
+under `streamer.checkpoint.reset.key.v2`.
+
+Passing a bare timestamp is rejected:
+
+```plain
+Illegal checkpoint key override `20250110120000000`. Valid format is either
+`resumeFromInstantRequestTime:<checkpoint value>` or
`resumeFromInstantCompletionTime:<checkpoint value>`.
+```
+
+:::note
+`S3EventsHoodieIncrSource` and `GcsEventsHoodieIncrSource` are excluded from
completion time checkpoints. They continue
+to take a plain `--checkpoint` value regardless of the table version.
+:::
+
+:::caution
+Do not pass `--checkpoint` or `--ignore-checkpoint` on the run that upgrades
or downgrades the table version while using
+a Hudi incremental source. Rather than risk applying the override under the
wrong checkpoint semantics, Hudi fails the
+job with a message asking you to drop those options. Let the upgrade finish
first, then reset the checkpoint on a later
+run.
+:::
+
### Transformers
Hudi Streamer supports custom transformation on records before writing to
storage. This is done by supplying
diff --git a/website/versioned_docs/version-1.2.0/hoodie_streaming_ingestion.md
b/website/versioned_docs/version-1.2.0/hoodie_streaming_ingestion.md
index 7886ff13251b..a4958454598a 100644
--- a/website/versioned_docs/version-1.2.0/hoodie_streaming_ingestion.md
+++ b/website/versioned_docs/version-1.2.0/hoodie_streaming_ingestion.md
@@ -312,14 +312,55 @@ Read more in depth about concurrency control in the
[concurrency control concept
Hudi Streamer uses checkpoints to keep track of what data has been read
already so it can resume without needing to reprocess all data.
When using a Kafka source, the checkpoint is the [Kafka
Offset](https://cwiki.apache.org/confluence/display/KAFKA/Offset+Management)
When using a DFS source, the checkpoint is the 'last modified' timestamp of
the latest file read.
-Checkpoints are saved in the .hoodie commit file as `streamer.checkpoint.key`.
+Checkpoints are saved in the .hoodie commit file. Completion time checkpoints
are stored under
+`streamer.checkpoint.key.v2` and request time checkpoints under
`deltastreamer.checkpoint.key`. Which one applies
+depends on the source, and for the Hudi incremental source also on the table
version — see below.
If you need to change the checkpoints for reprocessing or replaying data you
can use the following options:
-- `--checkpoint` will set `streamer.checkpoint.reset_key` in the commit file
to overwrite the current checkpoint. Format of checkpoint depends on
[KAFKA_CHECKPOINT_TYPE](configurations.md#hoodiestreamersourcekafkacheckpointtype).
By default (for type `string`), checkpoint should be provided as:
`topicName,0:offset0,1:offset1,2:offset2`. For type `timestamp`, checkpoint
should be provided as long value of desired timestamp. For type
`single_offset`, we assume that topic consists of a sing [...]
+- `--checkpoint` will set the matching reset key in the commit file to
overwrite the current checkpoint, either `streamer.checkpoint.reset.key.v2` or
`deltastreamer.checkpoint.reset_key`. Format of checkpoint depends on
[KAFKA_CHECKPOINT_TYPE](configurations.md#hoodiestreamersourcekafkacheckpointtype).
By default (for type `string`), checkpoint should be provided as:
`topicName,0:offset0,1:offset1,2:offset2`. For type `timestamp`, checkpoint
should be provided as long value of desired ti [...]
- `--source-limit` will set a maximum amount of data to read from the source.
For DFS sources, this is max # of bytes read.
For Kafka, this is the max # of events to read.
+#### Resetting the checkpoint for the Hudi incremental source
+
+When Hudi Streamer writes a target table at table version 8 or higher using
`HoodieIncrSource`, it tracks progress by
+**completion time** instead of requested instant time, and records it in the
commit metadata under
+`streamer.checkpoint.key.v2`. A bare timestamp would be ambiguous between the
two, so `--checkpoint` has to state which
+one it is. This form is required from Hudi 1.0.1 onward; on 1.0.0 the prefixes
are not recognised and `--checkpoint`
+takes a bare completion time.
+
+```shell
+# resume after the instant with this requested instant time
+--checkpoint resumeFromInstantRequestTime:20250110120000000
+
+# resume after the instant with this completion time
+--checkpoint resumeFromInstantCompletionTime:20250110120005000
+```
+
+Both forms resume from the same position. The request time form is translated
internally to the completion time of that
+same instant, and ingestion proceeds in completion time order either way.
Whichever value you pass is recorded verbatim
+under `streamer.checkpoint.reset.key.v2`.
+
+Passing a bare timestamp is rejected:
+
+```plain
+Illegal checkpoint key override `20250110120000000`. Valid format is either
+`resumeFromInstantRequestTime:<checkpoint value>` or
`resumeFromInstantCompletionTime:<checkpoint value>`.
+```
+
+:::note
+`S3EventsHoodieIncrSource` and `GcsEventsHoodieIncrSource` are excluded from
completion time checkpoints. They continue
+to take a plain `--checkpoint` value regardless of the table version.
+:::
+
+:::caution
+Do not pass `--checkpoint` or `--ignore-checkpoint` on the run that upgrades
or downgrades the table version while using
+a Hudi incremental source. Rather than risk applying the override under the
wrong checkpoint semantics, Hudi fails the
+job with a message asking you to drop those options. Let the upgrade finish
first, then reset the checkpoint on a later
+run.
+:::
+
### Transformers
Hudi Streamer supports custom transformation on records before writing to
storage. This is done by supplying
