deepakpanda93 opened a new pull request, #19573:
URL: https://github.com/apache/hudi/pull/19573
### Describe the issue this Pull Request addresses
Closes #16813. (JIRA: HUDI-8927.)
#12718 changed how `--checkpoint` is interpreted for the Hudi incremental
source once the target table is at table
version 8 or higher: the value must now be prefixed to say whether it is a
requested instant time or a completion time.
The **Checkpointing** section of the Hudi Streamer page still described only
the Kafka and DFS formats, so a user
resetting a checkpoint on such a table hits
```
Illegal checkpoint key override `20250110120000000`. Valid format is either
`resumeFromInstantRequestTime:<checkpoint value>` or
`resumeFromInstantCompletionTime:<checkpoint value>`.
```
with no documentation to consult. The PR that introduced this asked for the
docs follow-up in its own Documentation
Update section, which is what this addresses.
### Summary and Changelog
Added a `#### Resetting the checkpoint for the Hudi incremental source`
subsection under **Checkpointing** covering the
two prefixed forms, the error you get without a prefix, the S3/GCS
exclusion, the version floor, and the upgrade
restriction.
Also corrected an existing inaccuracy in the same section, described below.
### Reproduction — all six 1.x releases
Ran real `HoodieStreamer` jobs on Spark 3.5.7, each release using its own
`hudi-spark3.5-bundle` and
`hudi-utilities-slim-bundle`, with a source table built by that same
release's writer. The reset boundary is source
instant #2; the bare case was given that instant's requested time, and the
prefixed cases named the same instant by
requested time and by completion time.
| Version | Table ver | bare value | `resumeFromInstantRequestTime:` |
`resumeFromInstantCompletionTime:` | Rows ingested (prefixed) |
|---|---|---|---|---|---|
| **1.0.0** | 8 | :white_check_mark: accepted | :x: **fails** | :x:
**fails** | — |
| 1.0.1 | 8 | :x: rejected | :white_check_mark: | :white_check_mark: |
`31,32,41,42` (identical) |
| 1.0.2 | 8 | :x: rejected | :white_check_mark: | :white_check_mark: |
:white_check_mark: |
| 1.1.0 | 9 | :x: rejected | :white_check_mark: | :white_check_mark: |
:white_check_mark: |
| 1.1.1 | 9 | :x: rejected | :white_check_mark: | :white_check_mark: |
:white_check_mark: |
| 1.2.0 | 9 | :x: rejected | :white_check_mark: | :white_check_mark: |
`31,32,41,42` (identical) |
Every rejection is the `Illegal checkpoint key override` message above.
Every success wrote
`streamer.checkpoint.key.v2` plus `streamer.checkpoint.reset.key.v2`, the
latter holding the prefixed value verbatim.
Three things this settled that source reading alone would not have:
1. **The behaviour is completion time based today.** #12718's own note said
*"the hoodie incremental sources are still
using request time based handling in 1.x"*. That is no longer true: a run
with no override stored
`streamer.checkpoint.key.v2 = 20260810122008215`, the completion time of
the last source commit. Documenting from the
PR description alone would have got this backwards.
2. **Both prefixes name the same boundary.** They produced byte-identical
row sets on the earliest and the latest
affected release, confirming the requested time form is translated to
that instant's completion time rather than
being a separate mode.
3. **1.0.0 is not simply missing the feature, it behaves oppositely.** The
prefixed form dies with
`DateTimeParseException: Text 'resumeFromInstant' could not be parsed at
index 0`, while a bare value is accepted and
read as a *completion* time — so passing a requested time there silently
resumes from the wrong place (six records
ingested where 1.0.1+ ingests four). That ambiguity is exactly what
#12718 removed, and it is why the new text states
a 1.0.1 floor.
Also verified the `S3EventsHoodieIncrSource` exclusion: with a bare
`--checkpoint` the prefix parser never engages
(zero rejections) and the job proceeds to fail later on unrelated S3 event
schema, matching
`CheckpointUtils.DATASOURCES_NOT_SUPPORTED_WITH_CKPT_V2`.
### Correcting the checkpoint metadata key
The section said checkpoints are saved as `streamer.checkpoint.key`, which
is not a key Hudi writes. The actual keys are
| Checkpoint | Key | Defined at |
|---|---|---|
| request time | `deltastreamer.checkpoint.key`,
`deltastreamer.checkpoint.reset_key` | `StreamerCheckpointV1.java:29-30`,
written at `:55`/`:60` |
| completion time | `streamer.checkpoint.key.v2`,
`streamer.checkpoint.reset.key.v2` | `StreamerCheckpointV2.java:32-33`, written
at `:61`/`:66` |
selected by `CheckpointUtils.shouldTargetCheckpointV2` on the write table
version. Confirmed by running the streamer
both ways against the same source: the default version-9 target commit
carries `streamer.checkpoint.key.v2`, and a
target forced to `hoodie.write.table.version=6` carries
`deltastreamer.checkpoint.key" : "20260810134321000`.
`StreamerCheckpointV2` also has an `addV1Props()` method that would write
the v1 keys alongside the v2 ones, so I
checked before wording this as an either/or: it has no callers anywhere in
main or test at `release-1.2.0`, and no
commit produced during testing carried both sets.
One simplification worth flagging: the corrected sentence ties the split to
table version, which is what
`shouldTargetCheckpointV2` does, but that method also excludes the S3 and
GCS incremental sources, so those stay on the
v1 key even at version 8 and above. That carve-out is stated in the note a
few lines below, so the two read together
correctly. Happy to make the intro sentence carry the exception explicitly
if reviewers prefer.
### Version scope
Applied to `next`, 1.2.0, 1.1.1, 1.0.2 and 1.0.1 — every release where the
prefixed form works, verified by execution
rather than inference, and covering both table version 8 and 9.
**Not 1.0.0**, where the prefixed form fails outright.
`HoodieIncrSourceCheckpointValUtils` does not exist at
`release-1.0.0`, and the test above shows what a user would actually hit.
There is no `version-1.1.0` docs folder, so
although 1.1.0 was tested and behaves like the rest there is nothing to
change for it. 0.14.x and 0.15.x predate
checkpoint v2 entirely.
### Site verification
`npm run build` passes with the warning set byte-identical to a baseline
build of the same base commit — no new warnings
and no new broken anchors. Rendering checked under `npm run serve` on
`/docs`, `/docs/next`, `/docs/1.1.1`,
`/docs/1.0.2` and `/docs/1.0.1`: the new heading, anchor, TOC entry, both
admonitions, all three code fences, the
corrected key sentence and the 1.0.1 floor are all present. `/docs/1.0.0`
correctly shows none of it.
### Impact
Documentation only. No code, config, or behaviour change.
### Risk Level
none
### Documentation Update
This PR is the documentation update — the Hudi Streamer page,
`/docs/hoodie_streaming_ingestion` and
`/docs/next/hoodie_streaming_ingestion`.
### Contributor's checklist
- [x] Read through [contributor's
guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Enough context is provided in the sections above
- [x] Adequate tests were added if applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
