danny0405 opened a new pull request, #20323:
URL: https://github.com/apache/hudi/pull/20323
### Describe the issue this Pull Request addresses
A restored Flink writer currently resends checkpointed write metadata only
on attempt 0. If that delivery fails, later attempts send only an empty cleanup
event. The coordinator can therefore miss the metadata needed to commit a
successfully checkpointed batch, causing data loss.
Recovered metadata must be retried, while duplicate deliveries must not
recreate committed event buffers or trigger redundant recommits.
### Summary and Changelog
- Resend metadata for pending instants on every restored attempt, retaining
retry cleanup of events from failed attempts.
- Track the latest accepted bootstrap checkpoint per data/index writer in
`EventBuffers`. Ignore duplicate and older events, retain tracking after
commit, and reset it on global recovery. Tracking uses fixed-size arrays
proportional to writer parallelism.
- Update the coordinator's recovery flowchart and add coverage for later
attempts, duplicate/stale events, independent writers, and recovery resets.
Adapt existing failover tests to consume both recovered metadata and cleanup
events.
- No code was copied from another project.
### Impact
Fixes Flink sink recovery when the initial delivery of recovered metadata
fails. No user-facing API, configuration, checkpoint serialization format, or
table storage format changes. Additional metadata delivery occurs during
recovery; regular write processing is unchanged.
### Risk Level
medium
The change affects recovery and commit coordination. Per-writer tracking
relies on local retries restoring the same or a newer completed checkpoint;
global recovery resets tracking to permit replay.
Validated on JDK 17 with the default Flink 2.2 profile: **79 tests passed, 1
previously disabled test skipped**, with no failures or errors. This covers
`TestAbstractStreamWriteFunction`, `TestStreamWriteOperatorCoordinator`,
`TestEventBuffers`, and the existing `TestWriteCopyOnWrite` failover cases.
```shell
mvn -o -pl hudi-flink-datasource/hudi-flink -am \
'-Dtest=TestAbstractStreamWriteFunction,TestStreamWriteOperatorCoordinator,TestEventBuffers,TestWriteCopyOnWrite#testSubtaskFails+testAppendInsertAfterFailoverWithEmptyCheckpoint+testPartialFailover+testNonBlockedInstantRequestAfterFailover'
\
-Dsurefire.rerunFailingTestsCount=0 \
-Dsurefire.failIfNoSpecifiedTests=false -DfailIfNoTests=false test
```
`git diff --check` also passed. The full project test suite was not run.
### Documentation Update
Updated the coordinator Javadoc flowchart and bootstrap checkpoint tracking
documentation. No website update is needed: this restores the existing recovery
guarantee without adding a feature, configuration, or changing user-facing
interfaces.
### Contributor's checklist
- [x] Read through [contributor's
guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Enough context is provided in the sections above
- [x] Adequate tests were added if applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
