danny0405 opened a new pull request, #20756:
URL: https://github.com/apache/hudi/pull/20756
### Describe the issue this Pull Request addresses
Closes #20408. Merge after #20755, which completes the #20406 prerequisite;
#20403 is already closed.
The keyed-flow fixtures merged in #20496 and #20676 cover ordering,
delete/reinsertion and MOR log merging. This completes acceptance for existing
targets loading persisted settings and keyed batch replay after Hudi commit but
before Spark checkpoint completion, with metadata enabled.
### Summary and Changelog
- Seed the existing target with an unrelated row in partition `c`. Omit
`TBLPROPERTIES` from its SDP declaration so table type, record key, ordering
field and merger are loaded from persisted settings.
- Preserve that seed row throughout the six input stages, retry, restart and
five rejected configurations.
- Enable metadata indexing and require initialized metadata tables with
completed metadata writes for both targets.
- Model the commit/checkpoint interruption by removing each stopped query's
latest Spark commit-log entry while retaining source offsets, checkpoint
identities and Hudi timelines. Retry must restore Spark's entries, preserve
exact rows and add no Hudi data commits.
- Retain retry timeline evidence and assert that rejected configurations
also leave valid target timelines unchanged.
- Default the harness directory to the fixture's own directory and update
the runnable instructions.
Validation: a fresh Spark 4.1 / Scala 2.13 bundle build on JDK 17 passed,
including style and license checks; unit-test execution was skipped for the
bundle build. Both full COW and MOR CLI/Connect matrices passed with
`PYTHONOPTIMIZE=1`, including verification subprocesses. Each exercised new and
pre-existing targets, six input stages, retry, ordinary restart and five
negative cases. Both MOR targets retained the expected old base rows while
snapshots reflected updated log records. Python compilation and `git diff
--check` passed.
```sh
PYTHONOPTIMIZE=1 python3 hudi-integ-test/spark-pipelines/keyed_flows.py \
--bundle
packaging/hudi-spark-bundle/target/hudi-spark4.1-bundle_2.13-1.3.0-SNAPSHOT.jar
\
--table-type cow
# Repeat with --table-type mor.
```
This fixture models the checkpoint gap directly; real sink failure injection
and lifecycle coverage remain in the existing `TestStreamingSinkRecovery` suite.
### Impact
Completes the existing keyed SDP acceptance matrix using the production
metadata-enabled default. The fixture verifies persisted target configuration
and idempotent keyed replay through the real CLI/Connect path.
### Risk Level
low — Test and documentation changes only. Checkpoint-log removal is
restricted to the fixture's isolated storage after its AvailableNow queries
stop. Both table-type matrices passed with exact-row and timeline assertions.
### Documentation Update
Updated `hudi-integ-test/spark-pipelines/KEYED_FLOWS.md` with
persisted-setting reuse, the seed row, metadata checks, the checkpoint replay
model and runnable commands. No new runtime feature or configuration requires a
website change.
### Contributor's checklist
- [x] Read through [contributor's
guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Enough context is provided in the sections above
- [x] Adequate tests were added if applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]