danny0405 opened a new pull request, #20755:
URL: https://github.com/apache/hudi/pull/20755
### Describe the issue this Pull Request addresses
Closes #20406 for the single-writer, purge-style streaming-table scope.
The existing append/restart, lifecycle and post-commit recovery coverage
landed in #20493, #20495 and #20677. The remaining acceptance gaps were real
CLI full refresh and invalid table configuration.
Automatic pre-write fan-in rejection is explicitly deferred: stock Spark
4.1.1 does not expose a Hudi graph-validation extension. Multiple flows writing
one Hudi target remain unsupported. Timeline-preserving refresh and sink
checkpoint-generation isolation remain under #20404.
### Summary and Changelog
- Extend the existing Spark 4.1 CLI/Connect suite with COW and MOR cases:
two input batches, ordinary restart, full refresh, another restart, and new
input after refresh.
- Check exact insert rows, old timeline removal, a fresh streaming
checkpoint identity with committed batch 0, and no extra commits on quiescent
restarts.
- Reject a nonexistent primary-key column before any Hudi data commit.
- Retain each CLI run's input definitions and logs; refresh the verifier's
cached table listing between writes from separate Connect sessions.
- Document the acceptance coverage, single-writer constraint, and manual
Kafka connector setup.
Validation: Spark 4.1.1 / Scala 2.13 / JDK 17 reactor package build passed
with all 6 `TestHoodieSparkConnectPipelines` tests passing, zero failures or
cancellations. COW and MOR refresh cases use metadata enabled. Checkstyle,
Scalastyle, license checks and `git diff --check` passed.
```sh
HUDI_SDP_REQUIRE_PYTHON=true mvn package -Pjava17 -Dspark4.1 -Dscala-2.13 \
-Dtest=TestHoodieSparkConnectPipelines \
-DwildcardSuites=org.apache.spark.sql.connect.hudi.TestHoodieSparkConnectPipelines
\
-Dsurefire.failIfNoSpecifiedTests=false -DfailIfNoTests=false \
-DskipITs -DskipFTs -Dgpg.skip -Djacoco.skip -DshadeSources=false \
-am -pl hudi-spark-datasource/hudi-spark
```
The Kafka recipe was not executed against a broker. Existing recovery suites
are referenced as merged coverage; this change's local test execution targets
the CLI/Connect suite.
### Impact
Extends the existing Spark 4.1 CI suite and documents the supported
single-writer release scope. Full refresh is validated against the current
physical purge and SDP-managed checkpoint behavior.
### Risk Level
low — Changes are limited to integration tests and documentation. Additional
CLI runs increase this suite's runtime; the existing CI timeout remains the
gate for its total build and test cost.
### Documentation Update
Updated `hudi-integ-test/spark-pipelines/ACCEPTANCE.md` with the acceptance
matrix, deferred fan-in guard, and a manual Kafka recipe. Updated the layer-2
coverage table in `README.md`.
### Contributor's checklist
- [x] Read through [contributor's
guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Enough context is provided in the sections above
- [x] Adequate tests were added if applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]