danny0405 opened a new pull request, #20755:
URL: https://github.com/apache/hudi/pull/20755

   ### Describe the issue this Pull Request addresses
   
   Closes #20406 for the single-writer, purge-style streaming-table scope.
   
   The existing append/restart, lifecycle and post-commit recovery coverage 
landed in #20493, #20495 and #20677. The remaining acceptance gaps were real 
CLI full refresh and invalid table configuration.
   
   Automatic pre-write fan-in rejection is explicitly deferred: stock Spark 
4.1.1 does not expose a Hudi graph-validation extension. Multiple flows writing 
one Hudi target remain unsupported. Timeline-preserving refresh and sink 
checkpoint-generation isolation remain under #20404.
   
   ### Summary and Changelog
   
   - Extend the existing Spark 4.1 CLI/Connect suite with COW and MOR cases: 
two input batches, ordinary restart, full refresh, another restart, and new 
input after refresh.
   - Check exact insert rows, old timeline removal, a fresh streaming 
checkpoint identity with committed batch 0, and no extra commits on quiescent 
restarts.
   - Reject a nonexistent primary-key column before any Hudi data commit.
   - Retain each CLI run's input definitions and logs; refresh the verifier's 
cached table listing between writes from separate Connect sessions.
   - Document the acceptance coverage, single-writer constraint, and manual 
Kafka connector setup.
   
   Validation: Spark 4.1.1 / Scala 2.13 / JDK 17 reactor package build passed 
with all 6 `TestHoodieSparkConnectPipelines` tests passing, zero failures or 
cancellations. COW and MOR refresh cases use metadata enabled. Checkstyle, 
Scalastyle, license checks and `git diff --check` passed.
   
   ```sh
   HUDI_SDP_REQUIRE_PYTHON=true mvn package -Pjava17 -Dspark4.1 -Dscala-2.13 \
     -Dtest=TestHoodieSparkConnectPipelines \
     
-DwildcardSuites=org.apache.spark.sql.connect.hudi.TestHoodieSparkConnectPipelines
 \
     -Dsurefire.failIfNoSpecifiedTests=false -DfailIfNoTests=false \
     -DskipITs -DskipFTs -Dgpg.skip -Djacoco.skip -DshadeSources=false \
     -am -pl hudi-spark-datasource/hudi-spark
   ```
   
   The Kafka recipe was not executed against a broker. Existing recovery suites 
are referenced as merged coverage; this change's local test execution targets 
the CLI/Connect suite.
   
   ### Impact
   
   Extends the existing Spark 4.1 CI suite and documents the supported 
single-writer release scope. Full refresh is validated against the current 
physical purge and SDP-managed checkpoint behavior.
   
   ### Risk Level
   
   low — Changes are limited to integration tests and documentation. Additional 
CLI runs increase this suite's runtime; the existing CI timeout remains the 
gate for its total build and test cost.
   
   ### Documentation Update
   
   Updated `hudi-integ-test/spark-pipelines/ACCEPTANCE.md` with the acceptance 
matrix, deferred fan-in guard, and a manual Kafka recipe. Updated the layer-2 
coverage table in `README.md`.
   
   ### Contributor's checklist
   
   - [x] Read through [contributor's 
guide](https://hudi.apache.org/contribute/how-to-contribute)
   - [x] Enough context is provided in the sections above
   - [x] Adequate tests were added if applicable
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to