danny0405 opened a new pull request, #20515: URL: https://github.com/apache/hudi/pull/20515
### Describe the issue this Pull Request addresses Issue: #20407; parent #20401. The incremental Hudi streaming-table chaining path needs executable acceptance through the real Spark pipelines CLI and Connect server. This PR covers that path only and does not close #20407. ### Summary and Changelog - Add a standalone two-stage scenario adapted from the existing phase1_acceptance.py: external JSON input -> keyed Hudi upsert table -> Hudi insert table via FROM STREAM. - Assert exact business rows after duplicate-key input and a later update. Using insert downstream makes replay visible as extra rows rather than masking it with upserts. - Verify checkpoint resume and unchanged completed timelines after quiescence for COW and MOR. For MOR, require compaction and separately drain its maintenance instant while asserting no business-record replay. - Explicitly project business columns and retain generated SQL, pipeline configuration, and timeline evidence in the logs. This requires the #20453 harness checkout through --harness-dir. Reset/recreation detection, expired-history errors, reader-option conflicts, and coordinated rebuild acceptance remain outstanding under #20407. No new production source behavior is claimed. ### Impact Adds opt-in integration acceptance and runnable examples. No production behavior, public API, configuration default, or storage-format changes. The source remains latest-state incremental, without full hard-delete propagation or bounded admission control. ### Risk Level low: test and documentation changes only. The MOR scenario explicitly accounts for async compaction completing after AvailableNow captures its end offset; that catch-up can produce an empty downstream commit. ### Documentation Update Updated hudi-integ-test/spark-pipelines/ACCEPTANCE.md with invocation, assertions, compaction behavior, source limitations, and outstanding reset-safety work. No new public feature or configuration requiring a website change. ### Testing - Built the Spark 4.1 bundle from master c58c54e068b with JDK 17: mvn package -Pjava17 -Dspark4.1 -Dscala-2.13 -DskipTests -Dgpg.skip -Djacoco.skip -am -pl packaging/hudi-spark-bundle. - Real Spark 4.1.1 CLI/Connect scenario passed with --table-type cow and --table-type mor, both with metadata disabled. Each verifies exact rows across runs and unchanged quiescent restart timelines; MOR additionally verifies compaction catch-up. - Python syntax compilation and git diff --check passed. - Metadata-enabled chaining and CI have not been run. The Maven package command skipped unit-test execution. ### Contributor's checklist - [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [x] Enough context is provided in the sections above - [x] Adequate tests were added if applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
