danny0405 opened a new pull request, #20493: URL: https://github.com/apache/hudi/pull/20493
### Describe the issue this Pull Request addresses Part of #20401, #20406 and #20408: the Phase 1 acceptance criteria require exact insert/upsert rows over repeated real CLI runs, deduplication, no new Hudi commit after an unchanged restart, and isolation of sink failures from the Connect server. The earlier smoke test and flow-settings run did not establish all of these assertions. ### Summary and Changelog - Add a standalone Phase 1 scenario using the #20453 Connect harness via an explicit `--harness-dir` argument; no dependency on its unresolved CI wiring. - Feed duplicate-key JSON input followed by a newer version through two SQL flows capturing independent insert/upsert settings. - Verify exact results after each run and restart, and preserve before/after completed commit filenames as evidence of no new commit. - Fail a flow on a null record key, assert the CLI reports the key error and the server stays alive, then issue a successful query through a fresh Connect client. - Document local execution, configuration scope, retained examples/logs and coverage limits. Adapt the earlier local flow-settings scenario from this repository's #20403 work; no external-project code is copied. Validation: clean master `b0d16f94b3b`, Spark 4.1.1 / Scala 2.13 / JDK 17, real CLI with the #20453 harness at `2a0616a1c6f`: all scenario assertions passed. First input produces two insert rows and one deduplicated upsert row; second input produces three insert rows and the updated upsert row. Both tables have two completed commits before and after the no-input restart. Null-key failure is reported and a fresh remote query succeeds afterward. Candidate bundle build, Python syntax, `git diff --check` and PR compliance validation passed. ### Impact Test-only changes. No production behavior, public API, storage format or configuration default changes. Uses pinned PySpark 4.1.1 and JDK 17 through the harness. This PR does not add a CI job; moving acceptance into CI follows the #20453 discussion. ### Risk Level low. Client timeouts and process cleanup come from the existing harness. Each run uses isolated local input, warehouse and checkpoints. The scenario deliberately disables metadata indexing and does not claim metadata-writer lifecycle coverage. ### Documentation Update Add `hudi-integ-test/spark-pipelines/ACCEPTANCE.md` with reproducible commands and explicit limits. No website change is needed for this developer test scenario. ### Contributor's checklist - [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [x] Enough context is provided in the sections above - [x] Adequate tests were added if applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
