danny0405 opened a new issue, #20402: URL: https://github.com/apache/hudi/issues/20402
### Task Description Build a reusable integration harness that launches a separately configured Spark 4.1 Connect server and invokes the real spark-pipelines CLI from a Python environment with pyspark[pipelines,connect]==4.1.1. Use the candidate Hudi bundle, deterministic test input, isolated table/checkpoint directories, bounded client timeouts, and captured server logs. Configure Hudi extensions/catalog, serializer, bundle, warehouse, and storage access at server startup. Add a Spark 4.1 CI lane, checking the current .github/workflows/bot.yml matrix before enabling or extending it. Scala DataflowGraph tests are supplementary, not a substitute for remote execution. ### Acceptance criteria - [ ] A documented local command starts the server, runs a minimal pipeline, and reads the Hudi output. - [ ] The harness distinguishes static server configuration from per-flow session configuration. - [ ] Failure is surfaced with bounded timeouts and useful client/server logs. - [ ] Teardown removes test resources and terminates only the server/processes owned by the test. - [ ] CI runs the harness with the built bundle; dependent flow scenarios can be added by sibling tasks. ### Task Type Test enhancement ### Related Issues Parent feature issue: #20401 RFC: #19997 (under review; implementation must follow the agreed contract). This task can start independently; coordinate integration with the sibling tasks under #20401. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
