nchammas opened a new pull request, #59250: URL: https://github.com/apache/spark/pull/59250
### What changes were proposed in this pull request? Add a new test goal type: shell scripts. A shell script test goal runs a script at a specific stage of our test build and confirms it succeeds. Port the existing build test (`test-dependencies.sh`) over to this new test goal type. ### Why are the changes needed? We have many existing tests that fit the pattern of a shell script test goal. However, they are currently all expressed as custom functions in `dev/run-tests.py` with custom execution handling there and in our workflows. This makes it very hard to standardize the handling of these tests, to express test dependencies, or select/deselect tests based on what files are modified in a given PR. This PR migrates just the existing build test as a demonstration of its use, but in follow-up PRs I plan to migrate all of the following tests to this new test goal type: | Stage | Script | | --- | --- | | `pre_build` | `dev/check-license` | | `pre_build` | `dev/check-protos.py` | | `pre_build` | `dev/lint-java` | | `pre_build` | `dev/lint-js` | | `pre_build` | `dev/lint-python` | | `pre_build` | `dev/lint-scala` | | `pre_build` | `dev/structured_logging_style.py` | | `pre_build` | `dev/test-dependencies.sh` (done in this PR) | | `pre_build` | New UI Jest wrapper | | `post_build` | `dev/connect-jvm-client-mima-check` | | `post_build` | `dev/mima` | | `post_assembly_sbt` | `dev/sbt-checkstyle` | | `post_assembly` | `dev/run-pip-tests` | Some of these shell script tests require additional context to be passed in, like the active sbt profiles or a list of files modified by the PR. When I port these tests to the new test goal type, I will make the necessary API changes to `ShellScriptTestGoal`. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? New doctests added to `sparktestsupport`. ### Was this patch authored or co-authored using generative AI tooling? I wrote this with assistance from GitHub Copilot. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
