HyukjinKwon opened a new pull request #29086:
URL: https://github.com/apache/spark/pull/29086
### What changes were proposed in this pull request?
This PR mainly proposes to run only relevant tests just like Jenkins PR
builder does. Currently, GitHub Actions always run full tests which wastes the
resources.
In addition, this PR also fixes 3 more issues very closely related together
while I am here.
1. The main idea here is: It reuses the existing logic embedded in
`dev/run-tests.py` which Jenkins PR builder use in order to run only the
related test cases.
2. While I am here, I fixed SPARK-32292 too to run the doc tests. It was
because other references were not available when it is cloned via
`checkout@v2`. With `fetch-depth: 0`, the history is available.
3. In addition, it fixes the `dev/run-tests.py` to match with
`python/run-tests.py` in terms of its options. Environment variables such as
`TEST_ONLY_XXX` were moved as proper options. For example,
```bash
dev/run-tests.py --modules sql,core
```
which is consistent with `python/run-tests.py`, for example,
```bash
python/run-tests.py --modules pyspark-core,pyspark-ml
```
4. Lastly, also fixed the formatting issue in module specification in the
matrix:
```diff
- network_common, network_shuffle, repl, launcher
+ network-common, network-shuffle, repl, launcher,
```
which incorrectly build/test the modules.
### Why are the changes needed?
By running only related tests, we can hugely safe the resources and avoid
unrelated flaky tests, etc.
Also, now it runs the doctest of `dev/run-tests.py` properly, the usages are
similar between `dev/run-tests.py` and `python/run-tests.py`, and run
`network-common`, `network-shuffle`, `launcher` and `examples` modules too.
### Does this PR introduce _any_ user-facing change?
No, dev-only.
### How was this patch tested?
Manually tested in my own forked Spark:
https://github.com/HyukjinKwon/spark/pull/7
https://github.com/HyukjinKwon/spark/pull/8
https://github.com/HyukjinKwon/spark/pull/9
https://github.com/HyukjinKwon/spark/pull/10
https://github.com/HyukjinKwon/spark/pull/11
https://github.com/HyukjinKwon/spark/pull/12
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
