[
https://issues.apache.org/jira/browse/SPARK-59180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nicholas Chammas updated SPARK-59180:
-------------------------------------
Summary: Improve precision of CI test selection (was: Explicitly map every
repo path to a test module via `sparktestsupport`)
> Improve precision of CI test selection
> --------------------------------------
>
> Key: SPARK-59180
> URL: https://issues.apache.org/jira/browse/SPARK-59180
> Project: Spark
> Issue Type: Improvement
> Components: Build
> Affects Versions: 4.3.0
> Reporter: Nicholas Chammas
> Priority: Minor
>
> I would like `sparktestsupport` to be the single owner of deciding what tests
> to run based on what files are changed in a given pull request. I want it to
> be more judicious in selecting tests to run, and avoid running obviously
> unnecessary tests.
> To accomplish this, we need to:
> * Explicitly map all repo paths to test modules and remove the default of
> "unmapped file -> run all tests".
> * Create new test modules to cover concerns like linters, and reorganize
> some dev scripts so they are grouped more naturally for mapping to test
> modules.
> * Merge [this large block of inline
> shell|https://github.com/apache/spark/blob/3979ac9f6aad52d4e85edd1d4167a3bb29262b92/.github/workflows/build_and_test.yml#L106-L199]
> into `sparktestsupport`.
> * Merge `is-changed.py` into `sparktestsupport`.
> * Migrate all path specifications to a single format, likely [`PurePath`
> with some
> modifications|https://github.com/apache/spark/blob/3979ac9f6aad52d4e85edd1d4167a3bb29262b92/dev/sparktestsupport/modules.py#L27-L37].
> When this is done:
> * We will have one consistent approach and one central place to update when
> mapping diff paths to test modules.
> * Fewer unnecessary tests will run, saving us CI time and speeding up PR
> feedback.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
