[
https://issues.apache.org/jira/browse/SPARK-1455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14134856#comment-14134856
]
Nicholas Chammas commented on SPARK-1455:
-----------------------------------------
I think we could reuse some of [the logic
here|https://github.com/databricks/spark-pr-dashboard/blob/bf289c15ffd959eed0911f12591bb1b08672b2dd/sparkprs/models.py#L71]
to determine what tests to run from the files that were changed.
> Determine which test suites to run based on code changes
> --------------------------------------------------------
>
> Key: SPARK-1455
> URL: https://issues.apache.org/jira/browse/SPARK-1455
> Project: Spark
> Issue Type: Improvement
> Components: Project Infra
> Reporter: Patrick Wendell
> Fix For: 1.2.0
>
>
> Right now we run the entire set of tests for every change. This means the
> tests take a long time. Our pull request builder checks out the merge branch
> from git, so we could do a diff and figure out what source files were
> changed, and run a more isolated set of tests. We should just run tests in a
> way that reflects the inter-dependencies of the project. E.g:
> - If Spark core is modified, we should run all tests
> - If just SQL is modified, we should run only the SQL tests
> - If just Streaming is modified, we should run only the streaming tests
> - If just Pyspark is modified, we only run the PySpark tests.
> And so on. I think this would reduce the RTT of the tests a lot and it should
> be pretty easy to accomplish with some scripting foo.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
