[
https://issues.apache.org/jira/browse/SPARK-3431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14143684#comment-14143684
]
Josh Rosen commented on SPARK-3431:
-----------------------------------
[~nchammas] I'm not sure.
The different test suites depend on the same build artifacts, but it looks like
we call {{sbt assembly}} before running any tests. The GNU parallel approach
would certainly be easy to implement and it has the nice advantage of only
affecting Jenkins, but I have one concern about test reporting. How will
output from tests be printed and will the test report XML files be generated at
the same locations? It might be confusing to see the output of several test
suites interleaved in an arbitrary way. Do you know how maven / sbt plugins
handle this?
> Parallelize execution of tests
> ------------------------------
>
> Key: SPARK-3431
> URL: https://issues.apache.org/jira/browse/SPARK-3431
> Project: Spark
> Issue Type: Improvement
> Components: Build
> Reporter: Nicholas Chammas
>
> Running all the tests in {{dev/run-tests}} takes up to 2 hours. A common
> strategy to cut test time down is to parallelize the execution of the tests.
> Doing that may in turn require some prerequisite changes to be made to how
> certain tests run.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
