[
https://issues.apache.org/jira/browse/SPARK-3431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14143707#comment-14143707
]
Nicholas Chammas commented on SPARK-3431:
-----------------------------------------
{quote}
Do you know how maven / sbt plugins handle this?
{quote}
Not really. What I can do for starters is just experiment with GNU parallel and
see how it works.
{quote}
The GNU parallel approach ... has the nice advantage of only affecting Jenkins
{quote}
Well, if we are modifying {{dev/run-tests}} then developers should also be able
to use it locally. The contributing guide recommends running tests using that
script. If we do go the GNU parallel route, we can have it trigger only if it
detects GNU parallel on the host.
> Parallelize execution of tests
> ------------------------------
>
> Key: SPARK-3431
> URL: https://issues.apache.org/jira/browse/SPARK-3431
> Project: Spark
> Issue Type: Improvement
> Components: Build
> Reporter: Nicholas Chammas
>
> Running all the tests in {{dev/run-tests}} takes up to 2 hours. A common
> strategy to cut test time down is to parallelize the execution of the tests.
> Doing that may in turn require some prerequisite changes to be made to how
> certain tests run.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
