[jira] [Commented] (SPARK-3431) Parallelize Scala/Java test execution
[
https://issues.apache.org/jira/browse/SPARK-3431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14311270#comment-14311270
]
Sean Owen commented on SPARK-3431:
--
Haven't tried anything recently, no.
Parallelize Scala/Java test execution
-
Key: SPARK-3431
URL: https://issues.apache.org/jira/browse/SPARK-3431
Project: Spark
Issue Type: Improvement
Components: Build
Reporter: Nicholas Chammas
Assignee: Nicholas Chammas
Attachments: SPARK-3431-srowen-attempt.patch
Running all the tests in {{dev/run-tests}} takes up to 2 hours. A common
strategy to cut test time down is to parallelize the execution of the tests.
Doing that may in turn require some prerequisite changes to be made to how
certain tests run.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-3431) Parallelize Scala/Java test execution
[
https://issues.apache.org/jira/browse/SPARK-3431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14311135#comment-14311135
]
Nicholas Chammas commented on SPARK-3431:
-
[~srowen] - Have you tried anything recently with parallelizing tests with
Maven?
Parallelize Scala/Java test execution
-
Key: SPARK-3431
URL: https://issues.apache.org/jira/browse/SPARK-3431
Project: Spark
Issue Type: Improvement
Components: Build
Reporter: Nicholas Chammas
Assignee: Nicholas Chammas
Attachments: SPARK-3431-srowen-attempt.patch
Running all the tests in {{dev/run-tests}} takes up to 2 hours. A common
strategy to cut test time down is to parallelize the execution of the tests.
Doing that may in turn require some prerequisite changes to be made to how
certain tests run.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-3431) Parallelize Scala/Java test execution
[
https://issues.apache.org/jira/browse/SPARK-3431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14270324#comment-14270324
]
Nicholas Chammas commented on SPARK-3431:
-
Generic update:
* For those not following [the open
PR|https://github.com/apache/spark/pull/3564], I got through the problems with
sbt not passing configs properly to forked JVMs, and [~joshrosen] has been
methodically updating unit tests that don't play nice when run in parallel.
Only a few tests remain.
* [~srowen] has been looking into using Maven + Surefire and has fixed up our
execution of Java unit tests.
* I've created a separate issue, [SPARK-5161], to track parallelizing the
Python unit tests. This issue here will focus on Scala/Java test execution only.
Parallelize Scala/Java test execution
-
Key: SPARK-3431
URL: https://issues.apache.org/jira/browse/SPARK-3431
Project: Spark
Issue Type: Improvement
Components: Build
Reporter: Nicholas Chammas
Assignee: Nicholas Chammas
Attachments: SPARK-3431-srowen-attempt.patch
Running all the tests in {{dev/run-tests}} takes up to 2 hours. A common
strategy to cut test time down is to parallelize the execution of the tests.
Doing that may in turn require some prerequisite changes to be made to how
certain tests run.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
