Saikat Kanjilal commented on SPARK-9487:

[~holdenk] finally getting time to look at this, so I am starting small, I made 
the change inside ContextCleanerSuite and HeartbeatReceiverSuite from local[2] 
tp local[4], per the documentation here 
 I ran mvn -Phadoop2 -Dsuites=org.apache.spark.HeartbeatReceiverSuite test--- 
looks like everything worked

I then ran mvn -Phadoop2 -Dsuites=org.apache.spark.ContextCleanerSuite test-- 
looks like everything worked as well

See the attachments and let me know if this is not the right process to run 
single unit tests, if not I'll start making changes to the other Suites , how 
would you like to see the output, should I just have attachments or just do a 
pull request from the new branch that I created?

Another question, running single unit tests like this takes forever, are there 
flags I can set to speed up the builds, even on my 15 inch macbook pro with SSD 
the builds shouldnt take this long :(.  

Let me know next steps to get this into a PR.

> Use the same num. worker threads in Scala/Python unit tests
> -----------------------------------------------------------
>                 Key: SPARK-9487
>                 URL: https://issues.apache.org/jira/browse/SPARK-9487
>             Project: Spark
>          Issue Type: Improvement
>          Components: PySpark, Spark Core, SQL, Tests
>    Affects Versions: 1.5.0
>            Reporter: Xiangrui Meng
>              Labels: starter
>         Attachments: ContextCleanerSuiteResults, HeartbeatReceiverSuiteResults
> In Python we use `local[4]` for unit tests, while in Scala/Java we use 
> `local[2]` and `local` for some unit tests in SQL, MLLib, and other 
> components. If the operation depends on partition IDs, e.g., random number 
> generator, this will lead to different result in Python and Scala/Java. It 
> would be nice to use the same number in all unit tests.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

Reply via email to