[
https://issues.apache.org/jira/browse/SPARK-9487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15587627#comment-15587627
]
Saikat Kanjilal edited comment on SPARK-9487 at 10/19/16 4:24 AM:
------------------------------------------------------------------
[~holdenk] finally getting time to look at this, so I am starting small, I made
the change inside ContextCleanerSuite and HeartbeatReceiverSuite from local[2]
tp local[4], per the documentation here
(http://spark.apache.org/docs/latest/building-spark.html#specifying-the-hadoop-version)
I ran mvn -P hadoop2 -Dsuites= org.apache.spark.HeartbeatReceiverSuite test---
looks like everything worked
I then ran mvn -P hadoop2 -Dsuites= org.apache.spark.ContextCleanerSuite test--
looks like everything worked as well
See the attachments and let me know if this is the right process to run single
unit tests, I'll start making changes to the other Suites , how would you like
to see the output, should I just have attachments or just do a pull request
from the new branch that I created?
Thanks
PS
Another question, running single unit tests like this takes forever, are there
flags I can set to speed up the builds, even on my 15 inch macbook pro with SSD
the builds shouldnt take this long :(.
Let me know next steps to get this into a PR.
was (Author: kanjilal):
[~holdenk] finally getting time to look at this, so I am starting small, I made
the change inside ContextCleanerSuite and HeartbeatReceiverSuite from local[2]
tp local[4], per the documentation here
(http://spark.apache.org/docs/latest/building-spark.html#specifying-the-hadoop-version)
I ran mvn -P hadoop2 -Dsuites= org.apache.spark.HeartbeatReceiverSuite test---
looks like everything worked
I then ran mvn -P hadoop2 -Dsuites= org.apache.spark.ContextCleanerSuite test--
looks like everything worked as well
See the attachments and let me know if this is this is the right process to run
single unit tests, if not I'll start making changes to the other Suites , how
would you like to see the output, should I just have attachments or just do a
pull request from the new branch that I created?
Thanks
PS
Another question, running single unit tests like this takes forever, are there
flags I can set to speed up the builds, even on my 15 inch macbook pro with SSD
the builds shouldnt take this long :(.
Let me know next steps to get this into a PR.
> Use the same num. worker threads in Scala/Python unit tests
> -----------------------------------------------------------
>
> Key: SPARK-9487
> URL: https://issues.apache.org/jira/browse/SPARK-9487
> Project: Spark
> Issue Type: Improvement
> Components: PySpark, Spark Core, SQL, Tests
> Affects Versions: 1.5.0
> Reporter: Xiangrui Meng
> Labels: starter
> Attachments: ContextCleanerSuiteResults, HeartbeatReceiverSuiteResults
>
>
> In Python we use `local[4]` for unit tests, while in Scala/Java we use
> `local[2]` and `local` for some unit tests in SQL, MLLib, and other
> components. If the operation depends on partition IDs, e.g., random number
> generator, this will lead to different result in Python and Scala/Java. It
> would be nice to use the same number in all unit tests.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
