Re: Unit test best practice for Spark-derived projects

2014-08-07 Thread Madhu
this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/Unit-test-best-practice-for-Spark-derived-projects-tp7704p7731.html Sent from the Apache Spark Developers List mailing list archive at Nabble.com

Re: Unit test best practice for Spark-derived projects

2014-08-07 Thread Dmitriy Lyubimov
-practice-for-Spark-derived-projects-tp7704p7731.html Sent from the Apache Spark Developers List mailing list archive at Nabble.com. - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h

Re: Unit test best practice for Spark-derived projects

2014-08-07 Thread Patrick Wendell
up 30 seconds to start up locally. I think a hosts file entry is sufficient. - -- Madhu https://www.linkedin.com/in/msiddalingaiah -- View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/Unit-test-best-practice-for-Spark-derived-projects

Unit test best practice for Spark-derived projects

2014-08-05 Thread Dmitriy Lyubimov
Hello, I 've been switching Mahout from Spark 0.9 to Spark 1.0.x [1] and noticed that tests now run much slower compared to 0.9 with CPU running idle most of the time. I had to conclude that most of that time is spent on tearing down/resetting Spark context which apparently now takes