Re: Unit tests of spark application
Thanks. Spark-testing-base works pretty well. On Fri, Jul 10, 2015 at 3:23 PM, Burak Yavuz brk...@gmail.com wrote: I can +1 Holden's spark-testing-base package. Burak On Fri, Jul 10, 2015 at 12:23 PM, Holden Karau hol...@pigscanfly.ca wrote: Somewhat biased of course, but you can also use spark-testing-base from spark-packages.org as a basis for your unittests. On Fri, Jul 10, 2015 at 12:03 PM, Daniel Siegmann daniel.siegm...@teamaol.com wrote: On Fri, Jul 10, 2015 at 1:41 PM, Naveen Madhire vmadh...@umail.iu.edu wrote: I want to write junit test cases in scala for testing spark application. Is there any guide or link which I can refer. https://spark.apache.org/docs/latest/programming-guide.html#unit-testing Typically I create test data using SparkContext.parallelize and then call RDD.collect to get the results to assert. -- Cell : 425-233-8271 Twitter: https://twitter.com/holdenkarau Linked In: https://www.linkedin.com/in/holdenkarau
Unit tests of spark application
Hi, I want to write junit test cases in scala for testing spark application. Is there any guide or link which I can refer. Thank you very much. -Naveen
Re: Unit tests of spark application
Unless you had something specific in mind, it should be as simple as creating a SparkContext object using a master of local[2] in your tests On Fri, Jul 10, 2015 at 1:41 PM, Naveen Madhire vmadh...@umail.iu.edu wrote: Hi, I want to write junit test cases in scala for testing spark application. Is there any guide or link which I can refer. Thank you very much. -Naveen -- -- *Richard Marscher* Software Engineer Localytics Localytics.com http://localytics.com/ | Our Blog http://localytics.com/blog | Twitter http://twitter.com/localytics | Facebook http://facebook.com/localytics | LinkedIn http://www.linkedin.com/company/1148792?trk=tyah
Re: Unit tests of spark application
On Fri, Jul 10, 2015 at 1:41 PM, Naveen Madhire vmadh...@umail.iu.edu wrote: I want to write junit test cases in scala for testing spark application. Is there any guide or link which I can refer. https://spark.apache.org/docs/latest/programming-guide.html#unit-testing Typically I create test data using SparkContext.parallelize and then call RDD.collect to get the results to assert.
Re: Unit tests of spark application
Somewhat biased of course, but you can also use spark-testing-base from spark-packages.org as a basis for your unittests. On Fri, Jul 10, 2015 at 12:03 PM, Daniel Siegmann daniel.siegm...@teamaol.com wrote: On Fri, Jul 10, 2015 at 1:41 PM, Naveen Madhire vmadh...@umail.iu.edu wrote: I want to write junit test cases in scala for testing spark application. Is there any guide or link which I can refer. https://spark.apache.org/docs/latest/programming-guide.html#unit-testing Typically I create test data using SparkContext.parallelize and then call RDD.collect to get the results to assert. -- Cell : 425-233-8271 Twitter: https://twitter.com/holdenkarau Linked In: https://www.linkedin.com/in/holdenkarau
Re: Unit tests of spark application
I can +1 Holden's spark-testing-base package. Burak On Fri, Jul 10, 2015 at 12:23 PM, Holden Karau hol...@pigscanfly.ca wrote: Somewhat biased of course, but you can also use spark-testing-base from spark-packages.org as a basis for your unittests. On Fri, Jul 10, 2015 at 12:03 PM, Daniel Siegmann daniel.siegm...@teamaol.com wrote: On Fri, Jul 10, 2015 at 1:41 PM, Naveen Madhire vmadh...@umail.iu.edu wrote: I want to write junit test cases in scala for testing spark application. Is there any guide or link which I can refer. https://spark.apache.org/docs/latest/programming-guide.html#unit-testing Typically I create test data using SparkContext.parallelize and then call RDD.collect to get the results to assert. -- Cell : 425-233-8271 Twitter: https://twitter.com/holdenkarau Linked In: https://www.linkedin.com/in/holdenkarau
