Re: Unit tests of spark application

[Naveen Madhire]

Thanks. Spark-testing-base works pretty well.

On Fri, Jul 10, 2015 at 3:23 PM, Burak Yavuz  wrote:

> I can +1 Holden's spark-testing-base package.
>
> Burak
>
> On Fri, Jul 10, 2015 at 12:23 PM, Holden Karau 
> wrote:
>
>> Somewhat biased of course, but you can also use spark-testing-base from
>> spark-packages.org as a basis for your unittests.
>>
>> On Fri, Jul 10, 2015 at 12:03 PM, Daniel Siegmann <
>> daniel.siegm...@teamaol.com> wrote:
>>
>>> On Fri, Jul 10, 2015 at 1:41 PM, Naveen Madhire 
>>> wrote:
>>>
 I want to write junit test cases in scala for testing spark
 application. Is there any guide or link which I can refer.

>>>
>>> https://spark.apache.org/docs/latest/programming-guide.html#unit-testing
>>>
>>> Typically I create test data using SparkContext.parallelize and then
>>> call RDD.collect to get the results to assert.
>>>
>>
>>
>>
>> --
>> Cell : 425-233-8271
>> Twitter: https://twitter.com/holdenkarau
>> Linked In: https://www.linkedin.com/in/holdenkarau
>>
>
>

Re: Unit tests of spark application

[Burak Yavuz]

I can +1 Holden's spark-testing-base package.

Burak

On Fri, Jul 10, 2015 at 12:23 PM, Holden Karau  wrote:

> Somewhat biased of course, but you can also use spark-testing-base from
> spark-packages.org as a basis for your unittests.
>
> On Fri, Jul 10, 2015 at 12:03 PM, Daniel Siegmann <
> daniel.siegm...@teamaol.com> wrote:
>
>> On Fri, Jul 10, 2015 at 1:41 PM, Naveen Madhire 
>> wrote:
>>
>>> I want to write junit test cases in scala for testing spark application.
>>> Is there any guide or link which I can refer.
>>>
>>
>> https://spark.apache.org/docs/latest/programming-guide.html#unit-testing
>>
>> Typically I create test data using SparkContext.parallelize and then
>> call RDD.collect to get the results to assert.
>>
>
>
>
> --
> Cell : 425-233-8271
> Twitter: https://twitter.com/holdenkarau
> Linked In: https://www.linkedin.com/in/holdenkarau
>

Re: Unit tests of spark application

[Holden Karau]

Somewhat biased of course, but you can also use spark-testing-base from
spark-packages.org as a basis for your unittests.

On Fri, Jul 10, 2015 at 12:03 PM, Daniel Siegmann <
daniel.siegm...@teamaol.com> wrote:

> On Fri, Jul 10, 2015 at 1:41 PM, Naveen Madhire 
> wrote:
>
>> I want to write junit test cases in scala for testing spark application.
>> Is there any guide or link which I can refer.
>>
>
> https://spark.apache.org/docs/latest/programming-guide.html#unit-testing
>
> Typically I create test data using SparkContext.parallelize and then call
> RDD.collect to get the results to assert.
>



-- 
Cell : 425-233-8271
Twitter: https://twitter.com/holdenkarau
Linked In: https://www.linkedin.com/in/holdenkarau

Re: Unit tests of spark application

[Daniel Siegmann]

On Fri, Jul 10, 2015 at 1:41 PM, Naveen Madhire 
wrote:

> I want to write junit test cases in scala for testing spark application.
> Is there any guide or link which I can refer.
>

https://spark.apache.org/docs/latest/programming-guide.html#unit-testing

Typically I create test data using SparkContext.parallelize and then call
RDD.collect to get the results to assert.

Re: Unit tests of spark application

[Richard Marscher]

Unless you had something specific in mind, it should be as simple as
creating a SparkContext object using a master of local[2] in your tests

On Fri, Jul 10, 2015 at 1:41 PM, Naveen Madhire 
wrote:

> Hi,
>
> I want to write junit test cases in scala for testing spark application.
> Is there any guide or link which I can refer.
>
> Thank you very much.
>
> -Naveen
>
>
>
>


-- 
-- 
*Richard Marscher*
Software Engineer
Localytics
Localytics.com  | Our Blog
 | Twitter  |
Facebook  | LinkedIn