Re: testing frameworks

ithub.com/svenkreiss/pysparkling > This lib is useful in case you have RDD. > > Hope this helps, > > Hichame > > *From:* mmistr...@gmail.com > *Sent:* February 3, 2019 4:42 PM > *To:* radams...@gmail.com > *Cc:* la...@mapflat.com; bpru...@opentext.com; user@spark.apache.

Re: testing frameworks

ect: Re: testing frameworks Hi sorry to resurrect this thread Any spark libraries for testing code in pyspark? the github code above seems related to Scala following links in the original threads (and also LMGFY) i found out <https://pypi.org/project/pytest-spark/> pytest-spark · PyPI

Re: testing frameworks

Hi sorry to resurrect this thread Any spark libraries for testing code in pyspark? the github code above seems related to Scala following links in the original threads (and also LMGFY) i found out pytest-spark · PyPI w/kindest regards Marco On Tue,

Re: testing frameworks

We use spark testing base for unit testing. These tests execute on a very small amount of data that covers all paths the code can take (or most paths anyway). https://github.com/holdenk/spark-testing-base For integration testing we use automated routines to ensure that aggregate values match an

Re: testing frameworks

Hi, I wrote this answer to the same question a couple of years ago: https://www.mail-archive.com/user%40spark.apache.org/msg48032.html I have made a couple of presentations on the subject. Slides and video are linked on this page: http://www.mapflat.com/presentations/ You can find more material

Re: testing frameworks

Hello! Thank you very much for your helpful answer and for the very good job performed in spark-testing-base . I managed to perform unit testing with spark-testing-base library as the provided article and also get inspired from

Re: testing frameworks

So Jessie has an excellent blog post on how to use it with Java applications - http://www.jesse-anderson.com/2016/04/unit-testing-spark-with-java/ On Wed, May 30, 2018 at 4:14 AM Spico Florin wrote: > Hello! > I'm also looking for unit testing spark Java application. I've seen the > great

Re: testing frameworks

Hello! I'm also looking for unit testing spark Java application. I've seen the great work done in spark-testing-base but it seemed to me that I could not use for Spark Java applications. Only spark scala applications are supported? Thanks. Regards, Florin On Wed, May 23, 2018 at 8:07 AM,

Re: testing frameworks

Hi Steve, you can try out pytest-spark plugin if your writing programs using pyspark ,please find below link for reference. https://github.com/malexer/pytest-spark Thanks, Umar -- Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/

Re: [EXTERNAL] - Re: testing frameworks

> > > Thanks. > > > > *From:* Holden Karau [mailto:hol...@pigscanfly.ca] > *Sent:* Monday, May 21, 2018 11:32 AM > *To:* Steve Pruitt <bpru...@opentext.com> > *Cc:* user@spark.apache.org > *Subject:* [EXTERNAL] - Re: testing frameworks > > > >

RE: [EXTERNAL] - Re: testing frameworks

[mailto:hol...@pigscanfly.ca] Sent: Monday, May 21, 2018 11:32 AM To: Steve Pruitt <bpru...@opentext.com> Cc: user@spark.apache.org Subject: [EXTERNAL] - Re: testing frameworks So I’m biased as the author of spark-testing-base but I think it’s pretty ok. Are you looking for unit or integ

Re: testing frameworks

So I’m biased as the author of spark-testing-base but I think it’s pretty ok. Are you looking for unit or integration or something else? On Mon, May 21, 2018 at 5:24 AM Steve Pruitt wrote: > Hi, > > > > Can anyone recommend testing frameworks suitable for Spark jobs. >