Re: testing frameworks

ithub.com/svenkreiss/pysparkling > This lib is useful in case you have RDD. > > Hope this helps, > > Hichame > > *From:* mmistr...@gmail.com > *Sent:* February 3, 2019 4:42 PM > *To:* radams...@gmail.com > *Cc:* la...@mapflat.com; bpru...@opentext.com; user@spark.apache.

Re: testing frameworks

ect: Re: testing frameworks Hi sorry to resurrect this thread Any spark libraries for testing code in pyspark? the github code above seems related to Scala following links in the original threads (and also LMGFY) i found out <https://pypi.org/project/pytest-spark/> pytest-spark · PyPI

Re: testing frameworks

//twitter.com/lalleal >> +46 70 7687109 >> Calendar: http://www.mapflat.com/calendar >> >> >> On Mon, May 21, 2018 at 2:24 PM, Steve Pruitt >> wrote: >> > Hi, >> > >> > >> > >> > Can anyone recommend testi

Re: testing frameworks

gt; > Lars Albertsson > Data engineering consultant > www.mapflat.com > https://twitter.com/lalleal > +46 70 7687109 > Calendar: http://www.mapflat.com/calendar > > > On Mon, May 21, 2018 at 2:24 PM, Steve Pruitt > wrote: > > Hi, > > > > > > &g

Re: testing frameworks

Pruitt wrote: > Hi, > > > > Can anyone recommend testing frameworks suitable for Spark jobs. Something > that can be integrated into a CI tool would be great. > > > > Thanks. > > -

Re: testing frameworks

Hello! Thank you very much for your helpful answer and for the very good job performed in spark-testing-base . I managed to perform unit testing with spark-testing-base library as the provided article and also get inspired from https://github.com/holdenk/spark-testing-base/blob/master/src/test/1

Re: testing frameworks

So Jessie has an excellent blog post on how to use it with Java applications - http://www.jesse-anderson.com/2016/04/unit-testing-spark-with-java/ On Wed, May 30, 2018 at 4:14 AM Spico Florin wrote: > Hello! > I'm also looking for unit testing spark Java application. I've seen the > great work

Re: testing frameworks

Hello! I'm also looking for unit testing spark Java application. I've seen the great work done in spark-testing-base but it seemed to me that I could not use for Spark Java applications. Only spark scala applications are supported? Thanks. Regards, Florin On Wed, May 23, 2018 at 8:07 AM, umarg

Re: testing frameworks

Hi Steve, you can try out pytest-spark plugin if your writing programs using pyspark ,please find below link for reference. https://github.com/malexer/pytest-spark Thanks, Umar -- Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/

Re: [EXTERNAL] - Re: testing frameworks

gt; > > > *From:* Holden Karau [mailto:hol...@pigscanfly.ca] > *Sent:* Monday, May 21, 2018 11:32 AM > *To:* Steve Pruitt > *Cc:* user@spark.apache.org > *Subject:* [EXTERNAL] - Re: testing frameworks > > > > So I’m biased as the author of spark-testing-base but I think

RE: [EXTERNAL] - Re: testing frameworks

[mailto:hol...@pigscanfly.ca] Sent: Monday, May 21, 2018 11:32 AM To: Steve Pruitt Cc: user@spark.apache.org Subject: [EXTERNAL] - Re: testing frameworks So I’m biased as the author of spark-testing-base but I think it’s pretty ok. Are you looking for unit or integration or something else? On Mon

Re: testing frameworks

So I’m biased as the author of spark-testing-base but I think it’s pretty ok. Are you looking for unit or integration or something else? On Mon, May 21, 2018 at 5:24 AM Steve Pruitt wrote: > Hi, > > > > Can anyone recommend testing frameworks suitable for Spark jobs. > Som

testing frameworks

Hi, Can anyone recommend testing frameworks suitable for Spark jobs. Something that can be integrated into a CI tool would be great. Thanks.