Hi sorry to resurrect this thread Any spark libraries for testing code in pyspark? the github code above seems related to Scala following links in the original threads (and also LMGFY) i found out pytest-spark ยท PyPI <https://pypi.org/project/pytest-spark/>
w/kindest regards Marco On Tue, Jun 12, 2018 at 6:44 PM Ryan Adams <radams...@gmail.com> wrote: > We use spark testing base for unit testing. These tests execute on a very > small amount of data that covers all paths the code can take (or most paths > anyway). > > https://github.com/holdenk/spark-testing-base > > For integration testing we use automated routines to ensure that aggregate > values match an aggregate baseline. > > Ryan > > Ryan Adams > radams...@gmail.com > > On Tue, Jun 12, 2018 at 11:51 AM, Lars Albertsson <la...@mapflat.com> > wrote: > >> Hi, >> >> I wrote this answer to the same question a couple of years ago: >> https://www.mail-archive.com/user%40spark.apache.org/msg48032.html >> >> I have made a couple of presentations on the subject. Slides and video >> are linked on this page: http://www.mapflat.com/presentations/ >> >> You can find more material in this list of resources: >> http://www.mapflat.com/lands/resources/reading-list >> >> Happy testing! >> >> Regards, >> >> >> >> Lars Albertsson >> Data engineering consultant >> www.mapflat.com >> https://twitter.com/lalleal >> +46 70 7687109 >> Calendar: http://www.mapflat.com/calendar >> >> >> On Mon, May 21, 2018 at 2:24 PM, Steve Pruitt <bpru...@opentext.com> >> wrote: >> > Hi, >> > >> > >> > >> > Can anyone recommend testing frameworks suitable for Spark jobs. >> Something >> > that can be integrated into a CI tool would be great. >> > >> > >> > >> > Thanks. >> > >> > >> >> --------------------------------------------------------------------- >> To unsubscribe e-mail: user-unsubscr...@spark.apache.org >> >> >
