Re: Run a specific PySpark test or group of tests

Yes! This is very helpful! On Wed, Dec 5, 2018 at 9:21 PM Wenchen Fan wrote: > great job! thanks a lot! > > On Thu, Dec 6, 2018 at 9:39 AM Hyukjin Kwon wrote: > >> It's merged now and in developer tools page - >> http://spark.apache.org/developer-tools.html#individual-tests >> Have some func

Re: Run a specific PySpark test or group of tests

great job! thanks a lot! On Thu, Dec 6, 2018 at 9:39 AM Hyukjin Kwon wrote: > It's merged now and in developer tools page - > http://spark.apache.org/developer-tools.html#individual-tests > Have some func with PySpark testing! > > 2018년 12월 5일 (수) 오후 4:30, Hyukjin Kwon 님이 작성: > >> Hey all, I

Re: Run a specific PySpark test or group of tests

It's merged now and in developer tools page - http://spark.apache.org/developer-tools.html#individual-tests Have some func with PySpark testing! 2018년 12월 5일 (수) 오후 4:30, Hyukjin Kwon 님이 작성: > Hey all, I kind of met the goal with a minimised fix with keeping > available framework and options.

Re: Run a specific PySpark test or group of tests

Hey all, I kind of met the goal with a minimised fix with keeping available framework and options. See https://github.com/apache/spark/pull/23203 https://github.com/apache/spark-website/pull/161 I know it's not perfect and other Python testing framework provide many good other features but

Re: Run a specific PySpark test or group of tests

Looks like it doesn’t take too much work to get pytest working on our code base, since it knows how to run unittest tests. https://github.com/apache/spark/compare/master…nchammas:pytest For example I was able to do this from that

Re: Run a specific PySpark test or group of tests

This generally works for me to just run tests within a class or even a single test. Not as flexible as pytest -k, which would be nice.. $ SPARK_TESTING=1 bin/pyspark pyspark.sql.tests ArrowTests On Tue, Aug 15, 2017 at 5:49 AM, Nicholas Chammas < nicholas.cham...@gmail.com> wrote: > Pytest

Re: Run a specific PySpark test or group of tests

Pytest does support unittest-based tests , allowing for incremental adoption. I'll see how convenient it is to use with our current test layout. On Tue, Aug 15, 2017 at 1:03 AM Hyukjin Kwon wrote: > For me, I would like this

Re: Run a specific PySpark test or group of tests

For me, I would like this if this can be done with relatively small changes. How about adding more granular options, for example, specifying or filtering smaller set of test goals in the run-tests.py script? I think it'd be quite small change and we could roughly reach this goal if I understood

Run a specific PySpark test or group of tests

Say you’re working on something and you want to rerun the PySpark tests, focusing on a specific test or group of tests. Is there a way to do that? I know that you can test entire modules with this: ./python/run-tests --modules pyspark-sql But I’m looking for something more granular, like