HyukjinKwon commented on a change in pull request #23797:
[SPARK-26856][PYSPARK] Python support for from_avro and to_avro APIs
URL: https://github.com/apache/spark/pull/23797#discussion_r263658764
##########
File path: dev/sparktestsupport/modules.py
##########
@@ -178,6 +178,10 @@ def __hash__(self):
],
sbt_test_goals=[
"avro/test",
+ ],
+ python_test_goals=[
+ # doctests
+ "pyspark.sql.avro.functions"
Review comment:
I think we can remove this one too. `pyspark_sql` has `avro` as dependency.
So, if `avro` is detected, python ones will be tested as well.
FWIW, this can be checked as below:
```bash
$ cd dev
$ python
```
```python
runtests = __import__("run-tests")
import sparktestsupport.modules as modules
runtests.determine_modules_to_test([modules.avro])
```
Also, if we keep this `python_test_goals`, it shows avro as a separate
Python module in Python testing script:
```bash
cd python
```
```bash
$ ./run-tests --help
usage: run-tests [-h] [--python-executables PYTHON_EXECUTABLES]
[--modules MODULES] [-p PARALLELISM] [--verbose]
[--testnames TESTNAMES]
optional arguments:
-h, --help show this help message and exit
--python-executables PYTHON_EXECUTABLES
A comma-separated list of Python executables to test
against (default: python2.7,pypy)
--modules MODULES A comma-separated list of Python modules to test
(default: avro,pyspark-core,pyspark-ml,pyspark-mllib
,pyspark-sql,pyspark-streaming)
-p PARALLELISM, --parallelism PARALLELISM
The number of suites to test in parallel (default 4)
--verbose Enable additional debug logging
Developer Options:
--testnames TESTNAMES
A comma-separated list of specific modules, classes
and functions of doctest or unittest to test. For
example, 'pyspark.sql.foo' to run the module as
unittests or doctests, 'pyspark.sql.tests FooTests'
to
run the specific class of unittests,
'pyspark.sql.tests FooTests.test_foo' to run the
specific unittest in the class. '--modules' option is
ignored if they are given.
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
