GitHub user HyukjinKwon opened a pull request:
https://github.com/apache/spark/pull/20473
[SPARK-23300][TESTS] Prints out if Pandas and PyArrow are installed or not
in PySpark SQL tests
## What changes were proposed in this pull request?
This PR proposes to log if PyArrow and Pandas are installed or not so we
can check if related tests are going to be skipped or not.
## How was this patch tested?
Manually tested:
I don't have PyArrow installed in PyPy.
**Not printing the log**
```
./run-tests --python-executables=python3
```
```
...
Will test against the following Python executables: ['python3']
Will test the following Python modules: ['pyspark-core', 'pyspark-ml',
'pyspark-mllib', 'pyspark-sql', 'pyspark-streaming']
Starting test(python3): pyspark.mllib.tests
Starting test(python3): pyspark.streaming.tests
Starting test(python3): pyspark.tests
Starting test(python3): pyspark.sql.tests
```
```
./run-tests --modules=pyspark-streaming
```
```
...
Will test against the following Python executables: ['python2.7', 'pypy']
Will test the following Python modules: ['pyspark-streaming']
Starting test(pypy): pyspark.streaming.tests
Starting test(python2.7): pyspark.streaming.tests
Starting test(pypy): pyspark.streaming.util
Starting test(python2.7): pyspark.streaming.util
```
**Printing the log**
```
./run-tests
```
```
...
Will test against the following Python executables: ['python2.7', 'pypy']
Will test the following Python modules: ['pyspark-core', 'pyspark-ml',
'pyspark-mllib', 'pyspark-sql', 'pyspark-streaming']
PyArrow is not installed in Python executable 'pypy', skipping related
tests in 'pyspark-sql'.
Starting test(pypy): pyspark.sql.tests
Starting test(pypy): pyspark.tests
Starting test(pypy): pyspark.streaming.tests
Starting test(python2.7): pyspark.mllib.tests
```
```
./run-tests --modules=pyspark-sql --python-executables=pypy
```
```
...
Will test against the following Python executables: ['pypy']
Will test the following Python modules: ['pyspark-sql']
PyArrow is not installed in Python executable 'pypy', skipping related
tests in 'pyspark-sql'.
Starting test(pypy): pyspark.sql.tests
Starting test(pypy): pyspark.sql.catalog
Starting test(pypy): pyspark.sql.column
Starting test(pypy): pyspark.sql.conf
```
These cases below were manually tested after modifying some codes.
```
./run-tests
```
```
...
Will test against the following Python executables: ['python2.7', 'pypy']
Will test the following Python modules: ['pyspark-core', 'pyspark-ml',
'pyspark-mllib', 'pyspark-sql', 'pyspark-streaming']
Pandas is not installed in Python executable 'python2.7', skipping related
tests in 'pyspark-sql'.
PyArrow is not installed in Python executable 'pypy', skipping related
tests in 'pyspark-sql'.
Pandas is not installed in Python executable 'pypy', skipping related tests
in 'pyspark-sql'.
Starting test(pypy): pyspark.sql.tests
Starting test(pypy): pyspark.streaming.tests
Starting test(pypy): pyspark.tests
Starting test(python2.7): pyspark.mllib.tests
```
```
./run-tests-with-coverage
```
```
...
Will test against the following Python executables: ['python2.7', 'pypy']
Will test the following Python modules: ['pyspark-core', 'pyspark-ml',
'pyspark-mllib', 'pyspark-sql', 'pyspark-streaming']
Coverage is not installed in Python executable 'python2.7' but
'COVERAGE_PROCESS_START' environment variable is set, exiting.
```
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/HyukjinKwon/spark SPARK-23300
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/20473.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #20473
----
commit 026104543bc2a9ea39e710f1df52e0c6ba15faab
Author: hyukjinkwon <gurwls223@...>
Date: 2018-02-01T12:55:19Z
Prints out if Pandas and PyArrow are installed or not in PySpark SQL tests
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
