This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch branch-2.4
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-2.4 by this push:
new b615f22 [SPARK-27544][PYTHON][TEST][BRANCH-2.4] Fix Python test
script to work on Scala-2.12 build
b615f22 is described below
commit b615f2272f59304101973b2450f8470128b0bb20
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Tue Apr 23 07:00:17 2019 -0700
[SPARK-27544][PYTHON][TEST][BRANCH-2.4] Fix Python test script to work on
Scala-2.12 build
## What changes were proposed in this pull request?
Since [SPARK-27274](https://issues.apache.org/jira/browse/SPARK-27274)
deprecated Scala-2.11 at Spark 2.4.1, we need to test Scala-2.12 more. This PR
aims to fix the Python test script on Scala-2.12 build in `branch-2.4`.
**BEFORE**
```
$ dev/change-scala-version.sh 2.12
$ build/sbt -Pscala-2.12 package
$ python/run-tests.py --python-executables python2.7 --modules pyspark-sql
Traceback (most recent call last):
File "python/run-tests.py", line 70, in <module>
raise Exception("Cannot find assembly build directory, please build
Spark first.")
Exception: Cannot find assembly build directory, please build Spark first.
```
**AFTER**
```
$ python/run-tests.py --python-executables python2.7 --modules pyspark-sql
Running PySpark tests. Output is in
/Users/dongjoon/APACHE/spark/python/unit-tests.log
Will test against the following Python executables: ['python2.7']
Will test the following Python modules: ['pyspark-sql']
Starting test(python2.7): pyspark.sql.tests
...
```
## How was this patch tested?
Manually do the above procedure because Jenkins doesn't test Scala-2.12 in
`branch-2.4`.
Closes #24439 from dongjoon-hyun/SPARK-27544.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
---
python/run-tests.py | 4 +---
1 file changed, 1 insertion(+), 3 deletions(-)
diff --git a/python/run-tests.py b/python/run-tests.py
index 921fdc9..5059905 100755
--- a/python/run-tests.py
+++ b/python/run-tests.py
@@ -59,9 +59,7 @@ FAILURE_REPORTING_LOCK = Lock()
LOGGER = logging.getLogger()
# Find out where the assembly jars are located.
-# Later, add back 2.12 to this list:
-# for scala in ["2.11", "2.12"]:
-for scala in ["2.11"]:
+for scala in ["2.11", "2.12"]:
build_dir = os.path.join(SPARK_HOME, "assembly", "target", "scala-" +
scala)
if os.path.isdir(build_dir):
SPARK_DIST_CLASSPATH = os.path.join(build_dir, "jars", "*")
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
