This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 277a406 [SPARK-32422][SQL][TESTS] Use python3 executable instead of
python3.6 in IntegratedUDFTestUtils
277a406 is described below
commit 277a4063ef4ff6e91d999864b820607fa1a206aa
Author: HyukjinKwon <[email protected]>
AuthorDate: Sat Jul 25 03:06:45 2020 -0700
[SPARK-32422][SQL][TESTS] Use python3 executable instead of python3.6 in
IntegratedUDFTestUtils
### What changes were proposed in this pull request?
This PR uses `python3` instead of `python3.6` executable as a fallback in
`IntegratedUDFTestUtils`.
### Why are the changes needed?
Currently, GitHub Actions skips pandas UDFs. Python 3.8 is installed
explicitly but somehow `python3.6` looks available in GitHub Actions build
environment by default.
```
[info] - udf/postgreSQL/udf-case.sql - Scalar Pandas UDF is skipped because
pyspark,pandas and/or pyarrow were not available in [python3.6]. !!! IGNORED !!!
...
[info] - udf/postgreSQL/udf-select_having.sql - Scalar Pandas UDF is
skipped because pyspark,pandas and/or pyarrow were not available in
[python3.6]. !!! IGNORED !!!
...
```
It was chosen as `python3.6` for Jenkins to pick one Python explicitly;
however, looks we're already using `python3` here and there.
It will also reduce the overhead to fix when we deprecate or drop Python
versions.
### Does this PR introduce _any_ user-facing change?
No, dev-only.
### How was this patch tested?
It should be tested in Jenkins and GitHub Actions environments here.
Closes #29217 from HyukjinKwon/SPARK-32422.
Authored-by: HyukjinKwon <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
---
.../src/test/scala/org/apache/spark/sql/IntegratedUDFTestUtils.scala | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/IntegratedUDFTestUtils.scala
b/sql/core/src/test/scala/org/apache/spark/sql/IntegratedUDFTestUtils.scala
index 6391d56..80346b3 100644
--- a/sql/core/src/test/scala/org/apache/spark/sql/IntegratedUDFTestUtils.scala
+++ b/sql/core/src/test/scala/org/apache/spark/sql/IntegratedUDFTestUtils.scala
@@ -197,7 +197,7 @@ object IntegratedUDFTestUtils extends SQLHelper {
lazy val pythonExec: String = {
val pythonExec = sys.env.getOrElse(
- "PYSPARK_DRIVER_PYTHON", sys.env.getOrElse("PYSPARK_PYTHON",
"python3.6"))
+ "PYSPARK_DRIVER_PYTHON", sys.env.getOrElse("PYSPARK_PYTHON", "python3"))
if (TestUtils.testCommandAvailable(pythonExec)) {
pythonExec
} else {
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
