This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a commit to branch branch-3.2
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.2 by this push:
new 586f898 [SPARK-36345][INFRA] Update PySpark GitHubAction docker image
to 20210730
586f898 is described below
commit 586f89886fc8029d35aaa11021a4a88909c85804
Author: Dongjoon Hyun <dh...@apple.com>
AuthorDate: Sat Jul 31 07:20:17 2021 +0900
[SPARK-36345][INFRA] Update PySpark GitHubAction docker image to 20210730
### What changes were proposed in this pull request?
This PR aims to upgrade PySpark GitHub Action job to use the latest docker
image `20210730` having `sklearn` and `mlflow` additionally.
-
https://github.com/dongjoon-hyun/ApacheSparkGitHubActionImage/commit/5ca94453d1108dfe40bceb8872387a1b19b0c783
```
$ docker run -it --rm dongjoon/apache-spark-github-action-image:20210730
python3.9 -m pip list | grep mlflow
mlflow 1.19.0
$ docker run -it --rm dongjoon/apache-spark-github-action-image:20210730
python3.9 -m pip list | grep sklearn
sklearn 0.0
```
### Why are the changes needed?
This will save the installation time.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Pass the GitHub Action PySpark jobs.
Closes #33595 from dongjoon-hyun/SPARK-36345.
Authored-by: Dongjoon Hyun <dh...@apple.com>
Signed-off-by: Hyukjin Kwon <gurwls...@apache.org>
(cherry picked from commit 0e65ed5fb9c62671789a651a993abbb9f546367c)
Signed-off-by: Hyukjin Kwon <gurwls...@apache.org>
---
.github/workflows/build_and_test.yml | 4 +---
1 file changed, 1 insertion(+), 3 deletions(-)
diff --git a/.github/workflows/build_and_test.yml
b/.github/workflows/build_and_test.yml
index 3eb12f5..d247e6b 100644
--- a/.github/workflows/build_and_test.yml
+++ b/.github/workflows/build_and_test.yml
@@ -149,7 +149,7 @@ jobs:
name: "Build modules: ${{ matrix.modules }}"
runs-on: ubuntu-20.04
container:
- image: dongjoon/apache-spark-github-action-image:20210602
+ image: dongjoon/apache-spark-github-action-image:20210730
strategy:
fail-fast: false
matrix:
@@ -227,8 +227,6 @@ jobs:
# Run the tests.
- name: Run tests
run: |
- # TODO(SPARK-36345): Install mlflow>=1.0 and sklearn in Python 3.9 of
the base image
- python3.9 -m pip install 'mlflow>=1.0' sklearn
export PATH=$PATH:$HOME/miniconda/bin
./dev/run-tests --parallelism 1 --modules "$MODULES_TO_TEST"
- name: Upload test results to report
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
