This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push: new 0e65ed5 [SPARK-36345][INFRA] Update PySpark GitHubAction docker image to 20210730 0e65ed5 is described below commit 0e65ed5fb9c62671789a651a993abbb9f546367c Author: Dongjoon Hyun <dh...@apple.com> AuthorDate: Sat Jul 31 07:20:17 2021 +0900 [SPARK-36345][INFRA] Update PySpark GitHubAction docker image to 20210730 ### What changes were proposed in this pull request? This PR aims to upgrade PySpark GitHub Action job to use the latest docker image `20210730` having `sklearn` and `mlflow` additionally. - https://github.com/dongjoon-hyun/ApacheSparkGitHubActionImage/commit/5ca94453d1108dfe40bceb8872387a1b19b0c783 ``` $ docker run -it --rm dongjoon/apache-spark-github-action-image:20210730 python3.9 -m pip list | grep mlflow mlflow 1.19.0 $ docker run -it --rm dongjoon/apache-spark-github-action-image:20210730 python3.9 -m pip list | grep sklearn sklearn 0.0 ``` ### Why are the changes needed? This will save the installation time. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Pass the GitHub Action PySpark jobs. Closes #33595 from dongjoon-hyun/SPARK-36345. Authored-by: Dongjoon Hyun <dh...@apple.com> Signed-off-by: Hyukjin Kwon <gurwls...@apache.org> --- .github/workflows/build_and_test.yml | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/.github/workflows/build_and_test.yml b/.github/workflows/build_and_test.yml index 17908ff..58487a4 100644 --- a/.github/workflows/build_and_test.yml +++ b/.github/workflows/build_and_test.yml @@ -186,7 +186,7 @@ jobs: name: "Build modules: ${{ matrix.modules }}" runs-on: ubuntu-20.04 container: - image: dongjoon/apache-spark-github-action-image:20210602 + image: dongjoon/apache-spark-github-action-image:20210730 strategy: fail-fast: false matrix: @@ -252,8 +252,6 @@ jobs: # Run the tests. - name: Run tests run: | - # TODO(SPARK-36345): Install mlflow>=1.0 and sklearn in Python 3.9 of the base image - python3.9 -m pip install 'mlflow>=1.0' sklearn export PATH=$PATH:$HOME/miniconda/bin ./dev/run-tests --parallelism 1 --modules "$MODULES_TO_TEST" - name: Upload test results to report --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org