This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.0 by this push:
new f8d0688 [SPARK-35393][PYTHON][INFRA][TESTS] Recover pip packaging
test in Github Actions
f8d0688 is described below
commit f8d06883853ff711d5d20e895a5eecfb87bd0131
Author: Hyukjin Kwon <gurwls...@apache.org>
AuthorDate: Thu May 13 10:35:56 2021 -0700
[SPARK-35393][PYTHON][INFRA][TESTS] Recover pip packaging test in Github
Actions
Currently pip packaging test is being skipped:
```
========================================================================
Running PySpark packaging tests
========================================================================
Constructing virtual env for testing
Missing virtualenv & conda, skipping pip installability tests
Cleaning up temporary directory - /tmp/tmp.iILYWISPXW
```
See https://github.com/apache/spark/runs/2568923639?check_suite_focus=true
GitHub Actions's image has its default Conda installed at
`/usr/share/miniconda` but seems like the image we're using for PySpark does
not have it (which is legitimate).
This PR proposes to install Conda to use in pip packaging tests in GitHub
Actions.
To recover the test coverage.
No, dev-only.
It was tested in my fork:
https://github.com/HyukjinKwon/spark/runs/2575126882?check_suite_focus=true
```
========================================================================
Running PySpark packaging tests
========================================================================
Constructing virtual env for testing
Using conda virtual environments
Testing pip installation with python 3.6
Using /tmp/tmp.qPjTenqfGn for virtualenv
Collecting package metadata (current_repodata.json): ...working... done
Solving environment: ...working... failed with repodata from
current_repodata.json, will retry with next repodata source.
Collecting package metadata (repodata.json): ...working... done
Solving environment: ...working... done
environment location: /tmp/tmp.qPjTenqfGn/3.6
added / updated specs:
- numpy
- pandas
- pip
- python=3.6
- setuptools
...
Successfully ran pip sanity check
```
Closes #32537 from HyukjinKwon/SPARK-35393.
Authored-by: Hyukjin Kwon <gurwls...@apache.org>
Signed-off-by: Dongjoon Hyun <dh...@apple.com>
(cherry picked from commit 7d371d27f2a974b682ffa16b71576e61e9338c34)
Signed-off-by: Hyukjin Kwon <gurwls...@apache.org>
---
.github/workflows/build_and_test.yml | 11 ++++++-----
dev/run-pip-tests | 4 ----
2 files changed, 6 insertions(+), 9 deletions(-)
diff --git a/.github/workflows/build_and_test.yml
b/.github/workflows/build_and_test.yml
index 8749499..936a256 100644
--- a/.github/workflows/build_and_test.yml
+++ b/.github/workflows/build_and_test.yml
@@ -76,8 +76,6 @@ jobs:
INCLUDED_TAGS: ${{ matrix.included-tags }}
HADOOP_PROFILE: ${{ matrix.hadoop }}
HIVE_PROFILE: ${{ matrix.hive }}
- # GitHub Actions' default miniconda to use in pip packaging test.
- CONDA_PREFIX: /usr/share/miniconda
GITHUB_PREV_SHA: ${{ github.event.before }}
SPARK_LOCAL_IP: localhost
steps:
@@ -160,8 +158,6 @@ jobs:
MODULES_TO_TEST: ${{ matrix.modules }}
HADOOP_PROFILE: hadoop2.7
HIVE_PROFILE: hive2.3
- # GitHub Actions' default miniconda to use in pip packaging test.
- CONDA_PREFIX: /usr/share/miniconda
GITHUB_PREV_SHA: ${{ github.event.before }}
SPARK_LOCAL_IP: localhost
steps:
@@ -195,13 +191,18 @@ jobs:
with:
python-version: 2.7
architecture: x64
- - name: Install Python packages (Python 2.7 )
+ - name: Install Python packages (Python 2.7)
run: |
python2.7 -m pip install numpy 'pyarrow<3.0.0' pandas scipy xmlrunner
python2.7 -m pip list
+ - name: Install Conda for pip packaging test
+ run: |
+ curl -s
https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh >
miniconda.sh
+ bash miniconda.sh -b -p $HOME/miniconda
# Run the tests.
- name: Run tests
run: |
+ export PATH=$PATH:$HOME/miniconda/bin
./dev/run-tests --parallelism 2 --modules "$MODULES_TO_TEST"
- name: Upload test results to report
if: always()
diff --git a/dev/run-pip-tests b/dev/run-pip-tests
index b322d3f..cb64e88 100755
--- a/dev/run-pip-tests
+++ b/dev/run-pip-tests
@@ -80,10 +80,6 @@ for python in "${PYTHON_EXECS[@]}"; do
VIRTUALENV_PATH="$VIRTUALENV_BASE"/$python
rm -rf "$VIRTUALENV_PATH"
if [ -n "$USE_CONDA" ]; then
- if [ -f "$CONDA_PREFIX/etc/profile.d/conda.sh" ]; then
- # See also https://github.com/conda/conda/issues/7980
- source "$CONDA_PREFIX/etc/profile.d/conda.sh"
- fi
conda create -y -p "$VIRTUALENV_PATH" python=$python numpy pandas pip
setuptools
source activate "$VIRTUALENV_PATH" || conda activate "$VIRTUALENV_PATH"
else
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
