This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 80277eefae75 [SPARK-48745][INFRA][PYTHON][TESTS] Remove unnecessary
installation `miniconda` from `build_and_test.yml`
80277eefae75 is described below
commit 80277eefae75f2e509a096c15ecf2c60db361333
Author: panbingkun <[email protected]>
AuthorDate: Fri Jun 28 16:41:37 2024 +0900
[SPARK-48745][INFRA][PYTHON][TESTS] Remove unnecessary installation
`miniconda` from `build_and_test.yml`
### What changes were proposed in this pull request?
The pr aims to `remove` unnecessary installation `miniconda` from
`build_and_test.yml`(for `pyspark` tests).
### Why are the changes needed?
- To fix the issues encountered on `GA`, as following:
https://github.com/panbingkun/spark/actions/runs/9706791583/job/26791058294
<img width="994" alt="image"
src="https://github.com/apache/spark/assets/15246973/1bc31caa-e2e0-48c7-9396-89eb579a9215";>
- The `ubuntu-22.04` image we are currently using already contains the
`minicode`:
https://github.com/actions/runner-images/blob/main/images/ubuntu/Ubuntu2204-Readme.md
<img width="235" alt="image"
src="https://github.com/apache/spark/assets/15246973/eb1f6ac4-4c10-4c48-8abe-4520f9be3bf4";>
### Does this PR introduce _any_ user-facing change?
No, only for GA's PySpark tests.
### How was this patch tested?
Pass GA.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes #47138 from panbingkun/remove_miniconda.
Authored-by: panbingkun <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
---
.github/workflows/build_and_test.yml | 11 ++++-------
dev/run-pip-tests | 4 ++++
2 files changed, 8 insertions(+), 7 deletions(-)
diff --git a/.github/workflows/build_and_test.yml
b/.github/workflows/build_and_test.yml
index 881fb8cb0674..9dc9d85520c2 100644
--- a/.github/workflows/build_and_test.yml
+++ b/.github/workflows/build_and_test.yml
@@ -213,6 +213,8 @@ jobs:
INCLUDED_TAGS: ${{ matrix.included-tags }}
HADOOP_PROFILE: ${{ matrix.hadoop }}
HIVE_PROFILE: ${{ matrix.hive }}
+ # GitHub Actions' default miniconda to use in pip packaging test.
+ CONDA_PREFIX: /usr/share/miniconda
GITHUB_PREV_SHA: ${{ github.event.before }}
SPARK_LOCAL_IP: localhost
NOLINT_ON_COMPILE: true
@@ -400,6 +402,8 @@ jobs:
PYTHON_TO_TEST: 'python3.11'
HADOOP_PROFILE: ${{ inputs.hadoop }}
HIVE_PROFILE: hive2.3
+ # GitHub Actions' default miniconda to use in pip packaging test.
+ CONDA_PREFIX: /usr/share/miniconda
GITHUB_PREV_SHA: ${{ github.event.before }}
SPARK_LOCAL_IP: localhost
SKIP_UNIDOC: true
@@ -463,19 +467,12 @@ jobs:
echo $py
$py -m pip list
done
- - name: Install Conda for pip packaging test
- if: contains(matrix.modules, 'pyspark-errors')
- run: |
- curl -s
https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh >
miniconda.sh
- bash miniconda.sh -b -p $HOME/miniconda
- rm miniconda.sh
# Run the tests.
- name: Run tests
env: ${{ fromJSON(inputs.envs) }}
shell: 'script -q -e -c "bash {0}"'
run: |
if [[ "$MODULES_TO_TEST" == *"pyspark-errors"* ]]; then
- export PATH=$PATH:$HOME/miniconda/bin
export SKIP_PACKAGING=false
echo "Python Packaging Tests Enabled!"
fi
diff --git a/dev/run-pip-tests b/dev/run-pip-tests
index f8a547b0c917..91399ff1e25e 100755
--- a/dev/run-pip-tests
+++ b/dev/run-pip-tests
@@ -87,6 +87,10 @@ for python in "${PYTHON_EXECS[@]}"; do
VIRTUALENV_PATH="$VIRTUALENV_BASE"/$python
rm -rf "$VIRTUALENV_PATH"
if [ -n "$USE_CONDA" ]; then
+ if [ -f "$CONDA_PREFIX/etc/profile.d/conda.sh" ]; then
+ # See also https://github.com/conda/conda/issues/7980
+ source "$CONDA_PREFIX/etc/profile.d/conda.sh"
+ fi
conda create -y -p "$VIRTUALENV_PATH" python=$python numpy pandas pip
setuptools
source activate "$VIRTUALENV_PATH" || conda activate "$VIRTUALENV_PATH"
else
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
