This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 6dda3258533f [SPARK-47096][INFRA] Upgrade Python version in Maven
builds
6dda3258533f is described below
commit 6dda3258533fed149788cbf0d58e902ea78d51ee
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Mon Feb 19 16:58:01 2024 -0800
[SPARK-47096][INFRA] Upgrade Python version in Maven builds
### What changes were proposed in this pull request?
This PR upgrades Python version to make the build compatible with macos-14
### Why are the changes needed?
macos-14 build fails during Python 3.9 installation, see
https://github.com/apache/spark/actions/runs/7964626045/job/21742574260
```
Error: dyld[4738]: Library not loaded:
/usr/local/opt/gettext/lib/libintl.8.dylib
Referenced from: <E2FD7085-038F
Error: -3B8E-94C6-6649527BFDBE>
/Users/runner/hostedtoolcache/Python/3.9.18/x64/bin/python3.9
Reason: tried: '/usr/local/opt/gettext/lib/libintl.8.dylib' (no such
file),
'/System/Volumes/Preboot/Cryptexes/OS/usr/local/opt/gettext/lib/libintl.8.dylib'
(no such file), '/usr/local/opt/gettext/lib/libintl.8.dylib' (no such file),
'/usr/local/lib/libintl.8.dylib' (no such file), '/usr/lib/libintl.8.dylib' (no
such file, not in dyld cache)
./setup.sh: line 52: 4738 Abort trap: 6 ./python -m ensurepip
```
In fact, marcos-14 already has Python 3.11
(https://github.com/actions/runner-images/blob/main/images/macos/macos-14-Readme.md).
### Does this PR introduce _any_ user-facing change?
No, test-only.
### How was this patch tested?
Will monitor the scheduled build.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes #45172 from HyukjinKwon/SPARK-47096.
Lead-authored-by: Hyukjin Kwon <[email protected]>
Co-authored-by: Hyukjin Kwon <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
---
.github/workflows/maven_test.yml | 14 ++++++++------
1 file changed, 8 insertions(+), 6 deletions(-)
diff --git a/.github/workflows/maven_test.yml b/.github/workflows/maven_test.yml
index ed2af672d6e0..80898b3a507a 100644
--- a/.github/workflows/maven_test.yml
+++ b/.github/workflows/maven_test.yml
@@ -159,20 +159,22 @@ jobs:
with:
distribution: zulu
java-version: ${{ matrix.java }}
- - name: Install Python 3.9
+ - name: Install Python 3.11
uses: actions/setup-python@v5
# We should install one Python that is higher than 3+ for SQL and Yarn
because:
# - SQL component also has Python related tests, for example,
IntegratedUDFTestUtils.
# - Yarn has a Python specific test too, for example, YarnClusterSuite.
- if: contains(matrix.modules, 'resource-managers#yarn') ||
(contains(matrix.modules, 'sql#core')) || contains(matrix.modules, 'connect')
+ # macos (14) already has its Python installed, see also SPARK-47096 and
+ #
https://github.com/actions/runner-images/blob/main/images/macos/macos-14-Readme.md
+ if: contains(inputs.os, 'ubuntu') && (contains(matrix.modules,
'resource-managers#yarn') || (contains(matrix.modules, 'sql#core')) ||
contains(matrix.modules, 'connect'))
with:
- python-version: '3.9'
+ python-version: '3.11'
architecture: x64
- - name: Install Python packages (Python 3.9)
+ - name: Install Python packages (Python 3.11)
if: (contains(matrix.modules, 'sql#core')) || contains(matrix.modules,
'connect')
run: |
- python3.9 -m pip install 'numpy>=1.20.0' pyarrow pandas scipy
unittest-xml-reporting 'grpcio==1.59.3' 'grpcio-status==1.59.3'
'protobuf==4.25.1'
- python3.9 -m pip list
+ python3.11 -m pip install 'numpy>=1.20.0' pyarrow pandas scipy
unittest-xml-reporting 'grpcio==1.59.3' 'grpcio-status==1.59.3'
'protobuf==4.25.1'
+ python3.11 -m pip list
# Run the tests.
- name: Run tests
env: ${{ fromJSON(inputs.envs) }}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
