This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 76afa1cf2731 [SPARK-54221][INFRA] Add PyPy 3.11 GitHub Action job
76afa1cf2731 is described below
commit 76afa1cf2731a5df2a686794d7355a7cba5f4042
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Thu Nov 6 15:36:27 2025 -0800
[SPARK-54221][INFRA] Add PyPy 3.11 GitHub Action job
### What changes were proposed in this pull request?
This PR aims to add `PyPy 3.11` GitHub Action job.
### Why are the changes needed?
From PyPy v7.3.20, `Python 3.11` is the provided version. `Python 3.10` is
not supported.
<img width="216" height="244" alt="Screenshot 2025-11-06 at 08 24 37"
src="https://github.com/user-attachments/assets/1df6fe38-b3ae-4fa3-9219-8b60b310d424";
/>
- https://downloads.python.org/pypy/
### Does this PR introduce _any_ user-facing change?
No, this is a new addition to the test infra.
### How was this patch tested?
Pass the CIs.
For docker file, I manually tested like the following.
```
$ docker buildx build --platform linux/amd64 .
[+] Building 0.9s (10/10) FINISHED
docker-container:serene_visvesvaraya
=> [internal] load build definition from Dockerfile
0.0s
=> => transferring dockerfile: 2.62kB
0.0s
=> [internal] load metadata for docker.io/library/ubuntu:jammy-20240911.1
0.8s
=> [auth] library/ubuntu:pull token for registry-1.docker.io
0.0s
=> [internal] load .dockerignore
0.0s
=> => transferring context: 2B
0.0s
=> [1/6] FROM
docker.io/library/ubuntu:jammy-20240911.1sha256:0e5e4a57c2499249aafc3b40fcd541e9a456aab7296681a3994d631587203f97
0.0s
=> => resolve
docker.io/library/ubuntu:jammy-20240911.1sha256:0e5e4a57c2499249aafc3b40fcd541e9a456aab7296681a3994d631587203f97
0.0s
=> CACHED [2/6] RUN apt-get update && apt-get install -y
build-essential ca-certificates curl gfortran git gnupg
libcurl4-openssl-dev libfontconfig1-dev 0.0s
=> CACHED [3/6] RUN add-apt-repository ppa:pypy/ppa
0.0s
=> CACHED [4/6] RUN mkdir -p /usr/local/pypy/pypy3.11 && curl -sqL
https://downloads.python.org/pypy/pypy3.11-v7.3.20-linux64.tar.bz2 | tar xjf -
-C /usr/local/pypy/pypy3.11 --strip-co 0.0s
=> CACHED [5/6] RUN curl -sS https://bootstrap.pypa.io/get-pip.py | pypy3
0.0s
=> CACHED [6/6] RUN pypy3 -m pip install numpy 'six==1.16.0'
'pandas==2.3.3' scipy coverage matplotlib lxml
0.0s
WARNING: No output specified with docker-container driver. Build result
will only remain in the build cache. To push result image into registry use
--push or to load image into docker use --load
What's next:
View a summary of image vulnerabilities and recommendations → docker
scout quickview
```
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes #52921 from dongjoon-hyun/SPARK-54221.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
---
.github/workflows/build_infra_images_cache.yml | 14 +++++
.github/workflows/build_python_pypy3.11.yml | 47 +++++++++++++++++
dev/spark-test-image/pypy-311/Dockerfile | 72 ++++++++++++++++++++++++++
3 files changed, 133 insertions(+)
diff --git a/.github/workflows/build_infra_images_cache.yml
b/.github/workflows/build_infra_images_cache.yml
index 430903b570ea..455512212d2c 100644
--- a/.github/workflows/build_infra_images_cache.yml
+++ b/.github/workflows/build_infra_images_cache.yml
@@ -33,6 +33,7 @@ on:
- 'dev/spark-test-image/python-minimum/Dockerfile'
- 'dev/spark-test-image/python-ps-minimum/Dockerfile'
- 'dev/spark-test-image/pypy-310/Dockerfile'
+ - 'dev/spark-test-image/pypy-311/Dockerfile'
- 'dev/spark-test-image/python-310/Dockerfile'
- 'dev/spark-test-image/python-311/Dockerfile'
- 'dev/spark-test-image/python-311-classic-only/Dockerfile'
@@ -153,6 +154,19 @@ jobs:
- name: Image digest (PySpark with PyPy 3.10)
if: hashFiles('dev/spark-test-image/pypy-310/Dockerfile') != ''
run: echo ${{ steps.docker_build_pyspark_pypy_310.outputs.digest }}
+ - name: Build and push (PySpark with PyPy 3.11)
+ if: hashFiles('dev/spark-test-image/pypy-311/Dockerfile') != ''
+ id: docker_build_pyspark_pypy_311
+ uses: docker/build-push-action@v6
+ with:
+ context: ./dev/spark-test-image/pypy-311/
+ push: true
+ tags:
ghcr.io/apache/spark/apache-spark-github-action-image-pyspark-pypy-311-cache:${{
github.ref_name }}-static
+ cache-from:
type=registry,ref=ghcr.io/apache/spark/apache-spark-github-action-image-pyspark-pypy-311-cache:${{
github.ref_name }}
+ cache-to:
type=registry,ref=ghcr.io/apache/spark/apache-spark-github-action-image-pyspark-pypy-311-cache:${{
github.ref_name }},mode=max
+ - name: Image digest (PySpark with PyPy 3.11)
+ if: hashFiles('dev/spark-test-image/pypy-311/Dockerfile') != ''
+ run: echo ${{ steps.docker_build_pyspark_pypy_311.outputs.digest }}
- name: Build and push (PySpark with Python 3.10)
if: hashFiles('dev/spark-test-image/python-310/Dockerfile') != ''
id: docker_build_pyspark_python_310
diff --git a/.github/workflows/build_python_pypy3.11.yml
b/.github/workflows/build_python_pypy3.11.yml
new file mode 100644
index 000000000000..29794f89b64c
--- /dev/null
+++ b/.github/workflows/build_python_pypy3.11.yml
@@ -0,0 +1,47 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+name: "Build / Python-only (master, PyPy 3.11)"
+
+on:
+ schedule:
+ - cron: '0 15 * * *'
+ workflow_dispatch:
+
+jobs:
+ run-build:
+ permissions:
+ packages: write
+ name: Run
+ uses: ./.github/workflows/build_and_test.yml
+ if: github.repository == 'apache/spark'
+ with:
+ java: 17
+ branch: master
+ hadoop: hadoop3
+ envs: >-
+ {
+ "PYSPARK_IMAGE_TO_TEST": "pypy-311",
+ "PYTHON_TO_TEST": "pypy3"
+ }
+ jobs: >-
+ {
+ "pyspark": "true",
+ "pyspark-pandas": "true"
+ }
diff --git a/dev/spark-test-image/pypy-311/Dockerfile
b/dev/spark-test-image/pypy-311/Dockerfile
new file mode 100644
index 000000000000..120e7303851b
--- /dev/null
+++ b/dev/spark-test-image/pypy-311/Dockerfile
@@ -0,0 +1,72 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# Image for building and testing Spark branches. Based on Ubuntu 22.04.
+# See also in https://hub.docker.com/_/ubuntu
+FROM ubuntu:jammy-20240911.1
+LABEL org.opencontainers.image.authors="Apache Spark project
<[email protected]>"
+LABEL org.opencontainers.image.licenses="Apache-2.0"
+LABEL org.opencontainers.image.ref.name="Apache Spark Infra Image For PySpark
with PyPy 3.11"
+# Overwrite this label to avoid exposing the underlying Ubuntu OS version label
+LABEL org.opencontainers.image.version=""
+
+ENV FULL_REFRESH_DATE=20251106
+
+ENV DEBIAN_FRONTEND=noninteractive
+ENV DEBCONF_NONINTERACTIVE_SEEN=true
+
+RUN apt-get update && apt-get install -y \
+ build-essential \
+ ca-certificates \
+ curl \
+ gfortran \
+ git \
+ gnupg \
+ libcurl4-openssl-dev \
+ libfontconfig1-dev \
+ libfreetype6-dev \
+ libfribidi-dev \
+ libgit2-dev \
+ libharfbuzz-dev \
+ libjpeg-dev \
+ liblapack-dev \
+ libopenblas-dev \
+ libpng-dev \
+ libpython3-dev \
+ libssl-dev \
+ libtiff5-dev \
+ libwebp-dev \
+ libxml2-dev \
+ openjdk-17-jdk-headless \
+ pkg-config \
+ qpdf \
+ tzdata \
+ software-properties-common \
+ wget \
+ zlib1g-dev \
+ && apt-get autoremove --purge -y \
+ && apt-get clean \
+ && rm -rf /var/lib/apt/lists/*
+
+
+RUN add-apt-repository ppa:pypy/ppa
+RUN mkdir -p /usr/local/pypy/pypy3.11 && \
+ curl -sqL
https://downloads.python.org/pypy/pypy3.11-v7.3.20-linux64.tar.bz2 | tar xjf -
-C /usr/local/pypy/pypy3.11 --strip-components=1 && \
+ ln -sf /usr/local/pypy/pypy3.11/bin/pypy /usr/local/bin/pypy3.11 && \
+ ln -sf /usr/local/pypy/pypy3.11/bin/pypy /usr/local/bin/pypy3
+RUN curl -sS https://bootstrap.pypa.io/get-pip.py | pypy3
+RUN pypy3 -m pip install numpy 'six==1.16.0' 'pandas==2.3.3' scipy coverage
matplotlib lxml
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
