This is an automated email from the ASF dual-hosted git repository.
ruifengz pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new e4be5e67fefc [SPARK-50552][INFRA] Add a separate docker file for PyPy
3.10 daily build
e4be5e67fefc is described below
commit e4be5e67fefc73b600954352d10c1e1bce5e35d9
Author: Ruifeng Zheng <[email protected]>
AuthorDate: Thu Dec 12 16:09:18 2024 +0800
[SPARK-50552][INFRA] Add a separate docker file for PyPy 3.10 daily build
### What changes were proposed in this pull request?
Add a separate docker file for PyPy 3.10 daily build
### Why are the changes needed?
to isolate the environments
### Does this PR introduce _any_ user-facing change?
no, infra-only
### How was this patch tested?
PR builder with `env`:
```
default: '{"PYSPARK_IMAGE_TO_TEST": "pypy-310", "PYTHON_TO_TEST": "pypy3"}'
```
https://github.com/zhengruifeng/spark/runs/34293616750
### Was this patch authored or co-authored using generative AI tooling?
no
Closes #49160 from zhengruifeng/py_image_pypy_310.
Authored-by: Ruifeng Zheng <[email protected]>
Signed-off-by: Ruifeng Zheng <[email protected]>
---
.github/workflows/build_infra_images_cache.yml | 14 +++++
.github/workflows/build_python_pypy3.10.yml | 1 +
dev/spark-test-image/pypy-310/Dockerfile | 71 ++++++++++++++++++++++++++
3 files changed, 86 insertions(+)
diff --git a/.github/workflows/build_infra_images_cache.yml
b/.github/workflows/build_infra_images_cache.yml
index adccc64261f5..df927da57c24 100644
--- a/.github/workflows/build_infra_images_cache.yml
+++ b/.github/workflows/build_infra_images_cache.yml
@@ -30,6 +30,7 @@ on:
- 'dev/spark-test-image/docs/Dockerfile'
- 'dev/spark-test-image/lint/Dockerfile'
- 'dev/spark-test-image/sparkr/Dockerfile'
+ - 'dev/spark-test-image/pypy-310/Dockerfile'
- 'dev/spark-test-image/python-309/Dockerfile'
- 'dev/spark-test-image/python-310/Dockerfile'
- 'dev/spark-test-image/python-312/Dockerfile'
@@ -106,6 +107,19 @@ jobs:
- name: Image digest (SparkR)
if: hashFiles('dev/spark-test-image/sparkr/Dockerfile') != ''
run: echo ${{ steps.docker_build_sparkr.outputs.digest }}
+ - name: Build and push (PySpark with PyPy 3.10)
+ if: hashFiles('dev/spark-test-image/pypy-310/Dockerfile') != ''
+ id: docker_build_pyspark_pypy_310
+ uses: docker/build-push-action@v6
+ with:
+ context: ./dev/spark-test-image/pypy-310/
+ push: true
+ tags:
ghcr.io/apache/spark/apache-spark-github-action-image-pyspark-pypy-310-cache:${{
github.ref_name }}-static
+ cache-from:
type=registry,ref=ghcr.io/apache/spark/apache-spark-github-action-image-pyspark-pypy-310-cache:${{
github.ref_name }}
+ cache-to:
type=registry,ref=ghcr.io/apache/spark/apache-spark-github-action-image-pyspark-pypy-310-cache:${{
github.ref_name }},mode=max
+ - name: Image digest (PySpark with PyPy 3.10)
+ if: hashFiles('dev/spark-test-image/pypy-310/Dockerfile') != ''
+ run: echo ${{ steps.docker_build_pyspark_pypy_310.outputs.digest }}
- name: Build and push (PySpark with Python 3.9)
if: hashFiles('dev/spark-test-image/python-309/Dockerfile') != ''
id: docker_build_pyspark_python_309
diff --git a/.github/workflows/build_python_pypy3.10.yml
b/.github/workflows/build_python_pypy3.10.yml
index 163af2f4aec8..c590427f6f49 100644
--- a/.github/workflows/build_python_pypy3.10.yml
+++ b/.github/workflows/build_python_pypy3.10.yml
@@ -36,6 +36,7 @@ jobs:
hadoop: hadoop3
envs: >-
{
+ "PYSPARK_IMAGE_TO_TEST": "pypy-310",
"PYTHON_TO_TEST": "pypy3"
}
jobs: >-
diff --git a/dev/spark-test-image/pypy-310/Dockerfile
b/dev/spark-test-image/pypy-310/Dockerfile
new file mode 100644
index 000000000000..7d6ff26566de
--- /dev/null
+++ b/dev/spark-test-image/pypy-310/Dockerfile
@@ -0,0 +1,71 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# Image for building and testing Spark branches. Based on Ubuntu 22.04.
+# See also in https://hub.docker.com/_/ubuntu
+FROM ubuntu:jammy-20240911.1
+LABEL org.opencontainers.image.authors="Apache Spark project
<[email protected]>"
+LABEL org.opencontainers.image.licenses="Apache-2.0"
+LABEL org.opencontainers.image.ref.name="Apache Spark Infra Image For PySpark
with PyPy 3.10"
+# Overwrite this label to avoid exposing the underlying Ubuntu OS version label
+LABEL org.opencontainers.image.version=""
+
+ENV FULL_REFRESH_DATE 20241212
+
+ENV DEBIAN_FRONTEND noninteractive
+ENV DEBCONF_NONINTERACTIVE_SEEN true
+
+RUN apt-get update && apt-get install -y \
+ build-essential \
+ ca-certificates \
+ curl \
+ gfortran \
+ git \
+ gnupg \
+ libcurl4-openssl-dev \
+ libfontconfig1-dev \
+ libfreetype6-dev \
+ libfribidi-dev \
+ libgit2-dev \
+ libharfbuzz-dev \
+ libjpeg-dev \
+ liblapack-dev \
+ libopenblas-dev \
+ libpng-dev \
+ libpython3-dev \
+ libssl-dev \
+ libtiff5-dev \
+ libxml2-dev \
+ openjdk-17-jdk-headless \
+ pkg-config \
+ qpdf \
+ tzdata \
+ software-properties-common \
+ wget \
+ zlib1g-dev \
+ && apt-get autoremove --purge -y \
+ && apt-get clean \
+ && rm -rf /var/lib/apt/lists/*
+
+
+RUN add-apt-repository ppa:pypy/ppa
+RUN mkdir -p /usr/local/pypy/pypy3.10 && \
+ curl -sqL
https://downloads.python.org/pypy/pypy3.10-v7.3.17-linux64.tar.bz2 | tar xjf -
-C /usr/local/pypy/pypy3.10 --strip-components=1 && \
+ ln -sf /usr/local/pypy/pypy3.10/bin/pypy /usr/local/bin/pypy3.10 && \
+ ln -sf /usr/local/pypy/pypy3.10/bin/pypy /usr/local/bin/pypy3
+RUN curl -sS https://bootstrap.pypa.io/get-pip.py | pypy3
+RUN pypy3 -m pip install numpy 'six==1.16.0' 'pandas==2.2.3' scipy coverage
matplotlib lxml
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
