This is an automated email from the ASF dual-hosted git repository.
dongjoon-hyun pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new 29caede3dcca [SPARK-57771][INFRA] Use Python 3.11 in `spark-rm`
release image
29caede3dcca is described below
commit 29caede3dcca6f94a2ec1f7491593b9668c38925
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Tue Jun 30 13:10:37 2026 -0700
[SPARK-57771][INFRA] Use Python 3.11 in `spark-rm` release image
### What changes were proposed in this pull request?
This PR uses `Python 3.11` instead of 3.10 in
`dev/create-release/spark-rm/Dockerfile` for Apache Spark 4.3.0+.
### Why are the changes needed?
Since `Python 3.10` reaches end-of-life in October 2026 before Apache Spark
4.3.0 release, Apache Spark 3.4.0 dropped it.
- https://github.com/apache/spark/pull/55914
So, ideally, we need to use Python 3.11+ in the release script. However,
recently, on 2026-01-23, we share `Dockerfile.base` for all maintenance
releases and `ubuntu:jammy` has only `Python 3.11.0rc1`. Since `Python
3.11.0rc1` is better than the EOL `Python 3.10`, this PR upgrades it only for
Spark release.
- https://github.com/apache/spark/pull/53905
https://github.com/apache/spark/blob/e28edaf0480e19abca83cd2b215450b85fdaf438/dev/create-release/spark-rm/Dockerfile.base#L30
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Manual review.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 4.8
Closes #56885 from dongjoon-hyun/SPARK-57771.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
(cherry picked from commit a57c537e1a4898f6686c2202a30673186389c606)
Signed-off-by: Dongjoon Hyun <[email protected]>
---
dev/create-release/spark-rm/Dockerfile | 34 +++++++++++++++++-----------------
1 file changed, 17 insertions(+), 17 deletions(-)
diff --git a/dev/create-release/spark-rm/Dockerfile
b/dev/create-release/spark-rm/Dockerfile
index fffe7e387a31..d40e9de45741 100644
--- a/dev/create-release/spark-rm/Dockerfile
+++ b/dev/create-release/spark-rm/Dockerfile
@@ -18,7 +18,7 @@
# Spark master (4.2-SNAPSHOT) release image
# Extends the base image with:
# - Java 17
-# - Python 3.10 with required packages
+# - Python 3.11 with required packages
FROM spark-rm-base:latest
@@ -27,16 +27,16 @@ RUN apt-get update && apt-get install -y \
openjdk-17-jdk-headless \
&& rm -rf /var/lib/apt/lists/*
-# Install Python 3.10
+# Install Python 3.11
RUN apt-get update && apt-get install -y \
- python3.10 \
- python3.10-dev \
+ python3.11 \
+ python3.11-dev \
python3-psutil \
libpython3-dev \
&& rm -rf /var/lib/apt/lists/*
-# Install pip for Python 3.10
-RUN curl -sS https://bootstrap.pypa.io/get-pip.py | python3.10
+# Install pip for Python 3.11
+RUN curl -sS https://bootstrap.pypa.io/get-pip.py | python3.11
# Basic Python packages for master
ARG BASIC_PIP_PKGS="numpy pyarrow>=18.0.0 six==1.16.0 pandas==2.3.3 scipy
plotly<6.0.0 \
@@ -46,18 +46,18 @@ ARG BASIC_PIP_PKGS="numpy pyarrow>=18.0.0 six==1.16.0
pandas==2.3.3 scipy plotly
ARG CONNECT_PIP_PKGS="grpcio==1.76.0 grpcio-status==1.76.0 protobuf==6.33.5 \
googleapis-common-protos==1.71.0 graphviz==0.20.3"
-# Install Python 3.10 packages
-RUN python3.10 -m pip install --ignore-installed 'blinker>=1.6.2' && \
- python3.10 -m pip install --ignore-installed 'six==1.16.0' && \
- python3.10 -m pip install $BASIC_PIP_PKGS unittest-xml-reporting
$CONNECT_PIP_PKGS && \
- python3.10 -m pip install 'torch<2.6.0' torchvision --index-url
https://download.pytorch.org/whl/cpu && \
- python3.10 -m pip install deepspeed torcheval && \
- python3.10 -m pip cache purge
+# Install Python 3.11 packages
+RUN python3.11 -m pip install --ignore-installed 'blinker>=1.6.2' && \
+ python3.11 -m pip install --ignore-installed 'six==1.16.0' && \
+ python3.11 -m pip install $BASIC_PIP_PKGS unittest-xml-reporting
$CONNECT_PIP_PKGS && \
+ python3.11 -m pip install 'torch<2.6.0' torchvision --index-url
https://download.pytorch.org/whl/cpu && \
+ python3.11 -m pip install deepspeed torcheval && \
+ python3.11 -m pip cache purge
# Sphinx and documentation packages
# Should unpin 'sphinxcontrib-*' after upgrading sphinx>5
# See 'ipython_genutils' in SPARK-38517, 'docutils<0.18.0' in SPARK-39421
-RUN python3.10 -m pip install 'sphinx==4.5.0' mkdocs
'pydata_sphinx_theme>=0.13' \
+RUN python3.11 -m pip install 'sphinx==4.5.0' mkdocs
'pydata_sphinx_theme>=0.13' \
sphinx-copybutton nbsphinx numpydoc jinja2 markupsafe 'pyzmq<24.0.0' \
ipython ipython_genutils sphinx_plotly_directive 'numpy>=1.23.2' pyarrow
pandas \
'plotly>=4.8' 'docutils<0.18.0' 'flake8==3.9.0' 'mypy==1.19.1'
'pytest==7.1.3' \
@@ -67,9 +67,9 @@ RUN python3.10 -m pip install 'sphinx==4.5.0' mkdocs
'pydata_sphinx_theme>=0.13'
'sphinxcontrib-htmlhelp==2.0.1' 'sphinxcontrib-qthelp==1.0.3' \
'sphinxcontrib-serializinghtml==1.1.5'
-# Set Python 3.10 as the default
-RUN ln -sf "$(which python3.10)" "/usr/local/bin/python" && \
- ln -sf "$(which python3.10)" "/usr/local/bin/python3"
+# Set Python 3.11 as the default
+RUN ln -sf "$(which python3.11)" "/usr/local/bin/python" && \
+ ln -sf "$(which python3.11)" "/usr/local/bin/python3"
# Create user for release manager
ARG UID
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
