This is an automated email from the ASF dual-hosted git repository.
gaogaotiantian pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new 26a6794dd226 [SPARK-58832][INFRA] Cleanup the dependency list for
minimum docker image
26a6794dd226 is described below
commit 26a6794dd2264469d8f7af5d3a419f36c3810666
Author: Tian Gao <[email protected]>
AuthorDate: Mon Aug 24 20:27:14 2026 -0700
[SPARK-58832][INFRA] Cleanup the dependency list for minimum docker image
### What changes were proposed in this pull request?
Refer to `pyproject.toml` for CI dependency requirements instead of
hard-coding.
### Why are the changes needed?
To better manage the dependencies for CI.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
This is only used by scheduled CI so we need to wait for scheduled CI.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes #58060 from gaogaotiantian/dependency-minimum.
Authored-by: Tian Gao <[email protected]>
Signed-off-by: Tian Gao <[email protected]>
(cherry picked from commit c282a5b02ba10acf0ca5232e4970a63682bfc452)
Signed-off-by: Tian Gao <[email protected]>
---
.github/workflows/build_infra_images_cache.yml | 2 ++
dev/spark-test-image/python-minimum/Dockerfile | 7 ++++---
2 files changed, 6 insertions(+), 3 deletions(-)
diff --git a/.github/workflows/build_infra_images_cache.yml
b/.github/workflows/build_infra_images_cache.yml
index 003e0a9e5fdd..69af217f4534 100644
--- a/.github/workflows/build_infra_images_cache.yml
+++ b/.github/workflows/build_infra_images_cache.yml
@@ -121,6 +121,8 @@ jobs:
uses: docker/build-push-action@53b7df96c91f9c12dcc8a07bcb9ccacbed38856a
with:
context: ./dev/spark-test-image/python-minimum/
+ build-contexts: |
+ root=./
push: true
tags:
ghcr.io/apache/spark/apache-spark-github-action-image-pyspark-python-minimum-cache:${{
github.ref_name }}-static
cache-from:
type=registry,ref=ghcr.io/apache/spark/apache-spark-github-action-image-pyspark-python-minimum-cache:${{
github.ref_name }}
diff --git a/dev/spark-test-image/python-minimum/Dockerfile
b/dev/spark-test-image/python-minimum/Dockerfile
index 89da6f618124..292f4f328ced 100644
--- a/dev/spark-test-image/python-minimum/Dockerfile
+++ b/dev/spark-test-image/python-minimum/Dockerfile
@@ -64,8 +64,9 @@ ENV VIRTUAL_ENV=/opt/spark-venv
RUN python3.11 -m venv $VIRTUAL_ENV
ENV PATH="$VIRTUAL_ENV/bin:$PATH"
-ARG BASIC_PIP_PKGS="numpy==1.23.2 pyarrow==18.0.0 pandas==2.2.0 six==1.16.0
scipy scikit-learn coverage unittest-xml-reporting psutil"
-ARG CONNECT_PIP_PKGS="grpcio==1.76.0 grpcio-status==1.76.0
googleapis-common-protos==1.71.0 zstandard==0.25.0 graphviz==0.20
protobuf==6.33.5"
+COPY --from=root pyproject.toml ./pyproject.toml
-RUN python3.11 -m pip install --force $BASIC_PIP_PKGS $CONNECT_PIP_PKGS && \
+RUN python3.11 -m pip install -U pip
+
+RUN python3.11 -m pip install --group ci_classic_minimum --group
ci_connect_minimum && \
python3.11 -m pip cache purge
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
