This is an automated email from the ASF dual-hosted git repository.
gaogaotiantian pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new 205dfccc702e [SPARK-58771][INFRA] Clean up the dependency list for
classic-only docker image
205dfccc702e is described below
commit 205dfccc702e282046dcb40ca041f7793efeebe1
Author: Tian Gao <[email protected]>
AuthorDate: Mon Aug 17 19:59:35 2026 -0700
[SPARK-58771][INFRA] Clean up the dependency list for classic-only docker
image
### What changes were proposed in this pull request?
* Use `pyproject.toml` as the dependency reference for classic-only
* Remove the unnecessary pip bootstrap for the docker image
### Why are the changes needed?
To clean up docker dependency list.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
This docker image is only used by scheduled CI so we need to wait for the
scheduled CI.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes #58006 from gaogaotiantian/docker-classic-only.
Authored-by: Tian Gao <[email protected]>
Signed-off-by: Tian Gao <[email protected]>
(cherry picked from commit ff26d01d4c0130b02e9a448ff167271820823f9c)
Signed-off-by: Tian Gao <[email protected]>
---
dev/spark-test-image/python-312-classic-only/Dockerfile | 11 +++++------
1 file changed, 5 insertions(+), 6 deletions(-)
diff --git a/dev/spark-test-image/python-312-classic-only/Dockerfile
b/dev/spark-test-image/python-312-classic-only/Dockerfile
index ceb4694b2dc9..7dfa024f4aab 100644
--- a/dev/spark-test-image/python-312-classic-only/Dockerfile
+++ b/dev/spark-test-image/python-312-classic-only/Dockerfile
@@ -59,11 +59,10 @@ ENV VIRTUAL_ENV=/opt/spark-venv
RUN python3.12 -m venv $VIRTUAL_ENV
ENV PATH="$VIRTUAL_ENV/bin:$PATH"
-ARG BASIC_PIP_PKGS="numpy pyarrow>=23.0.0 pandas==2.3.3 plotly<6.0.0
matplotlib openpyxl memory-profiler>=0.61.0 mlflow>=2.8.1 scipy
scikit-learn>=1.3.2 pystack>=1.6.0 psutil"
-ARG TEST_PIP_PKGS="coverage unittest-xml-reporting"
+COPY --from=root pyproject.toml ./pyproject.toml
-RUN curl -sS https://bootstrap.pypa.io/get-pip.py | python3.12
-RUN python3.12 -m pip install $BASIC_PIP_PKGS $TEST_PIP_PKGS && \
- python3.12 -m pip install torch torchvision --index-url
https://download.pytorch.org/whl/cpu && \
- python3.12 -m pip install deepspeed torcheval && \
+RUN python3.12 -m pip install -U pip
+
+RUN python3.12 -m pip install --group ml_torch --index-url
https://download.pytorch.org/whl/cpu && \
+ python3.12 -m pip install --group ci_classic_standard && \
python3.12 -m pip cache purge
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
