This is an automated email from the ASF dual-hosted git repository.
gaogaotiantian pushed a commit to branch branch-4.1
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.1 by this push:
new 90d6f3ba0a4d [SPARK-57708][INFRA] Install zstd in CI container images
used by the branch-4.1 scheduler
90d6f3ba0a4d is described below
commit 90d6f3ba0a4d8df56e8c512b29c1053ba7c746f9
Author: Tian Gao <[email protected]>
AuthorDate: Wed Jul 1 15:25:33 2026 -0700
[SPARK-57708][INFRA] Install zstd in CI container images used by the
branch-4.1 scheduler
### What changes were proposed in this pull request?
Install `zstd` in the CI container image Dockerfiles that are actually
built by the `branch-4.1` scheduler (`branch41_scheduler.yml`):
`dev/infra/Dockerfile` and the `docs`, `lint`, `sparkr`, `python-311`,
`python-314`, and `pypy-310` images under `dev/spark-test-image/`.
### Why are the changes needed?
On `branch-4.1`, `build_and_test.yml` extracts the precompiled artifact
with `zstd -dc compile-artifact.tar.zst | tar -xf -`, but the container images
do not have `zstd` installed, so the step fails with `zstd: not found`. (The
same gap also defeats `actions/cache`, whose cache version embeds the
compression method; container images without `zstd` fall back to `gzip` and
never restore caches saved by host jobs.)
This is the `branch-4.1`-scoped counterpart of SPARK-57278, which fixed the
same problem on `master`. Only the images that the `branch-4.1` scheduler
builds are modified here; the other Dockerfiles under `dev/spark-test-image/`
are not used by any `branch-4.1` workflow.
### Does this PR introduce _any_ user-facing change?
No. CI-only.
### How was this patch tested?
CI on `branch-4.1`. The `Extract precompiled artifact` step now finds
`zstd` in the rebuilt images, and `actions/cache` restores the Coursier cache
in the container jobs.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (claude-opus-4-8)
Closes #56911 from gaogaotiantian/SPARK-57708-zstd-branch-4.1.
Authored-by: Tian Gao <[email protected]>
Signed-off-by: Tian Gao <[email protected]>
---
dev/infra/Dockerfile | 1 +
dev/spark-test-image/docs/Dockerfile | 1 +
dev/spark-test-image/lint/Dockerfile | 1 +
dev/spark-test-image/pypy-310/Dockerfile | 1 +
dev/spark-test-image/python-311/Dockerfile | 3 ++-
dev/spark-test-image/python-314/Dockerfile | 3 ++-
dev/spark-test-image/sparkr/Dockerfile | 1 +
7 files changed, 9 insertions(+), 2 deletions(-)
diff --git a/dev/infra/Dockerfile b/dev/infra/Dockerfile
index 444f0522ba5e..e4ca1a8e12cc 100644
--- a/dev/infra/Dockerfile
+++ b/dev/infra/Dockerfile
@@ -66,6 +66,7 @@ RUN apt-get update && apt-get install -y \
software-properties-common \
wget \
zlib1g-dev \
+ zstd \
&& rm -rf /var/lib/apt/lists/*
diff --git a/dev/spark-test-image/docs/Dockerfile
b/dev/spark-test-image/docs/Dockerfile
index 43c08a22b935..8d7bb69aa241 100644
--- a/dev/spark-test-image/docs/Dockerfile
+++ b/dev/spark-test-image/docs/Dockerfile
@@ -66,6 +66,7 @@ RUN apt-get update && apt-get install -y \
software-properties-common \
wget \
zlib1g-dev \
+ zstd \
&& rm -rf /var/lib/apt/lists/*
diff --git a/dev/spark-test-image/lint/Dockerfile
b/dev/spark-test-image/lint/Dockerfile
index 61165e320d08..b47a1de1d3cf 100644
--- a/dev/spark-test-image/lint/Dockerfile
+++ b/dev/spark-test-image/lint/Dockerfile
@@ -59,6 +59,7 @@ RUN apt-get update && apt-get install -y \
software-properties-common \
wget \
zlib1g-dev \
+ zstd \
&& rm -rf /var/lib/apt/lists/*
RUN Rscript -e "install.packages(c('remotes', 'knitr', 'markdown',
'rmarkdown', 'testthat'), repos='https://cloud.r-project.org/')" \
diff --git a/dev/spark-test-image/pypy-310/Dockerfile
b/dev/spark-test-image/pypy-310/Dockerfile
index c8672fc0ec06..460355cacc88 100644
--- a/dev/spark-test-image/pypy-310/Dockerfile
+++ b/dev/spark-test-image/pypy-310/Dockerfile
@@ -58,6 +58,7 @@ RUN apt-get update && apt-get install -y \
software-properties-common \
wget \
zlib1g-dev \
+ zstd \
&& apt-get autoremove --purge -y \
&& apt-get clean \
&& rm -rf /var/lib/apt/lists/*
diff --git a/dev/spark-test-image/python-311/Dockerfile
b/dev/spark-test-image/python-311/Dockerfile
index f8a9df5842ce..bc6244c2a324 100644
--- a/dev/spark-test-image/python-311/Dockerfile
+++ b/dev/spark-test-image/python-311/Dockerfile
@@ -57,7 +57,8 @@ RUN apt-get update && apt-get install -y \
tzdata \
software-properties-common \
wget \
- zlib1g-dev
+ zlib1g-dev \
+ zstd
# Install Python 3.11
RUN add-apt-repository ppa:deadsnakes/ppa
diff --git a/dev/spark-test-image/python-314/Dockerfile
b/dev/spark-test-image/python-314/Dockerfile
index f3da21e005b3..ec51f1a29563 100644
--- a/dev/spark-test-image/python-314/Dockerfile
+++ b/dev/spark-test-image/python-314/Dockerfile
@@ -57,7 +57,8 @@ RUN apt-get update && apt-get install -y \
tzdata \
software-properties-common \
wget \
- zlib1g-dev
+ zlib1g-dev \
+ zstd
# Install Python 3.14
RUN add-apt-repository ppa:deadsnakes/ppa
diff --git a/dev/spark-test-image/sparkr/Dockerfile
b/dev/spark-test-image/sparkr/Dockerfile
index 9b2d702fc2d6..26554d693a9c 100644
--- a/dev/spark-test-image/sparkr/Dockerfile
+++ b/dev/spark-test-image/sparkr/Dockerfile
@@ -60,6 +60,7 @@ RUN apt-get update && apt-get install -y \
software-properties-common \
wget \
zlib1g-dev \
+ zstd \
&& rm -rf /var/lib/apt/lists/*
RUN echo 'deb https://cloud.r-project.org/bin/linux/ubuntu jammy-cran40/' >>
/etc/apt/sources.list
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
