This is an automated email from the ASF dual-hosted git repository.
zhengruifeng pushed a commit to branch branch-4.2
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.2 by this push:
new c5259ac424c0 [SPARK-57330][INFRA] Switch shared CI compile artifacts
to zstd compression
c5259ac424c0 is described below
commit c5259ac424c0c4cd9af3cc18deeac2b60dbbb070
Author: Ruifeng Zheng <[email protected]>
AuthorDate: Tue Jun 9 09:40:15 2026 +0800
[SPARK-57330][INFRA] Switch shared CI compile artifacts to zstd compression
### What changes were proposed in this pull request?
Compress the shared compile artifacts that CI passes between jobs with
`zstd` instead of `gzip`, across the three reusable workflows that
produce/consume them:
- `build_and_test.yml` - `compile-artifact` (produced by `precompile`,
consumed by the `build`, `pyspark`, `sparkr`, `tpcds-1g`,
`docker-integration-tests` and `k8s-integration-tests` jobs)
- `python_hosted_runner_test.yml` - `compile-artifact` (macOS / ARM python
matrix)
- `maven_test.yml` - `compile-target` and `compile-m2-spark`
Each producer pipes `tar` through the `zstd` binary, and each consumer
decompresses with `zstd -dc`:
```bash
# create
... | tar --null -cf - -T - | zstd -c -T0 > compile-artifact.tar.zst
# extract
zstd -dc compile-artifact.tar.zst | tar -xf -
```
`zstd` is driven through the standalone binary rather than `tar --zstd` on
purpose: GitHub's macOS runners ship bsdtar, whose `--zstd` hangs, so letting
`tar` do plain (un)archiving and piping through the `zstd` binary keeps one
portable idiom across Linux hosts, the container images, `ubuntu-24.04-arm` and
macOS.
### Why are the changes needed?
`zstd` compresses and (especially) decompresses much faster than `gzip` at
a comparable or better ratio, and `-T0` parallelizes compression across all
cores. These artifacts are produced once and downloaded by up to 8 downstream
jobs per run, so faster (de)compression shortens the critical path of every
matrix entry. The `zstd` binary is already present on all relevant runners and
container images (the latter since SPARK-57278), so no new dependency is
introduced.
### Does this PR introduce _any_ user-facing change?
No. CI-only.
### How was this patch tested?
- Local validation: all three workflows parse as YAML, and no `*.tar.gz`
reference to these artifacts remains.
- The artifacts are ephemeral per run (`retention-days: 1`, names keyed by
`run_id`), so producer and consumer always run the same code; there is no
cross-version compatibility concern.
- The `build_and_test.yml` path is exercised by this PR's CI. The
`maven_test.yml` and `python_hosted_runner_test.yml` paths run via their
scheduled / `workflow_dispatch` callers and can be validated by dispatching
those workflows on the fork.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 4.8
Closes #56369 from zhengruifeng/shared-artifact-zstd-dev6.
Authored-by: Ruifeng Zheng <[email protected]>
Signed-off-by: Ruifeng Zheng <[email protected]>
(cherry picked from commit ac0c117cf280e1a44e8fdc0c00a91740bb9a6abf)
Signed-off-by: Ruifeng Zheng <[email protected]>
---
.github/workflows/build_and_test.yml | 30 ++++++++++++-------------
.github/workflows/maven_test.yml | 20 ++++++++---------
.github/workflows/python_hosted_runner_test.yml | 10 ++++-----
3 files changed, 30 insertions(+), 30 deletions(-)
diff --git a/.github/workflows/build_and_test.yml
b/.github/workflows/build_and_test.yml
index 89c98792b0fa..483f54e17754 100644
--- a/.github/workflows/build_and_test.yml
+++ b/.github/workflows/build_and_test.yml
@@ -409,8 +409,8 @@ jobs:
if: steps.download-precompiled.outcome == 'success'
continue-on-error: true
run: |
- tar -xzf compile-artifact.tar.gz
- rm compile-artifact.tar.gz
+ zstd -dc compile-artifact.tar.zst | tar -xf -
+ rm compile-artifact.tar.zst
# Run the tests.
- name: Run tests
env: ${{ fromJSON(inputs.envs) }}
@@ -620,13 +620,13 @@ jobs:
- name: Package compile output
run: |
find . -type d -name target -not -path './build/*' -not -path
'./.git/*' -print0 \
- | tar --null -czf compile-artifact.tar.gz -T -
- ls -lh compile-artifact.tar.gz
+ | tar --null -cf - -T - | zstd -c -T0 > compile-artifact.tar.zst
+ ls -lh compile-artifact.tar.zst
- name: Upload compile artifact
uses: actions/upload-artifact@v6
with:
name: spark-compile-${{ inputs.branch }}-${{ github.run_id }}
- path: compile-artifact.tar.gz
+ path: compile-artifact.tar.zst
retention-days: 1
if-no-files-found: error
@@ -764,8 +764,8 @@ jobs:
if: steps.download-precompiled.outcome == 'success'
continue-on-error: true
run: |
- tar -xzf compile-artifact.tar.gz
- rm compile-artifact.tar.gz
+ zstd -dc compile-artifact.tar.zst | tar -xf -
+ rm compile-artifact.tar.zst
# Run the tests.
- name: Run tests
env: ${{ fromJSON(inputs.envs) }}
@@ -901,8 +901,8 @@ jobs:
if: steps.download-precompiled.outcome == 'success'
continue-on-error: true
run: |
- tar -xzf compile-artifact.tar.gz
- rm compile-artifact.tar.gz
+ zstd -dc compile-artifact.tar.zst | tar -xf -
+ rm compile-artifact.tar.zst
- name: Run tests
env: ${{ fromJSON(inputs.envs) }}
run: |
@@ -1410,8 +1410,8 @@ jobs:
if: steps.download-precompiled.outcome == 'success'
continue-on-error: true
run: |
- tar -xzf compile-artifact.tar.gz
- rm compile-artifact.tar.gz
+ zstd -dc compile-artifact.tar.zst | tar -xf -
+ rm compile-artifact.tar.zst
- name: Cache TPC-DS generated data
id: cache-tpcds-sf-1
uses: actions/cache@v5
@@ -1542,8 +1542,8 @@ jobs:
if: steps.download-precompiled.outcome == 'success'
continue-on-error: true
run: |
- tar -xzf compile-artifact.tar.gz
- rm compile-artifact.tar.gz
+ zstd -dc compile-artifact.tar.zst | tar -xf -
+ rm compile-artifact.tar.zst
- name: Run tests
env: ${{ fromJSON(inputs.envs) }}
run: |
@@ -1633,8 +1633,8 @@ jobs:
if: steps.download-precompiled.outcome == 'success'
continue-on-error: true
run: |
- tar -xzf compile-artifact.tar.gz
- rm compile-artifact.tar.gz
+ zstd -dc compile-artifact.tar.zst | tar -xf -
+ rm compile-artifact.tar.zst
- name: Install R
run: |
sudo apt update
diff --git a/.github/workflows/maven_test.yml b/.github/workflows/maven_test.yml
index 49ab53324871..bae85c699b9f 100644
--- a/.github/workflows/maven_test.yml
+++ b/.github/workflows/maven_test.yml
@@ -122,18 +122,18 @@ jobs:
# and the connect entry rebuilds it via `mvn install -pl assembly`.
find . \( -path './build' -o -path './.git' -o -path './assembly' \)
-prune \
-o -type d -name target -print0 \
- | tar --null -czf compile-target.tar.gz -T -
+ | tar --null -cf - -T - | zstd -c -T0 > compile-target.tar.zst
if [ -d "$HOME/.m2/repository/org/apache/spark" ]; then
- tar -C "$HOME/.m2/repository/org/apache" -czf
compile-m2-spark.tar.gz spark
+ tar -C "$HOME/.m2/repository/org/apache" -cf - spark | zstd -c -T0
> compile-m2-spark.tar.zst
fi
- ls -lh compile-target.tar.gz compile-m2-spark.tar.gz
+ ls -lh compile-target.tar.zst compile-m2-spark.tar.zst
- name: Upload compile artifact
uses: actions/upload-artifact@v6
with:
name: spark-maven-compile-${{ inputs.branch }}-java${{ inputs.java
}}-${{ github.run_id }}
path: |
- compile-target.tar.gz
- compile-m2-spark.tar.gz
+ compile-target.tar.zst
+ compile-m2-spark.tar.zst
retention-days: 1
if-no-files-found: error
@@ -283,12 +283,12 @@ jobs:
if: steps.download-precompiled.outcome == 'success'
continue-on-error: true
run: |
- tar -xzf compile-target.tar.gz
- rm compile-target.tar.gz
- if [ -f compile-m2-spark.tar.gz ]; then
+ zstd -dc compile-target.tar.zst | tar -xf -
+ rm compile-target.tar.zst
+ if [ -f compile-m2-spark.tar.zst ]; then
mkdir -p "$HOME/.m2/repository/org/apache"
- tar -C "$HOME/.m2/repository/org/apache" -xzf
compile-m2-spark.tar.gz
- rm compile-m2-spark.tar.gz
+ zstd -dc compile-m2-spark.tar.zst | tar -C
"$HOME/.m2/repository/org/apache" -xf -
+ rm compile-m2-spark.tar.zst
fi
# Run the tests using script command.
# BSD's script command doesn't support -c option, and the usage is
different from Linux's one.
diff --git a/.github/workflows/python_hosted_runner_test.yml
b/.github/workflows/python_hosted_runner_test.yml
index 1427e6011f58..1ba0c5529d5a 100644
--- a/.github/workflows/python_hosted_runner_test.yml
+++ b/.github/workflows/python_hosted_runner_test.yml
@@ -119,13 +119,13 @@ jobs:
- name: Package compile output
run: |
find . -type d -name target -not -path './build/*' -not -path
'./.git/*' -print0 \
- | tar --null -czf compile-artifact.tar.gz -T -
- ls -lh compile-artifact.tar.gz
+ | tar --null -cf - -T - | zstd -c -T0 > compile-artifact.tar.zst
+ ls -lh compile-artifact.tar.zst
- name: Upload compile artifact
uses: actions/upload-artifact@v6
with:
name: spark-compile-${{ inputs.os }}-${{ inputs.branch }}-${{
github.run_id }}
- path: compile-artifact.tar.gz
+ path: compile-artifact.tar.zst
retention-days: 1
if-no-files-found: error
@@ -246,8 +246,8 @@ jobs:
if: steps.download-precompiled.outcome == 'success'
continue-on-error: true
run: |
- tar -xzf compile-artifact.tar.gz
- rm compile-artifact.tar.gz
+ zstd -dc compile-artifact.tar.zst | tar -xf -
+ rm compile-artifact.tar.zst
# Run the tests.
- name: Run tests
env: ${{ fromJSON(inputs.envs) }}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
