This is an automated email from the ASF dual-hosted git repository.
shahar1 pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git
The following commit(s) were added to refs/heads/main by this push:
new 53e43f29130 Speed up CI image restores with zstd-compressed stashes
(#73997)
53e43f29130 is described below
commit 53e43f291303e43018ac51a0f21236514ae62485
Author: Yossi Eliaz <[email protected]>
AuthorDate: Thu Oct 1 19:31:58 2026 +0300
Speed up CI image restores with zstd-compressed stashes (#73997)
Every CI-image consumer downloads the stash as a zlib-6 zip and inflates
about 8 GB before docker image load can start, and the producer spends
135-200 s zipping it. Compressing the tar once with zstd and storing it
without zlib removes the inflate and moves about 12% fewer bytes, so
restore+load per consumer drops by about 50 s and the producer gets
faster too.
The file keeps its name and stash key because docker image load detects
zstd by content: consumers on any branch, including release branches
whose Breeze only accepts ci-image-save-v3-*.tar, load it unchanged.
---
.github/workflows/ci-image-build.yml | 8 ++++++++
1 file changed, 8 insertions(+)
diff --git a/.github/workflows/ci-image-build.yml
b/.github/workflows/ci-image-build.yml
index 7817fd211c3..1541337bc5c 100644
--- a/.github/workflows/ci-image-build.yml
+++ b/.github/workflows/ci-image-build.yml
@@ -321,12 +321,18 @@ jobs:
docker rmi "${CACHE_FROM_IMAGE}"
shell: bash
if: always() && env.CACHE_FROM_IMAGE != ''
+ # zstd stands in for the stash's zlib (compression-level 0 below), which
every consumer inflates.
+ # The .tar name, which every Breeze expects, stays: docker image load
detects zstd by content.
- name: "Export CI docker image ${{ env.PYTHON_MAJOR_MINOR_VERSION }}"
env:
PLATFORM: ${{ inputs.platform }}
COMMIT_FILE: "/mnt/ci-image-commit-${{
env.PYTHON_MAJOR_MINOR_VERSION }}.txt"
run: |
breeze ci-image save --platform "${PLATFORM}" --image-file-dir "/mnt"
+ for image_file in
/mnt/ci-image-save-*-"${PYTHON_MAJOR_MINOR_VERSION}".tar; do
+ zstd -6 -T0 --rm -o "${image_file}.zst" "${image_file}"
+ mv "${image_file}.zst" "${image_file}"
+ done
git rev-parse HEAD > "${COMMIT_FILE}"
shell: bash
if: inputs.upload-image-artifact == 'true' &&
steps.stashed-image.outputs.reusable != 'true'
@@ -336,6 +342,7 @@ jobs:
key: ci-image-save-v3-${{ inputs.platform }}-${{
env.PYTHON_MAJOR_MINOR_VERSION }}
path: "/mnt/ci-image-save-*-${{ env.PYTHON_MAJOR_MINOR_VERSION
}}.tar"
if-no-files-found: 'error'
+ compression-level: '0'
retention-days: '2'
if: inputs.upload-image-artifact == 'true' && inputs.image-stash-ref
== ''
# The per-branch stash above is written by every build on this branch,
so an image built
@@ -350,6 +357,7 @@ jobs:
-${{ inputs.image-stash-ref }}"
path: "/mnt/ci-image-save-*-${{ env.PYTHON_MAJOR_MINOR_VERSION
}}.tar"
if-no-files-found: 'error'
+ compression-level: '0'
retention-days: '6'
if: >
inputs.upload-image-artifact == 'true' && inputs.image-stash-ref !=
'' &&
