github-actions[bot] opened a new pull request, #76183: URL: https://github.com/apache/airflow/pull/76183
* Restore the CI image from a snapshot of Docker's data directory Every job that prepares the CI image runs `docker image load` on the stash, which unpacks and checksums every layer of an 8 GB image again, about two minutes per job. Extracting a copy of the image store into a stopped daemon gives the same image in about half a minute on the same runners. The image stash stays as it is, for other consumers and as the fallback when a snapshot does not fit the daemon a job runs on. * Recover Docker safely when CI snapshot materialization fails Snapshot failures must not prevent the authoritative stash from loading, and an older branch snapshot must not substitute an image for the current checkout. * Scope CI snapshots to the current workflow run Raw daemon snapshots are job handoffs, so publishing them into branch caches unnecessarily exposes later runs to arbitrary checkout refs. * Publish authoritative caches before optional CI snapshot execution The snapshot script runs checkout code and prunes Docker state. Existing cache publications must finish before that optional execution. * Isolate CI snapshot creation from branch cache publishers The optional snapshot producer executes checkout code without publishing branch caches; its artifacts are available only to consumers in the same workflow run. * Bind optional snapshot jobs to the workflow commit Snapshot producers must not accept arbitrary checkout refs in privileged workflow contexts, and a partially failed Docker stop must still trigger restart cleanup. * Avoid reloading the CI image before snapshot creation Preparing the authoritative image in a second runner adds nearly three minutes before consumers can start. All branch publications must finish before the optional snapshot code prunes Docker state. * Execute the snapshot script from the immutable workflow commit The builder can check out a configurable ref before publishing caches. Snapshot execution must read the workflow commit itself rather than trust files left in that working tree. * Keep optional snapshots in an isolated read-only job Executing snapshot code inside the configurable-ref builder introduces a cache-poisoning finding. A separate job preserves the security boundary; its latency cost must remain explicit when measuring the runner-time benefit. * Preserve Docker state when snapshot inventory checks fail An unavailable daemon must not be mistaken for an empty image store before destructive snapshot materialization. * Retrigger CI after provider compatibility timeout * Create CI snapshots in the builder after publishing caches Reloading the freshly built image in a second job delays every consumer by nearly four minutes. Trusted workflow commands reuse the builder daemon without executing code from the configurable checkout. * Write CI snapshots off the Docker data disk The builder bind-mounts Docker from /mnt. Reading its image store and writing the compressed archive to that same disk competes for I/O; runner temp keeps the output on the root disk. * Cache CI dependencies separately from workspace source Ordinary source edits invalidate the dependency installation layer because it follows the full checkout copy. Reuse locked third-party dependencies while retaining the complete source installation and its resolution fallback. * Make dependency manifest extraction portable Use portable path-preserving shell commands for the BuildKit manifest stage and cover nested paths containing spaces.\n\nCo-authored-by: Copilot <[email protected]> * Run dependency manifest extraction with Bash Use the shell required by the manifest extraction stage so the BuildKit RUN command works with Docker's default shell.\n\nCo-authored-by: Copilot <[email protected]> * Harden dependency manifest extraction Restore the current UV and prek versions, propagate manifest discovery and copy failures, and cover failure injection in the extraction tests.\n\nCo-authored-by: Copilot <[email protected]> * Address CI snapshot review and separate dependency caching * Name the CI image snapshot artifact v1 There has never been an earlier version of this artifact on main, so the -v2 suffix left over from earlier branch revisions would puzzle the next reader. Snapshot artifacts are scoped to the workflow run that uploads them and kept for two days, so no consumer can see an artifact under the old name and no compatibility shim is needed. A test now checks that the upload and restore steps use the same artifact name, so the two cannot drift apart silently. * Keep CI image snapshots for one day A snapshot is only downloaded by consumer jobs of the run that uploaded it. Once it expires, a re-run restores the image stash instead, which keeps its two-day retention. One day halves the storage the snapshot adds on top of the stash. * Drop snapshot-ci-images assertion per review Co-authored-by: Yossi Eliaz <[email protected]> * Keep CI image snapshots faithful across reruns and check them before download - Archive and extract xattrs in every namespace: GNU tar restores only user.* by default, which dropped overlay2's trusted.overlay.opaque and file capabilities, so a restored image could differ from the built one. - Upload the snapshot with overwrite, plus a small metadata artifact that is uploaded even when the export fails; it reads "unavailable" until the archive is complete, so a rerun never restores an earlier attempt. - Check the metadata against the checkout and daemon before downloading the snapshot, and bound extraction with a timeout that falls back to the stash after cleaning partial state. - Load images for prepare_breeze_and_image and prepare_single_ci_image through one shared load_image action. Co-Authored-By: Claude Opus 5.5 <[email protected]> * Verify restored CI image snapshots and route them through breeze - Record the image ID and a hash of its merged file tree when creating the snapshot and compare both after restoring: a restore that lost an opaque directory marker keeps the image ID but changes the tree, so it now falls back to the stash instead of passing `bash -c true`. - Restore through `breeze ci-image load --snapshot-file`, keeping breeze's environment checks, image check, diagnostics and rebuild marker. - Bound the daemon stop and start as well as the extraction. - Download the snapshot metadata by pattern, which adds no error annotation when the builder made no snapshot. - Limit snapshots to GitHub-hosted runners, where self-hosted runners already restore the stash from their own storage. Co-Authored-By: Claude Opus 5.5 <[email protected]> * Check that an unsupported image store skips pruning and publishes only the unavailable marker Co-Authored-By: Claude Opus 5.5 <[email protected]> * Sort the snapshot file-tree listing outside the container Sorting the CI image's 265k-entry listing inside the container spilled to temporary files in /tmp while find was still walking it, so every run listed different files and every consumer rejected the snapshot. Co-Authored-By: Claude Opus 5.5 <[email protected]> * Create CI image snapshots through breeze ci-image save The builder now calls `breeze ci-image save --snapshot-file`, mirroring the `breeze ci-image load --snapshot-file` that consumers restore with, so both sides of the snapshot go through breeze and its environment checks. * Retire the snapshot metadata when its archive upload fails and fingerprint ownership and size The export writes metadata for a complete archive before the upload; if the upload then fails, that metadata would point consumers at an earlier attempt's archive. The tree fingerprint now also covers owner, group and size, and the artifact actions use the v7.0.2/v8.0.2 pins used elsewhere in the repository. Generated-by: Claude Opus 5 --------- (cherry picked from commit bf177e2948514f0807bf33c6462cde81ac43238e) Co-authored-by: Yossi Eliaz <[email protected]> Co-authored-by: Cursor Agent <[email protected]> Co-authored-by: Claude Opus 5.5 <[email protected]> Co-authored-by: Shahar Epstein <[email protected]> Co-authored-by: Jarek Potiuk <[email protected]> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
