potiuk opened a new issue, #43268: URL: https://github.com/apache/airflow/issues/43268
I've learned that as of artifact@v4 APIs for GitHub Actions, it should be possible to get rid of the GitHub Registry use as CI cache. In our CI/CD pipelines we are using "pull_request_target" workflow in order to be able to publish images to GitHub registry from Pull Requests coming from forks. This is done in order to speed up the workflows - our images take a lot of time to build and by reusing the cache, we are saving a lot of build time. However after following a discussion in Apache Software Foundation `#builds` channel https://the-asf.slack.com/archives/CF6PY5M4N/p1729595424091089 - with Pulsar and Netbeans CI/CD maintainers, it looks like we have a viable alternative - that will remove the need of using GH Registry for PRs - instead we could use artifacts. Our images are quite big (2GB) but Pulsar team successfully uses artifacts to store 1.7GB ones, so this should be quite possible for us to use the artifacts instead of registry. We dicussed also - with Pulsar and Netbeans CI maintainers about caching strategy we use and it seems entirely doable to switch to artifacts and get the same effect, without the need of having "pull_request_target" workflow. That would be a major win for security of our workflows to switch to it. This is especially atractive because the new artifact API v4 introduced in February 2024 is very fast, implements automated chunking of binaries uploaded/downloaded and parallell uploads/downloads - which according to Pulsar/Netbeans team had a huge impact on upload/download times for the images - see https://github.blog/news-insights/product-news/get-started-with-v4-of-github-actions-artifacts/ - there are a lot of details in the post, but the way uploading artifacts works with v4 is that you directly interact with Azure Blob Storage, the files are chunked etc. The examples they show in the blog post is speeding up the upload of 1GB file from 2 m to 3 s. Also v4 when `run-id` and other parameters have been added - making it suitable to implement similar caching solution we have, where artifacts from "canary" build could be used by all PRs as source of their image cache. Seems that artifacts have no individual size limits and they also do not have "total limits" with the ASF "Enterprise Account" level - there are things The way it **should** work: All workflows - PRs from apache and fork work in the same way: 1) Pull latest image (stored as artifact) from latest "main" successful build (we can use GH API to find the run-id to pull it from and the download-artifact@v4 action already supports pulling artifacts from other workflows and from other runs. 2) The PRs first unpack and push the "main" image to be present in local docker registry - no need to have GH Registry "PUSH" access. 3) The PRs then build the image locally not using `--cache-from` but to use such local registry cache. Achieving basically the same (or even bettter) caching speedups comparing to what we have now. 4) The successful "canary" build artifact could be uploaded at "finalize" time to become the new source of cache for all the PRs. 5) Breeze locak CI image could also be modified to pull latest image when rebuilding the image forcefully - otherwise rely on locally built images. 6) The `--image-tag` option of the local breeze (which pulls the image from a specific workflow run) could be easily adapted to be changed to `--pr PR#` - allowing to replicate a failure from specific PR locally without looking up the image. 7) The artifacts can have arbitrary set expiry time, so they could be available for a few days after the PR or "canary" build completed. The benefits: * we get rid of "Build image" "pull_request_target" workflow - that would be a huge security improvement * we get rid of the need of creating new images every time we create a new "release" branch. * we get rid of GitHub Registry * all the runs - commiter and non-commiter, from fork or apache would be identical, making it much easier to test changes in workflows * looking at the upload/download stats for artifacts, it is likely that switching to artifacts might get some speedup comparing to using the registry -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
