potiuk opened a new pull request, #45266:
URL: https://github.com/apache/airflow/pull/45266
For a long time we had used a sophisticated mechanism to speed up our CI
jobs by building the images in "pull_request_target" workflow and pushing them
to GitHub registry. That however had several drawbacks:
* CI image was complex when it comes to layer setup (we had to pre- cache
installed dependencies by installing them from branch tip
* The pull_request_target is a very dangerous workflow, we had a number of
security problems with it (and it's difficult to debug)
* Caching of `pip` and `uv` was not used because it increased size of the
image significantly
This PR significantly improves the caching mechanisms for the images
building of several advacements that were not possible before:
* The upload-artifacts@v4 action and improved stash action developed by
@assignUser and published in "apache/infrastructure-actions" allows us to store
all images (8GB per run) in artifacts rather than in registry - so we can do
the image build once and share it with all the jobs.
* The uv speed is "enough" to allow occasional installation of Airlfow
locally. This allows to utilize cache-mount and locally build uv cache, rather
than rely on "remote" cache when we are building local images for breeze. The
first time you build local breeze image it will take 2-5 more minutes
(depending on your network speed, but because we can utilise cache mounts,
every subsequent build should be very fast - even if all dependencies change.
Using uv also allows to "always" reinstall airflow when you build the image
even if single source file changed, because with cache it takes sub-seconds to
reinstall airflow and all dependencies.
* the cache mounts are not included in the image size, and since we can
export and import images in CI in artifacts and we do not need to rebuild them,
the images shared as compressed artifacts are relatively small (2GB) - cache of
`uv` is around 4GB on top of that so sharing image built in the "build image"
job with other jobs in the same workflow is fast.
* we are still using registry cache for the "non-python" parts of the image
- both CI and breeze image build speed benefit from using the image cache for
system dependencies, database clients etc.
Fixes: #42999
Fixes: #43268
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
In case of an existing issue, reference it using one of the following:
closes: #ISSUE
related: #ISSUE
How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->
<!-- Please keep an empty line above the dashes. -->
---
**^ Add meaningful description above**
Read the **[Pull Request
Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)**
for more information.
In case of fundamental code changes, an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals))
is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party
License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in a
newsfragment file, named `{pr_number}.significant.rst` or
`{issue_number}.significant.rst`, in
[newsfragments](https://github.com/apache/airflow/tree/main/newsfragments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
