potiuk commented on code in PR #45266:
URL: https://github.com/apache/airflow/pull/45266#discussion_r1899175050
##########
.github/workflows/ci-image-build.yml:
##########
@@ -137,42 +120,32 @@ ${{ inputs.do-build == 'true' && inputs.image-tag || ''
}}"
- name: "Cleanup repo"
shell: bash
run: docker run -v "${GITHUB_WORKSPACE}:/workspace" -u 0:0 bash -c "rm
-rf /workspace/*"
- if: inputs.do-build == 'true'
- name: "Checkout target branch"
uses: actions/checkout@v4
with:
persist-credentials: false
- - name: "Checkout target commit"
- uses: ./.github/actions/checkout_target_commit
- if: inputs.do-build == 'true'
- with:
- target-commit-sha: ${{ inputs.target-commit-sha }}
- pull-request-target: ${{ inputs.pull-request-target }}
- is-committer-build: ${{ inputs.is-committer-build }}
- name: "Cleanup docker"
run: ./scripts/ci/cleanup_docker.sh
- if: inputs.do-build == 'true'
- name: "Install Breeze"
uses: ./.github/actions/breeze
- if: inputs.do-build == 'true'
- - name: "Regenerate dependencies in case they were modified manually so
that we can build an image"
+ - name: "Restore CI docker image ${{ inputs.platform }}:${{
env.PYTHON_MAJOR_MINOR_VERSION }}"
+ uses:
apache/infrastructure-actions/stash/restore@c94b890bbedc2fc61466d28e6bd9966bc6c6643c
+ with:
+ key: "ci-image-save-${{ inputs.platform }}-${{
env.PYTHON_MAJOR_MINOR_VERSION }}"
+ path: "/tmp/"
+ id: restore-ci-image
+ - name: "Load CI image ${{ inputs.platform }}:${{
env.PYTHON_MAJOR_MINOR_VERSION }}"
+ run: breeze ci-image load --platform ${{ inputs.platform }}
shell: bash
- run: |
- pip install rich>=12.4.4 pyyaml
- python scripts/ci/pre_commit/update_providers_dependencies.py
- if: inputs.do-build == 'true' && inputs.upgrade-to-newer-dependencies
!= 'false'
- - name: "Start ARM instance"
- run: ./scripts/ci/images/ci_start_arm_instance_and_connect_to_docker.sh
- if: inputs.do-build == 'true' && inputs.platform == 'linux/arm64'
- - name: Login to ghcr.io
- run: echo "${{ env.GITHUB_TOKEN }}" | docker login ghcr.io -u ${{
github.actor }} --password-stdin
- if: inputs.do-build == 'true'
+ if: steps.restore-ci-image.stash-hit == 'true'
- name: >
Build ${{ inputs.push-image == 'true' && ' & push ' || '' }}
- ${{ inputs.platform }}:${{ matrix.python-version }}:${{
inputs.image-tag }}
+ ${{ inputs.platform }}:${{ env.PYTHON_MAJOR_MINOR_VERSION }} image
run: >
- breeze ci-image build --builder airflow_cache --tag-as-latest
--image-tag "${{ inputs.image-tag }}"
- --python "${{ matrix.python-version }}" --platform "${{
inputs.platform }}"
+ breeze ci-image build
+ --docker-cache local
Review Comment:
We are pushing it - but it's not used in CI. In CI we will solely base
builds on the "artifact" uploads.
The cache is only pushed (only in `canary` builds) so that when you build
image locally in breeze in your dev environment, you still use remote registry
cache as default (this is default `--docker-cache` setting.. Note that
`--cache-from` (which is used in this case) does not really prevent the local
cache to be used. When `--cache-from` is used, also local cache is used
additionally.
The way how buildx work - it will checj that your local layer is not
invalidated and will use it if you build the image once - and remote cache is
not used. But if you rebase and - for example - you get an incoming dockerfile
change - the local layer will get invalidated, and buildx will check in the
remote cache - and it's likely it will find the new cache pushed there (if
canary build managed to push the cache) - this is also why we do "early" cache
push in canary builds - to push the cache quicker, even if there will be
`canary` failure - this is **just** a cache, so worst thing it will take a
little longer to rebuild it locally.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
