potiuk commented on PR #24858: URL: https://github.com/apache/airflow/pull/24858#issuecomment-1175955271
This is really an optimisation in case we have another ARM build already running (pushing cache). Context: Currently we have set of optimisation that only build ARM image when we suspect it might break (i.e when we have changes in dependencies - setup.py has been changed, any of the provider.yaml files changed). I think we should also add it when Dockerfile* change, but I will add it when I will change the Dockerfile next time (this is happening very rarely recently as it has gotten very stable). Since - at least for now - we are only building the ARM image to verify if it builds (we do not use it for anything else), we want to run it at most once per build. I realized however that there is a scenario where we run ARM build twice (and it is not needed). You can see it here: https://github.com/apache/airflow/runs/7205909316?check_suite_focus=true. This is direct push of `v2-3-test` branch: <img width="331" alt="Screenshot 2022-07-06 at 10 26 57" src="https://user-images.githubusercontent.com/595491/177505783-2f05da16-045e-4d7a-9f85-68515769d685.png";> (there was a failure of pushing cache to AMD which I have to look at but it's another issue) Both "Build CI ARM Images 3.7...." and "Push Image Cache (linux/arm64)" perform an ARM image build. It has slighlty different build args in order to make the cache reusable for regular builds. This "case" of build is not a regular "PR" run (`needs.build-info.outputs.merge-run == 'true'`). This is when we run the build AFTER we merged something to main or AFTER we pushed a cherry-picl to v2-3-test or AFTER we merged to 'v2-3-stable'. So this build does not "gate" anything. It's rather a regular "verification" build. What it really checks is "did dependencies change? shoudl we upgrade? Are we still passing all the tests and building image after they did?" - and it starts failing if for example someone releases a new dependency version that breaks our image build. it is the same "case" that is run after we merge to "main" build. This is "Push merge" as described here https://github.com/apache/airflow/blob/main/CI.rst#tests-workflow - this workflow description is not yet (I forgot it) updated to add the ARM image building but I will do it shortly and update the CI.rst (and move it to dev because it belongs there). To summarize the important parts of the "push merge" build. Such builds run full combination of all python/backend versions and: * attempts to "upgrade to newer dependencies" when building images - to see if there are some new dependencies * if everything works it generates and push updated constraints with those dependencies * using the new constraints it building cache in CI (so that subsequent "PR builds" do not have to update to the newer version of the dependencies). What it also did (until this PR) is that it also run extra "Build ARM image" for verification. But since in this build we also run "Push Image Cache (linux/arm64)", there is no need to run the ARM image build separately. If the cache will fail, we will know it - by that time it's already too late, because someone released a dependency that broke our build and the only way we can react is to see the "main" build failing or see any of the "PRs" that update their own dependencies failing and we have to fix it. So removing of the "ARM build" in this particular case (if needs.build-info.outputs.merge-run == 'true') does not change anything in PR builds. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
![https://user-images.githubusercontent.com/595491/177505783-2f05da16-045e-4d7a-9f85-68515769d685.png"](https://user-images.githubusercontent.com/595491/177505783-2f05da16-045e-4d7a-9f85-68515769d685.png"){kind=link}