potiuk commented on PR #23104:
URL: https://github.com/apache/airflow/pull/23104#issuecomment-1104266008
Apologies for such a HUGE PR, but I could not find a better way of replacing
old with new Breeze's build/pull/push image.
I could keep on replacing piece by piece but keeping the two in parallel in
CI was really burdensome. That's why I replaced all the related build/pull/push
tasks in one go. I am still testing it (I will do some tests with "main" push
to my own repo). But it should be generally working.
It removes another 1000 lines of bash code (YAY!) and paves the way to
remove nearly all of them in subsequent PRs.
Most of all It unifies the way how CI and local development works - rather
than having scripts on CI and local breeze commands, everythong is done via
python-based breeze commands (including running stuff in parallel on CI which
does not use Gnu Parallel but Python Multiprocessing)
It's way neater, nicere and much easier to reproduce locally. Example - most
complex "building PROD image" is this:
````yaml
- name: "Build & Push PROD image ${{ matrix.python-version }}:${{
env.IMAGE_TAG_FOR_THE_BUILD }}"
run: |
rm -f "./dist/"*.{whl,tar.gz}
rm -f "./docker-context-files/"*.{whl,tar.gz}
breeze prepare-provider-packages \
--package-list-file scripts/ci/installed_providers.txt" \
--package-format wheel
breeze prepare-airflow-package --package-format wheel
mv "./dist/"*.{whl,tar.gz} "./docker-context-files"
breeze build-prod-image --push-image
--install-from-docker-context-files
env:
PYTHON_MAJOR_MINOR_VERSION: ${{ matrix.python-version }}
UPGRADE_TO_NEWER_DEPENDENCIES: ${{
needs.build-info.outputs.upgradeToNewerDependencies }}
DOCKER_CACHE: ${{ needs.build-info.outputs.cacheDirective }}
IMAGE_TAG: ${{ env.IMAGE_TAG_FOR_THE_BUILD }}
````
Which you can very easily reproduce locally with the new breeze. It combines
(for clarity of the commands) both evn-vars and flags, but all the relevant env
vars used are now added directly in each step rather than in job or in
workflow, which makes it much more obvious what's going on.
@mik-laj - I also removed the "special" preparation of virtualenv for
docker tests - the docker tests now can be executed via `breeze` command and
they will automatically use breeze's virtualenv (which already containts
pytest, pytest-xdistts and requests):
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
