potiuk commented on code in PR #45266:
URL: https://github.com/apache/airflow/pull/45266#discussion_r1899179239
##########
.github/workflows/prod-image-build.yml:
##########
@@ -231,57 +212,45 @@ ${{ inputs.do-build == 'true' && inputs.image-tag || ''
}}"
- name: "Cleanup repo"
shell: bash
run: docker run -v "${GITHUB_WORKSPACE}:/workspace" -u 0:0 bash -c "rm
-rf /workspace/*"
- if: inputs.do-build == 'true'
- name: "Checkout target branch"
uses: actions/checkout@v4
with:
persist-credentials: false
- - name: "Checkout target commit"
- uses: ./.github/actions/checkout_target_commit
- with:
- target-commit-sha: ${{ inputs.target-commit-sha }}
- pull-request-target: ${{ inputs.pull-request-target }}
- is-committer-build: ${{ inputs.is-committer-build }}
- if: inputs.do-build == 'true'
- name: "Cleanup docker"
run: ./scripts/ci/cleanup_docker.sh
- if: inputs.do-build == 'true'
- name: "Install Breeze"
uses: ./.github/actions/breeze
- if: inputs.do-build == 'true'
- - name: "Regenerate dependencies in case they was modified manually so
that we can build an image"
- shell: bash
- run: |
- pip install rich>=12.4.4 pyyaml
- python scripts/ci/pre_commit/update_providers_dependencies.py
- if: inputs.do-build == 'true' && inputs.upgrade-to-newer-dependencies
!= 'false'
- name: "Cleanup dist and context file"
shell: bash
run: rm -fv ./dist/* ./docker-context-files/*
- if: inputs.do-build == 'true'
+ - name: "Restore PROD docker image ${{ inputs.platform }}:${{
env.PYTHON_MAJOR_MINOR_VERSION }}"
+ uses:
apache/infrastructure-actions/stash/restore@c94b890bbedc2fc61466d28e6bd9966bc6c6643c
+ with:
+ key: "prod-image-save-${{ inputs.platform }}-${{
env.PYTHON_MAJOR_MINOR_VERSION }}"
+ path: "/tmp/"
+ id: restore-prod-image
+ - name: "Load PROD image ${{ inputs.platform }}:${{
env.PYTHON_MAJOR_MINOR_VERSION }}"
+ run: breeze prod-image load --platform ${{ inputs.platform }}
+ shell: bash
+ if: steps.restore-prod-image.stash-hit == 'true'
- name: "Download packages prepared as artifacts"
uses: actions/download-artifact@v4
with:
name: prod-packages
path: ./docker-context-files
- if: inputs.do-build == 'true'
- name: "Download constraints"
uses: actions/download-artifact@v4
with:
name: constraints
path: ./docker-context-files
- if: inputs.do-build == 'true'
- - name: Login to ghcr.io
- shell: bash
- run: echo "${{ env.GITHUB_TOKEN }}" | docker login ghcr.io -u ${{
github.actor }} --password-stdin
- if: inputs.do-build == 'true'
- - name: "Build PROD images w/ source providers ${{ matrix.python-version
}}:${{ inputs.image-tag }}"
+ - name: "Build PROD images w/ source providers ${{
env.PYTHON_MAJOR_MINOR_VERSION }}"
shell: bash
run: >
- breeze prod-image build --tag-as-latest --image-tag "${{
inputs.image-tag }}"
+ breeze prod-image build
+ --docker-cache local
--commit-sha "${{ github.sha }}"
--install-packages-from-context --airflow-constraints-mode
constraints-source-providers
- --use-constraints-for-context-packages --python "${{
matrix.python-version }}"
+ --use-constraints-for-context-packages
Review Comment:
Well. I think it's not that bad. Usually you locally already have **some**
python version set (or default) - we are already passing quite a number of
default values in the same way (and usually this should not be that difficult
to figure out you are trying different Python version)
Those are variables for example:
```
BACKEND: sqlite
PYTHON_MAJOR_MINOR_VERSION: ${{ matrix.python-version }}
DEFAULT_BRANCH: ${{ inputs.branch }}
DEFAULT_CONSTRAINTS_BRANCH: ${{ inputs.constraints-branch }}
VERSION_SUFFIX_FOR_PYPI: "dev0"
GITHUB_REPOSITORY: ${{ github.repository }}
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
GITHUB_USERNAME: ${{ github.actor }}
USE_UV: ${{ inputs.use-uv }}
VERBOSE: "true"
```
It would be rather tiresome to copy&paste all the flags to all the commands
that need them - this is why we have the env variables at the "job" level in
the first place.
Also ... This is already actually handled well. If you want to reproduce
what CI does 1-1 - you can already do it very, very easily. Just unfold the
breeze command run and you will see the env vars used:
<img width="700" alt="Screenshot 2024-12-29 at 19 56 57"
src="https://github.com/user-attachments/assets/ce38548f-8256-4974-bf45-5eeb3f861366";
/>
We have much, much, much less copy&paste in yaml files because of that ...
and I would hate having to trace it back and add those variables everywhere -
especially that not all breeze commands use those flags, so setting up the env
variable for "job" is equivalent to tell "set that flag for all breeze commands
that have the flag".
Yes. It's not perfect. But maybe we can do something else - every time when
breeze command is run in CI, we can print (in blue) the command to copy&paste
to reproduce it. We already do it for all the `docker-compose` commands that
are run "internally" in breeze. and we could do similar thing for breeze
commands (the docker compose commands are way longer as they have many more
variables to pass, but what they produce is a copy*pasteable command to run:
```bash
AIRFLOW_CI_IMAGE="ghcr.io/apache/airflow/main/ci/python3.9" \
AIRFLOW_CONSTRAINTS_LOCATION="" \
AIRFLOW_CONSTRAINTS_MODE="constraints-source-providers" \
AIRFLOW_CONSTRAINTS_REFERENCE="constraints-main" \
AIRFLOW_ENV="development" \
AIRFLOW_EXTRAS="" \
AIRFLOW_IMAGE_KUBERNETES="ghcr.io/apache/airflow/main/kubernetes/python3.9" \
AIRFLOW_SKIP_CONSTRAINTS="false" \
AIRFLOW_VERSION="3.0.0.dev0" \
AIRFLOW__CELERY__BROKER_URL="redis://redis:6379/0" \
AIRFLOW__CORE__EXECUTOR="LocalExecutor" \
ANSWER="" \
BASE_BRANCH="main" \
BREEZE="true" \
BREEZE_INIT_COMMAND="" \
CELERY_BROKER_URLS_MAP="{'rabbitmq': '***rabbitmq:5672', 'redis':
'redis://redis:6379/0'}" \
CELERY_FLOWER="false" \
CHICKEN_EGG_PROVIDERS="" \
CI_BUILD_ID="0" \
CI_EVENT_TYPE="pull_request" \
CI_JOB_ID="0" \
CI_TARGET_BRANCH="main" \
CI_TARGET_REPO="apache/airflow" \
CLEAN_AIRFLOW_INSTALLATION="false" \
COLLECT_ONLY="false" \
COMMIT_SHA="fcc8cc453a4ca6c1f3e25c358ad7ebcb4434452f" \
COMPOSE_FILE="/home/runner/work/airflow/airflow/scripts/ci/docker-compose/base.yml:/home/runner/work/airflow/airflow/scripts/ci/docker-compose/docker-socket.yml:/home/runner/work/airflow/airflow/scripts/ci/docker-compose/backend-mysql.yml:/home/runner/work/airflow/airflow/scripts/ci/docker-compose/files.yml"
\
DEFAULT_BRANCH="main" \
DEFAULT_CONSTRAINTS_BRANCH="constraints-main" \
DEV_MODE="false" \
DOCKER_IS_ROOTLESS="false" \
DRILL_HOST_PORT="28047" \
FASTAPI_API_HOST_PORT="29091" \
FLOWER_HOST_PORT="25555" \
HOST_GROUP_ID="128" \
HOST_OS="linux" \
HOST_USER_ID="128" \
INIT_SCRIPT_FILE="init.sh" \
INSTALL_AIRFLOW_PYTHON_CLIENT="false" \
INSTALL_AIRFLOW_VERSION="" \
INSTALL_AIRFLOW_WITH_CONSTRAINTS="false" \
ISSUE_ID="" \
LOAD_DEFAULT_CONNECTIONS="false" \
LOAD_EXAMPLES="false" \
MSSQL_HOST_PORT="21[433](https://github.com/apache/airflow/actions/runs/12537145069/job/34961056741?pr=45261#step:6:447)"
\
MYSQL_HOST_PORT="23306" \
MYSQL_VERSION="8.0" \
NUM_RUNS="" \
ONLY_MIN_VERSION_UPDATE="false" \
PACKAGE_FORMAT="wheel" \
POSTGRES_HOST_PORT="25433" \
POSTGRES_VERSION="13" \
PROVIDERS_CONSTRAINTS_MODE="constraints-source-providers" \
PROVIDERS_CONSTRAINTS_REFERENCE="" \
PROVIDERS_SKIP_CONSTRAINTS="false" \
PYTHONDONTWRITEBYTECODE="true" \
QUIET="false" \
REDIS_HOST_PORT="26379" \
REGENERATE_MISSING_DOCS="false" \
REMOVE_ARM_PACKAGES="false" \
RUN_TESTS="true" \
SKIP_ENVIRONMENT_INITIALIZATION="false" \
SKIP_SSH_SETUP="false" \
SQLALCHEMY_WARN_20="true" \
SQLITE_URL="sqlite:////root/airflow/sqlite/airflow.db" \
SSH_PORT="12322" \
STANDALONE_DAG_PROCESSOR="false" \
START_AIRFLOW="false" \
START_WEBSERVER_WITH_EXAMPLES="false" \
SUSPENDED_PROVIDERS_FOLDERS="" \
SYSTEM_TESTS_ENV_ID="" \
TEST_GROUP="providers" \
TEST_TYPE="All" \
USE_AIRFLOW_VERSION="" \
USE_PACKAGES_FROM_DIST="false" \
USE_UV="false" \
USE_XDIST="false" \
VERBOSE_COMMANDS="false" \
VERSION_SUFFIX_FOR_PYPI="" \
WEBSERVER_HOST_PORT="28080" \
_AIRFLOW_RUN_DB_TESTS_ONLY="true" \
_AIRFLOW_SKIP_DB_TESTS="false" \
docker compose pull
```
We could do something similar.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
