sdaberdaku opened a new pull request, #77123:
URL: https://github.com/apache/airflow/pull/77123
Worker sets route Celery queues to workers with different resources, but the
`worker` container of every set is rendered from the release-wide
`images.airflow`. Queues often need different task dependencies too (CUDA,
Spark, a heavy client library the rest of the deployment should not carry),
which today means one image carrying all of them, or patching the rendered
Deployments after Helm. We run one Kustomize patch per worker set that swaps
the `worker` container image, and every new set needs another one.
This adds `workers.celery.image` (`repository`, `tag`, `digest`,
`pullPolicy`), which worker sets inherit and override like any other
`workers.celery` value. It follows the per-component `image` design agreed in
the review of #68784 (an optional `image` section per Airflow component that
falls back to `images.airflow`), which also asked why workers were left out,
and the "Reshuffle image-related parameters into the components they belong to"
item of #64037.
Behaviour:
- All fields are unset by default, so existing releases render exactly as
before.
- Only the `worker` container uses it. The init containers
(`wait-for-airflow-migrations`, kerberos) and the log groomer sidecar keep
their usual images: they run chart-owned commands that need nothing beyond
Airflow itself.
- Unset fields fall back to `images.airflow` and then to
`defaultAirflowRepository` / `defaultAirflowTag`, except the digest: neither
`images.airflow.digest` nor `defaultAirflowDigest` is used once `repository`,
`tag` or `digest` is set here. A global digest pins the base build: combined
with another repository the image fails to pull, and combined with another tag
the digest wins and the worker silently runs the base image. If the `tag` and
`digest` fields are consolidated into one, as floated in #68784, plain
field-by-field fallback becomes safe.
- `pullPolicy` is per image because the right policy depends on the image: a
large image is best pulled only when missing, while one rebuilt under the same
tag needs `Always`.
- A set's `image` replaces `workers.celery.image` as a whole instead of
being deep-merged, for the same reason as the digest: a set's `repository`
combined with the parent's `digest` points at an image that does not exist.
`image: {}` returns a set to `images.airflow`.
- A set's `image` is validated against the same schema as
`workers.celery.image`. Worker set items are otherwise free-form, and here an
unquoted numeric tag (`tag: 3.2`) or a misspelt key would render a plausible
but wrong image instead of failing.
Compared to #63514, an earlier PR for the same feature that was closed as a
stale draft, this keeps the sidecars on `images.airflow` and does not inherit
the global digest. The review point there about duplicating `airflow_image` is
handled by a shared `image_reference` helper, which `airflow_image`,
`pod_template_image` and `default_airflow_image` now use as well.
related: #64037, #68784, #63514
Changes:
- `templates/_helpers.yaml`: `airflow_worker_image`, the shared
`image_reference` helper, and `image` added to the worker set full-overwrite
keys.
- `templates/workers/worker-deployment.yaml`: image and pull policy of the
`worker` container.
- `values.yaml`, `values.schema.json`: new `workers.celery.image` section;
its schema is a shared definition also applied to `workers.celery.sets[].image`.
- `docs/customizing-workers.rst`: per-set image example.
- `tests/helm_tests/airflow_core/test_worker.py`, `test_worker_sets.py`:
fallback rules including the chart defaults, pull policy, the other containers
keeping their images, a set's image replacing the parent's, invalid set images
rejected.
---
##### Was generative AI tooling used to co-author this PR?
- [X] Yes — Claude Code (Opus 5.5)
Generated-by: Claude Code (Opus 5.5) following [the
guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
---
* Read the **[Pull Request
Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)**
for more information. Note: commit author/co-author name and email in commits
become permanently public when merged.
* For fundamental code changes, an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals))
is needed.
* When adding dependency, check compliance with the [ASF 3rd Party License
Policy](https://www.apache.org/legal/resolved.html#category-x).
* For significant user-facing changes create newsfragment:
`{pr_number}.significant.rst`, in
[airflow-core/newsfragments](https://github.com/apache/airflow/tree/main/airflow-core/newsfragments).
You can add this file in a follow-up commit after the PR is created so you
know the PR number.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]