shahar1 opened a new issue, #70587:
URL: https://github.com/apache/airflow/issues/70587
## Human Summary
Since the providers registry build was introduced into the docs release
process, the release
process for the providers' docs became longer by about 20-45 minutes (median
~23).
The big issue is that this process is composed of sequential runs of
different GitHub Actions
workflows in different repositories (airflow, airflow-site, and
airflow-site-archive), and this
process shouldn't be interrupted. More run time means:
1. More time spent on watching the entire process.
2. More chance that things will go wrong in the critical path, and we might
need to do things all
over again.
In my opinion, the provider registry is not as critical a component as the
docs - which means that
it would be better separated into another workflow that will be initiated
separately.
related: #50648, #45641, #44373, #32491, #50570, #66436
<details>
<summary>Detailed investigation - measurements, code references and two open
questions (AI-generated)</summary>
During a provider release wave the release manager runs `breeze workflow-run
publish-docs` and has
to keep the terminal open and the machine awake until it returns — the whole
chain is driven from
the release manager's laptop, hence the warning at
[`dev/README_RELEASE_PROVIDERS.md:1769-1772`](https://github.com/apache/airflow/blob/379d4ac118071af99e62278fcaf0b1ba5b4f2c45/dev/README_RELEASE_PROVIDERS.md#L1769-L1772):
> Make sure to NOT close the terminal while the command is running and to
keep your computer from
> sleep. This command will run several (3!) workflows from your terminal and
this is important to
> keep it running until completion.
On the 2026-07-22 wave the `publish-docs-to-s3.yml` run alone took 90
minutes, and the docs had
been live on S3 since minute 47. The rest was the provider registry build,
which the release
manager gains nothing by waiting for. Two questions below about getting it
off that path.
#### Measured
Per-job timings, three recent wave runs:
| Job |
[30335779728](https://github.com/apache/airflow/actions/runs/30335779728) |
[30042123097](https://github.com/apache/airflow/actions/runs/30042123097) |
[29107680148](https://github.com/apache/airflow/actions/runs/29107680148) |
|---|---|---|---|
| Build Info | 0.7m | 0.7m | 0.7m |
| Build documentation | 41.0m | 34.2m | 41.8m |
| Publish documentation to S3 | 4.3m | 4.7m | 4.5m |
| — *docs are live from here* — | | | |
| Registry: Build CI image | **32.5m** | **13.3m** | **31.8m** |
| Registry: Build & Publish | 10.5m | 9.3m | 9.7m |
| **Total (wall clock)** | **90.2m** | **62.4m** | **88.9m** |
| *of which after docs were live* | *43.4m (48%)* | *22.7m (36%)* | *41.6m
(47%)* |
*(Total is run `createdAt`→`updatedAt`, so it exceeds the sum of the jobs by
the queue gaps.
`Build Java SDK Javadoc` — 0.2m, parallel with `Build documentation` — is
omitted.)*
Over the last 20 runs of the workflow, excluding one failed and one
cancelled run, split on whether
the `Update Provider Registry` job ran or was skipped:
| Cohort | n | Median total | Range |
|---|---|---|---|
| Registry ran | 11 | **59.9m** | 34.7–90.2m |
| Registry skipped | 7 | **21.2m** | 14.4–22.1m |
Not a clean A/B — the skipped runs are core version publishes plus one
dev-branch run, whose
`Build documentation` median is 16.1m against 32.6m for the registry-ran
cohort. Of the 38.7m
difference, roughly 23m is the registry and roughly 16m the larger docs
build.
The registry sub-workflow on its own: median **23.0m**, range 17.3–43.0m
(n=11) — ~38% of the
median run, and 100% additive because it is strictly serial after the S3
publish. Over half of it
(median 13.4m, range 8.0–33.1m) is spent building a CI image before `breeze
registry extract-data`
runs.
#### Question 1: why does the registry job build its own CI image?
[`registry-build.yml:59-94`](https://github.com/apache/airflow/blob/379d4ac118071af99e62278fcaf0b1ba5b4f2c45/.github/workflows/registry-build.yml#L59-L94)
calls `ci-image-build.yml` with `branch: "main"` and `push-image: "false"`.
`branch` only sets
`DEFAULT_BRANCH` — the image name and therefore the buildx cache source
([`ci-image-build.yml:115`](https://github.com/apache/airflow/blob/379d4ac118071af99e62278fcaf0b1ba5b4f2c45/.github/workflows/ci-image-build.yml#L115)).
No `checkout-ref` is passed, so
[`:125-129`](https://github.com/apache/airflow/blob/379d4ac118071af99e62278fcaf0b1ba5b4f2c45/.github/workflows/ci-image-build.yml#L125-L129)
checks out the workflow ref, i.e. the release tag. The job therefore builds
**the release tag's
sources under main's image name, against main's registry cache**. The hit
rate is unstable as a
result: the same tag rebuilt on 2026-07-23 and 2026-07-28 took 13.3m and
32.5m, the slow run
re-running `install_os_dependencies.sh` for 19.6m after #70285 landed on
main.
Meanwhile
[`publish-docs-to-s3.yml:301-318`](https://github.com/apache/airflow/blob/379d4ac118071af99e62278fcaf0b1ba5b4f2c45/.github/workflows/publish-docs-to-s3.yml#L301-L318)
builds a CI image from the *same ref, in the same run*, and never stashes
it. So every wave pays
for two full CI image builds from the same sources — the docs job's
`python3.10`
([`:101`](https://github.com/apache/airflow/blob/379d4ac118071af99e62278fcaf0b1ba5b4f2c45/.github/workflows/publish-docs-to-s3.yml#L101))
and the registry's `python3.12`
([`registry-build.yml:88`](https://github.com/apache/airflow/blob/379d4ac118071af99e62278fcaf0b1ba5b4f2c45/.github/workflows/registry-build.yml#L88)).
Could the registry job consume the docs job's image instead of rebuilding —
accepting that this
would move registry extraction to 3.10 — or, if main-tip dependencies are
acceptable, pull
`ghcr.io/apache/airflow/main/ci/python3.12`, which canary already pushes via
`finalize-tests.yml` → `push-image-cache.yml`? (For the pull variant note
that
`build-and-publish-registry` sets its own
[`permissions: contents:
read`](https://github.com/apache/airflow/blob/379d4ac118071af99e62278fcaf0b1ba5b4f2c45/.github/workflows/registry-build.yml#L112-L113),
which replaces the workflow-level default, so it would need `packages: read`
adding — though the
images are public.)
To be clear about what cannot be dropped: two of the three extractors need
the image.
`breeze registry extract-data` runs `extract_metadata.py`,
`extract_parameters.py` and
`extract_connections.py`
([`registry_commands.py:112-117`](https://github.com/apache/airflow/blob/379d4ac118071af99e62278fcaf0b1ba5b4f2c45/dev/breeze/src/airflow_breeze/commands/registry_commands.py#L112-L117)),
and module discovery uses runtime inspection inside Breeze where all
providers are installed —
`registry_commands.py:103-105` even `pip install`s suspended providers
first. The question is
build-vs-reuse, not image-vs-no-image. It is worth answering regardless of
question 2 — though
note the two interact: reusing the docs job's image assumes the current
serialisation, so if
question 2 goes the parallel way, only the ghcr-pull variant survives.
#### Question 2: does `update-registry` need `needs: [publish-docs-to-s3]`?
[`publish-docs-to-s3.yml:609`](https://github.com/apache/airflow/blob/379d4ac118071af99e62278fcaf0b1ba5b4f2c45/.github/workflows/publish-docs-to-s3.yml#L609)
is `needs: [publish-docs-to-s3, build-info]`. Dropping the first element —
keeping
`needs: [build-info]` — would run the registry *in parallel* with the docs
build. It stays a
`workflow_call`, so nothing changes about permissions, and `build-info`'s
`destination` /
`registry-providers` / `registry-full-build` outputs are still available —
`update-registry`
consumes nothing from the `publish-docs-to-s3` job. Recomputing the same 11
runs from their
per-job timings, the median run drops from 59.9m to ~39m and no run gets
longer, though on 5 of
the 11 the registry is the longer branch and becomes the critical path
itself. That is arithmetic
on historical timestamps, not an executed run — in particular it assumes the
two CI-image builds
can run concurrently without contending for runners, which the timestamps
cannot tell us.
The registry reads nothing the docs publish produces — its Sphinx
`objects.inv` fetch goes to
`apache-airflow-docs` (eu-central-1), the main-branch canary bucket written
by
[`ci-image-checks.yml:465`](https://github.com/apache/airflow/blob/379d4ac118071af99e62278fcaf0b1ba5b4f2c45/.github/workflows/ci-image-checks.yml#L465),
not the `live-docs-airflow-apache-org` bucket the publish job writes. The
two also write disjoint
prefixes (`…/docs/` vs `…/registry/`), so there is no clobbering.
What it *emits* is the real question. The docs URLs the wave build generates
are all `/stable/`:
```python
# dev/registry/extract_metadata.py:63
AIRFLOW_PROVIDER_DOCS_URL =
"https://airflow.apache.org/docs/{package_name}/stable/";
# dev/registry/extract_parameters.py:451
base_docs_url =
f"https://airflow.apache.org/docs/apache-airflow-providers-{provider_id}/stable";
```
and they are used for deep links, not just landing pages — per-class API
anchors
(`extract_parameters.py:463-468`) and per-connection pages
(`extract_metadata.py:317-326`). Since
the inventory comes from the canary bucket, it already knows about modules
newer than the release.
So publishing the registry before the docs gives a window in which
`/stable/` links point at the
previous version, and anything **new in this release** — new provider, new
module, new connection
page — 404s until the docs land.
Recomputing that window over the same 11 runs (registry starting after
`build-info` instead of
after the publish): in **5 of 11 the window is negative** — the registry
would still have landed
after the docs, so no exposure at all. On the other 6 it runs up to ~21
minutes, median ~3 minutes
across all 11. Note the shape: the largest windows (17–21m) are the waves
where the registry's
CI-image build hit cache and took 8–13m, while the two 90-minute waves —
where that build took
~32m — come out under 5 minutes. Fixing question 1 would make this window
*wider*, not narrower.
There is a second consequence that is not just ordering: `needs:
[publish-docs-to-s3]` currently
*gates* the registry, so a failed docs build or publish skips it. Remove the
edge and the registry
publishes to live regardless, and the window stops being bounded — the
registry would sit there
pointing at `/stable/` pages that never landed. If we do this, the gate
probably has to be
reinstated somewhere, e.g. on the S3 sync step rather than on the whole job.
Note that the GitHub source URLs *are* version-pinned
(`extract_metadata.py:64-66`,
`extract_parameters.py:450-452`) and gated on the `providers-<id>/<ver>` tag
existing
([`find_latest_released_version`,
`extract_metadata.py:507-521`](https://github.com/apache/airflow/blob/379d4ac118071af99e62278fcaf0b1ba5b4f2c45/dev/registry/extract_metadata.py#L507-L521)),
so the tag is a hard input — but the tag is pushed one step *before* docs
publishing
([`README_RELEASE_PROVIDERS.md:1706`](https://github.com/apache/airflow/blob/379d4ac118071af99e62278fcaf0b1ba5b4f2c45/dev/README_RELEASE_PROVIDERS.md#L1706)),
so it is already satisfied by the time the workflow starts.
So: is a stale window of that size on new modules acceptable in exchange for
taking the registry
off the critical path, and where should the failed-publish gate live? If the
window is not
acceptable, is there a cheaper ordering than "wait for the whole publish
job"?
Worth recording why the *fully* decoupled variant is harder, in case someone
proposes it: a
`gh workflow run` issued from inside Actions arrives as `workflow_dispatch`
with
`github.event.sender.login == github-actions[bot]`, which fails the
committer allowlist at
[`registry-build.yml:65-81`](https://github.com/apache/airflow/blob/379d4ac118071af99e62278fcaf0b1ba5b4f2c45/.github/workflows/registry-build.yml#L65-L81);
`build-and-publish-registry` needs that job, so the run would go green
having done nothing — the
same silent-skip class as #65188. That would need a bot entry in the
allowlist, which is what
#69460 proposes to delete. Neither of the questions above requires going
there.
#### Separately
The release manager's wait is also longer than the workflow for an unrelated
reason:
[`monitor_workflow_run`](https://github.com/apache/airflow/blob/379d4ac118071af99e62278fcaf0b1ba5b4f2c45/dev/breeze/src/airflow_breeze/utils/gh_workflow_utils.py#L150-L193)
polls **run**-level status, so the `apache/airflow-site` refresh does not
start until the whole run
finishes, registry included. Returning once `Publish documentation to S3`
succeeds would unblock the
terminal at ~47m instead of ~90m. The trade-off is failure visibility: today
a failed run exits
non-zero in the release manager's terminal, and #65188 is a case of a
registry problem going
unnoticed for weeks, so an early return needs a real signal rather than a
printed URL. Fixing
question 2 removes most of the need for this.
#### Why now, given #50648
[#50648](https://github.com/apache/airflow/issues/50648) asked for faster
docs publishing and was
closed on 2025-06-09 with:
> Yeah. I think release date is more important than speed of publishing.
Especially that it is not
> **that** slow and with the "publish-docs" breeze command it's fine to just
submit it in terminal
> and let it run until it gets completed..
>
> Closing
I am not reopening the artifact-reuse idea declined there — the release-date
argument against it
still holds. But that assessment predates the registry, merged in
[#62261](https://github.com/apache/airflow/pull/62261) on 2026-03-07.
#50648's body cites "12-15
minutes"; runs where the registry job fires now median 59.9m. "Submit it in
terminal and let it
run" is a different proposition at 90 minutes than at 15.
#### Related
- #62261 added the registry and the `needs:` chaining.
- #66436 (release-breaking), #65506 (surfaced a red check on every
airflow-ctl / task-sdk docs
publish) and #65188 (the registry job silently never ran) are three bugs
in the registry-input
derivation inside this workflow's `build-info` job. Neither question above
addresses that class.
- #69460 — replace the per-workflow committer allowlists with a protected
`release` environment.
It does not enumerate `registry-build.yml`, but the intent covers it.
</details>
### Committer
- [x] I acknowledge that I am a maintainer/committer of the Apache Airflow
project.
---
Drafted-by: Claude Code (Opus 5); reviewed by @shahar1 before posting
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
