This is an automated email from the ASF dual-hosted git repository.
shahar1 pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git
The following commit(s) were added to refs/heads/main by this push:
new 5dc96d65dd3 Fail publishing provider docs when a provider is missing
from the site index (#74941)
5dc96d65dd3 is described below
commit 5dc96d65dd3916dd902b592e1f5fd44d528af97e
Author: Jarek Potiuk <[email protected]>
AuthorDate: Sun Oct 18 07:43:52 2026 +0200
Fail publishing provider docs when a provider is missing from the site
index (#74941)
* Make the airflow-site index step for new providers hard to miss
The provider release docs already asked release managers to add new
providers to the airflow-site docs index, but the step was a sub-heading that
did not appear in the table of contents and gave no way to check the list.
DuckDB, IBM Db2, IBM MQ and Modal were released without it.
Promote the step to a top-level section with a command that lists released
providers missing from the index and the entry format, flag new providers for
it at the PyPI-history check, and remind the release manager from the
prepare-providers-documentation skill and the Magpie announce configuration.
Generated-by: Claude Opus 5
* Fail publishing provider docs when a provider is missing from the site
index
Documenting the airflow-site index step did not stop it from being skipped:
the step depends on the release manager remembering it, and nothing noticed
when they did not. Checking it in the command that publishes the docs stops the
wave before the docs go live, while the index PR can still be opened, and the
command then reminds the release manager to merge that PR once the docs are
live.
Generated-by: Claude Opus 5
---
.../prepare-providers-documentation/SKILL.md | 7 +
.../release-management-config.md | 9 +
dev/README_RELEASE_PROVIDERS.md | 65 +++++--
.../images/output_workflow-run_publish-docs.svg | 28 ++-
.../images/output_workflow-run_publish-docs.txt | 2 +-
.../airflow_breeze/commands/workflow_commands.py | 170 ++++++++++++++++-
.../commands/workflow_commands_config.py | 1 +
dev/breeze/tests/test_workflow_commands.py | 205 +++++++++++++++++++++
8 files changed, 457 insertions(+), 30 deletions(-)
diff --git a/.agents/skills/prepare-providers-documentation/SKILL.md
b/.agents/skills/prepare-providers-documentation/SKILL.md
index d51db0f3a4a..f706073229c 100644
--- a/.agents/skills/prepare-providers-documentation/SKILL.md
+++ b/.agents/skills/prepare-providers-documentation/SKILL.md
@@ -796,6 +796,13 @@ provider-by-provider:
packages and versions from the PR diff (Phase 4b). Reviewers reliably catch
all of these, so fix them before handing off.
- Flag anything where Phase 3.5 had to escalate, so the RM can double-check.
+- **List every provider in this wave that is an initial release** (no prior
+ release tag — the Phase 1 `note`/`error` case) and remind the release
+ manager that each needs an entry in the `apache/airflow-site` docs index:
+ open that PR before publishing the docs, merge it after.
+ `breeze workflow-run publish-docs --site-env live` fails while a provider is
+ neither in the index nor added by an open PR. Follow
+ [Update airflow-site index for new
providers](../../../dev/README_RELEASE_PROVIDERS.md#update-airflow-site-index-for-new-providers).
Stop here. Do not commit, do not push — the release manager opens the PR
themselves following the regular release workflow in
diff --git a/.apache-magpie-overrides/release-management-config.md
b/.apache-magpie-overrides/release-management-config.md
index af423112642..e981d3baf9f 100644
--- a/.apache-magpie-overrides/release-management-config.md
+++ b/.apache-magpie-overrides/release-management-config.md
@@ -98,3 +98,12 @@ step, and report in the "Status of testing Providers that
were prepared on
| `announce_list` | `[email protected]` |
| `announce_cc_lists` | `[email protected]`, `[email protected]` |
| `site_repo` | `apache/airflow-site` |
+
+When a wave contains a provider released for the first time, the
+`site_repo` change must also add that provider to the "Active providers"
+list in `landing-pages/site/content/en/docs/_index.md`. The list is
+maintained by hand. Open that PR before publishing the docs and merge it
+after them: `breeze workflow-run publish-docs --site-env live` fails while a
+published provider is neither in the index nor added by an open PR. See
+[`README_RELEASE_PROVIDERS.md` § Update airflow-site index for new
providers](../dev/README_RELEASE_PROVIDERS.md#update-airflow-site-index-for-new-providers)
+for the check command and the entry format.
diff --git a/dev/README_RELEASE_PROVIDERS.md b/dev/README_RELEASE_PROVIDERS.md
index be9b0e7cb62..ba745516126 100644
--- a/dev/README_RELEASE_PROVIDERS.md
+++ b/dev/README_RELEASE_PROVIDERS.md
@@ -53,6 +53,7 @@
- [Publish release to SVN](#publish-release-to-svn)
- [Publish the packages to PyPI](#publish-the-packages-to-pypi)
- [Add the final release tag in git](#add-the-final-release-tag-in-git)
+ - [Update airflow-site index for new
providers](#update-airflow-site-index-for-new-providers)
- [Publish documentation](#publish-documentation)
- [Update providers metadata](#update-providers-metadata)
- [Notify developers of release](#notify-developers-of-release)
@@ -259,6 +260,8 @@ and security team should be pinged to review and resolve
them.
## Check PyPI history of new providers
Do this before starting the wave, for every provider in it that has never been
released to PyPI.
+Note these providers down: each one also needs an entry in the airflow-site
docs index when the wave
+is published (see [Update airflow-site index for new
providers](#update-airflow-site-index-for-new-providers)).
PyPI never allows a filename to be reused, even after the file, its release or
its whole project was
deleted. If an earlier, deleted incarnation of
`apache-airflow-providers-<PROVIDER>` uploaded a version,
@@ -1990,6 +1993,52 @@ Then confirm each pushed tag resolves to the wave commit
(`^{}` dereferences the
git ls-remote upstream "refs/tags/providers-<PROVIDER>/<VERSION>^{}"
```
+## Update airflow-site index for new providers
+
+Do this before publishing the documentation of every wave that releases a
provider for the first
+time. This step is easy to miss: DuckDB, IBM Db2, IBM MQ and Modal were
released without it and stayed
+off the public index until it was fixed in
+[apache/airflow-site#1690](https://github.com/apache/airflow-site/pull/1690).
+
+The "Active providers" list on the [documentation
index](https://airflow.apache.org/docs/#providers-packages)
+comes from `landing-pages/site/content/en/docs/_index.md` in
+[`apache/airflow-site`](https://github.com/apache/airflow-site). Nothing
generates it, so a new
+provider does not appear there until someone adds it by hand.
+
+List the released providers missing from the index (run from the
`apache/airflow` checkout,
+with `airflow-site` cloned next to it):
+
+```shell script
+comm -23 \
+ <(git ls-files 'providers/*provider.yaml' | grep -v '/src/\|/tests/' \
+ | sed 's#^providers/##; s#/provider.yaml$##; s#/#-#g' | sort) \
+ <(grep -o 'apache-airflow-providers-[a-z0-9-]*'
../airflow-site/landing-pages/site/content/en/docs/_index.md \
+ | sed 's#^apache-airflow-providers-##' | sort -u)
+```
+
+Skip any listed provider that is not released yet (`state: not-ready` in its
`provider.yaml`,
+or not part of this wave). Add one entry per released provider, in
alphabetical order by
+the provider's `name:` from `provider.yaml`, keeping the blank line between
entries:
+
+```html
+ <li><a
href="/docs/apache-airflow-providers-<provider-id>/stable/index.html"><code><Provider
name></code></a></li>
+```
+
+`<provider-id>` is the provider path with `/` replaced by `-` (e.g. `ibm-db2`).
+
+Do these steps in this order:
+
+1. **Open the PR** in `apache/airflow-site` that adds the new providers to the
index. Do not merge it yet,
+ or the index links to docs that are not published.
+2. **Publish the documentation** ([Publish
documentation](#publish-documentation)). For `--site-env live`,
+ `breeze workflow-run publish-docs` fails before triggering anything when a
published provider is
+ neither in the index on `main` of `apache/airflow-site` nor added to it by
an open PR there. When an
+ open PR adds it, the command lists that PR at the end as the one to merge.
+3. **Merge the PR** once the docs are published.
+
+If you need to publish without this check (for example GitHub is unreachable
from your machine), add
+`--skip-site-index-check` and check the index by hand.
+
## Publish documentation
Documentation is an essential part of the product and should be made available
to users.
@@ -2021,6 +2070,10 @@ Run workflows:
breeze workflow-run publish-docs --ref providers/${RELEASE_DATE} --site-env
live all-providers
```
+If the wave releases a provider for the first time, open the
`apache/airflow-site` index PR first; the
+command fails otherwise, and lists the PR to merge when it completes - see
+[Update airflow-site index for new
providers](#update-airflow-site-index-for-new-providers).
+
If you need to exclude some providers from the documentation you need to add
`--exclude-providers` flag
with space separated list of excluded providers.
@@ -2057,18 +2110,6 @@ not be needed unless there is some problem with workflow
automation above)
> manual intervention. If you need to rebuild the registry independently,
> trigger the `registry-build.yml`
> workflow manually. See [`registry/README.md`](../registry/README.md) for
> details.
-### Update airflow-site index for new providers
-
-If this release adds a brand-new provider, also update
-`airflow-site/landing-pages/site/content/en/docs/_index.md` in the
-`apache/airflow-site` repository. The "Active providers" list there is
-maintained manually, so new providers will not appear on the public docs index
-until that file is updated and the live site refresh completes.
-
-To avoid publishing the provider in the site index too early, keep the
-`apache/airflow-site` change unmerged until the live release window, then merge
-it immediately before or together with the `live` refresh step above.
-
## Update providers metadata
Create PR and open it to be merged:
diff --git a/dev/breeze/doc/images/output_workflow-run_publish-docs.svg
b/dev/breeze/doc/images/output_workflow-run_publish-docs.svg
index e96627fd37c..c71ae5055ac 100644
--- a/dev/breeze/doc/images/output_workflow-run_publish-docs.svg
+++ b/dev/breeze/doc/images/output_workflow-run_publish-docs.svg
@@ -1,4 +1,4 @@
-<svg class="rich-terminal" viewBox="0 0 1482 1245.6"
xmlns="http://www.w3.org/2000/svg";>
+<svg class="rich-terminal" viewBox="0 0 1482 1294.3999999999999"
xmlns="http://www.w3.org/2000/svg";>
<!-- Generated with Rich https://www.textualize.io -->
<style>
@@ -45,7 +45,7 @@
<defs>
<clipPath id="breeze-workflow-run-publish-docs-clip-terminal">
- <rect x="0" y="0" width="1463.0" height="1194.6" />
+ <rect x="0" y="0" width="1463.0" height="1243.3999999999999" />
</clipPath>
<clipPath id="breeze-workflow-run-publish-docs-line-0">
<rect x="0" y="1.5" width="1464" height="24.65"/>
@@ -191,9 +191,15 @@
<clipPath id="breeze-workflow-run-publish-docs-line-47">
<rect x="0" y="1148.3" width="1464" height="24.65"/>
</clipPath>
+<clipPath id="breeze-workflow-run-publish-docs-line-48">
+ <rect x="0" y="1172.7" width="1464" height="24.65"/>
+ </clipPath>
+<clipPath id="breeze-workflow-run-publish-docs-line-49">
+ <rect x="0" y="1197.1" width="1464" height="24.65"/>
+ </clipPath>
</defs>
- <rect fill="#292929" stroke="rgba(255,255,255,0.35)" stroke-width="1"
x="1" y="1" width="1480" height="1243.6" rx="8"/><text
class="breeze-workflow-run-publish-docs-title" fill="#c5c8c6"
text-anchor="middle" x="740"
y="27">Command: workflow-run publish-docs</text>
+ <rect fill="#292929" stroke="rgba(255,255,255,0.35)" stroke-width="1"
x="1" y="1" width="1480" height="1292.4" rx="8"/><text
class="breeze-workflow-run-publish-docs-title" fill="#c5c8c6"
text-anchor="middle" x="740"
y="27">Command: workflow-run publish-docs</text>
<g transform="translate(26,22)">
<circle cx="0" cy="0" r="7" fill="#ff5f57"/>
<circle cx="22" cy="0" r="7" fill="#febc2e"/>
@@ -245,13 +251,15 @@
</text><text class="breeze-workflow-run-publish-docs-r5" x="0" y="971.6"
textLength="12.2"
clip-path="url(#breeze-workflow-run-publish-docs-line-39)">│</text><text
class="breeze-workflow-run-publish-docs-r1" x="402.6" y="971.6"
textLength="378.2"
clip-path="url(#breeze-workflow-run-publish-docs-line-39)">(example: apache.druid,google) </text><text
class="breeze-workflow-run-publish-docs-r8" x="780.8" y="971.6"
textLength="73.2" clip-path="url(#breeze-workflow-run-publish-docs-l [...]
</text><text class="breeze-workflow-run-publish-docs-r5" x="0" y="996"
textLength="12.2"
clip-path="url(#breeze-workflow-run-publish-docs-line-40)">│</text><text
class="breeze-workflow-run-publish-docs-r4" x="24.4" y="996" textLength="353.8"
clip-path="url(#breeze-workflow-run-publish-docs-line-40)">--site-env                   </text><text
class="breeze-workflow-run-publish-docs-r1" x="402.6" [...]
</text><text class="breeze-workflow-run-publish-docs-r5" x="0" y="1020.4"
textLength="12.2"
clip-path="url(#breeze-workflow-run-publish-docs-line-41)">│</text><text
class="breeze-workflow-run-publish-docs-r4" x="24.4" y="1020.4"
textLength="353.8"
clip-path="url(#breeze-workflow-run-publish-docs-line-41)">--skip-write-to-stable-folder</text><text
class="breeze-workflow-run-publish-docs-r1" x="402.6" y="1020.4"
textLength="366" clip-path="url(#breeze-workflow-run-publish-docs-line-41)">Sk
[...]
-</text><text class="breeze-workflow-run-publish-docs-r5" x="0" y="1044.8"
textLength="1464"
clip-path="url(#breeze-workflow-run-publish-docs-line-42)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text
class="breeze-workflow-run-publish-docs-r1" x="1464" y="1044.8"
textLength="12.2" clip-path="url(#breeze-workflow-run-publish-docs-line-42)">
-</text><text class="breeze-workflow-run-publish-docs-r5" x="0" y="1069.2"
textLength="24.4"
clip-path="url(#breeze-workflow-run-publish-docs-line-43)">╭─</text><text
class="breeze-workflow-run-publish-docs-r5" x="24.4" y="1069.2"
textLength="244"
clip-path="url(#breeze-workflow-run-publish-docs-line-43)"> Inventory handling </text><text
class="breeze-workflow-run-publish-docs-r5" x="268.4" y="1069.2"
textLength="1171.2" clip-path="url(#breeze-workflow-run-publish-docs-line [...]
-</text><text class="breeze-workflow-run-publish-docs-r5" x="0" y="1093.6"
textLength="12.2"
clip-path="url(#breeze-workflow-run-publish-docs-line-44)">│</text><text
class="breeze-workflow-run-publish-docs-r4" x="24.4" y="1093.6"
textLength="341.6"
clip-path="url(#breeze-workflow-run-publish-docs-line-44)">--ignore-missing-inventories</text><text
class="breeze-workflow-run-publish-docs-r1" x="390.4" y="1093.6"
textLength="695.4" clip-path="url(#breeze-workflow-run-publish-docs-line-44)">D
[...]
-</text><text class="breeze-workflow-run-publish-docs-r5" x="0" y="1118"
textLength="1464"
clip-path="url(#breeze-workflow-run-publish-docs-line-45)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text
class="breeze-workflow-run-publish-docs-r1" x="1464" y="1118"
textLength="12.2" clip-path="url(#breeze-workflow-run-publish-docs-line-45)">
-</text><text class="breeze-workflow-run-publish-docs-r5" x="0" y="1142.4"
textLength="24.4"
clip-path="url(#breeze-workflow-run-publish-docs-line-46)">╭─</text><text
class="breeze-workflow-run-publish-docs-r5" x="24.4" y="1142.4"
textLength="195.2"
clip-path="url(#breeze-workflow-run-publish-docs-line-46)"> Common options </text><text
class="breeze-workflow-run-publish-docs-r5" x="219.6" y="1142.4"
textLength="1220" clip-path="url(#breeze-workflow-run-publish-docs-line-46)
[...]
-</text><text class="breeze-workflow-run-publish-docs-r5" x="0" y="1166.8"
textLength="12.2"
clip-path="url(#breeze-workflow-run-publish-docs-line-47)">│</text><text
class="breeze-workflow-run-publish-docs-r4" x="24.4" y="1166.8"
textLength="73.2"
clip-path="url(#breeze-workflow-run-publish-docs-line-47)">--help</text><text
class="breeze-workflow-run-publish-docs-r9" x="122" y="1166.8"
textLength="24.4"
clip-path="url(#breeze-workflow-run-publish-docs-line-47)">-h</text><text
class="breez [...]
-</text><text class="breeze-workflow-run-publish-docs-r5" x="0" y="1191.2"
textLength="1464"
clip-path="url(#breeze-workflow-run-publish-docs-line-48)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text
class="breeze-workflow-run-publish-docs-r1" x="1464" y="1191.2"
textLength="12.2" clip-path="url(#breeze-workflow-run-publish-docs-line-48)">
+</text><text class="breeze-workflow-run-publish-docs-r5" x="0" y="1044.8"
textLength="12.2"
clip-path="url(#breeze-workflow-run-publish-docs-line-42)">│</text><text
class="breeze-workflow-run-publish-docs-r4" x="24.4" y="1044.8"
textLength="353.8"
clip-path="url(#breeze-workflow-run-publish-docs-line-42)">--skip-site-index-check      </text><text
class="breeze-workflow-run-publish-docs-r1" x="402.6" y="1044.8"
textLength="1037" clip-path="url(#breeze-workflo [...]
+</text><text class="breeze-workflow-run-publish-docs-r5" x="0" y="1069.2"
textLength="12.2"
clip-path="url(#breeze-workflow-run-publish-docs-line-43)">│</text><text
class="breeze-workflow-run-publish-docs-r1" x="402.6" y="1069.2"
textLength="1037"
clip-path="url(#breeze-workflow-run-publish-docs-line-43)">apache/airflow-site.                           
[...]
+</text><text class="breeze-workflow-run-publish-docs-r5" x="0" y="1093.6"
textLength="1464"
clip-path="url(#breeze-workflow-run-publish-docs-line-44)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text
class="breeze-workflow-run-publish-docs-r1" x="1464" y="1093.6"
textLength="12.2" clip-path="url(#breeze-workflow-run-publish-docs-line-44)">
+</text><text class="breeze-workflow-run-publish-docs-r5" x="0" y="1118"
textLength="24.4"
clip-path="url(#breeze-workflow-run-publish-docs-line-45)">╭─</text><text
class="breeze-workflow-run-publish-docs-r5" x="24.4" y="1118" textLength="244"
clip-path="url(#breeze-workflow-run-publish-docs-line-45)"> Inventory handling </text><text
class="breeze-workflow-run-publish-docs-r5" x="268.4" y="1118"
textLength="1171.2" clip-path="url(#breeze-workflow-run-publish-docs-line-45)">
[...]
+</text><text class="breeze-workflow-run-publish-docs-r5" x="0" y="1142.4"
textLength="12.2"
clip-path="url(#breeze-workflow-run-publish-docs-line-46)">│</text><text
class="breeze-workflow-run-publish-docs-r4" x="24.4" y="1142.4"
textLength="341.6"
clip-path="url(#breeze-workflow-run-publish-docs-line-46)">--ignore-missing-inventories</text><text
class="breeze-workflow-run-publish-docs-r1" x="390.4" y="1142.4"
textLength="695.4" clip-path="url(#breeze-workflow-run-publish-docs-line-46)">D
[...]
+</text><text class="breeze-workflow-run-publish-docs-r5" x="0" y="1166.8"
textLength="1464"
clip-path="url(#breeze-workflow-run-publish-docs-line-47)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text
class="breeze-workflow-run-publish-docs-r1" x="1464" y="1166.8"
textLength="12.2" clip-path="url(#breeze-workflow-run-publish-docs-line-47)">
+</text><text class="breeze-workflow-run-publish-docs-r5" x="0" y="1191.2"
textLength="24.4"
clip-path="url(#breeze-workflow-run-publish-docs-line-48)">╭─</text><text
class="breeze-workflow-run-publish-docs-r5" x="24.4" y="1191.2"
textLength="195.2"
clip-path="url(#breeze-workflow-run-publish-docs-line-48)"> Common options </text><text
class="breeze-workflow-run-publish-docs-r5" x="219.6" y="1191.2"
textLength="1220" clip-path="url(#breeze-workflow-run-publish-docs-line-48)
[...]
+</text><text class="breeze-workflow-run-publish-docs-r5" x="0" y="1215.6"
textLength="12.2"
clip-path="url(#breeze-workflow-run-publish-docs-line-49)">│</text><text
class="breeze-workflow-run-publish-docs-r4" x="24.4" y="1215.6"
textLength="73.2"
clip-path="url(#breeze-workflow-run-publish-docs-line-49)">--help</text><text
class="breeze-workflow-run-publish-docs-r9" x="122" y="1215.6"
textLength="24.4"
clip-path="url(#breeze-workflow-run-publish-docs-line-49)">-h</text><text
class="breez [...]
+</text><text class="breeze-workflow-run-publish-docs-r5" x="0" y="1240"
textLength="1464"
clip-path="url(#breeze-workflow-run-publish-docs-line-50)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text
class="breeze-workflow-run-publish-docs-r1" x="1464" y="1240"
textLength="12.2" clip-path="url(#breeze-workflow-run-publish-docs-line-50)">
</text>
</g>
</g>
diff --git a/dev/breeze/doc/images/output_workflow-run_publish-docs.txt
b/dev/breeze/doc/images/output_workflow-run_publish-docs.txt
index d37b2c6c017..481348f8775 100644
--- a/dev/breeze/doc/images/output_workflow-run_publish-docs.txt
+++ b/dev/breeze/doc/images/output_workflow-run_publish-docs.txt
@@ -1 +1 @@
-471bd4140e28d64d407386e57a726c50
+91396248ef3f0be998673d46b65b4aa8
diff --git a/dev/breeze/src/airflow_breeze/commands/workflow_commands.py
b/dev/breeze/src/airflow_breeze/commands/workflow_commands.py
index 55c3363fae9..a727884487f 100644
--- a/dev/breeze/src/airflow_breeze/commands/workflow_commands.py
+++ b/dev/breeze/src/airflow_breeze/commands/workflow_commands.py
@@ -25,12 +25,19 @@ import sys
import click
from airflow_breeze.commands.common_options import argument_doc_packages,
option_answer
+from airflow_breeze.global_constants import REGULAR_DOC_PACKAGES
from airflow_breeze.utils.click_utils import BreezeGroup
from airflow_breeze.utils.confirm import Answer, user_confirm
from airflow_breeze.utils.console import console_print
from airflow_breeze.utils.custom_param_types import BetterChoice
from airflow_breeze.utils.gh_workflow_utils import get_staging_only_commits,
trigger_workflow_and_monitor
from airflow_breeze.utils.github import run_gh_command
+from airflow_breeze.utils.packages import (
+ expand_all_provider_distributions,
+ get_available_distributions,
+ get_long_package_name,
+)
+from airflow_breeze.utils.shared_options import get_dry_run
WORKFLOW_NAME_MAPS = {
"publish-docs": "publish-docs-to-s3.yml",
@@ -47,6 +54,8 @@ RELEASE_VERSION_PATTERN =
re.compile(r"^\d+\.\d+\.\d+(rc\d+)?$")
APACHE_AIRFLOW_REPO = "apache/airflow"
APACHE_AIRFLOW_SITE_REPO = "apache/airflow-site"
APACHE_AIRFLOW_SITE_ARCHIVE_REPO = "apache/airflow-site-archive"
+# The "Active providers" list on https://airflow.apache.org/docs/ is
maintained by hand in this file.
+AIRFLOW_SITE_DOCS_INDEX_PATH = "landing-pages/site/content/en/docs/_index.md"
@click.group(cls=BreezeGroup, name="workflow-run", help="Tools to manage
Airflow repository workflows ")
@@ -118,6 +127,11 @@ def workflow_run_group():
help="Do not fail the build on missing third-party inventories.",
is_flag=True,
)
[email protected](
+ "--skip-site-index-check",
+ help="Do not fail when a published provider is missing from the docs index
in apache/airflow-site.",
+ is_flag=True,
+)
@argument_doc_packages
def workflow_run_publish(
ref: str,
@@ -131,6 +145,7 @@ def workflow_run_publish(
apply_commits: str | None = None,
workflow_branch: str | None = None,
ignore_missing_inventories: bool = False,
+ skip_site_index_check: bool = False,
):
# Default the workflow-definition ref to the ref being built, so the
workflow version
# matches the content. Running main's workflow against an older tag breaks
when the
@@ -185,6 +200,11 @@ def workflow_run_publish(
console_print("\nYou can add --skip-tag-validation to skip
this validation.")
sys.exit(1)
+ resolved_site_env = resolve_site_env(site_env, ref)
+ site_index_pr_urls: list[str] = []
+ if resolved_site_env == "live" and not skip_site_index_check:
+ site_index_pr_urls = check_providers_are_in_site_index(doc_packages,
exclude_docs)
+
console_print(
f"[blue]Triggering workflow {WORKFLOW_NAME_MAPS['publish-docs']}: at
{APACHE_AIRFLOW_REPO}[/blue]",
)
@@ -240,13 +260,7 @@ def workflow_run_publish(
**workflow_fields,
)
- if site_env == "auto":
- pattern = re.compile(r"^.*[0-9]+\.[0-9]+\.[0-9]+$")
- if pattern.match(ref):
- site_env = "live"
- else:
- site_env = "staging"
-
+ site_env = resolved_site_env
branch = "main" if site_env == "live" else "staging"
console_print(
@@ -281,6 +295,148 @@ def workflow_run_publish(
monitor=False,
)
+ if site_index_pr_urls:
+ console_print(
+ "\n[warning]The docs are published. Now merge the PRs that add the
new providers to the "
+ f"docs index of {APACHE_AIRFLOW_SITE_REPO}:[/warning]"
+ )
+ for url in site_index_pr_urls:
+ console_print(f" {url}")
+
+
+def resolve_site_env(site_env: str, ref: str) -> str:
+ if site_env != "auto":
+ return site_env
+ return "live" if re.match(r"^.*[0-9]+\.[0-9]+\.[0-9]+$", ref) else
"staging"
+
+
+def find_providers_missing_from_site_index(
+ doc_packages: tuple[str, ...], exclude_docs: str, site_index: str
+) -> list[str]:
+ # Removed and not-ready providers are not expected in the index of active
providers.
+ released_provider_ids = set(get_available_distributions())
+ # Split like S3DocsPublish.get_all_excluded_docs, so only docs the upload
skips are skipped here.
+ excluded = set(exclude_docs.split(" "))
+ # The publish workflow builds every package for "all".
+ doc_packages = tuple("all-providers" if package == "all" else package for
package in doc_packages)
+ provider_ids = sorted(
+ package
+ for package in expand_all_provider_distributions(doc_packages)
+ if package in released_provider_ids and package not in excluded
+ )
+ return [
+ provider_id
+ for provider_id in provider_ids
+ if f"/docs/{get_long_package_name(provider_id)}/" not in site_index
+ ]
+
+
+def read_site_index_from_main() -> str:
+ result = run_gh_command(
+ [
+ "gh",
+ "api",
+ "-H",
+ "Accept: application/vnd.github.raw",
+
f"repos/{APACHE_AIRFLOW_SITE_REPO}/contents/{AIRFLOW_SITE_DOCS_INDEX_PATH}?ref=main",
+ ],
+ capture_output=True,
+ check=True,
+ )
+ return result.stdout.decode("utf-8")
+
+
+def find_open_site_index_prs() -> dict[str, str]:
+ """Return the lines that open PRs add to the docs index, keyed by PR
URL."""
+ result = run_gh_command(
+ [
+ "gh",
+ "pr",
+ "list",
+ "--repo",
+ APACHE_AIRFLOW_SITE_REPO,
+ "--state",
+ "open",
+ "--limit",
+ "100",
+ "--json",
+ "number,url,files",
+ ],
+ capture_output=True,
+ check=True,
+ )
+ added_lines_by_pr: dict[str, str] = {}
+ for pr in json.loads(result.stdout):
+ if not any(file["path"] == AIRFLOW_SITE_DOCS_INDEX_PATH for file in
pr["files"]):
+ continue
+ diff = run_gh_command(
+ ["gh", "pr", "diff", str(pr["number"]), "--repo",
APACHE_AIRFLOW_SITE_REPO],
+ capture_output=True,
+ check=True,
+ )
+ added_lines_by_pr[pr["url"]] =
"\n".join(get_lines_added_to_site_index(diff.stdout.decode("utf-8")))
+ return added_lines_by_pr
+
+
+def get_lines_added_to_site_index(diff: str) -> list[str]:
+ added_lines = []
+ in_site_index = False
+ for line in diff.splitlines():
+ if line.startswith("diff --git "):
+ in_site_index = line.endswith(f" b/{AIRFLOW_SITE_DOCS_INDEX_PATH}")
+ elif in_site_index and line.startswith("+") and not
line.startswith("+++"):
+ added_lines.append(line)
+ return added_lines
+
+
+def check_providers_are_in_site_index(doc_packages: tuple[str, ...],
exclude_docs: str) -> list[str]:
+ """
+ Fail unless every published provider is in the docs index on main or added
to it by an open PR.
+
+ Returns the URLs of the open PRs that add providers, which must be merged
after the docs are published.
+ """
+ if all(package in REGULAR_DOC_PACKAGES for package in doc_packages):
+ return []
+ if get_dry_run():
+ console_print("[warning]Skipping the docs index check in dry-run
mode.[/warning]")
+ return []
+ console_print(
+ f"[blue]Checking that published providers are in the docs index of
{APACHE_AIRFLOW_SITE_REPO}[/blue]"
+ )
+ try:
+ missing = find_providers_missing_from_site_index(
+ doc_packages, exclude_docs, read_site_index_from_main()
+ )
+ pr_urls_to_merge = []
+ if missing:
+ for url, added_lines in find_open_site_index_prs().items():
+ still_missing = find_providers_missing_from_site_index(
+ tuple(missing), exclude_docs, added_lines
+ )
+ if len(still_missing) < len(missing):
+ pr_urls_to_merge.append(url)
+ missing = still_missing
+ except (subprocess.CalledProcessError, json.JSONDecodeError, KeyError) as
e:
+ console_print(
+ f"[red]Error: Could not check the docs index of
{APACHE_AIRFLOW_SITE_REPO}: {e!r}[/red]"
+ )
+ console_print("\nAdd --skip-site-index-check to publish without this
check.")
+ sys.exit(1)
+ if missing:
+ console_print(
+ f"[red]Error: These providers are missing from
{AIRFLOW_SITE_DOCS_INDEX_PATH} on main of "
+ f"{APACHE_AIRFLOW_SITE_REPO}, and no open PR there adds
them:[/red]"
+ )
+ for provider_id in missing:
+ console_print(f" {get_long_package_name(provider_id)}")
+ console_print(
+ "\nOpen a PR adding them to the index before publishing, and merge
it after the docs are "
+ "published - see 'Update airflow-site index for new providers' in "
+ "dev/README_RELEASE_PROVIDERS.md. Add --skip-site-index-check to
publish without this check."
+ )
+ sys.exit(1)
+ return pr_urls_to_merge
+
@workflow_run_group.command(
name="release-constraints",
diff --git a/dev/breeze/src/airflow_breeze/commands/workflow_commands_config.py
b/dev/breeze/src/airflow_breeze/commands/workflow_commands_config.py
index 6022fc3aaf8..f446a7c56bf 100644
--- a/dev/breeze/src/airflow_breeze/commands/workflow_commands_config.py
+++ b/dev/breeze/src/airflow_breeze/commands/workflow_commands_config.py
@@ -45,6 +45,7 @@ WORKFLOW_RUN_PARAMETERS: dict[str, list[dict[str, str |
list[str]]]] = {
"--exclude-docs",
"--site-env",
"--skip-write-to-stable-folder",
+ "--skip-site-index-check",
],
},
{
diff --git a/dev/breeze/tests/test_workflow_commands.py
b/dev/breeze/tests/test_workflow_commands.py
index cee0bf9ae88..65d0702aea2 100644
--- a/dev/breeze/tests/test_workflow_commands.py
+++ b/dev/breeze/tests/test_workflow_commands.py
@@ -26,6 +26,8 @@ import pytest
from click.testing import CliRunner
from airflow_breeze.commands.workflow_commands import (
+ find_providers_missing_from_site_index,
+ get_lines_added_to_site_index,
workflow_run_publish,
workflow_run_sync_staging_to_main,
)
@@ -107,6 +109,209 @@ class TestPublishDocsTagValidation:
mock_trigger.assert_called()
+def _site_index_with(*long_package_names: str) -> str:
+ return "\n".join(
+ f' <li><a
href="/docs/{name}/stable/index.html"><code>{name}</code></a></li>'
+ for name in long_package_names
+ )
+
+
+def _make_stdout_response(stdout: str) -> CompletedProcess:
+ return MagicMock(spec=CompletedProcess, returncode=0,
stdout=stdout.encode())
+
+
+SITE_INDEX_PATH = "landing-pages/site/content/en/docs/_index.md"
+SITE_INDEX_PR_URL = "https://github.com/apache/airflow-site/pull/1690";
+
+
+def _make_open_prs_response(*prs: tuple[int, str]) -> CompletedProcess:
+ return _make_stdout_response(
+ json.dumps(
+ [
+ {
+ "number": number,
+ "url":
f"https://github.com/apache/airflow-site/pull/{number}";,
+ "files": [{"path": path}],
+ }
+ for number, path in prs
+ ]
+ )
+ )
+
+
+def _make_diff(path: str, *added_long_package_names: str) -> str:
+ added = _site_index_with(*added_long_package_names).splitlines()
+ return "\n".join(
+ [
+ f"diff --git a/{path} b/{path}",
+ f"--- a/{path}",
+ f"+++ b/{path}",
+ *(f"+{line}" for line in added),
+ " unchanged",
+ ]
+ )
+
+
+def _make_pr_diff_response(*added_long_package_names: str) -> CompletedProcess:
+ return _make_stdout_response(_make_diff(SITE_INDEX_PATH,
*added_long_package_names))
+
+
+class TestPublishDocsSiteIndexCheck:
+ """Tests for the airflow-site docs index check in `breeze workflow-run
publish-docs`."""
+
+ def _invoke(self, site_env: str, doc_packages: list[str], extra_args:
list[str] | None = None):
+ args = ["--ref", "providers/2026-10-13", "--site-env", site_env,
*(extra_args or []), *doc_packages]
+ return CliRunner().invoke(workflow_run_publish, args,
catch_exceptions=False)
+
+
@patch("airflow_breeze.commands.workflow_commands.trigger_workflow_and_monitor",
autospec=True)
+ @patch("airflow_breeze.commands.workflow_commands.run_gh_command",
autospec=True)
+ def test_live_publish_fails_when_provider_missing_from_index_and_prs(
+ self, mock_run_gh_command, mock_trigger
+ ):
+ mock_run_gh_command.side_effect = [
+ _make_gh_response("refs/tags/providers/2026-10-13"),
+
_make_stdout_response(_site_index_with("apache-airflow-providers-amazon")),
+ _make_open_prs_response((1690, SITE_INDEX_PATH), (1638,
"README.md")),
+ _make_pr_diff_response("apache-airflow-providers-modal"),
+ ]
+
+ result = self._invoke("live", ["amazon", "ibm.db2"])
+
+ assert result.exit_code == 1
+ assert "apache-airflow-providers-ibm-db2" in result.output
+ assert "apache-airflow-providers-amazon" not in result.output
+ assert "--skip-site-index-check" in result.output
+ # Only the PR touching the index is diffed.
+ assert mock_run_gh_command.call_count == 4
+ assert mock_run_gh_command.call_args_list[3].args[0][3] == "1690"
+ mock_trigger.assert_not_called()
+
+
@patch("airflow_breeze.commands.workflow_commands.trigger_workflow_and_monitor",
autospec=True)
+ @patch("airflow_breeze.commands.workflow_commands.run_gh_command",
autospec=True)
+ def test_live_publish_proceeds_when_providers_are_in_index(self,
mock_run_gh_command, mock_trigger):
+ mock_run_gh_command.side_effect = [
+ _make_gh_response("refs/tags/providers/2026-10-13"),
+ _make_stdout_response(
+ _site_index_with("apache-airflow-providers-amazon",
"apache-airflow-providers-ibm-db2")
+ ),
+ ]
+
+ result = self._invoke("live", ["amazon", "ibm.db2"])
+
+ assert result.exit_code == 0
+ assert (
+ f"apache/airflow-site/contents/{SITE_INDEX_PATH}?ref=main"
+ in (mock_run_gh_command.call_args_list[1].args[0][-1])
+ )
+ assert "Now merge the PRs" not in result.output
+ mock_trigger.assert_called()
+
+
@patch("airflow_breeze.commands.workflow_commands.trigger_workflow_and_monitor",
autospec=True)
+ @patch("airflow_breeze.commands.workflow_commands.run_gh_command",
autospec=True)
+ def test_live_publish_proceeds_and_asks_to_merge_open_index_pr(self,
mock_run_gh_command, mock_trigger):
+ mock_run_gh_command.side_effect = [
+ _make_gh_response("refs/tags/providers/2026-10-13"),
+
_make_stdout_response(_site_index_with("apache-airflow-providers-amazon")),
+ _make_open_prs_response((1690, SITE_INDEX_PATH)),
+ _make_pr_diff_response("apache-airflow-providers-ibm-db2"),
+ ]
+
+ result = self._invoke("live", ["amazon", "ibm.db2"])
+
+ assert result.exit_code == 0
+ mock_trigger.assert_called()
+ assert "Now merge the PRs" in result.output
+ assert SITE_INDEX_PR_URL in result.output
+
+
@patch("airflow_breeze.commands.workflow_commands.trigger_workflow_and_monitor",
autospec=True)
+ @patch("airflow_breeze.commands.workflow_commands.run_gh_command",
autospec=True)
+ def test_live_publish_fails_when_index_cannot_be_read(self,
mock_run_gh_command, mock_trigger):
+ mock_run_gh_command.side_effect = [
+ _make_gh_response("refs/tags/providers/2026-10-13"),
+ subprocess.CalledProcessError(1, ["gh"]),
+ ]
+
+ result = self._invoke("live", ["amazon"])
+
+ assert result.exit_code == 1
+ assert "Could not check the docs index" in result.output
+ mock_trigger.assert_not_called()
+
+ @pytest.mark.parametrize(
+ ("site_env", "doc_packages", "extra_args"),
+ [
+ pytest.param("staging", ["ibm.db2"], [], id="staging"),
+ pytest.param("live", ["ibm.db2"], ["--skip-site-index-check"],
id="skip-flag"),
+ pytest.param("live", ["apache-airflow"], [], id="no-providers"),
+ ],
+ )
+
@patch("airflow_breeze.commands.workflow_commands.trigger_workflow_and_monitor",
autospec=True)
+ @patch("airflow_breeze.commands.workflow_commands.run_gh_command",
autospec=True)
+ def test_index_is_not_checked(
+ self, mock_run_gh_command, mock_trigger, site_env, doc_packages,
extra_args
+ ):
+ mock_run_gh_command.return_value =
_make_gh_response("refs/tags/providers/2026-10-13")
+
+ result = self._invoke(site_env, doc_packages, extra_args)
+
+ assert result.exit_code == 0
+ mock_run_gh_command.assert_called_once()
+ mock_trigger.assert_called()
+
+ @patch("airflow_breeze.commands.workflow_commands.get_dry_run",
autospec=True, return_value=True)
+
@patch("airflow_breeze.commands.workflow_commands.trigger_workflow_and_monitor",
autospec=True)
+ @patch("airflow_breeze.commands.workflow_commands.run_gh_command",
autospec=True)
+ def test_index_is_not_checked_in_dry_run(self, mock_run_gh_command,
mock_trigger, mock_get_dry_run):
+ result = self._invoke("live", ["ibm.db2"], ["--skip-tag-validation"])
+
+ assert result.exit_code == 0
+ assert "Skipping the docs index check in dry-run mode" in result.output
+ mock_run_gh_command.assert_not_called()
+ mock_trigger.assert_called()
+
+
+class TestFindProvidersMissingFromSiteIndex:
+ @pytest.mark.parametrize(
+ ("doc_packages", "exclude_docs", "expected"),
+ [
+ pytest.param(("amazon", "ibm.db2"), "no-docs-excluded",
["ibm.db2"], id="missing"),
+ pytest.param(("ibm.db2", "modal"), "ibm.db2 modal", [],
id="excluded"),
+ # The S3 upload splits exclusions on spaces only, so it still
publishes both.
+ pytest.param(("ibm.db2", "modal"), "ibm.db2,modal", ["ibm.db2",
"modal"], id="excluded-comma"),
+ pytest.param(("common.dataquality",), "no-docs-excluded", [],
id="not-ready"),
+ pytest.param(("apache-airflow", "amazon"), "no-docs-excluded", [],
id="non-provider"),
+ ],
+ )
+ def test_find_missing(self, doc_packages, exclude_docs, expected):
+ site_index = _site_index_with("apache-airflow-providers-amazon")
+
+ assert find_providers_missing_from_site_index(doc_packages,
exclude_docs, site_index) == expected
+
+ @pytest.mark.parametrize("all_packages", ["all-providers", "all"])
+ def test_all_providers_is_expanded(self, all_packages):
+ missing = find_providers_missing_from_site_index(
+ (all_packages,), "no-docs-excluded",
_site_index_with("apache-airflow-providers-amazon")
+ )
+
+ assert "ibm.db2" in missing
+ assert "amazon" not in missing
+ assert "common.dataquality" not in missing
+
+
+def test_only_lines_added_to_site_index_are_returned():
+ diff = "\n".join(
+ [
+ _make_diff("README.md", "apache-airflow-providers-modal"),
+ _make_diff(SITE_INDEX_PATH, "apache-airflow-providers-ibm-db2"),
+ ]
+ )
+
+ added = "\n".join(get_lines_added_to_site_index(diff))
+
+ assert "apache-airflow-providers-ibm-db2" in added
+ assert "apache-airflow-providers-modal" not in added
+
+
class TestSyncStagingToMain:
@pytest.mark.parametrize(
("answer", "expected_exit_code", "expected_triggered"),
