This is an automated email from the ASF dual-hosted git repository.
shahar1 pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git
The following commit(s) were added to refs/heads/main by this push:
new 986066e53d1 Record lessons from the October 2026 providers waves in
the release docs (#75006)
986066e53d1 is described below
commit 986066e53d159e7d42ffcb92726d02ccfa24c5a2
Author: Jarek Potiuk <[email protected]>
AuthorDate: Sun Oct 18 17:39:39 2026 +0200
Record lessons from the October 2026 providers waves in the release docs
(#75006)
The 2026-10-14 and 2026-10-17 waves hit steps the instructions did not
cover:
a provider outside the wave shipping with a common.compat floor below the
first release that has the symbol it imports, a candidate that had to be
rebuilt before its vote email went out, a build clone with no upstream
remote, an svn commit that cannot prompt from an agent session, tester
confirmations that only lived in issue comments, and a PyPI token prompt
that
needs a terminal an agent does not have.
Generated-by: Claude Opus 5
---
.../prepare-providers-documentation/SKILL.md | 8 ++
dev/README_RELEASE_PROVIDERS.md | 100 +++++++++++++++++++--
2 files changed, 101 insertions(+), 7 deletions(-)
diff --git a/.agents/skills/prepare-providers-documentation/SKILL.md
b/.agents/skills/prepare-providers-documentation/SKILL.md
index f706073229c..ecdb78a8db7 100644
--- a/.agents/skills/prepare-providers-documentation/SKILL.md
+++ b/.agents/skills/prepare-providers-documentation/SKILL.md
@@ -696,6 +696,14 @@ vote rather than re-cut in this release.
> doc preparation and PR creation, so it is easy to forget when the skill
> hands back to the regular release workflow.
+The command only resolves pins an author marked. While classifying, also
+note every PR that starts importing a symbol from another provider (most often
+`common.compat`) and check that the importing provider's floor is at least
+the first release that shipped it, even when the importing provider is not in
+this wave. If it is not, raise the floor in this PR and tell the release
+manager. The recipe is in "Update versions of dependent providers to the next
+version" in `dev/README_RELEASE_PROVIDERS.md`.
+
**Provider dependency-bump CI guard.** Every `>=` bump this produces (and any
inter-provider `>=` bump made during the wave, e.g. a `breaking` provider that
dependents must now require) trips the `check_provider_dependency_bumps`
diff --git a/dev/README_RELEASE_PROVIDERS.md b/dev/README_RELEASE_PROVIDERS.md
index ba745516126..af262adda32 100644
--- a/dev/README_RELEASE_PROVIDERS.md
+++ b/dev/README_RELEASE_PROVIDERS.md
@@ -42,6 +42,7 @@
- [Commit the source packages to Apache SVN
repo](#commit-the-source-packages-to-apache-svn-repo)
- [Publish the Regular distributions to PyPI (release
candidates)](#publish-the-regular-distributions-to-pypi-release-candidates)
- [Push the RC tags](#push-the-rc-tags)
+ - [Re-cut a candidate before the vote email is
sent](#re-cut-a-candidate-before-the-vote-email-is-sent)
- [Prepare documentation in Staging](#prepare-documentation-in-staging)
- [Prepare issue in GitHub to keep status of
testing](#prepare-issue-in-github-to-keep-status-of-testing)
- [Prepare voting email for Providers release
candidate](#prepare-voting-email-for-providers-release-candidate)
@@ -516,6 +517,22 @@ breeze release-management update-providers-next-version
> interactive breeze command instead of the skill. Run it regardless of
> whether you think any
> provider uses the `# use next version` comment — the command is a no-op when
> none do.
+The command only acts on lines the PR author marked. It does not notice when a
PR starts importing
+something from another provider (most often `common.compat`) without raising
the minimum version
+pin. That holds even for a provider that is **not** in this wave: its next
release ships with the
+old floor, and users who already have that older dependent provider installed
get an `ImportError`.
+For every new import from another provider in the wave's PRs, find the first
release that shipped
+the symbol and make sure the pin is at least that version:
+
+```shell script
+# The commit that introduced the symbol, then the first provider release tag
that contains it
+git log --reverse --format=%h -S "<symbol>" -- providers/common/compat/src |
head -1
+git tag --contains <commit> "providers-common-compat/*" --sort=v:refname |
grep -v rc | head -1
+```
+
+In the 2026-10-14 wave, `openai` imported `get_async_extra_dejson` but still
pinned
+`common.compat` below 1.21.0, the first release that has it.
+
## Create a PR with the changes
Make sure to set labels: `allow provider dependency bump` and `skip common
compat check` to the PR,
@@ -724,6 +741,11 @@ key you want to use.
```shell script
export RELEASE_DATE=$(date "+%Y-%m-%d%n")
export AIRFLOW_REPO_ROOT=$(pwd -P)
+# The merge commit of the release-prep PR. Tag and build exactly this commit,
never "whatever HEAD is".
+export WAVE_SHA=<sha>
+# The remote that `git remote -v` shows pushing to apache/airflow: `origin` in
a fresh clone
+export APACHE_REMOTE=<remote>
+git checkout ${WAVE_SHA}
rm -rf ${AIRFLOW_REPO_ROOT}/dist/*
```
@@ -732,13 +754,16 @@ rm -rf ${AIRFLOW_REPO_ROOT}/dist/*
These instructions assume the standard remote naming convention
(`upstream` → `apache/airflow`, `origin` → your fork — see
[`contributing-docs/10_working_with_git.rst`](../contributing-docs/10_working_with_git.rst#git-remote-naming-conventions)).
+A fresh `git clone` used for the build (see below) has only `origin`, pointing
at `apache/airflow`.
+Set `APACHE_REMOTE` below to whichever remote `git remote -v` shows pointing
at `apache/airflow` (a
+URL such as `[email protected]:apache/airflow.git` works too). Never push the
tags to your fork.
Set tags for the providers in the repo.
```shell script
echo "Tagging with providers/${RELEASE_DATE}"
-git tag -s providers/${RELEASE_DATE} -m "Tag providers for ${RELEASE_DATE}"
--force
-git push upstream providers/${RELEASE_DATE}
+git tag -s providers/${RELEASE_DATE} ${WAVE_SHA} -m "Tag providers for
${RELEASE_DATE}" --force
+git push ${APACHE_REMOTE} providers/${RELEASE_DATE}
breeze release-management prepare-provider-distributions
--include-removed-providers --distribution-format both --version-suffix ""
breeze release-management prepare-tarball --tarball-type
apache_airflow_providers --version "${RELEASE_DATE}"
```
@@ -768,8 +793,8 @@ if you only build few packages, run:
```shell script
echo "Tagging with providers/${RELEASE_DATE}"
-git tag -s providers/${RELEASE_DATE} -m "Tag providers for ${RELEASE_DATE}"
--force
-git push upstream providers/${RELEASE_DATE}
+git tag -s providers/${RELEASE_DATE} ${WAVE_SHA} -m "Tag providers for
${RELEASE_DATE}" --force
+git push ${APACHE_REMOTE} providers/${RELEASE_DATE}
breeze release-management prepare-provider-distributions
--include-removed-providers --distribution-format both --version-suffix ""
PACKAGE PACKAGE ....
breeze release-management prepare-tarball --tarball-type
apache_airflow_providers --version "${RELEASE_DATE}"
@@ -823,6 +848,10 @@ svn commit -m "Add artifacts for Airflow Providers
${RELEASE_DATE}"
cd "$AIRFLOW_REPO_ROOT"
```
+`svn commit` asks for your ASF password unless SVN has it cached, so run it in
your own terminal.
+From an agent session or a script without a terminal it fails with `E215004`
(authentication
+failed) instead of prompting.
+
Verify that the files are available in the ${RELEASE_DATE} folder under
[providers](https://dist.apache.org/repos/dist/dev/airflow/providers/)
@@ -914,6 +943,22 @@ twine check ${AIRFLOW_REPO_ROOT}/dist/*
read -rs TWINE_PASSWORD && export TWINE_PASSWORD # paste the token,
press Enter
```
+ `read -s` needs a terminal, so it cannot work when an agent or a script
runs the upload for
+ you. In that case, use Magpie's
+
[`secret-prompt`](https://github.com/apache/magpie/tree/main/tools/secret-prompt)
instead of the
+ two exports. It asks for the token in the terminal or, when there is
none, in a dialog (macOS
+ `osascript`, Linux `zenity`/`kdialog`, or `pinentry`). It shows the exact
command that will
+ receive the token and passes the token only to that command, so the token
never lands in your
+ shell, a file or the agent's transcript. Install it once, then prefix
`twine upload` in the
+ command below:
+
+ ```shell script
+ uv tool install
"git+https://github.com/apache/magpie#subdirectory=tools/secret-prompt";
+ # ... then, in the upload command below, replace "twine upload" with:
+ secret-prompt --env TWINE_PASSWORD --set TWINE_USERNAME=__token__
--prefix pypi- \
+ --title "PyPI upload" --prompt "Paste the single-use PyPI token" --
twine upload
+ ```
+
4. Run the upload (below).
5. **Delete the token** at
[Account settings → API
tokens](https://pypi.org/manage/account/#api-tokens) and run
@@ -972,8 +1017,41 @@ Verify the pushed tags point where you think they do.
Plain `git ls-remote` prin
SHA for annotated tags, which never matches the commit and looks alarming;
dereference it with `^{}`:
```shell script
-git ls-remote upstream "refs/tags/providers-<PROVIDER>/<VERSION>^{}"
-```
+git ls-remote ${APACHE_REMOTE} "refs/tags/providers-<PROVIDER>/<VERSION>^{}"
+```
+
+## Re-cut a candidate before the vote email is sent
+
+When a problem turns up after the candidates are built and staged but
**before** the vote email has
+gone out, you can rebuild the same wave instead of starting a new one. Nobody
has started verifying
+it yet. Merge the fix, then:
+
+* Keep `RELEASE_DATE`. The SVN folder, the wave tag and the vote email keep
their names.
+* Check that the fix is the only change to the wave's providers since the
original build:
+ `git log --oneline <old WAVE_SHA>..<new commit> -- providers/<provider>
...`. Anything else listed
+ would ship untested and missing from the changelog and the testing issue:
either fold it in
+ (changelog entries, testing-issue rows) or apply the fix on a branch cut
from the old `WAVE_SHA`
+ and tag that commit instead.
+* Set `WAVE_SHA` to the new commit and move the wave tag there:
+ `git tag -s -f providers/${RELEASE_DATE} ${WAVE_SHA} -m "Tag providers for
${RELEASE_DATE}"`, then
+ `git push --force ${APACHE_REMOTE} refs/tags/providers/${RELEASE_DATE}`.
+* Check out the new commit (`git checkout ${WAVE_SHA}`, and confirm with `git
rev-parse HEAD`), then
+ re-run the [build and
sign](#build-and-sign-the-source-and-convenience-packages) steps from the
+ cleanup of `dist/` onwards, minus the wave tag you just moved. Skip the
`export RELEASE_DATE=...`
+ line there and keep the original date, or a re-cut on a later day renames
the tarball.
+* In `dev/airflow/providers/${RELEASE_DATE}`, `svn rm` the files of any
provider or version that is no
+ longer in the wave, copy the new build over the rest, and commit (`svn add
--force .` picks up any
+ new files). The filenames carry no `rcN`, so unchanged providers' files are
simply replaced.
+* PyPI never lets you reuse a filename, so publish the next candidate number
(`rc2` → `rc3`), and pass
+ it explicitly as `--version-suffix rc3`: breeze only moves past an rcN whose
tag exists locally,
+ and the previous candidate's tags may not have been created yet. If the
wave's providers are at
+ different candidate numbers, build each group with its own next suffix. The
previous `rcN` stays on PyPI,
+ and its tags stay in git if they were pushed. Nobody voted on it, and the
vote email names only the
+ new one.
+* Push the new RC tags at `${WAVE_SHA}` and republish the staging docs.
+
+Once the vote email is out, do not do this: cancel the vote and run a new
wave, or remove the
+provider from it.
## Prepare documentation in Staging
@@ -1104,6 +1182,11 @@ sed -i '' -E "s/- \[ \] (.*\(#(${checked})\))/- [x] \1/"
files/provider_issue.md
sed -i -E "s/- \[ \] (.*\(#(${checked})\))/- [x] \1/" files/provider_issue.md
```
+Testers do not always tick the box: some confirm a PR in a comment instead
("tested #12345, works").
+Before you extract the ticks, read the previous issue's comments and tick any
PR confirmed there in
+the previous issue itself, so the record is complete and the extraction picks
it up. In the
+2026-10-14 wave, five such PRs were confirmed only in comments.
+
Review the resulting `[x]` lines in `files/provider_issue.md` (PRs only
present in the new wave stay
unchecked). If the issue was already created, apply the same local edit and
then edit the online issue as described in Step 3.
@@ -1933,6 +2016,9 @@ twine check ${AIRFLOW_REPO_ROOT}/dist/*.whl
${AIRFLOW_REPO_ROOT}/dist/*.tar.gz
read -rs TWINE_PASSWORD && export TWINE_PASSWORD # paste the token,
press Enter
```
+ When an agent or a script without a terminal runs the upload, use
`secret-prompt` instead, as
+ described in the [release candidate
upload](#publish-the-regular-distributions-to-pypi-release-candidates).
+
4. Run the upload (below).
5. **Delete the token** at
[Account settings → API
tokens](https://pypi.org/manage/account/#api-tokens) and run
@@ -1990,7 +2076,7 @@ breeze release-management tag-providers --release-date
${RELEASE_DATE}
Then confirm each pushed tag resolves to the wave commit (`^{}` dereferences
the annotated tag):
```shell script
-git ls-remote upstream "refs/tags/providers-<PROVIDER>/<VERSION>^{}"
+git ls-remote "${APACHE_REMOTE:?set it to the remote that pushes to
apache/airflow}" "refs/tags/providers-<PROVIDER>/<VERSION>^{}"
```
## Update airflow-site index for new providers
