This is an automated email from the ASF dual-hosted git repository.
shahar1 pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git
The following commit(s) were added to refs/heads/main by this push:
new bc2628e4f34 Auto-generate sorted packages.txt during providers PyPI
upload (#69356)
bc2628e4f34 is described below
commit bc2628e4f3455d76d4db90cb4650d1b55d6e27b3
Author: Shahar Epstein <[email protected]>
AuthorDate: Fri Jul 3 20:36:52 2026 +0300
Auto-generate sorted packages.txt during providers PyPI upload (#69356)
Release managers previously copied the PyPI URLs printed by twine out of the
terminal by hand, sorted them, and saved them off to the side for the vote
email and announcement. The completeness-check step also referenced
packages.txt before it existed, and the twine upload used dist/* globs that
swept in the ASF source tarball and the .asc/.sha512 files, which are
SVN-only
and make twine fail with 'Unknown distribution format'.
The upload now writes the log and a sorted, de-duplicated packages.txt into
the
git-ignored files/ directory, uploads only the provider wheels and sdists,
and
the completeness gate runs after the file is generated, right before the
vote
email is sent.
---
dev/README_RELEASE_PROVIDERS.md | 79 ++++++++++++++++++++++++++---------------
1 file changed, 50 insertions(+), 29 deletions(-)
diff --git a/dev/README_RELEASE_PROVIDERS.md b/dev/README_RELEASE_PROVIDERS.md
index 7a95c062dd1..7f4c1ab0350 100644
--- a/dev/README_RELEASE_PROVIDERS.md
+++ b/dev/README_RELEASE_PROVIDERS.md
@@ -592,20 +592,6 @@ svn commit -m "Add artifacts for Airflow Providers
${RELEASE_DATE}"
cd "$AIRFLOW_REPO_ROOT"
```
-* Before sending the vote email, gate on the same completeness check the PMC
verifiers run, so a
- missing artifact (e.g. the `-source.tar.gz` tarball) fails here instead of
in the vote thread.
- Put the package list from the upcoming vote email into `dev/packages.txt`,
then run:
-
-```shell script
-cd "$AIRFLOW_REPO_ROOT"
-breeze release-management check-release-files providers --release-date
"${RELEASE_DATE}" \
- --packages-file ./dev/packages.txt \
- --path-to-airflow-svn "$(cd ../asf-dist/dev/airflow && pwd -P)"
-```
-
- It exits non-zero and lists every missing file (including `.asc`/`.sha512`
variants) if anything is
- absent. Only proceed to the vote once it prints `All expected files are
present!`.
-
Verify that the files are available in the ${RELEASE_DATE} folder under
[providers](https://dist.apache.org/repos/dist/dev/airflow/providers/)
@@ -692,15 +678,27 @@ twine check ${AIRFLOW_REPO_ROOT}/dist/*
This is a defence-in-depth practice: the RM machine becomes a one-time
release vehicle, not a persistent point of compromise.
-* Upload the package to PyPI:
+* Upload the packages to PyPI. Only the provider wheels and sdists are
uploaded — the ASF
+ `-source.tar.gz` tarball and the `.asc`/`.sha512` files are SVN-only and
must not go to PyPI.
+ The output is teed to a log and the PyPI URLs twine prints are extracted
(sorted, de-duplicated)
+ into `files/packages.txt`. That file is what you paste into the vote email,
and what the
+ completeness gate ("Prepare voting email" section below) and the PMC
verifiers consume. Both files
+ land in `files/` (git-ignored), so they are never accidentally committed:
```shell script
-twine upload -r pypi ${AIRFLOW_REPO_ROOT}/dist/*
+mkdir -p "${AIRFLOW_REPO_ROOT}/files"
+# COLUMNS=200 stops rich (twine's printer) from wrapping long URLs when stdout
is a pipe.
+COLUMNS=200 twine upload -r pypi ${AIRFLOW_REPO_ROOT}/dist/*.whl \
+ $(ls ${AIRFLOW_REPO_ROOT}/dist/*.tar.gz | grep -v -- '-source.tar.gz') 2>&1 \
+ | tee "${AIRFLOW_REPO_ROOT}/files/twine-upload.log"
+# Only trust packages.txt if the upload above succeeded (PIPESTATUS[0] == 0):
+grep -oE 'https://pypi\.org/project/[^[:space:]]+'
"${AIRFLOW_REPO_ROOT}/files/twine-upload.log" \
+ | sort -u > "${AIRFLOW_REPO_ROOT}/files/packages.txt"
```
-* Confirm that the packages are available under the links printed and look
good.
-
-* Save these links for later, you'll need to paste them in the email you'll
send to [email protected]
+* Confirm that the packages are available under the links printed and look
good. The same links are
+ now saved, sorted, in `files/packages.txt` — you'll paste them into the vote
email you send to
+ [email protected].
## Push the RC tags
@@ -856,6 +854,21 @@ gh issue create --repo apache/airflow \
Make sure the packages are in
https://dist.apache.org/repos/dist/dev/airflow/providers/
+* Before sending the vote email, gate on the same completeness check the PMC
verifiers run, so a
+ missing artifact (e.g. the `-source.tar.gz` tarball) fails here instead of
in the vote thread.
+ `files/packages.txt` was already generated by the PyPI upload step ("Publish
the Regular
+ distributions to PyPI" above); run the check against it:
+
+```shell script
+cd "$AIRFLOW_REPO_ROOT"
+breeze release-management check-release-files providers --release-date
"${RELEASE_DATE}" \
+ --packages-file ./files/packages.txt \
+ --path-to-airflow-svn "$(cd ../asf-dist/dev/airflow && pwd -P)"
+```
+
+ It exits non-zero and lists every missing file (including `.asc`/`.sha512`
variants) if anything is
+ absent. Only proceed to the vote once it prints `All expected files are
present!`.
+
Send out a vote to the [email protected] mailing list. Here you can
prepare text of the
email.
@@ -1004,13 +1017,14 @@ to verify that all expected files are present in SVN.
This command will produce
may help with verifying installation of the packages.
Once you have cloned/updated the SVN repository, copy the PyPi URLs shared
-in the email to a file called `packages.txt` in the $AIRFLOW_REPO_ROOT/dev
-directory.
+in the email to a file called `packages.txt` in the $AIRFLOW_REPO_ROOT/files
+directory (git-ignored, so it won't be accidentally committed).
```shell script
cd "$AIRFLOW_REPO_ROOT"
+mkdir -p files
# Copy packages.txt extracted from the mail sent by the release manager here
-breeze release-management check-release-files providers --release-date
"${RELEASE_DATE}" --packages-file ./dev/packages.txt --path-to-airflow-svn
"${PATH_TO_AIRFLOW_SVN}"
+breeze release-management check-release-files providers --release-date
"${RELEASE_DATE}" --packages-file ./files/packages.txt --path-to-airflow-svn
"${PATH_TO_AIRFLOW_SVN}"
```
After the above command completes you can build `Dockerfile.pmc` to trigger an
installation of each provider
@@ -1566,17 +1580,24 @@ twine check ${AIRFLOW_REPO_ROOT}/dist/*.whl
${AIRFLOW_REPO_ROOT}/dist/*.tar.gz
This is a defence-in-depth practice: the RM machine becomes a one-time
release vehicle, not a persistent point of compromise.
-* Upload the package to PyPI:
+* Upload the packages to PyPI. Only the provider wheels and sdists are
uploaded — the ASF
+ `-source.tar.gz` tarball and the `.asc`/`.sha512` files are SVN-only. The
PyPI URLs twine prints
+ are extracted (sorted, de-duplicated) into `files/packages.txt` for the
announcement message; both
+ files land in `files/` (git-ignored):
```shell script
-twine upload -r pypi ${AIRFLOW_REPO_ROOT}/dist/*.whl
${AIRFLOW_REPO_ROOT}/dist/*.tar.gz
+mkdir -p "${AIRFLOW_REPO_ROOT}/files"
+# COLUMNS=200 stops rich (twine's printer) from wrapping long URLs when stdout
is a pipe.
+COLUMNS=200 twine upload -r pypi ${AIRFLOW_REPO_ROOT}/dist/*.whl \
+ $(ls ${AIRFLOW_REPO_ROOT}/dist/*.tar.gz | grep -v -- '-source.tar.gz') 2>&1 \
+ | tee "${AIRFLOW_REPO_ROOT}/files/twine-upload.log"
+# Only trust packages.txt if the upload above succeeded (PIPESTATUS[0] == 0):
+grep -oE 'https://pypi\.org/project/[^[:space:]]+'
"${AIRFLOW_REPO_ROOT}/files/twine-upload.log" \
+ | sort -u > "${AIRFLOW_REPO_ROOT}/files/packages.txt"
```
-* Verify that the packages are available under the links printed.
-
-Copy links to updated packages, sort it alphabetically and save it on the
side. You will need it for the announcement message.
-
-* Again, confirm that the packages are available under the links printed.
+* Confirm that the packages are available under the links printed. The sorted
list is saved in
+ `files/packages.txt` for the announcement message.
## Add the final release tag in git
