This is an automated email from the ASF dual-hosted git repository.
kaxil pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git
The following commit(s) were added to refs/heads/main by this push:
new 63a5966473b Check PyPI history before a new provider's first release
(#73887)
63a5966473b is described below
commit 63a5966473b9354a8dcf21403543c6e4dcef26f2
Author: Shahar Epstein <[email protected]>
AuthorDate: Wed Sep 30 22:22:29 2026 +0300
Check PyPI history before a new provider's first release (#73887)
PyPI never lets a filename be reused, even after the file or its whole
project was deleted, and it hides deleted files everywhere except its
public BigQuery dataset. The DuckDB provider hit this in the 2026-09-22
wave: its 0.1.0 upload failed after the vote had passed, because an
earlier, deleted project with the same name had used 0.1.0 and 0.1.1.
Checking the history before the wave starts lets the release manager
pick a free first version instead of dropping the provider from the wave.
Generated-by: Claude Code (Opus 5.5)
---
dev/README_RELEASE_PROVIDERS.md | 62 +++++++++++++++++++++++++++++++++++++++++
1 file changed, 62 insertions(+)
diff --git a/dev/README_RELEASE_PROVIDERS.md b/dev/README_RELEASE_PROVIDERS.md
index e71c211a96f..7061a16a622 100644
--- a/dev/README_RELEASE_PROVIDERS.md
+++ b/dev/README_RELEASE_PROVIDERS.md
@@ -31,6 +31,7 @@
- [Move provider into remove state](#move-provider-into-remove-state)
- [Prepare Regular Provider distributions
(RC)](#prepare-regular-provider-distributions-rc)
- [Perform review of security issues that are marked for the
release](#perform-review-of-security-issues-that-are-marked-for-the-release)
+ - [Check PyPI history of new providers](#check-pypi-history-of-new-providers)
- [Convert commits to changelog entries and bump provider
versions](#convert-commits-to-changelog-entries-and-bump-provider-versions)
- [Update versions of dependent providers to the next
version](#update-versions-of-dependent-providers-to-the-next-version)
- [Create a PR with the changes](#create-a-pr-with-the-changes)
@@ -254,6 +255,67 @@ Additionally, the [dependabot
alerts](https://github.com/apache/airflow/security
code [scanning
alerts](https://github.com/apache/airflow/security/code-scanning) should be
reviewed
and security team should be pinged to review and resolve them.
+## Check PyPI history of new providers
+
+Do this before starting the wave, for every provider in it that has never been
released to PyPI.
+
+PyPI never allows a filename to be reused, even after the file, its release or
its whole project was
+deleted. If an earlier, deleted incarnation of
`apache-airflow-providers-<PROVIDER>` uploaded a version,
+the upload of that version fails at the final release step, after the vote has
passed:
+
+```
+400 This filename was previously used by a file that has since been deleted.
Use a different version.
+```
+
+The provider then has to be excluded from the wave and go through another RC.
The PyPI project page, its
+JSON API and the project's "Security history" do not show files of a deleted
incarnation. The public
+PyPI dataset in BigQuery does:
+
+```shell script
+bq query --use_legacy_sql=false \
+ "SELECT version, filename, upload_time
+ FROM \`bigquery-public-data.pypi.distribution_metadata\`
+ WHERE name = 'apache-airflow-providers-<PROVIDER>'
+ ORDER BY upload_time"
+```
+
+Compare the result with the versions PyPI currently lists:
+
+```shell script
+curl -s https://pypi.org/pypi/apache-airflow-providers-<PROVIDER>/json \
+ | python3 -c "import json, sys; print(*json.load(sys.stdin)['releases'],
sep='\\n')"
+```
+
+A version that the query returns but PyPI does not list was deleted, and its
filenames can never be
+used again. Release candidates of this community, such as `0.1.0rc1`, are
listed by both and are not a
+problem.
+
+A single query like this processes about 2 GB, which is well within the free
tier of 1 TiB of queries
+per month, so it costs nothing. You can check the size first by adding
`--dry_run`. The query needs a
+Google Cloud account; the free [BigQuery
sandbox](https://cloud.google.com/bigquery/docs/sandbox) is
+enough. If you do not have a Google Cloud account or do not know how to use
BigQuery, ask another
+release manager who does to run the query for you.
+
+If deleted versions exist, choose the first version of the provider from the
highest deleted version,
+ignoring any pre-release suffix such as `rc1` or `a1`:
+
+* If the highest deleted version is `0.X.Y`, release the next minor version,
`0.(X+1).0`. For example,
+ if `0.1.0` and `0.1.1` were deleted, the first version is `0.2.0`.
+* If the highest deleted version is `1.X.Y` or higher, release the next major
version. For example,
+ if `1.0.1` was deleted, the first version is `2.0.0`.
+
+Set that version in the provider's `provider.yaml`, and add a warning at the
top of its changelog so
+that users do not mistake the deleted versions for releases of this community:
+
+```rst
+.. warning::
+
+ Versions ``<DELETED VERSIONS>`` of ``apache-airflow-providers-<PROVIDER>``
were uploaded to PyPI by
+ an earlier project with the same name and were later deleted. They were not
released by
+ the Apache Airflow community. Rely only on the versions listed in this
changelog, starting with
+ ``<FIRST VERSION>``.
+```
+
## Convert commits to changelog entries and bump provider versions
First thing that release manager has to do is to convert commits for each
provider into changelog entries
