This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git
The following commit(s) were added to refs/heads/main by this push:
new dda9d61ed94 Fix release process for Python Client (#58447)
dda9d61ed94 is described below
commit dda9d61ed94bcbdbfea11cc4f88e0b3c289ef8c2
Author: Jarek Potiuk <[email protected]>
AuthorDate: Wed Nov 19 02:04:43 2025 +0100
Fix release process for Python Client (#58447)
The Python Client of ours was released so far without sources, because
it was assumed it uses the same sources as Airflow - but it's not
necessarily the same and it can be released with a different cadence
and using different sources - for example to fix some client-only
issues.
This PR updates the release process to also include source tarball
snapshot as part of the release and updates breeze and release process
description to make sure sources are produced and verified.
---
dev/README_RELEASE_PYTHON_CLIENT.md | 97 ++++++++++++++++++++--
.../output_release-management_prepare-tarball.svg | 4 +-
.../output_release-management_prepare-tarball.txt | 2 +-
.../commands/release_candidate_command.py | 5 ++
dev/breeze/src/airflow_breeze/global_constants.py | 1 +
5 files changed, 98 insertions(+), 11 deletions(-)
diff --git a/dev/README_RELEASE_PYTHON_CLIENT.md
b/dev/README_RELEASE_PYTHON_CLIENT.md
index 5a487cef16a..36bdc103b0a 100644
--- a/dev/README_RELEASE_PYTHON_CLIENT.md
+++ b/dev/README_RELEASE_PYTHON_CLIENT.md
@@ -133,6 +133,8 @@ git log 2.8.0..HEAD --pretty=oneline --
airflow-core/src/airflow/api_fastapi/cor
cd ${AIRFLOW_REPO_ROOT}
rm dist/*
breeze release-management prepare-python-client --distribution-format both
--python-client-repo "${CLIENT_REPO_ROOT}" --version-suffix ""
+breeze release-management prepare-tarball --tarball-type
apache_airflow_python_client --version "${VERSION}" --version-suffix
"${VERSION_SUFFIX}"
+
```
- This should generate both sdist and .whl package in `dist` folder of the
Airflow repository. It should
@@ -339,15 +341,34 @@ Airflow Python client supports reproducible builds, which
means that the package
sources should produce binary identical packages in reproducible way. You
should check if the packages can be
binary-reproduced when built from the sources.
-Checkout airflow sources and build packages in dist folder (replace X.Y.Zrc1
with the version + rc candidate)
-you are checking):
+1) Set versions of the packages to be checked:
+
+```shell script
+cd <directory where airflow is checked out>
+VERSION=X.Y.Z
+VERSION_SUFFIX=rc1
+VERSION_RC=${VERSION}${VERSION_SUFFIX}
+```
+
+2) Change directory where your airflow sources are checked out
+
+```shell
+cd "${AIRFLOW_REPO_ROOT}"
+```
+
+3) Check out the ``python-client`` tag (assume apache is the remote name of
the repository):
+
+```shell
+git fetch apache --tags
+git checkout providers/${VERSION_RC}
+```
+
+4) Build the distribution and source tarball:
```shell script
-VERSION=X.Y.Zrc1
-git checkout python-client/${VERSION}
-export AIRFLOW_REPO_ROOT=$(pwd)
rm -rf dist/*
breeze release-management prepare-python-client --distribution-format both
--version-suffix ""
+breeze release-management prepare-tarball --tarball-type
apache_airflow_python_client --version "${VERSION}" --version-suffix
"${VERSION_SUFFIX}"
```
The last - build step - by default will use Dockerized build and building of
Python client packages
@@ -356,13 +377,16 @@ will be done in a docker container. However, if you have
`hatch` installed loc
```bash
breeze release-management prepare-python-client --distribution-format both
--use-local-hatch --version-suffix ""
+breeze release-management prepare-tarball --tarball-type
apache_airflow_python_client --version "${VERSION}" --version-suffix
"${VERSION_SUFFIX}"
```
This is generally faster and requires less resources/network bandwidth.
-Both commands should produce reproducible `.whl`, `.tar.gz` packages in dist
folder.
+Both commands should produce reproducible `.whl`, `.tar.gz` packages in dist
folder and "-source.tar.gz"
+file containing airflow sources in dist folder.
-Change to the directory where you have the packages from svn:
+4) Change to the directory where you have the packages from svn and check if
they are identical to the ones
+you just built:
```shell script
# First clone the repo if you do not have it
@@ -371,7 +395,7 @@ cd ..
svn update --set-depth=infinity asf-dist/dev/airflow/clients/python
# Then compare the packages
-cd asf-dist/dev/airflow/clients/python/${VERSION}
+cd asf-dist/dev/airflow/clients/python/${VERSION_RC}
for i in ${AIRFLOW_REPO_ROOT}/dist/*
do
echo "Checking if $(basename $i) is the same as $i"
@@ -386,6 +410,63 @@ In case the files are different, you should see:
Binary files apache_airflow-client-2.9.0.tar.gz and
.../apache_airflow-2.9.0.tar.gz differ
```
+### Licence check
+
+This can be done with the Apache RAT tool.
+
+Download the latest jar from https://creadur.apache.org/rat/download_rat.cgi
(unpack the binary, the jar is inside)
+
+You can run this command to do it for you:
+
+```shell script
+wget -qO-
https://dlcdn.apache.org//creadur/apache-rat-0.17/apache-rat-0.17-bin.tar.gz |
gunzip | tar -C /tmp -xvf -
+```
+
+Unpack the release source archive (the `<package + version>-source.tar.gz`
file) to a folder
+
+```shell script
+rm -rf /tmp/apache/airflow-python-client-src && mkdir -p
/tmp/apache-airflow-python-client-src && tar -xzf
${PATH_TO_SVN}/providers/${RELEASE_DATE}/apache_airflow_python_client-*-source.tar.gz
--strip-components 1 -C /tmp/apache-airflow-python-client-src
+```
+
+Run the check:
+
+```shell script
+java -jar /tmp/apache-rat-0.17/apache-rat-0.17.jar --input-exclude-file
/tmp/apache-airflow-python-client-src/.rat-excludes
/tmp/apache-airflow-python-client-src/ | grep -E "! |INFO: "
+```
+
+You should see no files reported as Unknown or with wrong licence and summary
of the check similar to:
+
+```
+INFO: Apache Creadur RAT 0.17 (Apache Software Foundation)
+INFO: Excluding patterns: .git-blame-ignore-revs, .github/*, .git ...
+INFO: Excluding MISC collection.
+INFO: Excluding HIDDEN_DIR collection.
+SLF4J(W): No SLF4J providers were found.
+SLF4J(W): Defaulting to no-operation (NOP) logger implementation
+SLF4J(W): See https://www.slf4j.org/codes.html#noProviders for further details.
+INFO: RAT summary:
+INFO: Approved: 15615
+INFO: Archives: 2
+INFO: Binaries: 813
+INFO: Document types: 5
+INFO: Ignored: 2392
+INFO: License categories: 2
+INFO: License names: 2
+INFO: Notices: 216
+INFO: Standards: 15609
+INFO: Unapproved: 0
+INFO: Unknown: 0
+```
+
+There should be no files reported as Unknown or Unapproved. The files that are
unknown or unapproved should be shown with a line starting with `!`.
+
+For example:
+
+```
+! Unapproved: 1 A count of unapproved licenses.
+! /CODE_OF_CONDUCT.md
+```
+
## Signature check
Make sure you have imported into your GPG the PGP key of the person signing
the release. You can find the valid keys in
diff --git
a/dev/breeze/doc/images/output_release-management_prepare-tarball.svg
b/dev/breeze/doc/images/output_release-management_prepare-tarball.svg
index 03e24473a25..278d0a18cd8 100644
--- a/dev/breeze/doc/images/output_release-management_prepare-tarball.svg
+++ b/dev/breeze/doc/images/output_release-management_prepare-tarball.svg
@@ -127,8 +127,8 @@
</text><text class="breeze-release-management-prepare-tarball-r1" x="1464"
y="117.6" textLength="12.2"
clip-path="url(#breeze-release-management-prepare-tarball-line-4)">
</text><text class="breeze-release-management-prepare-tarball-r5" x="0"
y="142" textLength="24.4"
clip-path="url(#breeze-release-management-prepare-tarball-line-5)">╭─</text><text
class="breeze-release-management-prepare-tarball-r5" x="24.4" y="142"
textLength="183"
clip-path="url(#breeze-release-management-prepare-tarball-line-5)"> Tarball flags </text><text
class="breeze-release-management-prepare-tarball-r5" x="207.4" y="142"
textLength="1232.2" clip-path="url(#breeze-r [...]
</text><text class="breeze-release-management-prepare-tarball-r5" x="0"
y="166.4" textLength="12.2"
clip-path="url(#breeze-release-management-prepare-tarball-line-6)">│</text><text
class="breeze-release-management-prepare-tarball-r4" x="24.4" y="166.4"
textLength="170.8"
clip-path="url(#breeze-release-management-prepare-tarball-line-6)">--tarball-type</text><text
class="breeze-release-management-prepare-tarball-r1" x="268.4" y="166.4"
textLength="1171.2" clip-path="url(#breeze-release-ma [...]
-</text><text class="breeze-release-management-prepare-tarball-r5" x="0"
y="190.8" textLength="12.2"
clip-path="url(#breeze-release-management-prepare-tarball-line-7)">│</text><text
class="breeze-release-management-prepare-tarball-r6" x="268.4" y="190.8"
textLength="1171.2"
clip-path="url(#breeze-release-management-prepare-tarball-line-7)">(apache_airflow | apache_airflow_ctl | apache_airflow_providers | apache_airflow_task_sdk |    &
[...]
-</text><text class="breeze-release-management-prepare-tarball-r5" x="0"
y="215.2" textLength="12.2"
clip-path="url(#breeze-release-management-prepare-tarball-line-8)">│</text><text
class="breeze-release-management-prepare-tarball-r6" x="268.4" y="215.2"
textLength="1171.2"
clip-path="url(#breeze-release-management-prepare-tarball-line-8)">helm-chart)                       &#
[...]
+</text><text class="breeze-release-management-prepare-tarball-r5" x="0"
y="190.8" textLength="12.2"
clip-path="url(#breeze-release-management-prepare-tarball-line-7)">│</text><text
class="breeze-release-management-prepare-tarball-r6" x="268.4" y="190.8"
textLength="1171.2"
clip-path="url(#breeze-release-management-prepare-tarball-line-7)">(apache_airflow | apache_airflow_ctl | apache_airflow_providers | apache_airflow_python_client |</text><text
class=" [...]
+</text><text class="breeze-release-management-prepare-tarball-r5" x="0"
y="215.2" textLength="12.2"
clip-path="url(#breeze-release-management-prepare-tarball-line-8)">│</text><text
class="breeze-release-management-prepare-tarball-r6" x="268.4" y="215.2"
textLength="1171.2"
clip-path="url(#breeze-release-management-prepare-tarball-line-8)">apache_airflow_task_sdk | helm-chart)                 &#
[...]
</text><text class="breeze-release-management-prepare-tarball-r5" x="0"
y="239.6" textLength="12.2"
clip-path="url(#breeze-release-management-prepare-tarball-line-9)">│</text><text
class="breeze-release-management-prepare-tarball-r5" x="268.4" y="239.6"
textLength="1171.2"
clip-path="url(#breeze-release-management-prepare-tarball-line-9)">[default: apache_airflow]                    &
[...]
</text><text class="breeze-release-management-prepare-tarball-r5" x="0"
y="264" textLength="12.2"
clip-path="url(#breeze-release-management-prepare-tarball-line-10)">│</text><text
class="breeze-release-management-prepare-tarball-r4" x="24.4" y="264"
textLength="109.8"
clip-path="url(#breeze-release-management-prepare-tarball-line-10)">--version</text><text
class="breeze-release-management-prepare-tarball-r1" x="268.4" y="264"
textLength="1171.2" clip-path="url(#breeze-release-management- [...]
</text><text class="breeze-release-management-prepare-tarball-r5" x="0"
y="288.4" textLength="12.2"
clip-path="url(#breeze-release-management-prepare-tarball-line-11)">│</text><text
class="breeze-release-management-prepare-tarball-r1" x="268.4" y="288.4"
textLength="1171.2"
clip-path="url(#breeze-release-management-prepare-tarball-line-11)">HEAD of current branch will be used and version will be retrieved from there. &
[...]
diff --git
a/dev/breeze/doc/images/output_release-management_prepare-tarball.txt
b/dev/breeze/doc/images/output_release-management_prepare-tarball.txt
index 45c6248ab2c..0435e785397 100644
--- a/dev/breeze/doc/images/output_release-management_prepare-tarball.txt
+++ b/dev/breeze/doc/images/output_release-management_prepare-tarball.txt
@@ -1 +1 @@
-7cb732b47aeebbb6b430cda72776a7a8
+1e525a6dd6717f0ce733a5f679c83489
diff --git
a/dev/breeze/src/airflow_breeze/commands/release_candidate_command.py
b/dev/breeze/src/airflow_breeze/commands/release_candidate_command.py
index 2e9564d9853..5f81a334491 100644
--- a/dev/breeze/src/airflow_breeze/commands/release_candidate_command.py
+++ b/dev/breeze/src/airflow_breeze/commands/release_candidate_command.py
@@ -332,6 +332,11 @@ def create_tarball_release(
if not version:
version = date.strftime(date.today(), "%Y-%m-%d")
console_print(f"\n[info]Using current date {version} as tarball
version\n")
+ elif tarball_type == TarBallType.PYTHON_CLIENT:
+ tag = f"python-client/{version + version_suffix}" if version else
"HEAD"
+ if not version:
+ version = date.strftime(date.today(), "%Y-%m-%d")
+ console_print(f"\n[info]Using current date {version} as tarball
version\n")
else: # pragma: no cover
console_print(f"[error]Unsupported tarball type: {tarball_type}")
exit(1)
diff --git a/dev/breeze/src/airflow_breeze/global_constants.py
b/dev/breeze/src/airflow_breeze/global_constants.py
index 3aeea527fb5..26a82821da5 100644
--- a/dev/breeze/src/airflow_breeze/global_constants.py
+++ b/dev/breeze/src/airflow_breeze/global_constants.py
@@ -243,6 +243,7 @@ class TarBallType(Enum):
PROVIDERS = "apache_airflow_providers"
TASK_SDK = "apache_airflow_task_sdk"
AIRFLOW_CTL = "apache_airflow_ctl"
+ PYTHON_CLIENT = "apache_airflow_python_client"
HELM_CHART = "helm-chart"
