Firstly, I released a difference while making breeze compatible with tool
install and enabling auto update in breeze. That made me realise we have a
parameter difference in verification and how the packages are built. So I
thought we should do the same thing and even committed to raising a PR to
sync the processes. Then, when I ran the verification a second time, it
created the packages. One we use version and another version suffix, but
when created haven't dug thinking that it should have been from
an environment variable. I always skipped tagging to prevent overriding the
tag created.
Verification:
https://github.com/apache/airflow/blob/d273d99262f93d9af601b3fb3c12f915f4ddbb27/dev/README_RELEASE_HELM_CHART.md?plain=1#L504
RC Creation:
https://github.com/apache/airflow/blob/d273d99262f93d9af601b3fb3c12f915f4ddbb27/dev/README_RELEASE_HELM_CHART.md?plain=1#L226
When I created it, I also saw the -dev suffix, but thought not a breaking
change for the release process. The Chart.yaml has also been updated
automatically while creating. I rerun the process again to double-check and
see that this also updates the doc links automatically.
Below is one running with `--skip-tagging` and seeing that updates the doc
references inside Chart.yaml. Even with an empty `VERSION_SUFFIX`, it seems
it is updating.
I ran the one in RC creation command first and then ran the one in
verification, which worked. I haven't seen any difference while checking
the packages. I thought only the Chart.yaml making it `-dev` while thinking
it isn't a great deal to stop the process made me overlook the doc URL,
also updated.
>From the print you shared, Jarek, I think we should use the Verification
approach then, while releasing, so that the doc references are also not
updated to staging. I also ran it once again in the verification process,
and that only updates the version in Chart.yaml.
Sharing the details of what was mentioned above after `---`.
---
*Verification Process*
``` bash
VERSION=1.20.0 breeze release-management prepare-helm-chart-tarball
--version-suffix ${VERSION_SUFFIX} --ignore-version-check --skip-tagging
```
Before
``` yaml
...
apiVersion: v2
name: airflow
version: 1.20.0-dev
appVersion: 3.1.8
description: The official Helm chart to deploy Apache Airflow, a platform to
programmatically author, schedule, and monitor workflows
home: https://airflow.apache.org/
...
```
After
``` yaml
...
apiVersion: v2
name: airflow
version: 1.20.0
appVersion: 3.1.8
description: The official Helm chart to deploy Apache Airflow, a platform to
programmatically author, schedule, and monitor workflows
home: https://airflow.apache.org/
...
```
*Release Process*
``` bash
breeze release-management prepare-helm-chart-tarball --version ${VERSION}
--version-suffix ${VERSION_SUFFIX} --skip-tagging
```
Before
``` yaml
...
apiVersion: v2
name: airflow
version: 1.20.0-dev
appVersion: 3.1.8
description: The official Helm chart to deploy Apache Airflow, a platform to
programmatically author, schedule, and monitor workflows
home: https://airflow.apache.org/
sources:
- https://github.com/apache/airflow
icon: https://airflow.apache.org/images/airflow_dark_bg.png
keywords:
- apache
- airflow
- workflow
- scheduler
dependencies:
- name: postgresql
version: 13.2.24
repository: https://charts.bitnami.com/bitnami
condition: postgresql.enabled
maintainers:
- email: [email protected]
name: Apache Airflow PMC
type: application
annotations:
artifacthub.io/links: |
- name: Documentation
url: https://airflow.apache.org/docs/helm-chart/1.20.0/
artifacthub.io/screenshots: |
- title: Home Page
url:
https://airflow.apache.org/docs/apache-airflow/3.1.8/_images/home_dark.png
- title: DAG Overview Dashboard
url:
https://airflow.apache.org/docs/apache-airflow/3.1.8/_images/dag_overview_dashboard.png
- title: DAGs View
url:
https://airflow.apache.org/docs/apache-airflow/3.1.8/_images/dags.png
- title: Assets View
url:
https://airflow.apache.org/docs/apache-airflow/3.1.8/_images/asset_view.png
- title: Grid View
url:
https://airflow.apache.org/docs/apache-airflow/3.1.8/_images/dag_overview_grid.png
- title: Graph View
url:
https://airflow.apache.org/docs/apache-airflow/3.1.8/_images/dag_overview_graph.png
- title: Variable View
url:
https://airflow.apache.org/docs/apache-airflow/3.1.8/_images/variable_hidden.png
- title: Code View
url:
https://airflow.apache.org/docs/apache-airflow/3.1.8/_images/dag_overview_code.png
...
```
After
``` yaml
...
apiVersion: v2
name: airflow
version: 1.20.0
appVersion: 3.1.8
description: The official Helm chart to deploy Apache Airflow, a platform to
programmatically author, schedule, and monitor workflows
home: https://airflow.staged.apache.org/
sources:
- https://github.com/apache/airflow
icon: https://airflow.staged.apache.org/images/airflow_dark_bg.png
keywords:
- apache
- airflow
- workflow
- scheduler
dependencies:
- name: postgresql
version: 13.2.24
repository: https://charts.bitnami.com/bitnami
condition: postgresql.enabled
maintainers:
- email: [email protected]
name: Apache Airflow PMC
type: application
annotations:
artifacthub.io/links: |
- name: Documentation
url: https://airflow.staged.apache.org/docs/helm-chart/1.20.0/
artifacthub.io/screenshots: |
- title: Home Page
url:
https://airflow.staged.apache.org/docs/apache-airflow/3.1.8/_images/home_dark.png
- title: DAG Overview Dashboard
url:
https://airflow.staged.apache.org/docs/apache-airflow/3.1.8/_images/dag_overview_dashboard.png
- title: DAGs View
url:
https://airflow.staged.apache.org/docs/apache-airflow/3.1.8/_images/dags.png
- title: Assets View
url:
https://airflow.staged.apache.org/docs/apache-airflow/3.1.8/_images/asset_view.png
- title: Grid View
url:
https://airflow.staged.apache.org/docs/apache-airflow/3.1.8/_images/dag_overview_grid.png
- title: Graph View
url:
https://airflow.staged.apache.org/docs/apache-airflow/3.1.8/_images/dag_overview_graph.png
- title: Variable View
url:
https://airflow.staged.apache.org/docs/apache-airflow/3.1.8/_images/variable_hidden.png
- title: Code View
url:
https://airflow.staged.apache.org/docs/apache-airflow/3.1.8/_images/dag_overview_code.png
...
```
On Sat, Mar 21, 2026 at 2:17 PM Jarek Potiuk <[email protected]> wrote:
> Hmm. For now it's
>
> -1: (binding) for me. I think we need to find out what happened here :)
> before making further decisions.
>
> I am not sure exactly how the packages were produced (I followed the
> verification procedure not the preparation procedure—but they look quite
> the same). Maybe? Possibly? I'm unaware of some subtle differences in the
> release process, or perhaps the tag was pushed differently. I can't really
> explain it currently—unless someone made manual modifications to the
> artifacts along the way to fix things (Jens?). This means our
> reproducibility check would **in fact** detect such a thing - even if it
> were an innocent attempt to fix the produced artifact (absolutely no doubt
> about no malicious intent from Jens):
>
> One thing is less important: when I use the `helm-chart/1.20.0rc1` - I
> can't produce the chart or tarball (breeze commands fail) unless I
> manually correct the version from 1.20.0-dev (currently in sources
>
> https://github.com/apache/airflow/blob/helm-chart/1.20.0rc1/chart/Chart.yaml
> #L22
> <https://github.com/apache/airflow/blob/helm-chart/1.20.0rc1/chart/Chart.yaml#L22>).
> When I manually change it to 1.20.0, both: `.tar.gz` and
> `-source.tar.gz` are produced, they have correct names, and the version in
> the chart is as expected (1.20.0).
>
> Documenting the process to prepare the chart to remove the dev suffix would
> be acceptable and should resolve the issue without cancelling the vote.
>
> However, there is one, more serious issue: even if I change the version,
> the resulting Helm chart is different (diff below). Essentially, the Helm
> chart in SVN links to "staged.apache.org" documentation, not "apache.org".
>
> That basically disqualifies what we currently have in SVN now from
> promoting to release because documentation links are wrong. Small, but
> important thing.
>
> I think there's a mishap regarding how the -dev prefix was added. We used
> to add it to the sources (usually) in the past, but we haven't done that
> for quite a while. We should probably agree on how to handle it (it might
> even be documented incorrectly somewhere). There is no mentioning of the
> prefix in the prefix, but I think Jed could have added it "automatically"
> here:
>
> https://github.com/apache/airflow/commit/370c1dc27961b5f36b738e4be27b07c3cd6e5313
>
> But basically the -dev suffix presence makes the links point to staged.
>
> So my hypothesis (Jens?) was that the version was modified manually, and we
> should improve the process to automate it rather than rely on manual
> modifications. But I am guessing now - however the chart in our SVN now is
> not really good for release.
>
> J.
>
> ---------
>
>
> ```
> --- /Users/jarekpotiuk/code/airflow-clone/dist/airflow-1.20.0.tgz
> +++
>
> /Users/jarekpotiuk/code/asf-dist/dev/airflow/helm-chart/1.20.0rc1/airflow-1.20.0.tgz
> ├── airflow-1.20.0.tgz-content
> │ ├── file list
> │ │ @@ -1,12 +1,12 @@
> │ │ drwx------ 0 root (0) root (0) 0 2026-03-17
> 21:55:32.000000 airflow/
> │ │ -rw------- 0 root (0) root (0) 1196 2026-03-17
> 21:55:32.000000 airflow/.helmignore
> │ │ -rw------- 0 root (0) root (0) 4148 2026-03-17
> 21:55:32.000000 airflow/.pre-commit-config.yaml
> │ │ -rw------- 0 root (0) root (0) 227 2026-03-17
> 21:55:32.000000 airflow/Chart.lock
> │ │ --rw------- 0 root (0) root (0) 7117 2026-03-17
> 21:55:32.000000 airflow/Chart.yaml
> │ │ +-rw------- 0 root (0) root (0) 7194 2026-03-17
> 21:55:32.000000 airflow/Chart.yaml
> │ │ -rw------- 0 root (0) root (0) 468 2026-03-17
> 21:55:32.000000 airflow/INSTALL
> │ │ -rw------- 0 root (0) root (0) 10850 2026-03-17
> 21:55:32.000000 airflow/LICENSE
> │ │ -rw------- 0 root (0) root (0) 863 2026-03-17
> 21:55:32.000000 airflow/NOTICE
> │ │ -rw------- 0 root (0) root (0) 3715 2026-03-17
> 21:55:32.000000 airflow/README.md
> │ │ -rw------- 0 root (0) root (0) 72480 2026-03-17
> 21:55:32.000000 airflow/RELEASE_NOTES.rst
> │ │ drwx------ 0 root (0) root (0) 0 2026-03-17
> 21:55:32.000000 airflow/charts/
> │ │ drwx------ 0 root (0) root (0) 0 2026-03-17
> 21:55:32.000000 airflow/charts/postgresql/
> │ ├── airflow/Chart.yaml
> │ │ @@ -142,43 +142,43 @@
> │ │ - description: Allow to use short SPDX licence identifier for
> selected files
> │ │ kind: fixed
> │ │ links:
> │ │ - name: '#62073'
> │ │ url: https://github.com/apache/airflow/pull/62073
> │ │ artifacthub.io/links: |
> │ │ - name: Documentation
> │ │ - url: https://airflow.apache.org/docs/helm-chart/1.20.0/
> │ │ + url: https://airflow.staged.apache.org/docs/helm-chart/1.20.0/
> │ │ artifacthub.io/screenshots: |
> │ │ - title: Home Page
> │ │ - url:
> https://airflow.apache.org/docs/apache-airflow/3.1.8/_images/home_dark.png
> │ │ + url:
>
> https://airflow.staged.apache.org/docs/apache-airflow/3.1.8/_images/home_dark.png
> │ │ - title: DAG Overview Dashboard
> │ │ - url:
>
> https://airflow.apache.org/docs/apache-airflow/3.1.8/_images/dag_overview_dashboard.png
> │ │ + url:
>
> https://airflow.staged.apache.org/docs/apache-airflow/3.1.8/_images/dag_overview_dashboard.png
> │ │ - title: DAGs View
> │ │ - url:
> https://airflow.apache.org/docs/apache-airflow/3.1.8/_images/dags.png
> │ │ + url:
>
> https://airflow.staged.apache.org/docs/apache-airflow/3.1.8/_images/dags.png
> │ │ - title: Assets View
> │ │ - url:
> https://airflow.apache.org/docs/apache-airflow/3.1.8/_images/asset_view.png
> │ │ + url:
>
> https://airflow.staged.apache.org/docs/apache-airflow/3.1.8/_images/asset_view.png
> │ │ - title: Grid View
> │ │ - url:
>
> https://airflow.apache.org/docs/apache-airflow/3.1.8/_images/dag_overview_grid.png
> │ │ + url:
>
> https://airflow.staged.apache.org/docs/apache-airflow/3.1.8/_images/dag_overview_grid.png
> │ │ - title: Graph View
> │ │ - url:
>
> https://airflow.apache.org/docs/apache-airflow/3.1.8/_images/dag_overview_graph.png
> │ │ + url:
>
> https://airflow.staged.apache.org/docs/apache-airflow/3.1.8/_images/dag_overview_graph.png
> │ │ - title: Variable View
> │ │ - url:
>
> https://airflow.apache.org/docs/apache-airflow/3.1.8/_images/variable_hidden.png
> │ │ + url:
>
> https://airflow.staged.apache.org/docs/apache-airflow/3.1.8/_images/variable_hidden.png
> │ │ - title: Code View
> │ │ - url:
>
> https://airflow.apache.org/docs/apache-airflow/3.1.8/_images/dag_overview_code.png
> │ │ + url:
>
> https://airflow.staged.apache.org/docs/apache-airflow/3.1.8/_images/dag_overview_code.png
> │ │ apiVersion: v2
> │ │ appVersion: 3.1.8
> │ │ dependencies:
> │ │ - condition: postgresql.enabled
> │ │ name: postgresql
> │ │ repository: https://charts.bitnami.com/bitnami
> │ │ version: 13.2.24
> │ │ description: The official Helm chart to deploy Apache Airflow, a
> platform to programmatically
> │ │ author, schedule, and monitor workflows
> │ │ -home: https://airflow.apache.org/
> │ │ -icon: https://airflow.apache.org/images/airflow_dark_bg.png
> │ │ +home: https://airflow.staged.apache.org/
> │ │ +icon: https://airflow.staged.apache.org/images/airflow_dark_bg.png
> │ │ keywords:
> │ │ - apache
> │ │ - airflow
> │ │ - workflow
> │ │ - scheduler
> │ │ maintainers:
> │ │ - email: [email protected]
> ```
>
>
>
>
> On Sat, Mar 21, 2026 at 12:32 PM Jens Scheffler <[email protected]>
> wrote:
>
> > Note/Reminder: Still looking for more VOTE responses in order to
> > validate release.
> >
> > All PMCs are on call for this (it is your duty!) and all other
> > contributers are also welcome to validate.
> >
> > On 18.03.26 23:34, Buğra Öztürk wrote:
> > > +1 (binding)
> > > I have checked,
> > > - SVN
> > > - Reproducible package builds
> > > - Licence
> > > - Signature
> > > - Checksums
> > >
> > > Best regards,
> > >
> > > Bugra Ozturk
> > >
> > > On Wed, Mar 18, 2026 at 12:37 AM Jens Scheffler <[email protected]>
> > wrote:
> > >
> > >> Hello Apache Airflow Community,
> > >>
> > >> This is a call for the vote to release Helm Chart version 1.20.0.
> > >>
> > >> The release candidate is available at:
> > >> https://dist.apache.org/repos/dist/dev/airflow/helm-chart/1.20.0rc1/
> > >>
> > >> airflow-chart-1.20.0-source.tar.gz - is the "main source release" that
> > >> comes with INSTALL instructions.
> > >> airflow-1.20.0.tgz - is the binary Helm Chart release.
> > >>
> > >> Public keys are available at:
> https://www.apache.org/dist/airflow/KEYS
> > >>
> > >> For convenience "index.yaml" has been uploaded (though excluded from
> > >> voting), so you can also run the below commands.
> > >>
> > >> helm repo add apache-airflow-dev
> > >> https://dist.apache.org/repos/dist/dev/airflow/helm-chart/1.20.0rc1/
> > >> helm repo update
> > >> helm install airflow apache-airflow-dev/airflow
> > >>
> > >> airflow-1.20.0.tgz.prov - is also uploaded for verifying Chart
> > >> Integrity, though not strictly required for releasing the artifact
> based
> > >> on ASF Guidelines.
> > >>
> > >> $ helm gpg verify airflow-1.20.0.tgz
> > >> gpg: Signature made Mi 18 Mär 2026 00:07:31 CET
> > >> gpg: using EDDSA key
> > >> EFDA915011C4751F8A3AD979AEFD29C8BA60C0F4
> > >> gpg: issuer "[email protected]"
> > >> gpg: Good signature from "Jens Scheffler <[email protected]>"
> > [ultimate]
> > >> plugin: Chart SHA verified.
> > >>
> sha256:50b9de4c9d363be0af893bf018f49b234be8af69bdabf84915a7789e6930d329
> > >>
> > >> The vote will be open for at least 72 hours (2026-03-20 23:36 UTC) or
> > >> until the necessary number of votes is reached.
> > >>
> > >>
> > >>
> >
> https://www.timeanddate.com/countdown/to?iso=20260320T2337&p0=136&font=cursive
> > >>
> > >> Please vote accordingly:
> > >>
> > >> [ ] +1 approve
> > >> [ ] +0 no opinion
> > >> [ ] -1 disapprove with the reason
> > >>
> > >> Only votes from PMC members are binding, but members of the community
> > are
> > >> encouraged to test the release and vote with "(non-binding)".
> > >>
> > >> Consider this my (binding) +1.
> > >>
> > >> For license checks, the .rat-excludes files is included, so you can
> run
> > >> the following to verify licenses (just update your path to rat):
> > >>
> > >> tar -xvf airflow-chart-1.20.0-source.tar.gz
> > >> cd airflow-chart-1.20.0
> > >> java -jar apache-rat-0.13.jar chart -E .rat-excludes
> > >>
> > >> Please note that the version number excludes the `rcX` string, so it's
> > >> now simply 1.20.0. This will allow us to rename the artifact without
> > >> modifying the artifact checksums when we actually release it.
> > >>
> > >> The status of testing the Helm Chart by the community is kept here:
> > >> https://github.com/apache/airflow/issues/63859
> > >>
> > >> Thanks,
> > >> Jens Scheffler
> > >>
> > >>
> > >> ---------------------------------------------------------------------
> > >> To unsubscribe, e-mail: [email protected]
> > >> For additional commands, e-mail: [email protected]
> > >>
> > >>
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: [email protected]
> > For additional commands, e-mail: [email protected]
> >
> >
>
