Elad, I would really love to solve this in a way that avoids those kinds of discussions and unnecessary "-1" in the future. So let's try to "solve" it for the future. And I am actually super-calm - no stress at all :) .
I am actually glad you raised it, because it shows some deficiencies in our process. I was just hugely surprised to see "-1" rather than discussion. It's pretty strange to discuss this in the VOTE thread, but ... here we are and the reason is your "-1" on those providers, Elad - nothing else. We could probably discuss it in the PR, but since those voting threads are records of the foundation - I think we should explain it here in detail. This way, board members reviewing our voting process will know we treat it seriously. They actually review our VOTE discussions in case you weren't aware :D) and sometimes ask questions about why something happened in a specific VOTE thread. VOTE is a legal act of foundation so we should treat it very seriously - and we should be very precise on why and how we vote here. Radu: > I mentioned this in the PR, that maybe it's OK to modify the CHANGELOG to say that paramiko was bumped because cryptography was bumped - and that happened because of security reasons? Na ..... that was not really it. I looked it up for details: We had **many** issues trying to add Vespa. and it's been - to be honest - one of the most difficult "cross-provider" interaction we had for a while. We had similar cases where we needed to bump botocore in order to avoid whole CI failing, we had cases where we had to bump 5 or 7 provider min-dependnecies, to avoid resolution going for hours - but Vespa was special one indeed. Here are the three errors we had: * We detected unrelated cargo failures, which were really caused by "cargo + uv + installing several rust providers in the same installation"—this caused changes in our Docker container (we added rustc in the CI image) * Cryptography bump was indeed needed * Then we had this one: This is from the track of PRs, and the root cause was this failure: https://github.com/apache/airflow/actions/runs/23802024315/job/69372913309?pr=63988#step:8:2556 ```__________________________________________________________________________ TestSSHHook.test_tunnel_with_private_key_passphrase ___________________________________________________________________________ providers/ssh/tests/unit/ssh/hooks/test_ssh.py:436: in test_tunnel_with_private_key_passphrase hook = SSHHook( providers/ssh/src/airflow/providers/ssh/hooks/ssh.py:183: in __init__ self.pkey = self._pkey_from_private_key(private_key, passphrase=private_key_passphrase) providers/ssh/src/airflow/providers/ssh/hooks/ssh.py:435: in _pkey_from_private_key raise AirflowException( E airflow.sdk.exceptions.AirflowException: Private key provided cannot be read by paramiko.Ensure key provided is valid for one of the followingkey formats: RSA, DSS, ECDSA, or Ed25519 ___________________________________________________________________ TestSSHHook.test_ssh_connection_with_private_key_passphrase_extra ____________________________________________________________________ providers/ssh/tests/unit/ssh/hooks/test_ssh.py:548: in test_ssh_connection_with_private_key_passphrase_extra hook = SSHHook( providers/ssh/src/airflow/providers/ssh/hooks/ssh.py:183: in __init__ self.pkey = self._pkey_from_private_key(private_key, passphrase=private_key_passphrase) providers/ssh/src/airflow/providers/ssh/hooks/ssh.py:435: in _pkey_from_private_key raise AirflowException( E airflow.sdk.exceptions.AirflowException: Private key provided cannot be read by paramiko.Ensure key provided is valid for one of the followingkey formats: RSA, DSS, ECDSA, or Ed25519``` ``` The root cause was that after adding vespa, when `uv` resolved lower-possible direct dependencies for ssh - because of the build heuristics—it also decided to lower-down the paramiko library. And this is difficult to assess - because we do not know `uv` heuristics and it's NP-complete problem in general, so `uv` does a number of shortcuts to figure out the right set of dependencies in as short time as possible, And it turned out that when you downgrade paramiko from 3.5.1 to 3.4.0 - which was the lowest possible version back then: ``` - paramiko==3.5.1 + paramiko==3.4.0 ``` It triggers unrelated issues with supported key formats. It wasn't caused by "vespa" per-se - it was caused by the interaction between the SSH provider, the SFTP provider, Vespa, and uv trying to find a good dependency resolution—apparently, just the presence of vespa and its dependencies in pyproject.toml, causes `uv` to choose this downgrade. And since paramiko 3.5.When version 1 was released back then, we figured the best way to fix our CI and potentially protect our users (who - by various combinations of installation steps could also trigger the same scenario) - we decided to bump paramiko to 3.5.1. This was the "thought train". I don't see any better solution, or one that could take less time to figure out. *Question Elad*: I understand you don't have a problem with the "solution," I described. but with its description of it and process it arrived - can you please confirm that ? Because I think discussion will be different if you have the problem with solution itsefl (but I do not know curreently how else it could be solved).. Now - the question of the "changelog": In this case, adding Vespa to our pyproject is the fact our pyproject.toml triggered it. This is why we bumped it. This is the root cause. There is no other. There is no direct interaction between "vespa" and "ssh" - the root cause is that we have a workspace where we want to install all providers togeether, and due to heuristics of how `uv` works, the issue has been triggered. However - it was there before. If someone installed ssh provider with paramiko==3.5.1 and then downgraded to "3.4.0" - The issue was triggered. I reproduced it. So, a better description would be: "Bumping to minimum version of 3.5.1 because when the user installs the provider with Paramiko 3.5.1 and then downgrades to 3.4.0, it triggers "uknown key" bug" *Elad* - would you be ok if we change it like that in a follow up release or do you have a better wording in mind? Maybe you can create a PR for that? *Very near future: * Also, following what Shahar wrote, I'm about to add "Provider doc preparation skill." That should completely replace (though I will leave it there for now) our "breeze release management prepare-providers-release" command. This is after learning from the "breeze pr auto-triage" conversion—I literally discussed it with Shahar yesterday on Slack And one of the things I planned there was to instruct the agent to search for the root cause of the issue and present it in "user facing way"—letting the LLM figure out the best way to explain the change to the user. And - to be honest - I even planned to re-run on ALL our changelogs to improve them. So maybe the whole discussion, especially the "-1," was unnecessary? Maybe we should just - I do not know - talk about it? J. On Thu, Apr 23, 2026 at 2:37 PM Shahar Epstein <[email protected]> wrote: > Just to put things into context from my perspective as a release manager: > 1. I was aware of the changes to SFTP and SSH resulting from Vespa's PR > while working on the release. I thought that it was an intended change - I > didn't go deeper into whether there was a discussion, as I assumed that it > had been agreed on during approval. > 2. I asked the LLM to modify the titles in the CHANGELOG if they are > unclear, and indeed it detected this issue and modified the entries on its > own. > 3. If we had added the upper bound "<4.0.0" as part of this PR, it would > have indeed been an issue - because users who have already upgraded > paramiko, might not be able to roll back. However, the upper bound had > already been set to "<4.0.0" *before* Vespa's PR, which just increased the > lower bound - and I agree here with Jarek that we do that all the time > < > https://github.com/apache/airflow/commits/main/?author=dependabot%5Bbot%5D > > > (even > without discussing it). > Nonetheless: > Specifically here I would have expected this matter to be discussed in the > PR as these changes seem unrelated, or even making them in separate PRs - > it was a bit confusing for me as a release manager to figure out why these > changes were needed. > > I'll pay more attention to this matter in future releases and let you know > beforehand if I encounter similar cases. > No changes to my vote. > > > Shahar > > On Thu, Apr 23, 2026, 14:15 Jarek Potiuk <[email protected]> wrote: > > > Hey Elad, > > > > But it's - IMHO - cler explained in the changelog: > > > > > > > https://airflow.staged.apache.org/docs/apache-airflow-providers-ssh/5.0.1/changelog.html > > > > > https://airflow.staged.apache.org/docs/apache-airflow-providers-sftp/stable/changelog.html > > > > SSH: > > > > 5.0.1¶ > > Release Date: 2026-04-22 > > > > Misc¶ > > Bump paramiko lower bound to >=3.5.1 due to adding Vespa provider > (#63988) > > > > SFTP" > > > > Changelog¶ > > 5.7.4¶ > > Release Date: 2026-04-22 > > > > Misc¶ > > Bump paramiko lower bound to >=3.5.1 due to adding Vespa provider > (#63988) > > > > Why do you think it needs more explanation? We usually do not do more > > explanation when we bump minimum version - and here the explanation is > > quite clear ? > > > > J. > > > > > > > > On Thu, Apr 23, 2026 at 1:02 PM Elad Kalif <[email protected]> wrote: > > > > > -1 (binding) for ssh and sftp providers. > > > the changes in these providers are bumping paramiko min version which > > came > > > from a PR that adds vespa provider > > > https://github.com/apache/airflow/pull/63988#discussion_r3130238616 > > > There is no indication into why version was bumped and seems like we > > missed > > > it during review. > > > Even if the version needs to be bumped that is not normally how we do > it > > > and generally lets leave this to when > > > https://github.com/apache/airflow/pull/64898 is resolved. > > > > > > On Thu, Apr 23, 2026 at 1:01 PM Wei Lee <[email protected]> wrote: > > > > > > > +1 (binding), checked: > > > > > > > > - SVN > > > > - in Docker installation > > > > - Reproducible package builds > > > > - Licence > > > > - Signature > > > > - Checksums > > > > > > > > Best regards, > > > > Wei Lee > > > > > > > > Shahar Epstein <[email protected]> 於 2026年4月23日週四 上午7:55寫道： > > > > > > > > > Hey all, > > > > > > > > > > I have just cut the new wave Airflow Providers packages with > release > > > > > preparation date 2026-04-21. This email is calling a vote on the > > > release, > > > > > which will last for 72 hours - which means that it will end on > > > 2026-04-25 > > > > > 23:54 UTC and until 3 binding +1 votes have been received. > > > > > > > > > > > > > > > Consider this my (binding) +1. > > > > > Please note that we have an RC for a new provider, Vespa - I'd like > > to > > > > ask > > > > > the stewards to test it well before shipping its first version > > > (v0.1.0). > > > > > > > > > > > > > > > Airflow Providers are available at: > > > > > > https://dist.apache.org/repos/dist/dev/airflow/providers/2026-04-21 > > > > > > > > > > *apache-airflow-providers-2026-04-21-source.tar.gz* is the full > > source > > > > > tarball of airflow repo - snapshot taken at the moment of > provider's > > > > > release. > > > > > > > > > > *apache-airflow-providers-<PROVIDER>-*.tar.gz* are the convenience > > > python > > > > > "sdist" distributions that we publish in PyPI > > > > > > > > > > *apache_airflow_providers_<PROVIDER>-*.whl are the convenience > Python > > > > > "wheel" distributions that we publish in PyPI. > > > > > > > > > > The test procedure for PMC members is described in > > > > > > > > > > > > > > > > > > > > https://github.com/apache/airflow/blob/main/dev/README_RELEASE_PROVIDERS.md#verify-the-release-candidate-by-pmc-members > > > > > > > > > > The test procedure for and Contributors who would like to test this > > RC > > > is > > > > > described in: > > > > > > > > > > > > > > > > > > > > https://github.com/apache/airflow/blob/main/dev/README_RELEASE_PROVIDERS.md#verify-the-release-candidate-by-contributors > > > > > > > > > > > > > > > Public keys are available at: > > > > > https://dist.apache.org/repos/dist/release/airflow/KEYS > > > > > > > > > > Please vote accordingly: > > > > > > > > > > [ ] +1 approve > > > > > [ ] +0 no opinion > > > > > [ ] -1 disapprove with the reason > > > > > > > > > > Only votes from PMC members are binding, but members of the > community > > > are > > > > > encouraged to test the release and vote with "(non-binding)". > > > > > > > > > > Please note that the version number excludes the 'rcX' string. > > > > > This will allow us to rename the artifact without modifying > > > > > the artifact checksums when we actually release it. > > > > > > > > > > The status of testing the providers by the community is kept here: > > > > > https://github.com/apache/airflow/issues/65702 > > > > > > > > > > The issue is also the easiest way to see important PRs included in > > the > > > RC > > > > > candidates. > > > > > Detailed changelog for the providers will be published in the > > > > documentation > > > > > after the > > > > > RC candidates are released. > > > > > > > > > > You can find the RC packages in PyPI following these links: > > > > > > > > > > > https://pypi.org/project/apache-airflow-providers-amazon/9.26.0rc1/ > > > > > > > > > > > > > > > https://pypi.org/project/apache-airflow-providers-apache-kafka/1.13.3rc1/ > > > > > > https://pypi.org/project/apache-airflow-providers-celery/3.19.0rc1/ > > > > > > > > > > > > > > > > > > > > https://pypi.org/project/apache-airflow-providers-cncf-kubernetes/10.16.1rc1/ > > > > > > > https://pypi.org/project/apache-airflow-providers-common-ai/0.1.1rc1/ > > > > > > > > > https://pypi.org/project/apache-airflow-providers-common-sql/1.35.0rc1/ > > > > > > > > > https://pypi.org/project/apache-airflow-providers-databricks/7.13.0rc1/ > > > > > https://pypi.org/project/apache-airflow-providers-edge3/3.5.0rc1/ > > > > > > > > > > > > > > > https://pypi.org/project/apache-airflow-providers-elasticsearch/6.5.3rc1/ > > > > > https://pypi.org/project/apache-airflow-providers-fab/3.6.2rc1/ > > > > > > https://pypi.org/project/apache-airflow-providers-google/21.2.0rc1/ > > > > > > > https://pypi.org/project/apache-airflow-providers-hashicorp/4.6.0rc1/ > > > > > https://pypi.org/project/apache-airflow-providers-jdbc/5.4.4rc1/ > > > > > > > > > > > > > > > > > > > > https://pypi.org/project/apache-airflow-providers-microsoft-azure/13.1.2rc1/ > > > > > > > > > https://pypi.org/project/apache-airflow-providers-openlineage/2.15.0rc1/ > > > > > > > https://pypi.org/project/apache-airflow-providers-opensearch/1.9.1rc1/ > > > > > > > https://pypi.org/project/apache-airflow-providers-papermill/3.13.0rc1/ > > > > > > https://pypi.org/project/apache-airflow-providers-sendgrid/4.2.3rc1/ > > > > > https://pypi.org/project/apache-airflow-providers-sftp/5.7.4rc1/ > > > > > https://pypi.org/project/apache-airflow-providers-smtp/3.0.0rc1/ > > > > > > > https://pypi.org/project/apache-airflow-providers-snowflake/6.12.2rc1/ > > > > > https://pypi.org/project/apache-airflow-providers-ssh/5.0.1rc1/ > > > > > https://pypi.org/project/apache-airflow-providers-vespa/0.1.0rc1/ > > > > > > > > > > Cheers, > > > > > Shahar Epstein > > > > > > > > > > > > > > >
