So just to close the discussion around the issues I raised. My concerns were addressed in the PR discussion and also with the information Jarek shared on this thread.
On Thu, Apr 23, 2026 at 4:30 PM Elad Kalif <[email protected]> wrote: > Jarek, > > I don't get what you are saying. This is a provider release vote. There is > no earlier discussion. > I don't know where you expected me to raise -1 prior to the vote? > From my point **at the time I casted the vote** there was no indication of > the reasoning. I'm sorry but both you and Radu added a lot of information > afterward. I am not a mind reader. > > > So maybe the whole discussion, especially the "-1," was unnecessary? > Maybe > we should just - I do not know - talk about it? > > -1 vote is a tool. It's a tool to say - stop. I am really against it which > is what I felt needed. > I don't understand why you are focusing on this. It's OK to disagree with > my vote choice but trying to say that I should have avoided casting a vote > on a vote thread is a very odd position. > > > We could probably discuss it in the PR, but since those voting threads > are > records of the foundation - I think we should explain it here in detail. > This way, board members reviewing our voting process will know we treat it > seriously. They actually review our VOTE discussions in case you weren't > aware :D) and sometimes ask questions about why something happened in a > specific VOTE thread. VOTE is a legal act of foundation so we should treat > it very seriously - and we should be very precise on why and how we vote > here. > > I don't get this. I provided reasoning which you can agree or not but you > can not say there was no reasoning. > > > I am limited on time so I will respond to the other > questions/comments later. > > > On Thu, Apr 23, 2026 at 3:45 PM Jarek Potiuk <[email protected]> wrote: > >> Elad, I would really love to solve this in a way that avoids those kinds >> of >> discussions and unnecessary "-1" in the future. So let's try to "solve" it >> for the future. >> And I am actually super-calm - no stress at all :) . >> >> I am actually glad you raised it, because it shows some deficiencies in >> our >> process. I was just hugely surprised to see "-1" rather than discussion. >> It's pretty strange to discuss this in the VOTE thread, but ... here we >> are >> and the reason is your "-1" on those providers, Elad - nothing else. >> >> We could probably discuss it in the PR, but since those voting threads are >> records of the foundation - I think we should explain it here in detail. >> This way, board members reviewing our voting process will know we treat it >> seriously. They actually review our VOTE discussions in case you weren't >> aware :D) and sometimes ask questions about why something happened in a >> specific VOTE thread. VOTE is a legal act of foundation so we should treat >> it very seriously - and we should be very precise on why and how we vote >> here. >> >> Radu: >> >> > I mentioned this in the PR, that maybe it's OK to modify the CHANGELOG >> to >> say that paramiko was bumped because cryptography was bumped - and that >> happened because of security reasons? >> >> Na ..... that was not really it. I looked it up for details: >> >> We had **many** issues trying to add Vespa. and it's been - to be honest - >> one of the most difficult "cross-provider" interaction we had for a while. >> We had similar cases where we needed to bump botocore in order to >> avoid whole CI failing, we had cases where we had to bump 5 or 7 provider >> min-dependnecies, to avoid resolution going for hours - but Vespa was >> special one indeed. >> >> Here are the three errors we had: >> >> * We detected unrelated cargo failures, which were really caused by >> "cargo + uv + installing several rust providers in the same >> installation"—this caused changes in our Docker container (we added rustc >> in the CI image) >> * Cryptography bump was indeed needed >> * Then we had this one: This is from the track of PRs, and the root >> cause was this failure: >> >> https://github.com/apache/airflow/actions/runs/23802024315/job/69372913309?pr=63988#step:8:2556 >> >> >> >> ```__________________________________________________________________________ >> TestSSHHook.test_tunnel_with_private_key_passphrase >> >> ___________________________________________________________________________ >> providers/ssh/tests/unit/ssh/hooks/test_ssh.py:436: in >> test_tunnel_with_private_key_passphrase >> hook = SSHHook( >> providers/ssh/src/airflow/providers/ssh/hooks/ssh.py:183: in __init__ >> self.pkey = self._pkey_from_private_key(private_key, >> passphrase=private_key_passphrase) >> providers/ssh/src/airflow/providers/ssh/hooks/ssh.py:435: in >> _pkey_from_private_key >> raise AirflowException( >> E airflow.sdk.exceptions.AirflowException: Private key provided cannot >> be >> read by paramiko.Ensure key provided is valid for one of the followingkey >> formats: RSA, DSS, ECDSA, or Ed25519 >> ___________________________________________________________________ >> TestSSHHook.test_ssh_connection_with_private_key_passphrase_extra >> ____________________________________________________________________ >> providers/ssh/tests/unit/ssh/hooks/test_ssh.py:548: in >> test_ssh_connection_with_private_key_passphrase_extra >> hook = SSHHook( >> providers/ssh/src/airflow/providers/ssh/hooks/ssh.py:183: in __init__ >> self.pkey = self._pkey_from_private_key(private_key, >> passphrase=private_key_passphrase) >> providers/ssh/src/airflow/providers/ssh/hooks/ssh.py:435: in >> _pkey_from_private_key >> raise AirflowException( >> E airflow.sdk.exceptions.AirflowException: Private key provided cannot >> be >> read by paramiko.Ensure key provided is valid for one of the followingkey >> formats: RSA, DSS, ECDSA, or Ed25519``` >> ``` >> >> The root cause was that after adding vespa, when `uv` resolved >> lower-possible direct dependencies for ssh - because of the build >> heuristics—it also decided to lower-down the paramiko library. And this is >> difficult to assess - because we do not know `uv` heuristics and it's >> NP-complete problem in general, so `uv` does a number of shortcuts to >> figure out the right set of dependencies in as short time as possible, >> >> And it turned out that when you downgrade paramiko from 3.5.1 to 3.4.0 - >> which was the lowest possible version back then: >> >> ``` >> - paramiko==3.5.1 >> + paramiko==3.4.0 >> ``` >> >> It triggers unrelated issues with supported key formats. It wasn't caused >> by "vespa" per-se - it was caused by the interaction between the SSH >> provider, the SFTP provider, Vespa, and uv trying to find a good >> dependency >> resolution—apparently, just the presence of vespa and its dependencies in >> pyproject.toml, causes `uv` to choose this downgrade. >> >> And since paramiko 3.5.When version 1 was released back then, we figured >> the best way to fix our CI and potentially protect our users (who - by >> various combinations of installation steps could also trigger the same >> scenario) - we decided to bump paramiko to 3.5.1. >> >> This was the "thought train". >> >> I don't see any better solution, or one that could take less time to >> figure >> out. >> >> *Question Elad*: I understand you don't have a problem with the >> "solution," >> I described. but with its description of it and process it arrived - can >> you please confirm that ? Because I think discussion will be different if >> you have the problem with solution itsefl (but I do not know curreently >> how >> else it could be solved).. >> >> Now - the question of the "changelog": >> >> In this case, adding Vespa to our pyproject is the fact our pyproject.toml >> triggered it. This is why we bumped it. This is the root cause. There is >> no >> other. There is no direct interaction between "vespa" and "ssh" - the root >> cause is that we have a workspace where we want to install all providers >> togeether, and due to heuristics of how `uv` works, the issue has been >> triggered. However - it was there before. If someone installed ssh >> provider >> with paramiko==3.5.1 and then downgraded to "3.4.0" - The issue was >> triggered. I reproduced it. So, a better description would be: >> >> "Bumping to minimum version of 3.5.1 because when the user installs the >> provider with Paramiko 3.5.1 and then downgrades to 3.4.0, it triggers >> "uknown key" bug" >> >> *Elad* - would you be ok if we change it like that in a follow up release >> or do you have a better wording in mind? Maybe you can create a PR for >> that? >> >> *Very near future: * >> >> Also, following what Shahar wrote, I'm about to add "Provider doc >> preparation skill." That should completely replace (though I will leave it >> there for now) our "breeze release management prepare-providers-release" >> command. This is after learning from the "breeze pr auto-triage" >> conversion—I literally discussed it with Shahar yesterday on Slack >> And one of the things I planned there was to instruct the agent to search >> for the root cause of the issue and present it in "user facing >> way"—letting >> the LLM figure out the best way to explain the change to the user. And - >> to >> be honest - I even planned to re-run on ALL our changelogs to improve >> them. >> >> So maybe the whole discussion, especially the "-1," was unnecessary? Maybe >> we should just - I do not know - talk about it? >> >> J. >> >> >> >> >> >> On Thu, Apr 23, 2026 at 2:37 PM Shahar Epstein <[email protected]> wrote: >> >> > Just to put things into context from my perspective as a release >> manager: >> > 1. I was aware of the changes to SFTP and SSH resulting from Vespa's PR >> > while working on the release. I thought that it was an intended change >> - I >> > didn't go deeper into whether there was a discussion, as I assumed that >> it >> > had been agreed on during approval. >> > 2. I asked the LLM to modify the titles in the CHANGELOG if they are >> > unclear, and indeed it detected this issue and modified the entries on >> its >> > own. >> > 3. If we had added the upper bound "<4.0.0" as part of this PR, it would >> > have indeed been an issue - because users who have already upgraded >> > paramiko, might not be able to roll back. However, the upper bound had >> > already been set to "<4.0.0" *before* Vespa's PR, which just increased >> the >> > lower bound - and I agree here with Jarek that we do that all the time >> > < >> > >> https://github.com/apache/airflow/commits/main/?author=dependabot%5Bbot%5D >> > > >> > (even >> > without discussing it). >> > Nonetheless: >> > Specifically here I would have expected this matter to be discussed in >> the >> > PR as these changes seem unrelated, or even making them in separate PRs >> - >> > it was a bit confusing for me as a release manager to figure out why >> these >> > changes were needed. >> > >> > I'll pay more attention to this matter in future releases and let you >> know >> > beforehand if I encounter similar cases. >> > No changes to my vote. >> > >> > >> > Shahar >> > >> > On Thu, Apr 23, 2026, 14:15 Jarek Potiuk <[email protected]> wrote: >> > >> > > Hey Elad, >> > > >> > > But it's - IMHO - cler explained in the changelog: >> > > >> > > >> > > >> > >> https://airflow.staged.apache.org/docs/apache-airflow-providers-ssh/5.0.1/changelog.html >> > > >> > > >> > >> https://airflow.staged.apache.org/docs/apache-airflow-providers-sftp/stable/changelog.html >> > > >> > > SSH: >> > > >> > > 5.0.1¶ >> > > Release Date: 2026-04-22 >> > > >> > > Misc¶ >> > > Bump paramiko lower bound to >=3.5.1 due to adding Vespa provider >> > (#63988) >> > > >> > > SFTP" >> > > >> > > Changelog¶ >> > > 5.7.4¶ >> > > Release Date: 2026-04-22 >> > > >> > > Misc¶ >> > > Bump paramiko lower bound to >=3.5.1 due to adding Vespa provider >> > (#63988) >> > > >> > > Why do you think it needs more explanation? We usually do not do more >> > > explanation when we bump minimum version - and here the explanation is >> > > quite clear ? >> > > >> > > J. >> > > >> > > >> > > >> > > On Thu, Apr 23, 2026 at 1:02 PM Elad Kalif <[email protected]> >> wrote: >> > > >> > > > -1 (binding) for ssh and sftp providers. >> > > > the changes in these providers are bumping paramiko min version >> which >> > > came >> > > > from a PR that adds vespa provider >> > > > https://github.com/apache/airflow/pull/63988#discussion_r3130238616 >> > > > There is no indication into why version was bumped and seems like we >> > > missed >> > > > it during review. >> > > > Even if the version needs to be bumped that is not normally how we >> do >> > it >> > > > and generally lets leave this to when >> > > > https://github.com/apache/airflow/pull/64898 is resolved. >> > > > >> > > > On Thu, Apr 23, 2026 at 1:01 PM Wei Lee <[email protected]> >> wrote: >> > > > >> > > > > +1 (binding), checked: >> > > > > >> > > > > - SVN >> > > > > - in Docker installation >> > > > > - Reproducible package builds >> > > > > - Licence >> > > > > - Signature >> > > > > - Checksums >> > > > > >> > > > > Best regards, >> > > > > Wei Lee >> > > > > >> > > > > Shahar Epstein <[email protected]> 於 2026年4月23日週四 上午7:55寫道： >> > > > > >> > > > > > Hey all, >> > > > > > >> > > > > > I have just cut the new wave Airflow Providers packages with >> > release >> > > > > > preparation date 2026-04-21. This email is calling a vote on the >> > > > release, >> > > > > > which will last for 72 hours - which means that it will end on >> > > > 2026-04-25 >> > > > > > 23:54 UTC and until 3 binding +1 votes have been received. >> > > > > > >> > > > > > >> > > > > > Consider this my (binding) +1. >> > > > > > Please note that we have an RC for a new provider, Vespa - I'd >> like >> > > to >> > > > > ask >> > > > > > the stewards to test it well before shipping its first version >> > > > (v0.1.0). >> > > > > > >> > > > > > >> > > > > > Airflow Providers are available at: >> > > > > > >> > https://dist.apache.org/repos/dist/dev/airflow/providers/2026-04-21 >> > > > > > >> > > > > > *apache-airflow-providers-2026-04-21-source.tar.gz* is the full >> > > source >> > > > > > tarball of airflow repo - snapshot taken at the moment of >> > provider's >> > > > > > release. >> > > > > > >> > > > > > *apache-airflow-providers-<PROVIDER>-*.tar.gz* are the >> convenience >> > > > python >> > > > > > "sdist" distributions that we publish in PyPI >> > > > > > >> > > > > > *apache_airflow_providers_<PROVIDER>-*.whl are the convenience >> > Python >> > > > > > "wheel" distributions that we publish in PyPI. >> > > > > > >> > > > > > The test procedure for PMC members is described in >> > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> https://github.com/apache/airflow/blob/main/dev/README_RELEASE_PROVIDERS.md#verify-the-release-candidate-by-pmc-members >> > > > > > >> > > > > > The test procedure for and Contributors who would like to test >> this >> > > RC >> > > > is >> > > > > > described in: >> > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> https://github.com/apache/airflow/blob/main/dev/README_RELEASE_PROVIDERS.md#verify-the-release-candidate-by-contributors >> > > > > > >> > > > > > >> > > > > > Public keys are available at: >> > > > > > https://dist.apache.org/repos/dist/release/airflow/KEYS >> > > > > > >> > > > > > Please vote accordingly: >> > > > > > >> > > > > > [ ] +1 approve >> > > > > > [ ] +0 no opinion >> > > > > > [ ] -1 disapprove with the reason >> > > > > > >> > > > > > Only votes from PMC members are binding, but members of the >> > community >> > > > are >> > > > > > encouraged to test the release and vote with "(non-binding)". >> > > > > > >> > > > > > Please note that the version number excludes the 'rcX' string. >> > > > > > This will allow us to rename the artifact without modifying >> > > > > > the artifact checksums when we actually release it. >> > > > > > >> > > > > > The status of testing the providers by the community is kept >> here: >> > > > > > https://github.com/apache/airflow/issues/65702 >> > > > > > >> > > > > > The issue is also the easiest way to see important PRs included >> in >> > > the >> > > > RC >> > > > > > candidates. >> > > > > > Detailed changelog for the providers will be published in the >> > > > > documentation >> > > > > > after the >> > > > > > RC candidates are released. >> > > > > > >> > > > > > You can find the RC packages in PyPI following these links: >> > > > > > >> > > > > > >> > https://pypi.org/project/apache-airflow-providers-amazon/9.26.0rc1/ >> > > > > > >> > > > > >> > > > >> > > >> > >> https://pypi.org/project/apache-airflow-providers-apache-kafka/1.13.3rc1/ >> > > > > > >> > https://pypi.org/project/apache-airflow-providers-celery/3.19.0rc1/ >> > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> https://pypi.org/project/apache-airflow-providers-cncf-kubernetes/10.16.1rc1/ >> > > > > > >> > > https://pypi.org/project/apache-airflow-providers-common-ai/0.1.1rc1/ >> > > > > > >> > > > >> > https://pypi.org/project/apache-airflow-providers-common-sql/1.35.0rc1/ >> > > > > > >> > > > >> > https://pypi.org/project/apache-airflow-providers-databricks/7.13.0rc1/ >> > > > > > >> https://pypi.org/project/apache-airflow-providers-edge3/3.5.0rc1/ >> > > > > > >> > > > > >> > > > >> > > >> > >> https://pypi.org/project/apache-airflow-providers-elasticsearch/6.5.3rc1/ >> > > > > > https://pypi.org/project/apache-airflow-providers-fab/3.6.2rc1/ >> > > > > > >> > https://pypi.org/project/apache-airflow-providers-google/21.2.0rc1/ >> > > > > > >> > > https://pypi.org/project/apache-airflow-providers-hashicorp/4.6.0rc1/ >> > > > > > >> https://pypi.org/project/apache-airflow-providers-jdbc/5.4.4rc1/ >> > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> https://pypi.org/project/apache-airflow-providers-microsoft-azure/13.1.2rc1/ >> > > > > > >> > > > >> > >> https://pypi.org/project/apache-airflow-providers-openlineage/2.15.0rc1/ >> > > > > > >> > > >> https://pypi.org/project/apache-airflow-providers-opensearch/1.9.1rc1/ >> > > > > > >> > > >> https://pypi.org/project/apache-airflow-providers-papermill/3.13.0rc1/ >> > > > > > >> > https://pypi.org/project/apache-airflow-providers-sendgrid/4.2.3rc1/ >> > > > > > >> https://pypi.org/project/apache-airflow-providers-sftp/5.7.4rc1/ >> > > > > > >> https://pypi.org/project/apache-airflow-providers-smtp/3.0.0rc1/ >> > > > > > >> > > >> https://pypi.org/project/apache-airflow-providers-snowflake/6.12.2rc1/ >> > > > > > https://pypi.org/project/apache-airflow-providers-ssh/5.0.1rc1/ >> > > > > > >> https://pypi.org/project/apache-airflow-providers-vespa/0.1.0rc1/ >> > > > > > >> > > > > > Cheers, >> > > > > > Shahar Epstein >> > > > > > >> > > > > >> > > > >> > > >> > >> >
