Yeah, this is an excellent proposal. The part that the release manager doesn't have to be added to the Apache Airflow organization in PYPI before they can publish a package would be a significant improvement.
> small difference is that we will also have to upload RC/BETA packages to SVN - we are currently not doing it) I think we currently publish RC/BETA packages for core airflow releases to SVN. No? - Ephraim On Tue, 25 Jun 2024 at 08:42, Jarek Potiuk <ja...@potiuk.com> wrote: > Hello everyone, > > TL;DR; I wanted to propose switching to PyPI Trusted Publishing workflow > for Airflow releases with Github Actions being the trusted publisher. > > A bit of background: > > In the security team and also in the security committee of ASF w have been > discussing ways how we can plugin into Trusted Publishing [1] feature of > PyPI to publish our artifacts more securely in PyPI and after some initial > back/forth discussions (I also discussed it with PyPI Security Engineer, > Mike Fiedler at PyCon in US) we settled on a proposal to simply use Github > Trusted Publishing (another option was to make ASF a trusted publisher - > and it still might happen, but not in a short term, and we can always > change it in the future. > > What is the model of authentication used today? > > Currently when release managers upload releases to PyPI, they have to > authenticate with their API key. Those API keys are generated by the > individuals using their personal credentials. All release managers are > added to Apache Airflow organisation and this is what determines if they > are able to upload releases to PyPI. The keys are long-living, and even if > you use 2FA (it's mandated in our organisation). So theoretically, someone > stealing RM API keys could upload a malicious release to PyPI. > > What does it mean to use trusted publishing for PyPI for uploads? > > Trusted Publisher, changes the authentication model - and delegates the > authentication of such upload (and only that) to a "Trusted Publisher" > entity. In the proposal, such Trusted Publisher is GitHub and we will be > able to use GitHub Actions to upload the packages - for security it can be > done from a separate "Github Actions Environment" [2] and only selected > maintainers (i.e. release managers) of Airflow might be added to have > access to that environment and up to 6 people can be designated to be > reviewers and workflows run there will need to be approved by one of those > 6 people. Only workflows from the Airflow repository and designated > environment will be able to upload packages. This has the added benefit > that you might disable "self-review" and any upload to PyPI will require at > least 2 people - including one reviewer. Also we can audit and see the > history of all such uploads in Github Action logs. We will disable the > possibility of "manual" uploads once we get it working and tested. > > Are we going to change how packages are built? > > No. We will continue to build them using the current process. Release > managers will built the packages locally on their machines and upload them > to the ASF SVN in the same way as today (small difference is that we will > also have to upload RC/BETA packages to SVN - we are currently not doing > it). The publishing workflow will be just pulling the artifacts from ASF > SVN and publishing them via GitHub Action. They will not be built in Github > Actions. Thanks to binary reproducibility we have, we will be able to track > and verify provenance of those - all the artifacts in SVN and PyPI will be > binary identical. > > I started a discussion some time ago about it in the ASF > Security-discuss/builds discussion lists [3] and there are no problems with > that approach. We might even be able to create a reusable action for other > projects in the ASF (but only if it will make sense). We might also need a > few workflows / ways to manage exceptional cases (yanking, removing > releases, creating new projects). We will work it out while implementing > it. I don't expect surprises there - this is a solved problem. There are > already multiple thousands of projects using the Trusted Publishing > workflow in PyPI, it's available for more than 6 months already. > > Future work: > > Also ASF is working - separately on a new (codename) "Artifact Distribution > Platform" where we will be able to build and distribute the packages on ASF > infrastructure. This does not change the approach for Trusted Publishing - > we will still be able to use Github Actions to pull the artifacts from the > "official" ASF URL and publish it in PyPI. > > Even more future work: > > In the even more distant future ASF might become a trusted publisher on its > own, but it's a bit complicated and it also might not be needed. There is > the draft PEP-770 [4] being worked on by the packaging team which will > allow to attach digital signatures to PyPI artifacts, which will mean that > the signatures and checksums generated by release manager (and in the > future by the Artifact Distribution Platform) - and it will allow our users > not only to verify binary reproducibility but also provenance of the > packages (i.e. signatures, indicating that the packages were generated by > the official means of the Apache Software Foundation). So regardless of > which Trusted Publisher is used to upload the packages, the attestations of > provenance will provide independent verification on who created the > packages - effectively decoupling package creation from publishing. > > I am looking for feedback on that one, I've been following it and > discussing details for quite some time (including discussing it at PyCon > with THE Python security team), so I am happy to answer any questions > anyone might have. > > If there will be (as I suspect) - no objections after any explanations > needed, I will run a lazy consensus and get it implemented. > > J. > > > [1] Trusted Publishers: https://docs.pypi.org/trusted-publishers/ > [2] Github Actions Environments: > > https://docs.github.com/en/actions/deployment/targeting-different-environments/using-environments-for-deployment > [3] Discussion about Trusted Publishing in security-discuss@a.o: > https://lists.apache.org/thread/byszw8vhq1cwtc9xhq0v9q5mkc5mz09n > [4] PEP-740 Index support for Digital Attestations: > > https://discuss.python.org/t/pep-740-index-support-for-digital-attestations/44498 >
