Jarek, Just to clarify, my understanding is that Infra is typically responsible for creating the signing key only for TLP releases and for projects participating in the Automated TLP Releases (ATR) program. I might be wrong but that’s my understanding.
Regards, JB Le mer. 10 déc. 2025 à 11:11, Jean-Baptiste Onofré <[email protected]> a écrit : > Hi, > > I would like to clarify why I do not see the concerns about the release > process being intrinsically different from a regular release: > > 1. Release Execution: This was a semi-automatic release, not fully > automatic. The process was initiated by a release manager (Pierre) who > executed the script; it was not triggered by a simple button click on > GitHub Actions. > 2. Key Management: The script uses a key that is published in the KEYS > file, available at: > https://dist.apache.org/repos/dist/release/incubator/polaris/KEYS. The > corresponding private key and password are securely stored as GitHub > secrets, a practice also adopted by other Apache projects ( > https://github.com/apache/iceberg/tree/main/dev for instance). I don't > think the infra has to review all Apache release scripts (not really > different from a mvn release:prepare/perform for instance). > 3. Verification Process: The verification steps remain identical to > previous releases. Voters can still verify the signature, checksum, and > other artifacts. The documentation on the website is still valid. > > I do not see the key not belonging to the release manager as an issue, > given that the key is publicly registered and securely managed. > > As a side note, Polaris is also volunteering for the Automated TLP > Releases (ATR) program. > > Regards, > JB > > On Wed, Dec 10, 2025 at 10:26 AM Jarek Potiuk <[email protected]> wrote: > >> Hello here, >> >> Just lurking around here and JBO asked me to take a look. So I did :). JBO >> - I hope you will not regret it, but for me it is: -1 currently. Mostly >> because I believe that the automation setup - even if very mature and well >> designed - is not done in an ASF / INFRA compliant way, and because I >> think the verification process is not well communicated (and we do not >> know >> if it has been followed). >> I might be mistaken of course - and it could be that I could not find the >> necessary information - so I might change the vote once I got more >> feedback. Sorry if I missed something obvious, I tried to be as diligent >> as >> possible to find all the information - this is my first message here, so I >> might have missed **something**, >> >> Automation of signing the release is only allowed when infra reviews your >> automation and when the build reproducibility is part of the verification >> done by the (P)PMC. This is clearly described in the Automated Release >> Signing step of release signing policy by infra [1]. >> I think while the signing process is technically sound, those conditions >> are not met. >> >> It's the first time for me to look at that as I am a new Incubator mentor, >> but I did my homework and tried to find all the answers - including >> reading >> the automation process thread [5] and the process proposal [6] that I >> found >> after JBO's comment. >> >> As a side comment - this is a fantastic job and way to set it up - with >> devlist discussion, proposal, prototypes etc. You definitely rock when it >> comes to maturity of your automation, process and description of what you >> have done. This was a fantastic read. >> >> However I think some crucial steps are missing in the "approval" and >> "communication" parts from [1]: >> >> 1) There must be a JIRA ticket in Infra where they reviewed the >> signing process and agreed to it and provided you with the key (and you >> never had access to the key). I actually looked at JIRA and tried to find >> the Polaris ticket about it and could not - I saw quite a few "setup" >> tickets - including Dockerhub secrets creation - but I could not see the >> key signing ticket (maybe I missed it and you can refer to it here). From >> the policy: >> > The project must request a signing key through an Infra Jira ticket, and >> Infra will provide a signing key for the project: >> >> 2) There must be a step (and it has to be followed) in the release process >> where (P)PMC member giving +1 builds the artifacts locally and gets a >> binary identical result as the one that is automatically signed - i.e. >> reproducibility check. It does not have to be binary reproducibility - in >> case package contains only sources and no compiled code, it's enough to >> compare if sources are identical, some binary properties like timestamps, >> permissions are not necessary (but binary reproducibility check is >> generally easier if your process is binary-build-reproducible). This is >> explained in the incubator release checklist [2] as "Do the contents of >> the >> release match with what's tagged in version control?". I tried to find >> such >> process and find out what level of reproducibility Polaris has and how you >> verify it - all I found was the (very nice) description of the release >> process >> >> https://github.com/apache/polaris/tree/apache-polaris-1.3.0-incubating-rc2/releasey >> and the release scipts mentioned in the doc, but I could not find a step >> describing the process of verification - so I could not find how you >> verify >> reproducibility. The reproducibility / verification scripts are mentioned >> in [7] as an idea of "Release verification script" and reproducibility is >> even mentioned there. But again - this is my first time here, so I could >> have missed it. >> >> I looked at the release email - and I cannot see the verification process >> referred to. So as someone who would give +1 - I am not sure if I know the >> process to follow. And I have no idea what any of the people giving +1 >> did. >> >> Also - this is actually required - each of the people casting their vote >> should specify what they did - which of the checks they did (signatures, >> checksums, licences, reproducibility). I see only +1s without explaining >> what has been done which is considered generally a bad practice (I cannot >> find a reference here - but maybe some other mentors might help heere >> >> Example of similar processes we do in Airflow [3] -> this is where the >> release verification "steps" are described. And when we vote, we mention >> the steps we have done - example here [4] (usually we do all of them as >> PMC >> members). There you can see: >> >> * link to the release verification process >> * each PMC member voting explicitly states what they have done >> * reproducibility is specifically checked (even if we do not have yet >> automation of releases - we do it to verify Release Manager's work). >> >> And just to explain why this is needed, so that you do not think it's some >> "cargo cult". It's all but that. >> >> This is needed from the legal standpoint. Voting on the release is a legal >> act of Foundation - which means that the three (P)PMC members voting on >> it, >> make the Foundation potentially liable in case someone has a legal >> complaint - for example harm done by using the software. That's why it's >> super important that we follow the process and confirm it. Reproducibility >> check in this case is a proof that this release has been prepared from the >> source code that is stored in our git repository - otherwise it is >> possible >> that release manager (if the build is prepared manually) or malicious >> actor >> who broke into the machine/system that build the artifacts (in case of >> automated process) has not injected anything. And it's not >> theoretical threat - this is exactly what happened in the infamous xz case >> [5] where maintainer got release manager status turned out to be a >> malicious actor and injected backdoor that almost made it to major >> distributions because they used the "official" packaged artifact published >> by the project and it contained malicious modification of the build >> process >> that was not in the source code. >> >> Since this is PPMC and not yet TLP, possibly not all those requirements >> have to be met, but that will not qualify this release as "ASF cm >> >> [1] Infra Signing Policy >> https://infra.apache.org/release-signing.html#automated-release-signing >> [2] Icubator Release Checklist >> >> https://cwiki.apache.org/confluence/display/INCUBATOR/Incubator+Release+Checklist >> [3] Airflow's Release verification process: >> >> https://github.com/apache/airflow/blob/main/dev/README_RELEASE_AIRFLOW.md#verify-the-release-candidate-by-pmc-members >> >> [4] Airflow Voting mail example with +1s and stating the actions: >> https://lists.apache.org/thread/crsg4bxhovyyz3gjorv5dcm85d7ytm5x >> [5] XZ Utils backdoor: https://en.wikipedia.org/wiki/XZ_Utils_backdoor >> [6] Polaris devlist discussion on automated releases: >> https://lists.apache.org/thread/7tlmzwpdc9m4p4dx79jrydxs3x60m875 >> [7] Polaris release automation process: >> >> https://docs.google.com/document/d/1MpovNXHFxqC1rxw_WhekQujDSj_zOAl4AN2QyFkX23k/edit?tab=t.0#heading=h.itmh6zawe02p >> >> J. >> >> >> >> On Wed, Dec 10, 2025, 05:04 Jean-Baptiste Onofré <[email protected]> wrote: >> >> > Hi >> > >> > Yes the key is in GitHub Secret. >> > As other ASF projects, the podling uses several GitHub Secrets to >> automate >> > release. >> > It has been described in the podling dev mailing list and also the >> > semi-automated script was on a PR and reviewed. >> > >> > Regards >> > JB >> > >> > Le mar. 9 déc. 2025 à 21:53, Ryan Blue <[email protected]> a écrit : >> > >> > > +0 >> > > >> > > More information is needed for me to update to +1. >> > > >> > > I see this signature: >> > > >> > > ``` >> > > [blue@dev tmp]$ gpg --verify >> apache-polaris-1.3.0-incubating.tar.gz.asc >> > > gpg: assuming signed data in 'apache-polaris-1.3.0-incubating.tar.gz' >> > > gpg: Signature made Tue 25 Nov 2025 01:08:54 AM PST >> > > gpg: using RSA key >> > 6A6532EAD1AE4441ACE054870E971D601C4AD16F >> > > gpg: Good signature from "Apache Polaris <[email protected]>" >> > > [unknown] >> > > gpg: WARNING: This key is not certified with a trusted signature! >> > > gpg: There is no indication that the signature belongs to the >> > > owner. >> > > Primary key fingerprint: 6A65 32EA D1AE 4441 ACE0 5487 0E97 1D60 1C4A >> > D16F >> > > ``` >> > > >> > > Since that is not a release manager, this must have been produced by >> the >> > > release automation scripts that were discussed on the dev list. I >> took a >> > > quick look, but I don't see how the private key is protected. The >> release >> > > guide covers manual releases. I'm assuming that this is stored as a >> > github >> > > secret and is only accessible in a workflow that authorized users have >> > > access to? (I see that it must be run by a committer from the thread >> on >> > the >> > > dev list.) >> > > >> > > I'll change to +1 if someone can let me know how the keys are managed. >> > > >> > > On Tue, Dec 9, 2025 at 9:27 AM Jean-Baptiste Onofré <[email protected]> >> > > wrote: >> > > >> > > > Dear IPMC members, >> > > > >> > > > This is a gentle reminder that the Apache Polaris 1.3.0-incubating >> > (RC2) >> > > > vote still requires a third binding vote to pass. >> > > > >> > > > Thank you, >> > > > >> > > > Regards >> > > > JB >> > > > >> > > > On Mon, Dec 1, 2025 at 9:35 AM Pierre Laporte < >> [email protected]> >> > > > wrote: >> > > > >> > > > > Hello everyone, >> > > > > >> > > > > The Apache Polaris community has voted and approved the release of >> > > Apache >> > > > > Polaris 1.3.0-incubating (RC2). We now kindly request the IPMC >> > members >> > > > > review and vote for this release. >> > > > > >> > > > > Polaris community vote thread: >> > > > > * >> https://lists.apache.org/thread/fw8xhobpnoy3mvvw8hxd3r7kw5of4kos >> > > > > >> > > > > Vote result thread: >> > > > > * >> https://lists.apache.org/thread/xjwb04c33oo387g5gjdx674bw7t9bhz2 >> > > > > >> > > > > This corresponds to the tag: apache-polaris-1.3.0-incubating-rc2 >> > > > > * >> > > > > >> > > > > >> > > > >> > > >> > >> https://github.com/apache/polaris/commits/apache-polaris-1.3.0-incubating-rc2 >> > > > > * >> > > > > >> > > > > >> > > > >> > > >> > >> https://github.com/apache/polaris/tree/308134d6440f8167afd563a885187e238c21048a >> > > > > >> > > > > The release tarball, signature, and checksums are here: >> > > > > * >> > > > > >> > > > >> > > >> > >> https://dist.apache.org/repos/dist/dev/incubator/polaris/1.3.0-incubating >> > > > > >> > > > > Helm charts are available on: >> > > > > * >> > > > > >> > > > > >> > > > >> > > >> > >> https://dist.apache.org/repos/dist/dev/incubator/polaris/helm-chart/1.3.0-incubating/ >> > > > > NB: you have to build the Docker images locally in order to test >> Helm >> > > > > charts. >> > > > > >> > > > > You can find the KEYS file here: >> > > > > * https://downloads.apache.org/incubator/polaris/KEYS >> > > > > >> > > > > Convenience binary artifacts are staged on Nexus. The Maven >> > repository >> > > > URL >> > > > > is: >> > > > > * >> > > > >> > >> https://repository.apache.org/content/repositories/orgapachepolaris-1046 >> > > > > >> > > > > Please download, verify and test. >> > > > > >> > > > > Please vote in the next 72 hours. >> > > > > >> > > > > [ ] +1 approve >> > > > > [ ] +0 no opinion >> > > > > [ ] -1 disapprove with the reason >> > > > > >> > > > > To learn more about apache Polaris, please see >> > > > https://polaris.apache.org/ >> > > > > >> > > > > Thanks, >> > > > > >> > > > > -- >> > > > > >> > > > > Pierre >> > > > > >> > > > >> > > >> > >> >
