Jarek, That is a good point regarding the "owned and controlled hardware" requirement for a release signed by an RM, which is a major difference from our current semi-automated process.
If the requirement is that INFRA must be responsible for creating the key and managing the GitHub secrets—and that this is the practice we must strictly follow for all incubating project releases—then we can cancel this release and remove the secrets. Could you clarify if this requirement (INFRA key management) applies equally to all resources used in the release process, such as the Docker account, or if it is specific to the signing key? I did create a JIRA ticket for INFRA a while ago requesting a release account for the podling. Regards, JB Le mer. 10 déc. 2025 à 12:11, Jarek Potiuk <[email protected]> a écrit : > Just another clarification from a short discussion on slack. JBO mentioned > that this is not an automated release, but semi-automated, triggered by RM > but this is not entirely how I see it. The main difference between "signed > by RM" and "signed by automation" is where the release is signed (not even > "technically" when the release is built). > The key here is that when RM builds the release (not automation) they have > to use. From the release policy > https://www.apache.org/legal/release-policy.html#owned-controlled-hardware > : > > > That means hardware the committer has physical possession and control > of and exclusively full administrative/superuser access to. That's because > only such hardware is qualified to hold a PGP private key, and the release > should be verified on the machine the private key lives on or on a machine > as trusted as that. > > This Is NOT what happens here. > > J. > > > > On Wed, Dec 10, 2025 at 12:02 PM Jean-Baptiste Onofré <[email protected]> > wrote: > >> Also, another clarification: I created the key (as mentor/PPMC member) and >> added it to the GitHub secrets. Only PPMC members can do it. >> >> Regards, >> JB >> >> Le mer. 10 déc. 2025 à 11:49, Jean-Baptiste Onofré <[email protected]> a >> écrit : >> >> > Jarek, >> > >> > Just to clarify, my understanding is that Infra is typically responsible >> > for creating the signing key only for TLP releases and for projects >> > participating in the Automated TLP Releases (ATR) program. >> > I might be wrong but that’s my understanding. >> > >> > Regards, >> > JB >> > >> > Le mer. 10 déc. 2025 à 11:11, Jean-Baptiste Onofré <[email protected]> a >> > écrit : >> > >> >> Hi, >> >> >> >> I would like to clarify why I do not see the concerns about the release >> >> process being intrinsically different from a regular release: >> >> >> >> 1. Release Execution: This was a semi-automatic release, not fully >> >> automatic. The process was initiated by a release manager (Pierre) who >> >> executed the script; it was not triggered by a simple button click on >> >> GitHub Actions. >> >> 2. Key Management: The script uses a key that is published in the KEYS >> >> file, available at: >> >> https://dist.apache.org/repos/dist/release/incubator/polaris/KEYS. The >> >> corresponding private key and password are securely stored as GitHub >> >> secrets, a practice also adopted by other Apache projects ( >> >> https://github.com/apache/iceberg/tree/main/dev for instance). I don't >> >> think the infra has to review all Apache release scripts (not really >> >> different from a mvn release:prepare/perform for instance). >> >> 3. Verification Process: The verification steps remain identical to >> >> previous releases. Voters can still verify the signature, checksum, and >> >> other artifacts. The documentation on the website is still valid. >> >> >> >> I do not see the key not belonging to the release manager as an issue, >> >> given that the key is publicly registered and securely managed. >> >> >> >> As a side note, Polaris is also volunteering for the Automated TLP >> >> Releases (ATR) program. >> >> >> >> Regards, >> >> JB >> >> >> >> On Wed, Dec 10, 2025 at 10:26 AM Jarek Potiuk <[email protected]> >> wrote: >> >> >> >>> Hello here, >> >>> >> >>> Just lurking around here and JBO asked me to take a look. So I did :). >> >>> JBO >> >>> - I hope you will not regret it, but for me it is: -1 currently. >> Mostly >> >>> because I believe that the automation setup - even if very mature and >> >>> well >> >>> designed - is not done in an ASF / INFRA compliant way, and because I >> >>> think the verification process is not well communicated (and we do not >> >>> know >> >>> if it has been followed). >> >>> I might be mistaken of course - and it could be that I could not find >> the >> >>> necessary information - so I might change the vote once I got more >> >>> feedback. Sorry if I missed something obvious, I tried to be as >> diligent >> >>> as >> >>> possible to find all the information - this is my first message here, >> so >> >>> I >> >>> might have missed **something**, >> >>> >> >>> Automation of signing the release is only allowed when infra reviews >> your >> >>> automation and when the build reproducibility is part of the >> verification >> >>> done by the (P)PMC. This is clearly described in the Automated Release >> >>> Signing step of release signing policy by infra [1]. >> >>> I think while the signing process is technically sound, those >> conditions >> >>> are not met. >> >>> >> >>> It's the first time for me to look at that as I am a new Incubator >> >>> mentor, >> >>> but I did my homework and tried to find all the answers - including >> >>> reading >> >>> the automation process thread [5] and the process proposal [6] that I >> >>> found >> >>> after JBO's comment. >> >>> >> >>> As a side comment - this is a fantastic job and way to set it up - >> with >> >>> devlist discussion, proposal, prototypes etc. You definitely rock >> when it >> >>> comes to maturity of your automation, process and description of what >> >>> you >> >>> have done. This was a fantastic read. >> >>> >> >>> However I think some crucial steps are missing in the "approval" and >> >>> "communication" parts from [1]: >> >>> >> >>> 1) There must be a JIRA ticket in Infra where they reviewed the >> >>> signing process and agreed to it and provided you with the key (and >> you >> >>> never had access to the key). I actually looked at JIRA and tried to >> find >> >>> the Polaris ticket about it and could not - I saw quite a few "setup" >> >>> tickets - including Dockerhub secrets creation - but I could not see >> the >> >>> key signing ticket (maybe I missed it and you can refer to it here). >> From >> >>> the policy: >> >>> > The project must request a signing key through an Infra Jira ticket, >> >>> and >> >>> Infra will provide a signing key for the project: >> >>> >> >>> 2) There must be a step (and it has to be followed) in the release >> >>> process >> >>> where (P)PMC member giving +1 builds the artifacts locally and gets a >> >>> binary identical result as the one that is automatically signed - i.e. >> >>> reproducibility check. It does not have to be binary reproducibility >> - in >> >>> case package contains only sources and no compiled code, it's enough >> to >> >>> compare if sources are identical, some binary properties like >> timestamps, >> >>> permissions are not necessary (but binary reproducibility check is >> >>> generally easier if your process is binary-build-reproducible). This >> is >> >>> explained in the incubator release checklist [2] as "Do the contents >> of >> >>> the >> >>> release match with what's tagged in version control?". I tried to find >> >>> such >> >>> process and find out what level of reproducibility Polaris has and how >> >>> you >> >>> verify it - all I found was the (very nice) description of the release >> >>> process >> >>> >> >>> >> https://github.com/apache/polaris/tree/apache-polaris-1.3.0-incubating-rc2/releasey >> >>> and the release scipts mentioned in the doc, but I could not find a >> step >> >>> describing the process of verification - so I could not find how you >> >>> verify >> >>> reproducibility. The reproducibility / verification scripts are >> mentioned >> >>> in [7] as an idea of "Release verification script" and >> reproducibility is >> >>> even mentioned there. But again - this is my first time here, so I >> could >> >>> have missed it. >> >>> >> >>> I looked at the release email - and I cannot see the verification >> process >> >>> referred to. So as someone who would give +1 - I am not sure if I know >> >>> the >> >>> process to follow. And I have no idea what any of the people giving +1 >> >>> did. >> >>> >> >>> Also - this is actually required - each of the people casting their >> vote >> >>> should specify what they did - which of the checks they did >> (signatures, >> >>> checksums, licences, reproducibility). I see only +1s without >> explaining >> >>> what has been done which is considered generally a bad practice (I >> cannot >> >>> find a reference here - but maybe some other mentors might help heere >> >>> >> >>> Example of similar processes we do in Airflow [3] -> this is where the >> >>> release verification "steps" are described. And when we vote, we >> mention >> >>> the steps we have done - example here [4] (usually we do all of them >> as >> >>> PMC >> >>> members). There you can see: >> >>> >> >>> * link to the release verification process >> >>> * each PMC member voting explicitly states what they have done >> >>> * reproducibility is specifically checked (even if we do not have yet >> >>> automation of releases - we do it to verify Release Manager's work). >> >>> >> >>> And just to explain why this is needed, so that you do not think it's >> >>> some >> >>> "cargo cult". It's all but that. >> >>> >> >>> This is needed from the legal standpoint. Voting on the release is a >> >>> legal >> >>> act of Foundation - which means that the three (P)PMC members voting >> on >> >>> it, >> >>> make the Foundation potentially liable in case someone has a legal >> >>> complaint - for example harm done by using the software. That's why >> it's >> >>> super important that we follow the process and confirm it. >> >>> Reproducibility >> >>> check in this case is a proof that this release has been prepared from >> >>> the >> >>> source code that is stored in our git repository - otherwise it is >> >>> possible >> >>> that release manager (if the build is prepared manually) or malicious >> >>> actor >> >>> who broke into the machine/system that build the artifacts (in case of >> >>> automated process) has not injected anything. And it's not >> >>> theoretical threat - this is exactly what happened in the infamous xz >> >>> case >> >>> [5] where maintainer got release manager status turned out to be a >> >>> malicious actor and injected backdoor that almost made it to major >> >>> distributions because they used the "official" packaged artifact >> >>> published >> >>> by the project and it contained malicious modification of the build >> >>> process >> >>> that was not in the source code. >> >>> >> >>> Since this is PPMC and not yet TLP, possibly not all those >> requirements >> >>> have to be met, but that will not qualify this release as "ASF cm >> >>> >> >>> [1] Infra Signing Policy >> >>> >> https://infra.apache.org/release-signing.html#automated-release-signing >> >>> [2] Icubator Release Checklist >> >>> >> >>> >> https://cwiki.apache.org/confluence/display/INCUBATOR/Incubator+Release+Checklist >> >>> [3] Airflow's Release verification process: >> >>> >> >>> >> https://github.com/apache/airflow/blob/main/dev/README_RELEASE_AIRFLOW.md#verify-the-release-candidate-by-pmc-members >> >>> >> >>> [4] Airflow Voting mail example with +1s and stating the actions: >> >>> https://lists.apache.org/thread/crsg4bxhovyyz3gjorv5dcm85d7ytm5x >> >>> [5] XZ Utils backdoor: >> https://en.wikipedia.org/wiki/XZ_Utils_backdoor >> >>> [6] Polaris devlist discussion on automated releases: >> >>> https://lists.apache.org/thread/7tlmzwpdc9m4p4dx79jrydxs3x60m875 >> >>> [7] Polaris release automation process: >> >>> >> >>> >> https://docs.google.com/document/d/1MpovNXHFxqC1rxw_WhekQujDSj_zOAl4AN2QyFkX23k/edit?tab=t.0#heading=h.itmh6zawe02p >> >>> >> >>> J. >> >>> >> >>> >> >>> >> >>> On Wed, Dec 10, 2025, 05:04 Jean-Baptiste Onofré <[email protected]> >> >>> wrote: >> >>> >> >>> > Hi >> >>> > >> >>> > Yes the key is in GitHub Secret. >> >>> > As other ASF projects, the podling uses several GitHub Secrets to >> >>> automate >> >>> > release. >> >>> > It has been described in the podling dev mailing list and also the >> >>> > semi-automated script was on a PR and reviewed. >> >>> > >> >>> > Regards >> >>> > JB >> >>> > >> >>> > Le mar. 9 déc. 2025 à 21:53, Ryan Blue <[email protected]> a écrit : >> >>> > >> >>> > > +0 >> >>> > > >> >>> > > More information is needed for me to update to +1. >> >>> > > >> >>> > > I see this signature: >> >>> > > >> >>> > > ``` >> >>> > > [blue@dev tmp]$ gpg --verify >> >>> apache-polaris-1.3.0-incubating.tar.gz.asc >> >>> > > gpg: assuming signed data in >> 'apache-polaris-1.3.0-incubating.tar.gz' >> >>> > > gpg: Signature made Tue 25 Nov 2025 01:08:54 AM PST >> >>> > > gpg: using RSA key >> >>> > 6A6532EAD1AE4441ACE054870E971D601C4AD16F >> >>> > > gpg: Good signature from "Apache Polaris <[email protected] >> >" >> >>> > > [unknown] >> >>> > > gpg: WARNING: This key is not certified with a trusted signature! >> >>> > > gpg: There is no indication that the signature belongs to >> >>> the >> >>> > > owner. >> >>> > > Primary key fingerprint: 6A65 32EA D1AE 4441 ACE0 5487 0E97 1D60 >> >>> 1C4A >> >>> > D16F >> >>> > > ``` >> >>> > > >> >>> > > Since that is not a release manager, this must have been produced >> by >> >>> the >> >>> > > release automation scripts that were discussed on the dev list. I >> >>> took a >> >>> > > quick look, but I don't see how the private key is protected. The >> >>> release >> >>> > > guide covers manual releases. I'm assuming that this is stored as >> a >> >>> > github >> >>> > > secret and is only accessible in a workflow that authorized users >> >>> have >> >>> > > access to? (I see that it must be run by a committer from the >> thread >> >>> on >> >>> > the >> >>> > > dev list.) >> >>> > > >> >>> > > I'll change to +1 if someone can let me know how the keys are >> >>> managed. >> >>> > > >> >>> > > On Tue, Dec 9, 2025 at 9:27 AM Jean-Baptiste Onofré < >> [email protected] >> >>> > >> >>> > > wrote: >> >>> > > >> >>> > > > Dear IPMC members, >> >>> > > > >> >>> > > > This is a gentle reminder that the Apache Polaris >> 1.3.0-incubating >> >>> > (RC2) >> >>> > > > vote still requires a third binding vote to pass. >> >>> > > > >> >>> > > > Thank you, >> >>> > > > >> >>> > > > Regards >> >>> > > > JB >> >>> > > > >> >>> > > > On Mon, Dec 1, 2025 at 9:35 AM Pierre Laporte < >> >>> [email protected]> >> >>> > > > wrote: >> >>> > > > >> >>> > > > > Hello everyone, >> >>> > > > > >> >>> > > > > The Apache Polaris community has voted and approved the >> release >> >>> of >> >>> > > Apache >> >>> > > > > Polaris 1.3.0-incubating (RC2). We now kindly request the >> IPMC >> >>> > members >> >>> > > > > review and vote for this release. >> >>> > > > > >> >>> > > > > Polaris community vote thread: >> >>> > > > > * >> >>> https://lists.apache.org/thread/fw8xhobpnoy3mvvw8hxd3r7kw5of4kos >> >>> > > > > >> >>> > > > > Vote result thread: >> >>> > > > > * >> >>> https://lists.apache.org/thread/xjwb04c33oo387g5gjdx674bw7t9bhz2 >> >>> > > > > >> >>> > > > > This corresponds to the tag: >> apache-polaris-1.3.0-incubating-rc2 >> >>> > > > > * >> >>> > > > > >> >>> > > > > >> >>> > > > >> >>> > > >> >>> > >> >>> >> https://github.com/apache/polaris/commits/apache-polaris-1.3.0-incubating-rc2 >> >>> > > > > * >> >>> > > > > >> >>> > > > > >> >>> > > > >> >>> > > >> >>> > >> >>> >> https://github.com/apache/polaris/tree/308134d6440f8167afd563a885187e238c21048a >> >>> > > > > >> >>> > > > > The release tarball, signature, and checksums are here: >> >>> > > > > * >> >>> > > > > >> >>> > > > >> >>> > > >> >>> > >> >>> >> https://dist.apache.org/repos/dist/dev/incubator/polaris/1.3.0-incubating >> >>> > > > > >> >>> > > > > Helm charts are available on: >> >>> > > > > * >> >>> > > > > >> >>> > > > > >> >>> > > > >> >>> > > >> >>> > >> >>> >> https://dist.apache.org/repos/dist/dev/incubator/polaris/helm-chart/1.3.0-incubating/ >> >>> > > > > NB: you have to build the Docker images locally in order to >> test >> >>> Helm >> >>> > > > > charts. >> >>> > > > > >> >>> > > > > You can find the KEYS file here: >> >>> > > > > * https://downloads.apache.org/incubator/polaris/KEYS >> >>> > > > > >> >>> > > > > Convenience binary artifacts are staged on Nexus. The Maven >> >>> > repository >> >>> > > > URL >> >>> > > > > is: >> >>> > > > > * >> >>> > > > >> >>> > >> >>> >> https://repository.apache.org/content/repositories/orgapachepolaris-1046 >> >>> > > > > >> >>> > > > > Please download, verify and test. >> >>> > > > > >> >>> > > > > Please vote in the next 72 hours. >> >>> > > > > >> >>> > > > > [ ] +1 approve >> >>> > > > > [ ] +0 no opinion >> >>> > > > > [ ] -1 disapprove with the reason >> >>> > > > > >> >>> > > > > To learn more about apache Polaris, please see >> >>> > > > https://polaris.apache.org/ >> >>> > > > > >> >>> > > > > Thanks, >> >>> > > > > >> >>> > > > > -- >> >>> > > > > >> >>> > > > > Pierre >> >>> > > > > >> >>> > > > >> >>> > > >> >>> > >> >>> >> >> >> >
