Hello folks Thanks for the thorough review. It raises a few very interesting points. I hope my answers can help clarify the misunderstandings.
On Wed, Dec 10, 2025 at 10:27 AM Jarek Potiuk <[email protected]> wrote: > I believe that the automation setup - even if very mature and well > designed - is not done in an ASF / INFRA compliant way, and because I > think the verification process is not well communicated (and we do not know > if it has been followed). > I might be mistaken of course - and it could be that I could not find the > necessary information - so I might change the vote once I got more > feedback. Sorry if I missed something obvious, I tried to be as diligent as > possible to find all the information - this is my first message here, so I > might have missed **something**, I am not sure whether you refer to the verification process of a release, in which case I can point you to the `How to verify a release` section on our release guide page [1]. Or are you saying that release automation workflows must be verified by ASF Infra before they can be used? To give you some more context, the current workflows are almost a 1:1 mapping of the release guide, so that manual mistakes and copy-paste issues can be avoided. We have ideas on how to push this even further but those have not been developed yet. If we need review from ASF infra, please let me know so that we can include this step in our development process. > Automation of signing the release is only allowed when infra reviews your > automation and when the build reproducibility is part of the verification > done by the (P)PMC. This is clearly described in the Automated Release > Signing step of release signing policy by infra [1]. > I think while the signing process is technically sound, those conditions > are not met. > Regarding reproducible builds, it is a very good point. Currently, Polaris does not meet the reproducible builds property. Robert Stupp has identified quite a few items that prevent our builds from being bit-by-bit identical, even when they are created against the same Git SHA and on the same hardware [2] [3] [4] [5] [6]. The roadblocks range from Maven pom.xml generation, git modification timestamp and even the need to upgrade to Helm 4 which was released less than a month ago (because Helm 3 does not support reproducible builds). All of this to say, it is our goal to get to fully reproducible builds. But indeed we are not there yet. I was under the impression that the release process should be the same, regardless of whether a release is done manually or in a semi-automated fashion. The intent for this release is to be one step further towards fully automated and reproducible releases. The release was indeed performed and signed on Github Runners, and not on a personal/company laptop. Regarding verification, Robert has created a script [7] that performs the following checks: * GPG signature and checksum verifications * All expected artifacts are present * Build artifacts are reproducible (minus known exceptions) * jar files * Main distribution zip/tarball * Helm chart * Build passes. * DISCLAIMER/LICENSE/NOTICE files are present in artifacts that require those As you can see, we are not considering this as a done topic. 2) There must be a step (and it has to be followed) in the release process > where (P)PMC member giving +1 builds the artifacts locally and gets a > binary identical result as the one that is automatically signed - i.e. > reproducibility check. It does not have to be binary reproducibility - in > case package contains only sources and no compiled code, it's enough to > compare if sources are identical, some binary properties like timestamps, > permissions are not necessary (but binary reproducibility check is > generally easier if your process is binary-build-reproducible). This is > explained in the incubator release checklist [2] as "Do the contents of the > release match with what's tagged in version control?". I tried to find such > process and find out what level of reproducibility Polaris has and how you > verify it - all I found was the (very nice) description of the release > process > > https://github.com/apache/polaris/tree/apache-polaris-1.3.0-incubating-rc2/releasey > and the release scipts mentioned in the doc, but I could not find a step > describing the process of verification - so I could not find how you verify > reproducibility. The reproducibility / verification scripts are mentioned > in [7] as an idea of "Release verification script" and reproducibility is > even mentioned there. But again - this is my first time here, so I could > have missed it. > Your assessment is correct. So far, we have implemented the semi-automated release process. We are working on reproducible builds (see the many tickets I linked) and on facilitating release verification. But those are separate efforts. I looked at the release email - and I cannot see the verification process > referred to. So as someone who would give +1 - I am not sure if I know the > process to follow. And I have no idea what any of the people giving +1 did. > > Also - this is actually required - each of the people casting their vote > should specify what they did - which of the checks they did (signatures, > checksums, licences, reproducibility). I see only +1s without explaining > what has been done which is considered generally a bad practice (I cannot > find a reference here - but maybe some other mentors might help heere > Thanks for the pointer, I think we could leverage some of it to improve our release verification page. > Example of similar processes we do in Airflow [3] -> this is where the > release verification "steps" are described. And when we vote, we mention > the steps we have done - example here [4] (usually we do all of them as PMC > members). There you can see: > > * link to the release verification process > * each PMC member voting explicitly states what they have done > * reproducibility is specifically checked (even if we do not have yet > automation of releases - we do it to verify Release Manager's work). This is beyond the current discussion, but AFAICT Airflow's Helm Chart release guide requires Helm 3.14.0 which does not support reproducible builds (see helm#31323 [8]). I am interested to know more about this, as this is one of the issues we identified. > Since this is PPMC and not yet TLP, possibly not all those requirements > have to be met, but that will not qualify this release as "ASF cm As JB mentioned, one of our goals is to also participate in the ATR program. I think that most of the issues we are talking about, like key management and trusted hardware, are going to be solved by ATR. Thanks for the information. If anything, it means that we should bump the priority of fully reproducible builds and release verification. I will monitor this thread to understand better if there are specific rules for TLP projects or if they must strictly be met by incubating projects as well. [1] https://polaris.apache.org/community/release-guide/ [2] https://github.com/apache/polaris/pull/3075 [3] https://github.com/apache/polaris/issues/3086 [4] https://github.com/apache/polaris/pull/3143 [5] https://github.com/apache/polaris/pull/2826 [6] https://github.com/apache/polaris/pull/3089 [7] https://github.com/apache/polaris/pull/2824 [8] https://github.com/helm/helm/pull/31323 -- Pierre
