I'm well aware of the policies you linked and quoted: I directly participated in the discussions which led up to the most recent change last year. Since I'm worried about repeating myself and producing another wall of text in another unsuccessful attempt to clarify all the details, I'll try to summarize with two points:
1. *We are 100% compliant* with the policy; the existence of the sha1/md5 files in Nexus are their own, separate Maven-specific thing, and 2. Checking them is *not* a waste of time; they ensure the integrity of the Maven staging area to be published to Maven Central. On Tue, Apr 2, 2019 at 1:44 PM Michael Wall <mjw...@apache.org> wrote: > > Thanks for the clarification. That was a much longer response than I > anticipated, so I think I am having trouble communicating over email. > > I was referencing "https://www.apache.org/dev/release-distribution.html"; > which says > > <snip> > > For every artifact distributed to the public through Apache channels, the > PMC > > - MUST supply a valid > <https://www.apache.org/dev/release-signing#verifying-signature> > OpenPGP-compatible > ASCII-armored detached signature > <https://www.apache.org/dev/release-signing#openpgp-ascii-detach-sig> > file > - MUST supply at least one checksum file > - SHOULD supply a SHA-256 and/or SHA-512 > <https://www.apache.org/dev/release-signing#sha-checksum> checksum file > - SHOULD NOT supply a MD5 or SHA-1 checksum file (because these are > deprecated) > > </snip> > > Our template generates an email with the following > > <snip> > (Append ".sha1", ".md5", or ".asc" to download the signature/hash for a > given artifact.) > > In addition to the tarballs, and their signatures, the following checksum > files will be added to the dist/release SVN area after release: > accumulo-1.9.3-src.tar.gz.sha512 will contain: > SHA512 (accumulo-1.9.3-src.tar.gz) > =b366b89295b1835038cb242f8ad46b1d8455753a987333f0e15e3d89749540f2cd59db1bc6cf7100fc9050d3d0bc7340a3b661381549d40f2f0223d4120fd809 > > accumulo-1.9.3-bin.tar.gz.sha512 will contain: > SHA512 (accumulo-1.9.3-bin.tar.gz) > =cc909296d9bbd12e08064fccaf21e81b754c183a8264dfa2575762c76705fd0c580b50c2b224c60feaeec120bd618fba4d6176d0f53e96e1ca9da0d9e2556f1f > > Signing keys are available at https://www.apache.org/dist/accumulo/KEYS > (Expected fingerprint: 8CC4F8A2B29C2B040F2B835D6F0CDAE700B6899D) > > Release notes (in progress) can be found at: > https://accumulo.apache.org/release/accumulo-1.9.3/ > > Release testing instructions: > https://accumulo.apache.org/contributor/verifying-release > </snip> > > That last link has the following > > <snip> > > OpenPGP is an asymmetric encryption scheme which lends itself well to the > globally distributed nature of Apache. Verification of a release artifact > can be done using the signature and the release-maker’s public key. Hashes > can be verified using the appropriate command (e.g. sha1sum, md5sum) > > </snip> > So putting that all together, I spent time verify the sha1 and md5 when I > didn't need to. I was not suggesting we add .sha512 files. Not suggesting > we change Maven. The only thing I was suggesting was to make it harder for > me (and others maybe) to make the mistake of checking the sha1 and md5 > again. I do think it is wording on the template, but the inaccuracies in > my suggestion were distracting. Maybe it is a change to > https://accumulo.apache.org/contributor/verifying-release#foundation-level-requirements > as well. I'll look for the template in code and make some PRs there and on > the website after I give it some more thought. > > Mike > > > On Tue, Apr 2, 2019 at 11:42 AM Christopher <ctubb...@apache.org> wrote: > > > On Tue, Apr 2, 2019 at 9:32 AM Michael Wall <mjw...@gmail.com> wrote: > > > > > > If I understand this correctly, you are saying the sha1 and md5 are > > created > > > by nexus? I do recall the discussion about moving to the stronger > > hashes, > > > so I was surprised to see sha1 and md5 still listed in the VOTE thread. > > > > They are actually created by one of the Maven plugins and uploaded to > > Nexus. Nexus might generate its own to compare/validate them, but that > > would be internal to Nexus. In any case, there is movement in the > > Maven community to move these from one plugin to another (from > > maven-install-plugin to maven-deploy-plugin, IIRC). Currently, Nexus > > upload validation rules require .sha1 and .md5, and the Maven > > dependency resolver uses these to validate downloads on the client. > > These aren't just requirements for ASF's Nexus, but requirements to > > sync artifacts to Maven Central also, and are core parts of the Maven2 > > repository format. > > > > In order for Maven to discontinue these, 3 components need to > > simultaneously (or coordinated-ly) update: maven-deploy-plugin, maven > > dependency resolver, and Nexus. I'm sure that will happen at some > > point, but it's going to take some time, and as I've said... all that > > is outside our control. > > > > > > > > When verifying a release, I want to verify the signatures are correct > > and I > > > want to use the asc and sha512 that will show up on the downloads page at > > > https://accumulo.apache.org/downloads/. > > > > > > What about something like "Nexus generates an md5 and sha1 file but those > > > are no longer recommended by Apache for verification of artifacts. For > > > release verification use the 512 signature below" or something. > > > > > > > Inaccuracies not withstanding (see above for clarification regarding > > which component generates these), the situation is more complex than > > merely not being recommended. First, it is currently not easy or > > reasonable to stage checksums of better quality in Nexus at all. > > Second, the release distribution policy applies to the ASF mirrors, > > not Nexus. Since we're staging in Nexus, the SHA512 have been provided > > directly in the email for the artifacts which will be placed on the > > mirrors. Third, for all artifacts that are not going on the mirrors > > (all the "convenience binary" jars that are only published to Maven), > > the *only* way to verify these is with their corresponding .sha1/.md5 > > (or .asc) files uploaded to Nexus. > > > > This current situation is complex... and it's very hard to encapsulate > > all the relevant details in a vote email, but I believe we are doing > > the right thing: > > > > 1. We are in compliance with the updated release distribution policy > > (as I said, one of the first to be), > > 2. We include SHA512 sums directly in our vote thread for provenance, > > which almost nobody else in ASF does (b/c we don't need to: the GPG > > signatures satisfy anything these could, plus some, and they satisfy > > their purpose just as well detached), and > > 3. We acknowledge the existence of the location of the Maven hashes, > > for all artifacts stored in Maven to ensure there are no errors with > > the staging area, and for other Maven-specific purposes. > > > > Short of explaining this over and over and over again, so that > > everybody else is as familiar with all these details as I am (which I > > can do, but it is time consuming, and lengthy), I don't know what we > > can do better. The only thing I can think of is possibly improve the > > wording in the template, but it's hard to be accurate without being > > excessively verbose. When I wrote the original wording for the > > template, I opted for brevity, but it seems to have only raised > > questions from those unfamiliar with all the relevant complexities, > > since this is the second vote thread in which we've discussed this. > > I'm open to suggestions, but I think your above suggestion diverges > > from accuracy a bit too far, since: > > > > 1) Nexus doesn't generate them, > > 2) it's not the case that the relevant issue is whether they are > > recommended by Apache, > > 3) it's further not the case that Apache has completely discontinued > > use of sha1/md5 for all purposes, since the policy update only applies > > to the mirrors, not Nexus, and > > 4) the *only* mandatory mechanism required by ASF that can be used to > > verify a release is the cryptographic signature (GPG) > > > > http://www.apache.org/legal/release-policy.html#what-must-every-release-contain > > ; all others are requirements of the distribution mirrors and are > > fixed characteristics of the release artifacts, not something that is > > itself voted on beyond the contents of the artifacts (a checksum is a > > fixed characteristic, like file size; we vote on the contents of the > > artifacts, not these traits); those other requirements for the mirrors > > are not release vote requirements (which makes sense... we're voting > > on the contents of the artifacts... not fixed, deterministic traits > > like their file size or checksums). > > > > > > > Thanks Christopher. > > > > > > No problem. I'm happy to explain further or clarify anything, as > > needed. And, if I've said something wrong, I welcome corrections. > > > > > > > > > > On Mon, Apr 1, 2019 at 10:21 PM Christopher <ctubb...@apache.org> wrote: > > > > > > > Apologies, but I was confused because you said the template was > > > > wrong... but you quoted a portion of the template which was not wrong, > > > > and which I've already explained in my original response to Mike. > > > > > > > > Once again, those are references to files generated by the Maven > > > > tooling, outside our control. Granted, we don't have to mention them > > > > at all. However, the template merely acknowledges their existence. > > > > Providing information for how those files are named still has value in > > > > the Maven context, and is useful for any artifact downloaded from > > > > Maven, not just our tarballs. > > > > > > > > Would there be less confusion over this if the template was a bit more > > > > verbose, saying something like: > > > > (Append ".asc" to download an artifact's corresponding GPG signature, > > > > or ".sha1" or ".md5" to verify the checksums generated by Maven.) > > > > > > > > If you'd prefer this, or an alternative wording, please change it in > > > > the repo... or let me know and I'll change it. > > > > > > > > On Mon, Apr 1, 2019 at 2:22 PM Josh Elser <els...@apache.org> wrote: > > > > > > > > > > Again, like I included earlier: > > > > > > > > > > > (Append ".sha1", ".md5", or ".asc" to download the signature/hash > > for > > > > > a given artifact.) > > > > > > > > > > On 4/1/19 1:56 PM, Christopher wrote: > > > > > > In what way? > > > > > > > > > > > > On Mon, Apr 1, 2019 at 1:54 PM Josh Elser <els...@apache.org> > > wrote: > > > > > >> > > > > > >> Your email template is wrong. > > > > > >> > > > > > >> On 4/1/19 1:33 PM, Christopher wrote: > > > > > >>> Sorry, I don't understand what you mean by 'retelling of > > "checksums > > > > of old"'. > > > > > >>> > > > > > >>> On Mon, Apr 1, 2019 at 12:30 PM Josh Elser <els...@apache.org> > > > > wrote: > > > > > >>>> > > > > > >>>> I think Mike's point was your VOTE template does not reflect the > > > > > >>>> retelling of "checksums of old" > > > > > >>>> > > > > > >>>> > (Append ".sha1", ".md5", or ".asc" to download the > > > > signature/hash for > > > > > >>>> a given artifact.) > > > > > >>>> > > > > > >>>> On 3/31/19 10:54 PM, Christopher wrote: > > > > > >>>>> Mike, > > > > > >>>>> > > > > > >>>>> We already stopped using md5 and sha1 for the release > > artifacts on > > > > the > > > > > >>>>> mirrors. I did this some time ago, and we discussed it on list > > on > > > > > >>>>> previous vote threads (last year)... which resulted in me > > changing > > > > the > > > > > >>>>> release candidate build script automated tooling to embed the > > > > SHA512 > > > > > >>>>> sums for the tarballs directly in the release vote message. I > > even > > > > > >>>>> went back and updated the downloads page for the previous > > releases > > > > and > > > > > >>>>> updated the mirrors to be SHA512 only. Because of these steps I > > > > took, > > > > > >>>>> Accumulo was one of the first projects across the entire ASF > > who > > > > were > > > > > >>>>> 100% compliant immediately after INFRA VP updated the release > > > > > >>>>> distribution policy you linked. > > > > > >>>>> > > > > > >>>>> *This is a resolved action for Accumulo.* > > > > > >>>>> > > > > > >>>>> FWIW, SHA512 was also used as the hash algorithm in the GPG > > > > signature > > > > > >>>>> (same as every RC I've ever prepped for ASF). The only > > remaining > > > > md5 > > > > > >>>>> and sha1 reference are Maven-specific tooling, and we have no > > > > control > > > > > >>>>> over that tooling. We could change the vote template to no > > longer > > > > > >>>>> mention them, but I don't see the point since they're still > > > > relevant > > > > > >>>>> within the context of Maven artifact hosting, and that's the > > > > context > > > > > >>>>> in which they are presented in the vote email. > > > > > >>>>> > > > > > >>>>> On Sun, Mar 31, 2019 at 1:59 PM Michael Wall <mjw...@gmail.com > > > > > > > wrote: > > > > > >>>>>> > > > > > >>>>>> -1 for the issue with commons config > > > > > >>>>>> > > > > > >>>>>> I check the signatures, they are good. We should stop using > > md5 > > > > and sha1 > > > > > >>>>>> though, see > > > > https://www.apache.org/dev/release-distribution#sigs-and-sums. > > > > > >>>>>> Has anyone looked at moving to sha256 and/org sha512? > > > > > >>>>>> Successful run of mvn clean verify -Psunny > > > > > >>>>>> > > > > > >>>>>> On Sat, Mar 30, 2019 at 11:31 PM Keith Turner < > > ke...@deenlo.com> > > > > wrote: > > > > > >>>>>> > > > > > >>>>>>> I completed a continuous ingest run on a 10 node cluster > > running > > > > > >>>>>>> Centos 7. I used the native map. I had to rebuild Accumulo > > to > > > > work > > > > > >>>>>>> around #1065 inorder to get the verify M/R job to run. > > > > > >>>>>>> > > > > > >>>>>>> > > > > org.apache.accumulo.test.continuous.ContinuousVerify$Counts > > > > > >>>>>>> REFERENCED=34417110819 > > > > > >>>>>>> UNREFERENCED=9097524 > > > > > >>>>>>> > > > > > >>>>>>> On Wed, Mar 27, 2019 at 7:57 PM Christopher < > > ctubb...@apache.org> > > > > wrote: > > > > > >>>>>>>> > > > > > >>>>>>>> Accumulo Developers, > > > > > >>>>>>>> > > > > > >>>>>>>> Please consider the following candidate for Apache Accumulo > > > > 1.9.3. > > > > > >>>>>>>> > > > > > >>>>>>>> This supersedes RC1 and contains the following change: > > > > > >>>>>>>> https://github.com/apache/accumulo/pull/1057 > > > > > >>>>>>>> > > > > > >>>>>>>> Git Commit: > > > > > >>>>>>>> 94f9782242a1f336e176c282f0f90063a21e361d > > > > > >>>>>>>> Branch: > > > > > >>>>>>>> 1.9.3-rc2 > > > > > >>>>>>>> > > > > > >>>>>>>> If this vote passes, a gpg-signed tag will be created using: > > > > > >>>>>>>> git tag -f -m 'Apache Accumulo 1.9.3' -s rel/1.9.3 \ > > > > > >>>>>>>> 94f9782242a1f336e176c282f0f90063a21e361d > > > > > >>>>>>>> > > > > > >>>>>>>> Staging repo: > > > > > >>>>>>> > > > > > > https://repository.apache.org/content/repositories/orgapacheaccumulo-1077 > > > > > >>>>>>>> Source (official release artifact): > > > > > >>>>>>>> > > > > > >>>>>>> > > > > > > https://repository.apache.org/content/repositories/orgapacheaccumulo-1077/org/apache/accumulo/accumulo/1.9.3/accumulo-1.9.3-src.tar.gz > > > > > >>>>>>>> Binary: > > > > > >>>>>>> > > > > > > https://repository.apache.org/content/repositories/orgapacheaccumulo-1077/org/apache/accumulo/accumulo/1.9.3/accumulo-1.9.3-bin.tar.gz > > > > > >>>>>>>> (Append ".sha1", ".md5", or ".asc" to download the > > > > signature/hash for > > > > > >>>>>>>> a given artifact.) > > > > > >>>>>>>> > > > > > >>>>>>>> In addition to the tarballs, and their signatures, the > > > > following checksum > > > > > >>>>>>>> files will be added to the dist/release SVN area after > > release: > > > > > >>>>>>>> accumulo-1.9.3-src.tar.gz.sha512 will contain: > > > > > >>>>>>>> SHA512 (accumulo-1.9.3-src.tar.gz) = > > > > > >>>>>>>> > > > > > >>>>>>> > > > > > > b366b89295b1835038cb242f8ad46b1d8455753a987333f0e15e3d89749540f2cd59db1bc6cf7100fc9050d3d0bc7340a3b661381549d40f2f0223d4120fd809 > > > > > >>>>>>>> accumulo-1.9.3-bin.tar.gz.sha512 will contain: > > > > > >>>>>>>> SHA512 (accumulo-1.9.3-bin.tar.gz) = > > > > > >>>>>>>> > > > > > >>>>>>> > > > > > > cc909296d9bbd12e08064fccaf21e81b754c183a8264dfa2575762c76705fd0c580b50c2b224c60feaeec120bd618fba4d6176d0f53e96e1ca9da0d9e2556f1f > > > > > >>>>>>>> > > > > > >>>>>>>> Signing keys are available at > > > > https://www.apache.org/dist/accumulo/KEYS > > > > > >>>>>>>> (Expected fingerprint: > > 8CC4F8A2B29C2B040F2B835D6F0CDAE700B6899D) > > > > > >>>>>>>> > > > > > >>>>>>>> Release notes (in progress) can be found at: > > > > > >>>>>>>> https://accumulo.apache.org/release/accumulo-1.9.3/ > > > > > >>>>>>>> > > > > > >>>>>>>> Release testing instructions: > > > > > >>>>>>>> https://accumulo.apache.org/contributor/verifying-release > > > > > >>>>>>>> > > > > > >>>>>>>> Please vote one of: > > > > > >>>>>>>> [ ] +1 - I have verified and accept... > > > > > >>>>>>>> [ ] +0 - I have reservations, but not strong enough to vote > > > > against... > > > > > >>>>>>>> [ ] -1 - Because..., I do not accept... > > > > > >>>>>>>> ... these artifacts as the 1.9.3 release of Apache Accumulo. > > > > > >>>>>>>> > > > > > >>>>>>>> This vote will remain open until at least Sun Mar 31 > > 00:00:00 > > > > UTC 2019. > > > > > >>>>>>>> (Sat Mar 30 20:00:00 EDT 2019 / Sat Mar 30 17:00:00 PDT > > 2019) > > > > > >>>>>>>> Voting can continue after this deadline until the release > > > > manager > > > > > >>>>>>>> sends an email ending the vote. > > > > > >>>>>>>> > > > > > >>>>>>>> Thanks! > > > > > >>>>>>>> > > > > > >>>>>>>> P.S. Hint: download the whole staging repo with > > > > > >>>>>>>> wget -erobots=off -r -l inf -np -nH \ > > > > > >>>>>>>> > > > > > >>>>>>> > > > > > > https://repository.apache.org/content/repositories/orgapacheaccumulo-1077/ > > > > > >>>>>>>> # note the trailing slash is needed > > > > > >>>>>>> > > > > > >
