Thank you very much for the response! The content is very comprehensive and valuable.
I will prepare hadoop-3.4.0-RC1 according to the instructions provided by you, and after RC1 is packaged, I will use validate-hadoop-client-artifacts for validation. Best Regards, Shilun Fan. On Tue, Jan 16, 2024 at 12:34 AM Steve Loughran <ste...@cloudera.com.invalid> wrote: > -1 I'm afraid, just due to staging/packaging issues. > > This took me a few goes to get right myself, so nothing unusual. > > Note I used my validator project which is set to retrieve binaries, check > signatures, run maven builds against staged artifacts *and clean up any > local copies first*and more. > > This uses apache ant to manage all this: > > https://github.com/steveloughran/validate-hadoop-client-artifacts > > Here's the initial build.properties:file I used to try and manage this > > ###### build.properties: > hadoop.version=3.4.0 > rc=RC0 > amd.src.dir=https://home.apache.org/~slfan1989/hadoop-3.4.0-RC0-amd64/ > http.source=https://home.apache.org/~slfan1989/hadoop-3.4.0-RC0-amd64 > <https://home.apache.org/~slfan1989/hadoop-3.4.0-RC0-amd64/http.source=https://home.apache.org/~slfan1989/hadoop-3.4.0-RC0-amd64> > > release=hadoop-${hadoop.version}-RC0 > rc.dirname=${release} > release.native.binaries=false > git.commit.id=cdb8af4f22ec > nexus.staging.url= > https://repository.apache.org/content/repositories/orgapachehadoop-1391/ > hadoop.source.dir=${local.dev.dir}/hadoop-trunk > ###### > > When I did my own builds, all the artifacts created were without the RC0 > suffix. It is critical this happens because the .sha512 checksums include > that in their paths > > > cat hadoop-3.4.0-RC0.tar.gz.sha512 > SHA512 (hadoop-3.4.0-RC0.tar.gz) = > > e50e68aecb36867c610db8309ccd3aae812184da21354b50d2a461b29c73f21d097fb27372c73c150e1c035003bb99a61c64db26c090fe0fb9e7ed6041722eab > > > Maven artifacts: staging problems > > Couldn't build with a -Pstaging profile as the staging repository wasn't > yet closed -I tried to do that myself. > > This failed with some rule problem > > Event: Failed: Checksum Validation > Monday, January 15, 2024 14:37:13 GMT (GMT+0000) > typeId checksum-staging > failureMessage INVALID SHA-1: > > '/org/apache/hadoop/hadoop-mapreduce-client-jobclient/3.4.0/hadoop-mapreduce-client-jobclient-3.4.0-tests.jar.sha1' > failureMessage Requires one-of SHA-1: > > /org/apache/hadoop/hadoop-mapreduce-client-jobclient/3.4.0/hadoop-mapreduce-client-jobclient-3.4.0-tests.jar.sha1, > SHA-256: > > /org/apache/hadoop/hadoop-mapreduce-client-jobclient/3.4.0/hadoop-mapreduce-client-jobclient-3.4.0-tests.jar.sha256, > SHA-512: > > /org/apache/hadoop/hadoop-mapreduce-client-jobclient/3.4.0/hadoop-mapreduce-client-jobclient-3.4.0-tests.jar.sha512 > > I don't know precisely what this means...my guess is that the upload didn't > include everything. > > Note my client-validator module can check this; just run its maven test > commands > > mvn clean test -U -P3.4 -Pstaging > > GPG signing: all good. > > Picked your key up from the site ( ant gpg.keys ) ... first validation with > ant gpg.verify was unhappy as your key wasn't trusted. I've signed it and > pushed that signature up, so people who trust me get some reassurance about > you. > > My build then failed as the gpg code couldn't find the > hadoop-3.4.0-aarch64.tar.gz.asc > > The problem here is that although we want separate arm and x86 tar files, > we don't really want separate binaries as it only creates different jars in > the wild. > > The way I addressed that was after creating that x86 release on an ec2 vm > and downloading it, I then did a local arm64 build and then created an arm > .tar.gz file, copied it into the same dir as the amd66 binaries but with > the arm64 .tar.gz filename, .asc and .sha512 checksum files all renamed > (checksum file patches to match the name). > > > https://github.com/steveloughran/validate-hadoop-client-artifacts?tab=readme-ov-file#arm64-binaries >
