Checked out the release tag. commit *6da346a358c * Seeing below error while compiling :
Duplicate classes found: Found in: org.apache.hadoop:hadoop-client-api:jar:3.3.2:compile org.apache.hadoop:hadoop-client-minicluster:jar:3.3.2:compile Duplicate classes: org/apache/hadoop/io/serializer/avro/AvroRecord.class org/apache/hadoop/io/serializer/avro/AvroRecord$Builder.class org/apache/hadoop/io/serializer/avro/AvroRecord$1.class [*INFO*] *------------------------------------------------------------------------* [*INFO*] *Reactor Summary for Apache Hadoop Client Test Minicluster 3.3.2:* [*INFO*] [*INFO*] Apache Hadoop Client Test Minicluster .............. *SUCCESS* [02:17 min] [*INFO*] Apache Hadoop Client Packaging Invariants for Test . *FAILURE* [ 0.221 s] [*INFO*] Apache Hadoop Client Packaging Integration Tests ... *SKIPPED* [*INFO*] Apache Hadoop Distribution ......................... *SKIPPED* [*INFO*] Apache Hadoop Client Modules ....................... *SKIPPED* [*INFO*] Apache Hadoop Tencent COS Support .................. *SKIPPED* [*INFO*] Apache Hadoop Cloud Storage ........................ *SKIPPED* [*INFO*] Apache Hadoop Cloud Storage Project ................ *SKIPPED* [*INFO*] *------------------------------------------------------------------------* [*INFO*] *BUILD FAILURE* [*INFO*] *------------------------------------------------------------------------* [*INFO*] Total time: 02:18 min [*INFO*] Finished at: 2022-01-21T18:06:11+05:30 [*INFO*] *------------------------------------------------------------------------* [*ERROR*] Failed to execute goal org.apache.maven.plugins:maven-enforcer-plugin:3.0.0-M1:enforce *(enforce-banned-dependencies)* on project hadoop-client-check-test-invariants: *Some Enforcer rules have failed. Look above for specific messages explaining why the rule failed.* -> *[Help 1]* On Fri, Jan 21, 2022 at 9:38 AM Wei-Chiu Chuang <weic...@apache.org> wrote: > I'll find time to check out the RC bits. > I just feel bad that the tarball is now more than 600MB in size. > > On Fri, Jan 21, 2022 at 2:23 AM Steve Loughran <ste...@cloudera.com.invalid > > > wrote: > > > *+1 binding.* > > > > reviewed binaries, source, artifacts in the staging maven repository in > > downstream builds. all good. > > > > *## test run* > > > > checked out the asf github repo at commit 6da346a358c into a location > > already set up with aws and azure test credentials > > > > ran the hadoop-aws tests with -Dparallel-tests -DtestsThreadCount=6 > > -Dmarkers=delete -Dscale > > and hadoop-azure against azure cardiff with -Dparallel-tests=abfs > > -DtestsThreadCount=6 > > > > all happy > > > > > > > > *## binary* > > downloaded KEYS and imported, so adding your key to my list (also signed > > this and updated the key servers) > > > > downloaded rc tar and verified > > ``` > > > gpg2 --verify hadoop-3.3.2.tar.gz.asc hadoop-3.3.2.tar.gz > > gpg: Signature made Sat Jan 15 23:41:10 2022 GMT > > gpg: using RSA key > DE7FA241EB298D027C97B2A1D8F1A97BE51ECA98 > > gpg: Good signature from "Chao Sun (CODE SIGNING KEY) < > sunc...@apache.org > > >" > > [full] > > > > > > > cat hadoop-3.3.2.tar.gz.sha512 > > SHA512 (hadoop-3.3.2.tar.gz) = > > > > > cdd3d9298ba7d6e63ed63f93c159729ea14d2b7d5e3a0640b1761c86c7714a721f88bdfa8cb1d8d3da316f616e4f0ceaace4f32845ee4441e6aaa7a12b8c647d > > > > > shasum -a 512 hadoop-3.3.2.tar.gz > > > > > cdd3d9298ba7d6e63ed63f93c159729ea14d2b7d5e3a0640b1761c86c7714a721f88bdfa8cb1d8d3da316f616e4f0ceaace4f32845ee4441e6aaa7a12b8c647d > > hadoop-3.3.2.tar.gz > > ``` > > > > > > *# cloudstore against staged artifacts* > > ``` > > cd ~/.m2/repository/org/apache/hadoop > > find . -name \*3.3.2\* -print | xargs rm -r > > ``` > > ensures no local builds have tainted the repo. > > > > in cloudstore mvn build without tests > > ``` > > mci -Pextra -Phadoop-3.3.2 -Psnapshots-and-staging > > ``` > > this fetches all from asf staging > > > > ``` > > Downloading from ASF Staging: > > > > > https://repository.apache.org/content/groups/staging/org/apache/hadoop/hadoop-client/3.3.2/hadoop-client-3.3.2.pom > > Downloaded from ASF Staging: > > > > > https://repository.apache.org/content/groups/staging/org/apache/hadoop/hadoop-client/3.3.2/hadoop-client-3.3.2.pom > > (11 kB at 20 kB/s) > > ``` > > there's no tests there, but it did audit the download process. FWIW, that > > project has switched to logback, so I now have all hadoop imports > excluding > > slf4j and log4j. it takes too much effort right now. > > > > build works. > > > > tested abfs and s3a storediags, all happy > > > > > > > > > > *### google GCS against staged artifacts* > > > > gcs is now java 11 only, so I had to switch JVMs here. > > > > had to add a snapshots and staging profile, after which I could build and > > test. > > > > ``` > > -Dhadoop.three.version=3.3.2 -Psnapshots-and-staging > > ``` > > two test failures were related to auth failures where the tests were > trying > > to raise exceptions but things failed differently > > ``` > > [ERROR] Failures: > > [ERROR] > > > > > GoogleHadoopFileSystemTest.eagerInitialization_fails_withInvalidCredentialsConfiguration:122 > > unexpected exception type thrown; expected:<java.io > .FileNotFoundException> > > but was:<java.lang.IllegalArgumentException> > > [ERROR] > > > > > GoogleHadoopFileSystemTest.lazyInitialization_deleteCall_fails_withInvalidCredentialsConfiguration:100 > > value of: throwable.getMessage() > > expected: Failed to create GCS FS > > but was : A JSON key file may not be specified at the same time as > > credentials via configuration. > > > > ``` > > > > I'm not worried here. > > > > ran cloudstore's diagnostics against gcs. > > > > Nice to see they are now collecting IOStatistics on their input streams. > we > > really need to get this collected through the parquet/orc libs and then > > through the query engines. > > > > ``` > > > bin/hadoop jar $CLOUDSTORE storediag gs://stevel-london/ > > > > ... > > 2022-01-20 17:52:47,447 [main] INFO diag.StoreDiag > > (StoreDurationInfo.java:<init>(56)) - Starting: Reading a file > > gs://stevel-london/dir-9cbfc774-76ff-49c0-b216-d7800369c3e1/file > > input stream summary: org.apache.hadoop.fs.FSDataInputStream@6cfd9a54: > > com.google.cloud.hadoop.fs.gcs.GoogleHadoopFSInputStream@78c1372d > > {counters=((stream_read_close_operations=1) > > (stream_read_seek_backward_operations=0) (stream_read_total_bytes=7) > > (stream_read_bytes=7) (stream_read_exceptions=0) > > (stream_read_seek_operations=0) (stream_read_seek_bytes_skipped=0) > > (stream_read_operations=3) (stream_read_bytes_backwards_on_seek=0) > > (stream_read_seek_forward_operations=0) > > (stream_read_operations_incomplete=1)); > > gauges=(); > > minimums=(); > > maximums=(); > > means=(); > > } > > ... > > ``` > > > > *### source* > > > > once I'd done builds and tests which fetched from staging, I did a local > > build and test > > > > repeated download/validate of source tarball, unzip/untar > > > > build with java11. > > > > I've not done the test run there, because that directory tree doesn't > have > > the credentials, and this mornings run was good. > > > > altogether then: very happy. tests good, downstream libraries building > and > > linking. > > > > On Wed, 19 Jan 2022 at 17:50, Chao Sun <sunc...@apache.org> wrote: > > > > > Hi all, > > > > > > I've put together Hadoop 3.3.2 RC2 below: > > > > > > The RC is available at: > > > http://people.apache.org/~sunchao/hadoop-3.3.2-RC2/ > > > The RC tag is at: > > > https://github.com/apache/hadoop/releases/tag/release-3.3.2-RC2 > > > The Maven artifacts are staged at: > > > > https://repository.apache.org/content/repositories/orgapachehadoop-1332 > > > > > > You can find my public key at: > > > https://downloads.apache.org/hadoop/common/KEYS > > > > > > I've done the following tests and they look good: > > > - Ran all the unit tests > > > - Started a single node HDFS cluster and tested a few simple commands > > > - Ran all the tests in Spark using the RC2 artifacts > > > > > > Please evaluate the RC and vote, thanks! > > > > > > Best, > > > Chao > > > > > >