Afraid I have to say -1 to this iteration, but I promise I'll help address the issues
First, I've cherrypicked a few final changes from branch-3.4 in, including this major one HADOOP-19153. hadoop-common exports logback as a transitive dependency (#6999) This broke parquet hadoop-test runs without an explicit exclusion downstream -it'll help others upgrade. Second, the cherrypick branch test run showed parquet JDK incompatibilities https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6997/1/testReport/ java.lang.NoSuchMethodError: java.nio.ByteBuffer.position(I)Ljava/nio/ByteBuffer; at org.apache.hadoop.thirdparty.protobuf.IterableByteBufferInputStream.read(IterableByteBufferInputStream.java:143) at org.apache.hadoop.thirdparty.protobuf.CodedInputStream$StreamDecoder.read(CodedInputStream.java:2080) at org.apache.hadoop.thirdparty.protobuf.CodedInputStream$StreamDecoder.tryRefillBuffer(CodedInputStream.java:2831) at org.apache.hadoop.thirdparty.protobuf.CodedInputStream$StreamDecoder.refillBuffer(CodedInputStream.java:2777) at org.apache.hadoop.thirdparty.protobuf.CodedInputStream$StreamDecoder.readRawByte(CodedInputStream.java:2859) at org.apache.hadoop.thirdparty.protobuf.CodedInputStream$StreamDecoder.readRawVarint64SlowPath(CodedInputStream.java:2648) at org.apache.hadoop.thirdparty.protobuf.CodedInputStream$StreamDecoder.readRawVarint64(CodedInputStream.java:2641) at org.apache.hadoop.thirdparty.protobuf.CodedInputStream$StreamDecoder.readSInt64(CodedInputStream.java:2497) at org.apache.hadoop.hdfs.protocol.BlockListAsLongs$BufferDecoder$1.next(BlockListAsLongs.java:419) at org.apache.hadoop.hdfs.protocol.BlockListAsLongs$BufferDecoder$1.next(BlockListAsLongs.java:397) at org.apache.hadoop.hdfs.protocol.BlockListAsLongs$BufferDecoder.getBlockListAsLongs(BlockListAsLongs.java:375) at org.apache.hadoop.hdfs.protocol.TestBlockListAsLongs.checkReport(TestBlockListAsLongs.java:156) at org.apache.hadoop.hdfs.protocol.TestBlockListAsLongs.testFuzz(TestBlockListAsLongs.java:139) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306) at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100) This is addressed by "HADOOP-19163. Use protobuf-java 3.25.3", which bumped up the shaded version then modified hadoop dependencies to match. I'm going to release thirdparty jar 1.3.0 with the relevant updates, upgrade 3.4.1+ to use it once that's out (I'll have the pending PRs up) HADOOP-19252. Release Hadoop Third-Party 1.3.0 https://issues.apache.org/jira/browse/HADOOP-19252 The other thing we should all look at is making sure we are current with all dependency updates in trunk *without doing any last minute update of jar versions entirely* I've got a PR of the kafka update: https://github.com/apache/hadoop/pull/7000 ; will merge if yetus doesn't complain On Thu, 8 Aug 2024 at 19:06, Mukund Madhav Thakur <mtha...@cloudera.com.invalid> wrote: > Apache Hadoop 3.4.1 > > > > I with help of Steve have put together a release candidate (RC1) for Hadoop > 3.4.1. > > > > What we would like is for anyone who can to verify the tarballs, especially > > anyone who can try the arm64 binaries as we want to include them too. > > > > The RC is available at: > > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.1-RC1/ > > > > The git tag is release-3.4.1-RC1, commit > 247daf0f827adc96a3847bb40e0fec3fc85f33bd > > > > The maven artifacts are staged at > > https://repository.apache.org/content/repositories/orgapachehadoop-1417 > > > > You can find my public key at: > > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS > > > > Change log > > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.1-RC1/CHANGELOG.md > > > > Release notes > > > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.1-RC1/RELEASENOTES.md > > > > This is off branch-3.4. > > > > Key changes include > > > > * Bulk Delete API. https://issues.apache.org/jira/browse/HADOOP-18679 > > * Fixes and enhancements in Vectored IO API. > > * Improvements in Hadoop Azure connector. > > * Fixes and improvements post upgrade to AWS V2 SDK in S3AConnector. > > * This release includes Arm64 binaries. Please can anyone with > > compatible systems validate these. > > > > Note, because the arm64 binaries are built separately on a different > > platform and JVM, their jar files may not match those of the x86 > > release -and therefore the maven artifacts. I don't think this is > > an issue (the ASF actually releases source tarballs, the binaries are > > there for help only, though with the maven repo that's a bit blurred). > > > > The only way to be consistent would actually untar the x86.tar.gz, > > overwrite its binaries with the arm stuff, retar, sign and push out > > for the vote. Even automating that would be risky. > > > > As this is just a first try to get this out, there might be issues, Please > try the release and let me know. Also let me know if you would like to add > somethings in 3.4.1 > > > Also found two issues in hadoop-yarn-ui while building the arm binaries > > https://issues.apache.org/jira/browse/YARN-11712 > > https://issues.apache.org/jira/browse/YARN-11713 > > > > Thanks, > > Mukund >
