[VOTE] Release Apache Hadoop 3.4.0 (RC2)
Hi folks, Xiaoqiao He and I have put together a release candidate (RC2) for Hadoop 3.4.0. What we would like is for anyone who can to verify the tarballs, especially anyone who can try the arm64 binaries as we want to include them too. The RC is available at: https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC2 The git tag is release-3.4.0-RC2, commit 88fbe62f27e The maven artifacts are staged at https://repository.apache.org/content/repositories/orgapachehadoop-1402 You can find my public key at: https://dist.apache.org/repos/dist/release/hadoop/common/KEYS Change log https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC2/CHANGELOG.md Release notes https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC2/RELEASENOTES.md This is off branch-3.4.0 and is the first big release since 3.3.6. Key changes include * S3A: Upgrade AWS SDK to V2 * HDFS DataNode Split one FsDatasetImpl lock to volume grain locks * YARN Federation improvements * YARN Capacity Scheduler improvements * HDFS RBF: Code Enhancements, New Features, and Bug Fixes * HDFS EC: Code Enhancements and Bug Fixes * Transitive CVE fixes Differences from RC0 & RC1 * We've improved Hadoop 3.4.0 Highlight big features and improvements. * Confirmed the JIRA status of Hadoop, HDFS, YARN, and MAPREDUCE modules. * Use validate-hadoop-client-artifacts[1] for packaging and verification. * Use hadoop-thirdparty-1.2.0 Note, because the arm64 binaries are built separately on a different platform and JVM, their jar files may not match those of the x86 release -and therefore the maven artifacts. I don't think this is an issue (the ASF actually releases source tarballs, the binaries are there for help only, though with the maven repo that's a bit blurred). The only way to be consistent would actually untar the x86.tar.gz, overwrite its binaries with the arm stuff, retar, sign and push out for the vote. Even automating that would be risky. [1] validate-hadoop-client-artifacts: https://github.com/steveloughran/validate-hadoop-client-artifacts Thanks to steve for providing validate-hadoop-client-artifacts. Best Regards, Shilun Fan.
Re: [VOTE] Release Apache Hadoop 3.4.0 (RC1)
We will backport all the PRs from the branch-3.4 branch to branch-3.4.0, and use this to release hadoop-3.4.0-RC2. It should include the s3a checksum and region stuff. Best Regards, Shilun Fan. On Mon, Feb 12, 2024 at 7:10 PM Steve Loughran wrote: > so it'll have the s3a checksum and region stuff? that'd be wonderful! I've > been assuming i'd have to push out a 3.4.1 release for that alone. > > On Sat, 10 Feb 2024 at 23:36, slfan1989 wrote: > >> We will end the voting for hadoop-3.4.0-RC1 and will open the voting for >> hadoop-3.4.0-RC2 in 1-2 days. >> >> * hadoop-3.4.0-RC2 will contain all commits in branch-3.4, and newly >> submitted(in branch-3.4) commits will be cherry-picked to branch-3.4.0. >> * hadoop-3.4.0-RC2 will use hadoop-thirdparty-1.2.0. >> >> Best Regards, >> Shilun Fan. >> >> On Mon, Jan 29, 2024 at 10:33 PM slfan1989 wrote: >> >> > Apache Hadoop 3.4.0 >> > >> > Xiaoqiao He and I have put together a release candidate (RC1) for Hadoop >> > 3.4.0. >> > >> > What we would like is for anyone who can to verify the tarballs, >> especially >> > anyone who can try the arm64 binaries as we want to include them too. >> > >> > The RC is available at: >> > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC1/ >> > >> > The git tag is release-3.4.0-RC1, commit 7e2edd8c5d1 >> > >> > The maven artifacts are staged at >> > >> https://repository.apache.org/content/repositories/orgapachehadoop-1395/ >> > >> > You can find my public key at: >> > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS >> > >> > Change log >> > >> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC1/CHANGELOG.md >> > >> > Release notes >> > >> > >> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC1/RELEASENOTES.md >> > >> > This is off branch-3.4.0 and is the first big release since 3.3.6. >> > >> > Key changes include >> > >> > * S3A: Upgrade AWS SDK to V2 >> > * HDFS DataNode Split one FsDatasetImpl lock to volume grain locks >> > * YARN Federation improvements >> > * YARN Capacity Scheduler improvements >> > * HDFS RBF: Code Enhancements, New Features, and Bug Fixes >> > * HDFS EC: Code Enhancements and Bug Fixes >> > * Transitive CVE fixes >> > >> > Differences from RC0 >> > >> > * We've improved Hadoop 3.4.0 Highlight big features and improvements. >> > * Confirmed the JIRA status of Hadoop, HDFS, YARN, and MAPREDUCE >> modules. >> > * Use validate-hadoop-client-artifacts[1] for packaging and >> verification. >> > >> > Note, because the arm64 binaries are built separately on a different >> > platform and JVM, their jar files may not match those of the x86 >> > release -and therefore the maven artifacts. I don't think this is >> > an issue (the ASF actually releases source tarballs, the binaries are >> > there for help only, though with the maven repo that's a bit blurred). >> > >> > The only way to be consistent would actually untar the x86.tar.gz, >> > overwrite its binaries with the arm stuff, retar, sign and push out >> > for the vote. Even automating that would be risky. >> > >> > [1] validate-hadoop-client-artifacts: >> > https://github.com/steveloughran/validate-hadoop-client-artifacts >> > Thanks to steve for providing validate-hadoop-client-artifacts. >> > >> > Best Regards, >> > Shilun Fan. >> > >> >
Re: [VOTE] Release Apache Hadoop 3.4.0 (RC1)
so it'll have the s3a checksum and region stuff? that'd be wonderful! I've been assuming i'd have to push out a 3.4.1 release for that alone. On Sat, 10 Feb 2024 at 23:36, slfan1989 wrote: > We will end the voting for hadoop-3.4.0-RC1 and will open the voting for > hadoop-3.4.0-RC2 in 1-2 days. > > * hadoop-3.4.0-RC2 will contain all commits in branch-3.4, and newly > submitted(in branch-3.4) commits will be cherry-picked to branch-3.4.0. > * hadoop-3.4.0-RC2 will use hadoop-thirdparty-1.2.0. > > Best Regards, > Shilun Fan. > > On Mon, Jan 29, 2024 at 10:33 PM slfan1989 wrote: > > > Apache Hadoop 3.4.0 > > > > Xiaoqiao He and I have put together a release candidate (RC1) for Hadoop > > 3.4.0. > > > > What we would like is for anyone who can to verify the tarballs, > especially > > anyone who can try the arm64 binaries as we want to include them too. > > > > The RC is available at: > > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC1/ > > > > The git tag is release-3.4.0-RC1, commit 7e2edd8c5d1 > > > > The maven artifacts are staged at > > https://repository.apache.org/content/repositories/orgapachehadoop-1395/ > > > > You can find my public key at: > > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS > > > > Change log > > > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC1/CHANGELOG.md > > > > Release notes > > > > > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC1/RELEASENOTES.md > > > > This is off branch-3.4.0 and is the first big release since 3.3.6. > > > > Key changes include > > > > * S3A: Upgrade AWS SDK to V2 > > * HDFS DataNode Split one FsDatasetImpl lock to volume grain locks > > * YARN Federation improvements > > * YARN Capacity Scheduler improvements > > * HDFS RBF: Code Enhancements, New Features, and Bug Fixes > > * HDFS EC: Code Enhancements and Bug Fixes > > * Transitive CVE fixes > > > > Differences from RC0 > > > > * We've improved Hadoop 3.4.0 Highlight big features and improvements. > > * Confirmed the JIRA status of Hadoop, HDFS, YARN, and MAPREDUCE modules. > > * Use validate-hadoop-client-artifacts[1] for packaging and verification. > > > > Note, because the arm64 binaries are built separately on a different > > platform and JVM, their jar files may not match those of the x86 > > release -and therefore the maven artifacts. I don't think this is > > an issue (the ASF actually releases source tarballs, the binaries are > > there for help only, though with the maven repo that's a bit blurred). > > > > The only way to be consistent would actually untar the x86.tar.gz, > > overwrite its binaries with the arm stuff, retar, sign and push out > > for the vote. Even automating that would be risky. > > > > [1] validate-hadoop-client-artifacts: > > https://github.com/steveloughran/validate-hadoop-client-artifacts > > Thanks to steve for providing validate-hadoop-client-artifacts. > > > > Best Regards, > > Shilun Fan. > > >
Apache Hadoop qbt Report: branch-2.10+JDK7 on Linux/x86_64
For more details, see https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1300/ No changes -1 overall The following subsystems voted -1: asflicense hadolint mvnsite pathlen unit The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck whitespace The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: Failed junit tests : hadoop.fs.TestFileUtil hadoop.contrib.bkjournal.TestBookKeeperHACheckpoints hadoop.hdfs.server.blockmanagement.TestReplicationPolicyWithUpgradeDomain hadoop.hdfs.TestFileLengthOnClusterRestart hadoop.hdfs.TestDFSInotifyEventInputStream hadoop.hdfs.qjournal.server.TestJournalNodeRespectsBindHostKeys hadoop.hdfs.server.federation.router.TestRouterQuota hadoop.hdfs.server.federation.router.TestRouterNamenodeHeartbeat hadoop.hdfs.server.federation.resolver.order.TestLocalResolver hadoop.hdfs.server.federation.resolver.TestMultipleDestinationResolver hadoop.contrib.bkjournal.TestBookKeeperHACheckpoints hadoop.mapreduce.lib.input.TestLineRecordReader hadoop.mapred.TestLineRecordReader hadoop.mapreduce.jobhistory.TestHistoryViewerPrinter hadoop.resourceestimator.service.TestResourceEstimatorService hadoop.resourceestimator.solver.impl.TestLpSolver hadoop.yarn.sls.TestSLSRunner hadoop.yarn.server.nodemanager.containermanager.linux.resources.TestNumaResourceAllocator hadoop.yarn.server.nodemanager.containermanager.linux.resources.TestNumaResourceHandlerImpl hadoop.yarn.server.resourcemanager.TestClientRMService hadoop.yarn.server.resourcemanager.recovery.TestFSRMStateStore hadoop.yarn.server.resourcemanager.monitor.invariants.TestMetricsInvariantChecker cc: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1300/artifact/out/diff-compile-cc-root.txt [4.0K] javac: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1300/artifact/out/diff-compile-javac-root.txt [488K] checkstyle: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1300/artifact/out/diff-checkstyle-root.txt [14M] hadolint: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1300/artifact/out/diff-patch-hadolint.txt [4.0K] mvnsite: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1300/artifact/out/patch-mvnsite-root.txt [572K] pathlen: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1300/artifact/out/pathlen.txt [12K] pylint: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1300/artifact/out/diff-patch-pylint.txt [20K] shellcheck: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1300/artifact/out/diff-patch-shellcheck.txt [72K] whitespace: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1300/artifact/out/whitespace-eol.txt [12M] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1300/artifact/out/whitespace-tabs.txt [1.3M] javadoc: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1300/artifact/out/patch-javadoc-root.txt [36K] unit: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1300/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt [220K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1300/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt [452K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1300/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt [36K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1300/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs_src_contrib_bkjournal.txt [16K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1300/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt [104K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1300/artifact/out/patch-unit-hadoop-tools_hadoop-azure.txt [20K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1300/artifact/out/patch-unit-hadoop-tools_hadoop-resourceestimator.txt [16K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1300/artifact/out/patch-unit-hadoop-tools_hadoop-sls.txt [28K]
