[ANNOUNCE] Apache Hadoop 3.2.4 release
On behalf of the Apache Hadoop Project Management Committee, I'm pleased to announce that Hadoop 3.2.4 has been released. This is the third stable release of Apache Hadoop 3.2 line. It contains 153 bug fixes, improvements and enhancements since 3.2.3. Users are encouraged to read the overview of major changes[1] since 3.2.3. For details of 153 bug fixes, improvements, and other enhancements since the previous 3.2.3 release, please check release notes[2] and changelog[3]. [1]: https://hadoop.apache.org/docs/r3.2.4/index.html [2]: https://hadoop.apache.org/docs/r3.2.4/hadoop-project-dist/hadoop-common/release/3.2.4/RELEASENOTES.3.2.4.html [3]: https://hadoop.apache.org/docs/r3.2.4/hadoop-project-dist/hadoop-common/release/3.2.4/CHANGELOG.3.2.4.html As the release notes highlight, this release contains HADOOP-18088 "Replace log4j 1.x with reload4j" https://issues.apache.org/jira/browse/HADOOP-18088 This ensures that the version of log4j shipped is free of known CVEs. the standard log4j 1.2.17 has some known CVEs in classes which were never uses; reload4j cuts them out. Audit scanning tools should stop highlighting perceived risks here. If you are using maven exclusions to manage logging libraries, or were otherwise replacing the log4j artifacts in deployments, note the different library/artifact names which need to be handled. Many thanks to everyone who helped in this release by supplying patches, reviewing them, helping get this release building and testing reviewing the final artifacts. Masatake Iwasaki - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
Apache Hadoop qbt Report: branch-3.3+JDK8 on Linux/x86_64
For more details, see https://ci-hadoop.apache.org/job/hadoop-qbt-branch-3.3-java8-linux-x86_64/65/ [Jul 14, 2022, 12:54:15 PM] (noreply) HADOOP-18074 - Partial/Incomplete groups list can be returned in LDAP. (#4503) [Jul 16, 2022, 5:18:52 PM] (noreply) HADOOP-18332. Remove rs-api dependency by downgrading jackson to 2.12.7. (#4552) -1 overall The following subsystems voted -1: blanks pathlen unit xml The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: XML : Parsing Error(s): hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-excerpt.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags2.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-sample-output.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/fair-scheduler-invalid.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/yarn-site-with-invalid-allocation-file-ref.xml Failed junit tests : hadoop.hdfs.server.namenode.TestReconstructStripedBlocks hadoop.hdfs.server.balancer.TestBalancerRPCDelay hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA hadoop.hdfs.server.namenode.ha.TestUpdateBlockTailing hadoop.hdfs.server.federation.router.TestRouterRpc hadoop.hdfs.server.federation.router.TestRouterRpcMultiDestination hadoop.yarn.server.resourcemanager.TestLeaderElectorService cc: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-3.3-java8-linux-x86_64/65/artifact/out/results-compile-cc-root.txt [48K] javac: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-3.3-java8-linux-x86_64/65/artifact/out/results-compile-javac-root.txt [376K] blanks: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-3.3-java8-linux-x86_64/65/artifact/out/blanks-eol.txt [13M] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-3.3-java8-linux-x86_64/65/artifact/out/blanks-tabs.txt [2.0M] checkstyle: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-3.3-java8-linux-x86_64/65/artifact/out/results-checkstyle-root.txt [14M] pathlen: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-3.3-java8-linux-x86_64/65/artifact/out/results-pathlen.txt [16K] pylint: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-3.3-java8-linux-x86_64/65/artifact/out/results-pylint.txt [20K] shellcheck: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-3.3-java8-linux-x86_64/65/artifact/out/results-shellcheck.txt [20K] xml: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-3.3-java8-linux-x86_64/65/artifact/out/xml.txt [24K] javadoc: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-3.3-java8-linux-x86_64/65/artifact/out/results-javadoc-javadoc-root.txt [1.1M] unit: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-3.3-java8-linux-x86_64/65/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt [548K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-3.3-java8-linux-x86_64/65/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt [60K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-3.3-java8-linux-x86_64/65/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt [168K] Powered by Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
Apache Hadoop qbt Report: trunk+JDK11 on Linux/x86_64
For more details, see https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java11-linux-x86_64/337/ [Jul 19, 2022, 4:07:22 PM] (noreply) HDFS-16464. Create only libhdfspp static libraries for Windows (#4571) [Jul 19, 2022, 4:09:06 PM] (noreply) HDFS-16665. Fix duplicate sources for HDFS test (#4573) [Jul 20, 2022, 4:15:39 PM] (noreply) HADOOP-18333.Upgrade jetty version to 9.4.48.v20220622 (#4553) [Jul 20, 2022, 4:27:28 PM] (noreply) HDFS-16667. Use malloc for buffer allocation in uriparser2 (#4576) -1 overall The following subsystems voted -1: blanks pathlen spotbugs unit xml The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: XML : Parsing Error(s): hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-excerpt.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags2.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-sample-output.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/fair-scheduler-invalid.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/yarn-site-with-invalid-allocation-file-ref.xml spotbugs : module:hadoop-hdfs-project/hadoop-hdfs Redundant nullcheck of oldLock, which is known to be non-null in org.apache.hadoop.hdfs.server.datanode.DataStorage.isPreUpgradableLayout(Storage$StorageDirectory)) Redundant null check at DataStorage.java:is known to be non-null in org.apache.hadoop.hdfs.server.datanode.DataStorage.isPreUpgradableLayout(Storage$StorageDirectory)) Redundant null check at DataStorage.java:[line 695] Redundant nullcheck of metaChannel, which is known to be non-null in org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.MappableBlockLoader.verifyChecksum(long, FileInputStream, FileChannel, String) Redundant null check at MappableBlockLoader.java:is known to be non-null in org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.MappableBlockLoader.verifyChecksum(long, FileInputStream, FileChannel, String) Redundant null check at MappableBlockLoader.java:[line 138] Redundant nullcheck of blockChannel, which is known to be non-null in org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.MemoryMappableBlockLoader.load(long, FileInputStream, FileInputStream, String, ExtendedBlockId) Redundant null check at MemoryMappableBlockLoader.java:is known to be non-null in org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.MemoryMappableBlockLoader.load(long, FileInputStream, FileInputStream, String, ExtendedBlockId) Redundant null check at MemoryMappableBlockLoader.java:[line 75] Redundant nullcheck of blockChannel, which is known to be non-null in org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.NativePmemMappableBlockLoader.load(long, FileInputStream, FileInputStream, String, ExtendedBlockId) Redundant null check at NativePmemMappableBlockLoader.java:is known to be non-null in org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.NativePmemMappableBlockLoader.load(long, FileInputStream, FileInputStream, String, ExtendedBlockId) Redundant null check at NativePmemMappableBlockLoader.java:[line 85] Redundant nullcheck of metaChannel, which is known to be non-null in org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.NativePmemMappableBlockLoader.verifyChecksumAndMapBlock(NativeIO$POSIX$$PmemMappedRegion,, long, FileInputStream, FileChannel, String) Redundant null check at NativePmemMappableBlockLoader.java:is known to be non-null in org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.NativePmemMappableBlockLoader.verifyChecksumAndMapBlock(NativeIO$POSIX$$PmemMappedRegion,, long, FileInputStream, FileChannel, String) Redundant null check at NativePmemMappableBlockLoader.java:[line 130] org.apache.hadoop.hdfs.server.namenode.top.window.RollingWindowManager$UserCounts doesn't override java.util.ArrayList.equals(Object) At RollingWindowManager.java:At RollingWindowManager.java:[line 1] spotbugs : module:hadoop-yarn-project/hadoop-yarn Redundant nullcheck of it, which is known to be non-null in org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService.recoverTrackerResources(LocalResourcesTracker, NMStateStoreService$LocalResourceTrackerState)) Redundant null check at ResourceLocalizationService.java:is
Re: [VOTE] Release Apache Hadoop 3.2.4 - RC0
The vote passed with the following result. Binding PMC members: +1 Chris Nauroth +1 Ayush Saxena +1 Masatake Iwasaki Non binding votes: +1 Ashutosh Gupta I'm going to publish the artifacts, site docs, maven artifacts, then announce. Thank you to all who helped to push this release out. Masatake Iwasaki On 2022/07/22 4:58, Ayush Saxena wrote: +1(Binding) * Built from Source * Successful native build on Ubuntu 18.04 * Verified Checksums * Verified Signatures * Successful RAT check * Ran Basic HDFS shell commands * Ran basic MR example Jobs (TeraGen/TeraSort & TeraValidate) * Browsed through UI(NN, DN, RM, NM & JHS) * Skimmed through the contents of ChangeLog & ReleaseNotes. Look Good Thanx Masatake for driving the release, Good Luck!!! -Ayush On Thu, 21 Jul 2022 at 23:28, Chris Nauroth wrote: I'm changing my vote to +1 (binding). Masatake and Ashutosh, thank you for investigating. I reran tests without the parallel options, and that mostly addressed the failures. Maybe the tests in question are just not sufficiently isolated to support parallel execution. That looks to be the case for TestFsck, where the failure was caused by missing audit log entries. This test works by toggling global logging state, so I can see why multi-threaded execution might confuse the test. Chris Nauroth On Thu, Jul 21, 2022 at 12:01 AM Ashutosh Gupta < ashutoshgupta...@gmail.com> wrote: +1(non-binding) * Builds from source look good. * Checksums and signatures are correct. * Running basic HDFS and MapReduce commands looks good. * TestAMRMProxy - Not able to reproduce in local * TestFsck - I can see failure only I can see is TestFsck.testFsckListCorruptSnapshotFiles which passed after applying HDFS-15038 * TestSLSStreamAMSynth - Not able to reproduce in local * TestServiceAM - Not able to reproduce in local Thanks Masatake for driving this release. On Thu, Jul 21, 2022 at 5:51 AM Masatake Iwasaki < iwasak...@oss.nttdata.com> wrote: Hi developers, I'm still waiting for your vote. I'm considering the intermittent test failures mentioned by Chris are not blocker. Please file a JIRA and let me know if you find a blocker issue. I will appreciate your help for the release process. Regards, Masatake Iwasaki On 2022/07/20 14:50, Masatake Iwasaki wrote: TestServiceAM I can see the reported failure of TestServiceAM in some "Apache Hadoop qbt Report: branch-3.2+JDK8 on Linux/x86_64". 3.3.0 and above might be fixed by YARN-8867 which added guard using GenericTestUtils#waitFor for stabilizing the testContainersReleasedWhenPreLaunchFails. YARN 8867 did not modified other code under hadoop-yarn-services. If it is the case, TestServiceAM can be tagged as flaky in branch-3.2. On 2022/07/20 14:21, Masatake Iwasaki wrote: Thanks for testing the RC0, Chris. The following are new test failures for me on 3.2.4: * TestAMRMProxy * TestFsck * TestSLSStreamAMSynth * TestServiceAM I could not reproduce the test failures on my local. For TestFsck, if the failed test case is testFsckListCorruptSnapshotFiles, cherry-picking HDFS-15038 (fixing only test code) could be the fix. The failure of TestSLSStreamAMSynth looks frequently reported by "Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86_64". It could be tagged as known flaky test. On 2022/07/20 9:15, Chris Nauroth wrote: -0 (binding) * Verified all checksums. * Verified all signatures. * Built from source, including native code on Linux. * mvn clean package -Pnative -Psrc -Drequire.openssl -Drequire.snappy -Drequire.zstd -DskipTests * Tests mostly passed, but see below. * mvn --fail-never clean test -Pnative -Dparallel-tests -Drequire.snappy -Drequire.zstd -Drequire.openssl -Dsurefire.rerunFailingTestsCount=3 -DtestsThreadCount=8 The following are new test failures for me on 3.2.4: * TestAMRMProxy * TestFsck * TestSLSStreamAMSynth * TestServiceAM The following tests also failed, but they also fail for me on 3.2.3, so they aren't likely to be related to this release candidate: * TestCapacitySchedulerNodeLabelUpdate * TestFrameworkUploader * TestSLSGenericSynth * TestSLSRunner * test_libhdfs_threaded_hdfspp_test_shim_static I'm not voting a full -1, because I haven't done any root cause analysis on these new test failures. I don't know if it's a quirk to my environment, though I'm using the start-build-env.sh Docker container, so any build dependencies should be consistent. I'd be comfortable moving ahead if others are seeing these tests pass. Chris Nauroth On Thu, Jul 14, 2022 at 7:57 AM Masatake Iwasaki < iwasak...@oss.nttdata.com> wrote: +1 from myself. * skimmed the contents of site documentation. * built the source tarball on Rocky Linux 8 (x86_64) by OpenJDK 8 with `-Pnative`. * launched pseudo distributed cluster including kms and httpfs with Kerberos and SSL enabled. * created encryption zone, put and read files via httpfs. * ran example MR wordcount over encryption zone. * launched
[jira] [Created] (HADOOP-18354) upgrade reload4j due to XXE vulnerability
PJ Fanning created HADOOP-18354: --- Summary: upgrade reload4j due to XXE vulnerability Key: HADOOP-18354 URL: https://issues.apache.org/jira/browse/HADOOP-18354 Project: Hadoop Common Issue Type: Improvement Reporter: PJ Fanning https://github.com/qos-ch/reload4j/issues/53 fixed in reload4j 1.2.22 -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86_64
For more details, see https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/928/ [Jul 20, 2022, 4:15:39 PM] (noreply) HADOOP-18333.Upgrade jetty version to 9.4.48.v20220622 (#4553) [Jul 20, 2022, 4:27:28 PM] (noreply) HDFS-16667. Use malloc for buffer allocation in uriparser2 (#4576) -1 overall The following subsystems voted -1: blanks pathlen xml The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: XML : Parsing Error(s): hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-excerpt.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags2.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-sample-output.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/fair-scheduler-invalid.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/yarn-site-with-invalid-allocation-file-ref.xml cc: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/928/artifact/out/results-compile-cc-root.txt [96K] javac: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/928/artifact/out/results-compile-javac-root.txt [540K] blanks: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/928/artifact/out/blanks-eol.txt [13M] https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/928/artifact/out/blanks-tabs.txt [2.0M] checkstyle: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/928/artifact/out/results-checkstyle-root.txt [14M] pathlen: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/928/artifact/out/results-pathlen.txt [16K] pylint: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/928/artifact/out/results-pylint.txt [20K] shellcheck: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/928/artifact/out/results-shellcheck.txt [28K] xml: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/928/artifact/out/xml.txt [24K] javadoc: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/928/artifact/out/results-javadoc-javadoc-root.txt [400K] Powered by Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
Re: [VOTE] Release Apache Hadoop 3.2.4 - RC0
+1(Binding) * Built from Source * Successful native build on Ubuntu 18.04 * Verified Checksums * Verified Signatures * Successful RAT check * Ran Basic HDFS shell commands * Ran basic MR example Jobs (TeraGen/TeraSort & TeraValidate) * Browsed through UI(NN, DN, RM, NM & JHS) * Skimmed through the contents of ChangeLog & ReleaseNotes. Look Good Thanx Masatake for driving the release, Good Luck!!! -Ayush On Thu, 21 Jul 2022 at 23:28, Chris Nauroth wrote: > I'm changing my vote to +1 (binding). > > Masatake and Ashutosh, thank you for investigating. > > I reran tests without the parallel options, and that mostly addressed the > failures. Maybe the tests in question are just not sufficiently isolated to > support parallel execution. That looks to be the case for TestFsck, where > the failure was caused by missing audit log entries. This test works by > toggling global logging state, so I can see why multi-threaded execution > might confuse the test. > > Chris Nauroth > > > On Thu, Jul 21, 2022 at 12:01 AM Ashutosh Gupta < > ashutoshgupta...@gmail.com> > wrote: > > > +1(non-binding) > > > > * Builds from source look good. > > * Checksums and signatures are correct. > > * Running basic HDFS and MapReduce commands looks good. > > > > > * TestAMRMProxy - Not able to reproduce in local > > > * TestFsck - I can see failure only I can see is > > TestFsck.testFsckListCorruptSnapshotFiles which passed after applying > > HDFS-15038 > > > * TestSLSStreamAMSynth - Not able to reproduce in local > > > * TestServiceAM - Not able to reproduce in local > > > > Thanks Masatake for driving this release. > > > > On Thu, Jul 21, 2022 at 5:51 AM Masatake Iwasaki < > > iwasak...@oss.nttdata.com> > > wrote: > > > > > Hi developers, > > > > > > I'm still waiting for your vote. > > > I'm considering the intermittent test failures mentioned by Chris are > not > > > blocker. > > > Please file a JIRA and let me know if you find a blocker issue. > > > > > > I will appreciate your help for the release process. > > > > > > Regards, > > > Masatake Iwasaki > > > > > > On 2022/07/20 14:50, Masatake Iwasaki wrote: > > > >> TestServiceAM > > > > > > > > I can see the reported failure of TestServiceAM in some "Apache > Hadoop > > > qbt Report: branch-3.2+JDK8 on Linux/x86_64". > > > > 3.3.0 and above might be fixed by YARN-8867 which added guard using > > > GenericTestUtils#waitFor for stabilizing the > > > testContainersReleasedWhenPreLaunchFails. > > > > YARN 8867 did not modified other code under hadoop-yarn-services. > > > > If it is the case, TestServiceAM can be tagged as flaky in > branch-3.2. > > > > > > > > > > > > On 2022/07/20 14:21, Masatake Iwasaki wrote: > > > >> Thanks for testing the RC0, Chris. > > > >> > > > >>> The following are new test failures for me on 3.2.4: > > > >>> * TestAMRMProxy > > > >>> * TestFsck > > > >>> * TestSLSStreamAMSynth > > > >>> * TestServiceAM > > > >> > > > >> I could not reproduce the test failures on my local. > > > >> > > > >> For TestFsck, if the failed test case is > > > testFsckListCorruptSnapshotFiles, > > > >> cherry-picking HDFS-15038 (fixing only test code) could be the fix. > > > >> > > > >> The failure of TestSLSStreamAMSynth looks frequently reported by > > > >> "Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86_64". > > > >> It could be tagged as known flaky test. > > > >> > > > >> On 2022/07/20 9:15, Chris Nauroth wrote: > > > >>> -0 (binding) > > > >>> > > > >>> * Verified all checksums. > > > >>> * Verified all signatures. > > > >>> * Built from source, including native code on Linux. > > > >>> * mvn clean package -Pnative -Psrc -Drequire.openssl > > > -Drequire.snappy > > > >>> -Drequire.zstd -DskipTests > > > >>> * Tests mostly passed, but see below. > > > >>> * mvn --fail-never clean test -Pnative -Dparallel-tests > > > >>> -Drequire.snappy -Drequire.zstd -Drequire.openssl > > > >>> -Dsurefire.rerunFailingTestsCount=3 -DtestsThreadCount=8 > > > >>> > > > >>> The following are new test failures for me on 3.2.4: > > > >>> * TestAMRMProxy > > > >>> * TestFsck > > > >>> * TestSLSStreamAMSynth > > > >>> * TestServiceAM > > > >>> > > > >>> The following tests also failed, but they also fail for me on > 3.2.3, > > so > > > >>> they aren't likely to be related to this release candidate: > > > >>> * TestCapacitySchedulerNodeLabelUpdate > > > >>> * TestFrameworkUploader > > > >>> * TestSLSGenericSynth > > > >>> * TestSLSRunner > > > >>> * test_libhdfs_threaded_hdfspp_test_shim_static > > > >>> > > > >>> I'm not voting a full -1, because I haven't done any root cause > > > analysis on > > > >>> these new test failures. I don't know if it's a quirk to my > > > environment, > > > >>> though I'm using the start-build-env.sh Docker container, so any > > build > > > >>> dependencies should be consistent. I'd be comfortable moving ahead > if > > > >>> others are seeing these tests pass. > > > >>> > > > >>> Chris Nauroth > > > >>> > > > >>> > > > >>> On Thu, Jul 14, 2022 at 7:57 AM
[VOTE] Release Apache Hadoop 3.3.4
I have put together a release candidate (RC0) for Hadoop 3.3.4 The RC is available at: https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.4-RC0/ The git tag is release-3.3.4-RC0, commit c679bc76d26 The maven artifacts are staged at https://repository.apache.org/content/repositories/orgapachehadoop-1356/ You can find my public key at: https://dist.apache.org/repos/dist/release/hadoop/common/KEYS Change log https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.4-RC0/CHANGELOG.md Release notes https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.4-RC0/RELEASENOTES.md There's a very small number of changes, primarily critical code/packaging issues and security fixes. See the release notes for details. Please try the release and vote. The vote will run for 5 days. -Steve
Re: [VOTE] Release Apache Hadoop 3.2.4 - RC0
I'm changing my vote to +1 (binding). Masatake and Ashutosh, thank you for investigating. I reran tests without the parallel options, and that mostly addressed the failures. Maybe the tests in question are just not sufficiently isolated to support parallel execution. That looks to be the case for TestFsck, where the failure was caused by missing audit log entries. This test works by toggling global logging state, so I can see why multi-threaded execution might confuse the test. Chris Nauroth On Thu, Jul 21, 2022 at 12:01 AM Ashutosh Gupta wrote: > +1(non-binding) > > * Builds from source look good. > * Checksums and signatures are correct. > * Running basic HDFS and MapReduce commands looks good. > > > * TestAMRMProxy - Not able to reproduce in local > > * TestFsck - I can see failure only I can see is > TestFsck.testFsckListCorruptSnapshotFiles which passed after applying > HDFS-15038 > > * TestSLSStreamAMSynth - Not able to reproduce in local > > * TestServiceAM - Not able to reproduce in local > > Thanks Masatake for driving this release. > > On Thu, Jul 21, 2022 at 5:51 AM Masatake Iwasaki < > iwasak...@oss.nttdata.com> > wrote: > > > Hi developers, > > > > I'm still waiting for your vote. > > I'm considering the intermittent test failures mentioned by Chris are not > > blocker. > > Please file a JIRA and let me know if you find a blocker issue. > > > > I will appreciate your help for the release process. > > > > Regards, > > Masatake Iwasaki > > > > On 2022/07/20 14:50, Masatake Iwasaki wrote: > > >> TestServiceAM > > > > > > I can see the reported failure of TestServiceAM in some "Apache Hadoop > > qbt Report: branch-3.2+JDK8 on Linux/x86_64". > > > 3.3.0 and above might be fixed by YARN-8867 which added guard using > > GenericTestUtils#waitFor for stabilizing the > > testContainersReleasedWhenPreLaunchFails. > > > YARN 8867 did not modified other code under hadoop-yarn-services. > > > If it is the case, TestServiceAM can be tagged as flaky in branch-3.2. > > > > > > > > > On 2022/07/20 14:21, Masatake Iwasaki wrote: > > >> Thanks for testing the RC0, Chris. > > >> > > >>> The following are new test failures for me on 3.2.4: > > >>> * TestAMRMProxy > > >>> * TestFsck > > >>> * TestSLSStreamAMSynth > > >>> * TestServiceAM > > >> > > >> I could not reproduce the test failures on my local. > > >> > > >> For TestFsck, if the failed test case is > > testFsckListCorruptSnapshotFiles, > > >> cherry-picking HDFS-15038 (fixing only test code) could be the fix. > > >> > > >> The failure of TestSLSStreamAMSynth looks frequently reported by > > >> "Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86_64". > > >> It could be tagged as known flaky test. > > >> > > >> On 2022/07/20 9:15, Chris Nauroth wrote: > > >>> -0 (binding) > > >>> > > >>> * Verified all checksums. > > >>> * Verified all signatures. > > >>> * Built from source, including native code on Linux. > > >>> * mvn clean package -Pnative -Psrc -Drequire.openssl > > -Drequire.snappy > > >>> -Drequire.zstd -DskipTests > > >>> * Tests mostly passed, but see below. > > >>> * mvn --fail-never clean test -Pnative -Dparallel-tests > > >>> -Drequire.snappy -Drequire.zstd -Drequire.openssl > > >>> -Dsurefire.rerunFailingTestsCount=3 -DtestsThreadCount=8 > > >>> > > >>> The following are new test failures for me on 3.2.4: > > >>> * TestAMRMProxy > > >>> * TestFsck > > >>> * TestSLSStreamAMSynth > > >>> * TestServiceAM > > >>> > > >>> The following tests also failed, but they also fail for me on 3.2.3, > so > > >>> they aren't likely to be related to this release candidate: > > >>> * TestCapacitySchedulerNodeLabelUpdate > > >>> * TestFrameworkUploader > > >>> * TestSLSGenericSynth > > >>> * TestSLSRunner > > >>> * test_libhdfs_threaded_hdfspp_test_shim_static > > >>> > > >>> I'm not voting a full -1, because I haven't done any root cause > > analysis on > > >>> these new test failures. I don't know if it's a quirk to my > > environment, > > >>> though I'm using the start-build-env.sh Docker container, so any > build > > >>> dependencies should be consistent. I'd be comfortable moving ahead if > > >>> others are seeing these tests pass. > > >>> > > >>> Chris Nauroth > > >>> > > >>> > > >>> On Thu, Jul 14, 2022 at 7:57 AM Masatake Iwasaki < > > iwasak...@oss.nttdata.com> > > >>> wrote: > > >>> > > +1 from myself. > > > > * skimmed the contents of site documentation. > > > > * built the source tarball on Rocky Linux 8 (x86_64) by OpenJDK 8 > with > > `-Pnative`. > > > > * launched pseudo distributed cluster including kms and httpfs with > > Kerberos and SSL enabled. > > > > * created encryption zone, put and read files via httpfs. > > * ran example MR wordcount over encryption zone. > > > > * launched 3-node docker cluster with NN-HA and RM-HA enabled and > ran > > some > > example MR jobs. > > > > * built HBase 2.4.11, Hive 3.1.2 and Spark 3.1.2 against
[jira] [Resolved] (HADOOP-18221) stream warns Not all bytes were read from the S3ObjectInputStream when closed
[ https://issues.apache.org/jira/browse/HADOOP-18221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ahmar Suhail resolved HADOOP-18221. --- Resolution: Fixed > stream warns Not all bytes were read from the S3ObjectInputStream when closed > - > > Key: HADOOP-18221 > URL: https://issues.apache.org/jira/browse/HADOOP-18221 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Ahmar Suhail >Assignee: Ahmar Suhail >Priority: Minor > Labels: pull-request-available > Time Spent: 1h 40m > Remaining Estimate: 0h > > Issue: [https://github.com/aws/aws-sdk-java/issues/1211] has resurfaced in > the prefetching stream when it is closed before reading for blocks is > complete. This can be fixed by draining the stream before closing -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-18247) Tests in ITestS3AOpenCost are failing
[ https://issues.apache.org/jira/browse/HADOOP-18247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ahmar Suhail resolved HADOOP-18247. --- Resolution: Fixed > Tests in ITestS3AOpenCost are failing > - > > Key: HADOOP-18247 > URL: https://issues.apache.org/jira/browse/HADOOP-18247 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Ahmar Suhail >Assignee: Ahmar Suhail >Priority: Minor > > After rebasing, when prefetching is enabled testOpenFileLongerLength & > testOpenFileShorterLength fail -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
Apache Hadoop qbt Report: branch-2.10+JDK7 on Linux/x86_64
For more details, see https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/729/ No changes -1 overall The following subsystems voted -1: hadolint mvnsite pathlen unit The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck whitespace The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: Failed junit tests : hadoop.fs.TestTrash hadoop.fs.TestFileUtil hadoop.hdfs.qjournal.server.TestJournalNodeRespectsBindHostKeys hadoop.hdfs.server.blockmanagement.TestReplicationPolicyWithUpgradeDomain hadoop.contrib.bkjournal.TestBookKeeperHACheckpoints hadoop.contrib.bkjournal.TestBookKeeperHACheckpoints hadoop.hdfs.server.federation.router.TestRouterNamenodeHeartbeat hadoop.hdfs.server.federation.router.TestRouterQuota hadoop.hdfs.server.federation.resolver.TestMultipleDestinationResolver hadoop.hdfs.server.federation.resolver.order.TestLocalResolver hadoop.yarn.server.resourcemanager.TestClientRMService hadoop.yarn.server.resourcemanager.monitor.invariants.TestMetricsInvariantChecker hadoop.yarn.applications.distributedshell.TestDistributedShell hadoop.mapreduce.jobhistory.TestHistoryViewerPrinter hadoop.mapreduce.lib.input.TestLineRecordReader hadoop.mapred.TestLineRecordReader hadoop.mapreduce.v2.TestSpeculativeExecution hadoop.mapreduce.v2.TestMRJobsWithProfiler hadoop.yarn.sls.TestSLSRunner hadoop.resourceestimator.solver.impl.TestLpSolver hadoop.resourceestimator.service.TestResourceEstimatorService cc: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/729/artifact/out/diff-compile-cc-root.txt [4.0K] javac: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/729/artifact/out/diff-compile-javac-root.txt [488K] checkstyle: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/729/artifact/out/diff-checkstyle-root.txt [14M] hadolint: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/729/artifact/out/diff-patch-hadolint.txt [4.0K] mvnsite: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/729/artifact/out/patch-mvnsite-root.txt [568K] pathlen: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/729/artifact/out/pathlen.txt [12K] pylint: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/729/artifact/out/diff-patch-pylint.txt [20K] shellcheck: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/729/artifact/out/diff-patch-shellcheck.txt [72K] whitespace: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/729/artifact/out/whitespace-eol.txt [12M] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/729/artifact/out/whitespace-tabs.txt [1.3M] javadoc: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/729/artifact/out/patch-javadoc-root.txt [40K] unit: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/729/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt [224K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/729/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt [428K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/729/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs_src_contrib_bkjournal.txt [16K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/729/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt [36K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/729/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt [20K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/729/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt [116K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/729/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-applications_hadoop-yarn-applications-distributedshell.txt [12K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/729/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt [104K]
Re: [VOTE] Release Apache Hadoop 3.2.4 - RC0
+1(non-binding) * Builds from source look good. * Checksums and signatures are correct. * Running basic HDFS and MapReduce commands looks good. > * TestAMRMProxy - Not able to reproduce in local > * TestFsck - I can see failure only I can see is TestFsck.testFsckListCorruptSnapshotFiles which passed after applying HDFS-15038 > * TestSLSStreamAMSynth - Not able to reproduce in local > * TestServiceAM - Not able to reproduce in local Thanks Masatake for driving this release. On Thu, Jul 21, 2022 at 5:51 AM Masatake Iwasaki wrote: > Hi developers, > > I'm still waiting for your vote. > I'm considering the intermittent test failures mentioned by Chris are not > blocker. > Please file a JIRA and let me know if you find a blocker issue. > > I will appreciate your help for the release process. > > Regards, > Masatake Iwasaki > > On 2022/07/20 14:50, Masatake Iwasaki wrote: > >> TestServiceAM > > > > I can see the reported failure of TestServiceAM in some "Apache Hadoop > qbt Report: branch-3.2+JDK8 on Linux/x86_64". > > 3.3.0 and above might be fixed by YARN-8867 which added guard using > GenericTestUtils#waitFor for stabilizing the > testContainersReleasedWhenPreLaunchFails. > > YARN 8867 did not modified other code under hadoop-yarn-services. > > If it is the case, TestServiceAM can be tagged as flaky in branch-3.2. > > > > > > On 2022/07/20 14:21, Masatake Iwasaki wrote: > >> Thanks for testing the RC0, Chris. > >> > >>> The following are new test failures for me on 3.2.4: > >>> * TestAMRMProxy > >>> * TestFsck > >>> * TestSLSStreamAMSynth > >>> * TestServiceAM > >> > >> I could not reproduce the test failures on my local. > >> > >> For TestFsck, if the failed test case is > testFsckListCorruptSnapshotFiles, > >> cherry-picking HDFS-15038 (fixing only test code) could be the fix. > >> > >> The failure of TestSLSStreamAMSynth looks frequently reported by > >> "Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86_64". > >> It could be tagged as known flaky test. > >> > >> On 2022/07/20 9:15, Chris Nauroth wrote: > >>> -0 (binding) > >>> > >>> * Verified all checksums. > >>> * Verified all signatures. > >>> * Built from source, including native code on Linux. > >>> * mvn clean package -Pnative -Psrc -Drequire.openssl > -Drequire.snappy > >>> -Drequire.zstd -DskipTests > >>> * Tests mostly passed, but see below. > >>> * mvn --fail-never clean test -Pnative -Dparallel-tests > >>> -Drequire.snappy -Drequire.zstd -Drequire.openssl > >>> -Dsurefire.rerunFailingTestsCount=3 -DtestsThreadCount=8 > >>> > >>> The following are new test failures for me on 3.2.4: > >>> * TestAMRMProxy > >>> * TestFsck > >>> * TestSLSStreamAMSynth > >>> * TestServiceAM > >>> > >>> The following tests also failed, but they also fail for me on 3.2.3, so > >>> they aren't likely to be related to this release candidate: > >>> * TestCapacitySchedulerNodeLabelUpdate > >>> * TestFrameworkUploader > >>> * TestSLSGenericSynth > >>> * TestSLSRunner > >>> * test_libhdfs_threaded_hdfspp_test_shim_static > >>> > >>> I'm not voting a full -1, because I haven't done any root cause > analysis on > >>> these new test failures. I don't know if it's a quirk to my > environment, > >>> though I'm using the start-build-env.sh Docker container, so any build > >>> dependencies should be consistent. I'd be comfortable moving ahead if > >>> others are seeing these tests pass. > >>> > >>> Chris Nauroth > >>> > >>> > >>> On Thu, Jul 14, 2022 at 7:57 AM Masatake Iwasaki < > iwasak...@oss.nttdata.com> > >>> wrote: > >>> > +1 from myself. > > * skimmed the contents of site documentation. > > * built the source tarball on Rocky Linux 8 (x86_64) by OpenJDK 8 with > `-Pnative`. > > * launched pseudo distributed cluster including kms and httpfs with > Kerberos and SSL enabled. > > * created encryption zone, put and read files via httpfs. > * ran example MR wordcount over encryption zone. > > * launched 3-node docker cluster with NN-HA and RM-HA enabled and ran > some > example MR jobs. > > * built HBase 2.4.11, Hive 3.1.2 and Spark 3.1.2 against Hadoop 3.2.4 > RC0 > on CentOS 7 (x86_64) by using Bigtop branch-3.1 and ran > smoke-tests. > https://github.com/apache/bigtop/pull/942 > > * Hive needs updating exclusion rule to address HADOOP-18088 > (migration > to reload4j). > > * built Spark 3.3.0 against Hadoop 3.2.4 RC0 using the staging > repository:: > > > staged > staged-releases > > > https://repository.apache.org/content/repositories/orgapachehadoop-1354 > > > true > > > true > > > > Thanks, > Masatake Iwasaki > > On 2022/07/13 1:14, Masatake Iwasaki wrote: > > Hi all, > > > > Here's Hadoop 3.2.4 release
