Re: [VOTE] Release Apache Hadoop 3.2.4 - RC0
The vote passed with the following result. Binding PMC members: +1 Chris Nauroth +1 Ayush Saxena +1 Masatake Iwasaki Non binding votes: +1 Ashutosh Gupta I'm going to publish the artifacts, site docs, maven artifacts, then announce. Thank you to all who helped to push this release out. Masatake Iwasaki On 2022/07/22 4:58, Ayush Saxena wrote: +1(Binding) * Built from Source * Successful native build on Ubuntu 18.04 * Verified Checksums * Verified Signatures * Successful RAT check * Ran Basic HDFS shell commands * Ran basic MR example Jobs (TeraGen/TeraSort & TeraValidate) * Browsed through UI(NN, DN, RM, NM & JHS) * Skimmed through the contents of ChangeLog & ReleaseNotes. Look Good Thanx Masatake for driving the release, Good Luck!!! -Ayush On Thu, 21 Jul 2022 at 23:28, Chris Nauroth wrote: I'm changing my vote to +1 (binding). Masatake and Ashutosh, thank you for investigating. I reran tests without the parallel options, and that mostly addressed the failures. Maybe the tests in question are just not sufficiently isolated to support parallel execution. That looks to be the case for TestFsck, where the failure was caused by missing audit log entries. This test works by toggling global logging state, so I can see why multi-threaded execution might confuse the test. Chris Nauroth On Thu, Jul 21, 2022 at 12:01 AM Ashutosh Gupta < ashutoshgupta...@gmail.com> wrote: +1(non-binding) * Builds from source look good. * Checksums and signatures are correct. * Running basic HDFS and MapReduce commands looks good. * TestAMRMProxy - Not able to reproduce in local * TestFsck - I can see failure only I can see is TestFsck.testFsckListCorruptSnapshotFiles which passed after applying HDFS-15038 * TestSLSStreamAMSynth - Not able to reproduce in local * TestServiceAM - Not able to reproduce in local Thanks Masatake for driving this release. On Thu, Jul 21, 2022 at 5:51 AM Masatake Iwasaki < iwasak...@oss.nttdata.com> wrote: Hi developers, I'm still waiting for your vote. I'm considering the intermittent test failures mentioned by Chris are not blocker. Please file a JIRA and let me know if you find a blocker issue. I will appreciate your help for the release process. Regards, Masatake Iwasaki On 2022/07/20 14:50, Masatake Iwasaki wrote: TestServiceAM I can see the reported failure of TestServiceAM in some "Apache Hadoop qbt Report: branch-3.2+JDK8 on Linux/x86_64". 3.3.0 and above might be fixed by YARN-8867 which added guard using GenericTestUtils#waitFor for stabilizing the testContainersReleasedWhenPreLaunchFails. YARN 8867 did not modified other code under hadoop-yarn-services. If it is the case, TestServiceAM can be tagged as flaky in branch-3.2. On 2022/07/20 14:21, Masatake Iwasaki wrote: Thanks for testing the RC0, Chris. The following are new test failures for me on 3.2.4: * TestAMRMProxy * TestFsck * TestSLSStreamAMSynth * TestServiceAM I could not reproduce the test failures on my local. For TestFsck, if the failed test case is testFsckListCorruptSnapshotFiles, cherry-picking HDFS-15038 (fixing only test code) could be the fix. The failure of TestSLSStreamAMSynth looks frequently reported by "Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86_64". It could be tagged as known flaky test. On 2022/07/20 9:15, Chris Nauroth wrote: -0 (binding) * Verified all checksums. * Verified all signatures. * Built from source, including native code on Linux. * mvn clean package -Pnative -Psrc -Drequire.openssl -Drequire.snappy -Drequire.zstd -DskipTests * Tests mostly passed, but see below. * mvn --fail-never clean test -Pnative -Dparallel-tests -Drequire.snappy -Drequire.zstd -Drequire.openssl -Dsurefire.rerunFailingTestsCount=3 -DtestsThreadCount=8 The following are new test failures for me on 3.2.4: * TestAMRMProxy * TestFsck * TestSLSStreamAMSynth * TestServiceAM The following tests also failed, but they also fail for me on 3.2.3, so they aren't likely to be related to this release candidate: * TestCapacitySchedulerNodeLabelUpdate * TestFrameworkUploader * TestSLSGenericSynth * TestSLSRunner * test_libhdfs_threaded_hdfspp_test_shim_static I'm not voting a full -1, because I haven't done any root cause analysis on these new test failures. I don't know if it's a quirk to my environment, though I'm using the start-build-env.sh Docker container, so any build dependencies should be consistent. I'd be comfortable moving ahead if others are seeing these tests pass. Chris Nauroth On Thu, Jul 14, 2022 at 7:57 AM Masatake Iwasaki < iwasak...@oss.nttdata.com> wrote: +1 from myself. * skimmed the contents of site documentation. * built the source tarball on Rocky Linux 8 (x86_64) by OpenJDK 8 with `-Pnative`. * launched pseudo distributed cluster including kms and httpfs with Kerberos and SSL enabled. * created encryption zone, put and read files via httpfs. * ran example MR wordcount over encryption zone. * launched
Re: [VOTE] Release Apache Hadoop 3.2.4 - RC0
+1(Binding) * Built from Source * Successful native build on Ubuntu 18.04 * Verified Checksums * Verified Signatures * Successful RAT check * Ran Basic HDFS shell commands * Ran basic MR example Jobs (TeraGen/TeraSort & TeraValidate) * Browsed through UI(NN, DN, RM, NM & JHS) * Skimmed through the contents of ChangeLog & ReleaseNotes. Look Good Thanx Masatake for driving the release, Good Luck!!! -Ayush On Thu, 21 Jul 2022 at 23:28, Chris Nauroth wrote: > I'm changing my vote to +1 (binding). > > Masatake and Ashutosh, thank you for investigating. > > I reran tests without the parallel options, and that mostly addressed the > failures. Maybe the tests in question are just not sufficiently isolated to > support parallel execution. That looks to be the case for TestFsck, where > the failure was caused by missing audit log entries. This test works by > toggling global logging state, so I can see why multi-threaded execution > might confuse the test. > > Chris Nauroth > > > On Thu, Jul 21, 2022 at 12:01 AM Ashutosh Gupta < > ashutoshgupta...@gmail.com> > wrote: > > > +1(non-binding) > > > > * Builds from source look good. > > * Checksums and signatures are correct. > > * Running basic HDFS and MapReduce commands looks good. > > > > > * TestAMRMProxy - Not able to reproduce in local > > > * TestFsck - I can see failure only I can see is > > TestFsck.testFsckListCorruptSnapshotFiles which passed after applying > > HDFS-15038 > > > * TestSLSStreamAMSynth - Not able to reproduce in local > > > * TestServiceAM - Not able to reproduce in local > > > > Thanks Masatake for driving this release. > > > > On Thu, Jul 21, 2022 at 5:51 AM Masatake Iwasaki < > > iwasak...@oss.nttdata.com> > > wrote: > > > > > Hi developers, > > > > > > I'm still waiting for your vote. > > > I'm considering the intermittent test failures mentioned by Chris are > not > > > blocker. > > > Please file a JIRA and let me know if you find a blocker issue. > > > > > > I will appreciate your help for the release process. > > > > > > Regards, > > > Masatake Iwasaki > > > > > > On 2022/07/20 14:50, Masatake Iwasaki wrote: > > > >> TestServiceAM > > > > > > > > I can see the reported failure of TestServiceAM in some "Apache > Hadoop > > > qbt Report: branch-3.2+JDK8 on Linux/x86_64". > > > > 3.3.0 and above might be fixed by YARN-8867 which added guard using > > > GenericTestUtils#waitFor for stabilizing the > > > testContainersReleasedWhenPreLaunchFails. > > > > YARN 8867 did not modified other code under hadoop-yarn-services. > > > > If it is the case, TestServiceAM can be tagged as flaky in > branch-3.2. > > > > > > > > > > > > On 2022/07/20 14:21, Masatake Iwasaki wrote: > > > >> Thanks for testing the RC0, Chris. > > > >> > > > >>> The following are new test failures for me on 3.2.4: > > > >>> * TestAMRMProxy > > > >>> * TestFsck > > > >>> * TestSLSStreamAMSynth > > > >>> * TestServiceAM > > > >> > > > >> I could not reproduce the test failures on my local. > > > >> > > > >> For TestFsck, if the failed test case is > > > testFsckListCorruptSnapshotFiles, > > > >> cherry-picking HDFS-15038 (fixing only test code) could be the fix. > > > >> > > > >> The failure of TestSLSStreamAMSynth looks frequently reported by > > > >> "Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86_64". > > > >> It could be tagged as known flaky test. > > > >> > > > >> On 2022/07/20 9:15, Chris Nauroth wrote: > > > >>> -0 (binding) > > > >>> > > > >>> * Verified all checksums. > > > >>> * Verified all signatures. > > > >>> * Built from source, including native code on Linux. > > > >>> * mvn clean package -Pnative -Psrc -Drequire.openssl > > > -Drequire.snappy > > > >>> -Drequire.zstd -DskipTests > > > >>> * Tests mostly passed, but see below. > > > >>> * mvn --fail-never clean test -Pnative -Dparallel-tests > > > >>> -Drequire.snappy -Drequire.zstd -Drequire.openssl > > > >>> -Dsurefire.rerunFailingTestsCount=3 -DtestsThreadCount=8 > > > >>> > > > >>> The following are new test failures for me on 3.2.4: > > > >>> * TestAMRMProxy > > > >>> * TestFsck > > > >>> * TestSLSStreamAMSynth > > > >>> * TestServiceAM > > > >>> > > > >>> The following tests also failed, but they also fail for me on > 3.2.3, > > so > > > >>> they aren't likely to be related to this release candidate: > > > >>> * TestCapacitySchedulerNodeLabelUpdate > > > >>> * TestFrameworkUploader > > > >>> * TestSLSGenericSynth > > > >>> * TestSLSRunner > > > >>> * test_libhdfs_threaded_hdfspp_test_shim_static > > > >>> > > > >>> I'm not voting a full -1, because I haven't done any root cause > > > analysis on > > > >>> these new test failures. I don't know if it's a quirk to my > > > environment, > > > >>> though I'm using the start-build-env.sh Docker container, so any > > build > > > >>> dependencies should be consistent. I'd be comfortable moving ahead > if > > > >>> others are seeing these tests pass. > > > >>> > > > >>> Chris Nauroth > > > >>> > > > >>> > > > >>> On Thu, Jul 14, 2022 at 7:57 AM
Re: [VOTE] Release Apache Hadoop 3.2.4 - RC0
I'm changing my vote to +1 (binding). Masatake and Ashutosh, thank you for investigating. I reran tests without the parallel options, and that mostly addressed the failures. Maybe the tests in question are just not sufficiently isolated to support parallel execution. That looks to be the case for TestFsck, where the failure was caused by missing audit log entries. This test works by toggling global logging state, so I can see why multi-threaded execution might confuse the test. Chris Nauroth On Thu, Jul 21, 2022 at 12:01 AM Ashutosh Gupta wrote: > +1(non-binding) > > * Builds from source look good. > * Checksums and signatures are correct. > * Running basic HDFS and MapReduce commands looks good. > > > * TestAMRMProxy - Not able to reproduce in local > > * TestFsck - I can see failure only I can see is > TestFsck.testFsckListCorruptSnapshotFiles which passed after applying > HDFS-15038 > > * TestSLSStreamAMSynth - Not able to reproduce in local > > * TestServiceAM - Not able to reproduce in local > > Thanks Masatake for driving this release. > > On Thu, Jul 21, 2022 at 5:51 AM Masatake Iwasaki < > iwasak...@oss.nttdata.com> > wrote: > > > Hi developers, > > > > I'm still waiting for your vote. > > I'm considering the intermittent test failures mentioned by Chris are not > > blocker. > > Please file a JIRA and let me know if you find a blocker issue. > > > > I will appreciate your help for the release process. > > > > Regards, > > Masatake Iwasaki > > > > On 2022/07/20 14:50, Masatake Iwasaki wrote: > > >> TestServiceAM > > > > > > I can see the reported failure of TestServiceAM in some "Apache Hadoop > > qbt Report: branch-3.2+JDK8 on Linux/x86_64". > > > 3.3.0 and above might be fixed by YARN-8867 which added guard using > > GenericTestUtils#waitFor for stabilizing the > > testContainersReleasedWhenPreLaunchFails. > > > YARN 8867 did not modified other code under hadoop-yarn-services. > > > If it is the case, TestServiceAM can be tagged as flaky in branch-3.2. > > > > > > > > > On 2022/07/20 14:21, Masatake Iwasaki wrote: > > >> Thanks for testing the RC0, Chris. > > >> > > >>> The following are new test failures for me on 3.2.4: > > >>> * TestAMRMProxy > > >>> * TestFsck > > >>> * TestSLSStreamAMSynth > > >>> * TestServiceAM > > >> > > >> I could not reproduce the test failures on my local. > > >> > > >> For TestFsck, if the failed test case is > > testFsckListCorruptSnapshotFiles, > > >> cherry-picking HDFS-15038 (fixing only test code) could be the fix. > > >> > > >> The failure of TestSLSStreamAMSynth looks frequently reported by > > >> "Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86_64". > > >> It could be tagged as known flaky test. > > >> > > >> On 2022/07/20 9:15, Chris Nauroth wrote: > > >>> -0 (binding) > > >>> > > >>> * Verified all checksums. > > >>> * Verified all signatures. > > >>> * Built from source, including native code on Linux. > > >>> * mvn clean package -Pnative -Psrc -Drequire.openssl > > -Drequire.snappy > > >>> -Drequire.zstd -DskipTests > > >>> * Tests mostly passed, but see below. > > >>> * mvn --fail-never clean test -Pnative -Dparallel-tests > > >>> -Drequire.snappy -Drequire.zstd -Drequire.openssl > > >>> -Dsurefire.rerunFailingTestsCount=3 -DtestsThreadCount=8 > > >>> > > >>> The following are new test failures for me on 3.2.4: > > >>> * TestAMRMProxy > > >>> * TestFsck > > >>> * TestSLSStreamAMSynth > > >>> * TestServiceAM > > >>> > > >>> The following tests also failed, but they also fail for me on 3.2.3, > so > > >>> they aren't likely to be related to this release candidate: > > >>> * TestCapacitySchedulerNodeLabelUpdate > > >>> * TestFrameworkUploader > > >>> * TestSLSGenericSynth > > >>> * TestSLSRunner > > >>> * test_libhdfs_threaded_hdfspp_test_shim_static > > >>> > > >>> I'm not voting a full -1, because I haven't done any root cause > > analysis on > > >>> these new test failures. I don't know if it's a quirk to my > > environment, > > >>> though I'm using the start-build-env.sh Docker container, so any > build > > >>> dependencies should be consistent. I'd be comfortable moving ahead if > > >>> others are seeing these tests pass. > > >>> > > >>> Chris Nauroth > > >>> > > >>> > > >>> On Thu, Jul 14, 2022 at 7:57 AM Masatake Iwasaki < > > iwasak...@oss.nttdata.com> > > >>> wrote: > > >>> > > +1 from myself. > > > > * skimmed the contents of site documentation. > > > > * built the source tarball on Rocky Linux 8 (x86_64) by OpenJDK 8 > with > > `-Pnative`. > > > > * launched pseudo distributed cluster including kms and httpfs with > > Kerberos and SSL enabled. > > > > * created encryption zone, put and read files via httpfs. > > * ran example MR wordcount over encryption zone. > > > > * launched 3-node docker cluster with NN-HA and RM-HA enabled and > ran > > some > > example MR jobs. > > > > * built HBase 2.4.11, Hive 3.1.2 and Spark 3.1.2 against
Re: [VOTE] Release Apache Hadoop 3.2.4 - RC0
+1(non-binding) * Builds from source look good. * Checksums and signatures are correct. * Running basic HDFS and MapReduce commands looks good. > * TestAMRMProxy - Not able to reproduce in local > * TestFsck - I can see failure only I can see is TestFsck.testFsckListCorruptSnapshotFiles which passed after applying HDFS-15038 > * TestSLSStreamAMSynth - Not able to reproduce in local > * TestServiceAM - Not able to reproduce in local Thanks Masatake for driving this release. On Thu, Jul 21, 2022 at 5:51 AM Masatake Iwasaki wrote: > Hi developers, > > I'm still waiting for your vote. > I'm considering the intermittent test failures mentioned by Chris are not > blocker. > Please file a JIRA and let me know if you find a blocker issue. > > I will appreciate your help for the release process. > > Regards, > Masatake Iwasaki > > On 2022/07/20 14:50, Masatake Iwasaki wrote: > >> TestServiceAM > > > > I can see the reported failure of TestServiceAM in some "Apache Hadoop > qbt Report: branch-3.2+JDK8 on Linux/x86_64". > > 3.3.0 and above might be fixed by YARN-8867 which added guard using > GenericTestUtils#waitFor for stabilizing the > testContainersReleasedWhenPreLaunchFails. > > YARN 8867 did not modified other code under hadoop-yarn-services. > > If it is the case, TestServiceAM can be tagged as flaky in branch-3.2. > > > > > > On 2022/07/20 14:21, Masatake Iwasaki wrote: > >> Thanks for testing the RC0, Chris. > >> > >>> The following are new test failures for me on 3.2.4: > >>> * TestAMRMProxy > >>> * TestFsck > >>> * TestSLSStreamAMSynth > >>> * TestServiceAM > >> > >> I could not reproduce the test failures on my local. > >> > >> For TestFsck, if the failed test case is > testFsckListCorruptSnapshotFiles, > >> cherry-picking HDFS-15038 (fixing only test code) could be the fix. > >> > >> The failure of TestSLSStreamAMSynth looks frequently reported by > >> "Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86_64". > >> It could be tagged as known flaky test. > >> > >> On 2022/07/20 9:15, Chris Nauroth wrote: > >>> -0 (binding) > >>> > >>> * Verified all checksums. > >>> * Verified all signatures. > >>> * Built from source, including native code on Linux. > >>> * mvn clean package -Pnative -Psrc -Drequire.openssl > -Drequire.snappy > >>> -Drequire.zstd -DskipTests > >>> * Tests mostly passed, but see below. > >>> * mvn --fail-never clean test -Pnative -Dparallel-tests > >>> -Drequire.snappy -Drequire.zstd -Drequire.openssl > >>> -Dsurefire.rerunFailingTestsCount=3 -DtestsThreadCount=8 > >>> > >>> The following are new test failures for me on 3.2.4: > >>> * TestAMRMProxy > >>> * TestFsck > >>> * TestSLSStreamAMSynth > >>> * TestServiceAM > >>> > >>> The following tests also failed, but they also fail for me on 3.2.3, so > >>> they aren't likely to be related to this release candidate: > >>> * TestCapacitySchedulerNodeLabelUpdate > >>> * TestFrameworkUploader > >>> * TestSLSGenericSynth > >>> * TestSLSRunner > >>> * test_libhdfs_threaded_hdfspp_test_shim_static > >>> > >>> I'm not voting a full -1, because I haven't done any root cause > analysis on > >>> these new test failures. I don't know if it's a quirk to my > environment, > >>> though I'm using the start-build-env.sh Docker container, so any build > >>> dependencies should be consistent. I'd be comfortable moving ahead if > >>> others are seeing these tests pass. > >>> > >>> Chris Nauroth > >>> > >>> > >>> On Thu, Jul 14, 2022 at 7:57 AM Masatake Iwasaki < > iwasak...@oss.nttdata.com> > >>> wrote: > >>> > +1 from myself. > > * skimmed the contents of site documentation. > > * built the source tarball on Rocky Linux 8 (x86_64) by OpenJDK 8 with > `-Pnative`. > > * launched pseudo distributed cluster including kms and httpfs with > Kerberos and SSL enabled. > > * created encryption zone, put and read files via httpfs. > * ran example MR wordcount over encryption zone. > > * launched 3-node docker cluster with NN-HA and RM-HA enabled and ran > some > example MR jobs. > > * built HBase 2.4.11, Hive 3.1.2 and Spark 3.1.2 against Hadoop 3.2.4 > RC0 > on CentOS 7 (x86_64) by using Bigtop branch-3.1 and ran > smoke-tests. > https://github.com/apache/bigtop/pull/942 > > * Hive needs updating exclusion rule to address HADOOP-18088 > (migration > to reload4j). > > * built Spark 3.3.0 against Hadoop 3.2.4 RC0 using the staging > repository:: > > > staged > staged-releases > > > https://repository.apache.org/content/repositories/orgapachehadoop-1354 > > > true > > > true > > > > Thanks, > Masatake Iwasaki > > On 2022/07/13 1:14, Masatake Iwasaki wrote: > > Hi all, > > > > Here's Hadoop 3.2.4 release
Re: [VOTE] Release Apache Hadoop 3.2.4 - RC0
Hi developers, I'm still waiting for your vote. I'm considering the intermittent test failures mentioned by Chris are not blocker. Please file a JIRA and let me know if you find a blocker issue. I will appreciate your help for the release process. Regards, Masatake Iwasaki On 2022/07/20 14:50, Masatake Iwasaki wrote: TestServiceAM I can see the reported failure of TestServiceAM in some "Apache Hadoop qbt Report: branch-3.2+JDK8 on Linux/x86_64". 3.3.0 and above might be fixed by YARN-8867 which added guard using GenericTestUtils#waitFor for stabilizing the testContainersReleasedWhenPreLaunchFails. YARN 8867 did not modified other code under hadoop-yarn-services. If it is the case, TestServiceAM can be tagged as flaky in branch-3.2. On 2022/07/20 14:21, Masatake Iwasaki wrote: Thanks for testing the RC0, Chris. The following are new test failures for me on 3.2.4: * TestAMRMProxy * TestFsck * TestSLSStreamAMSynth * TestServiceAM I could not reproduce the test failures on my local. For TestFsck, if the failed test case is testFsckListCorruptSnapshotFiles, cherry-picking HDFS-15038 (fixing only test code) could be the fix. The failure of TestSLSStreamAMSynth looks frequently reported by "Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86_64". It could be tagged as known flaky test. On 2022/07/20 9:15, Chris Nauroth wrote: -0 (binding) * Verified all checksums. * Verified all signatures. * Built from source, including native code on Linux. * mvn clean package -Pnative -Psrc -Drequire.openssl -Drequire.snappy -Drequire.zstd -DskipTests * Tests mostly passed, but see below. * mvn --fail-never clean test -Pnative -Dparallel-tests -Drequire.snappy -Drequire.zstd -Drequire.openssl -Dsurefire.rerunFailingTestsCount=3 -DtestsThreadCount=8 The following are new test failures for me on 3.2.4: * TestAMRMProxy * TestFsck * TestSLSStreamAMSynth * TestServiceAM The following tests also failed, but they also fail for me on 3.2.3, so they aren't likely to be related to this release candidate: * TestCapacitySchedulerNodeLabelUpdate * TestFrameworkUploader * TestSLSGenericSynth * TestSLSRunner * test_libhdfs_threaded_hdfspp_test_shim_static I'm not voting a full -1, because I haven't done any root cause analysis on these new test failures. I don't know if it's a quirk to my environment, though I'm using the start-build-env.sh Docker container, so any build dependencies should be consistent. I'd be comfortable moving ahead if others are seeing these tests pass. Chris Nauroth On Thu, Jul 14, 2022 at 7:57 AM Masatake Iwasaki wrote: +1 from myself. * skimmed the contents of site documentation. * built the source tarball on Rocky Linux 8 (x86_64) by OpenJDK 8 with `-Pnative`. * launched pseudo distributed cluster including kms and httpfs with Kerberos and SSL enabled. * created encryption zone, put and read files via httpfs. * ran example MR wordcount over encryption zone. * launched 3-node docker cluster with NN-HA and RM-HA enabled and ran some example MR jobs. * built HBase 2.4.11, Hive 3.1.2 and Spark 3.1.2 against Hadoop 3.2.4 RC0 on CentOS 7 (x86_64) by using Bigtop branch-3.1 and ran smoke-tests. https://github.com/apache/bigtop/pull/942 * Hive needs updating exclusion rule to address HADOOP-18088 (migration to reload4j). * built Spark 3.3.0 against Hadoop 3.2.4 RC0 using the staging repository:: staged staged-releases https://repository.apache.org/content/repositories/orgapachehadoop-1354 true true Thanks, Masatake Iwasaki On 2022/07/13 1:14, Masatake Iwasaki wrote: Hi all, Here's Hadoop 3.2.4 release candidate #0: The RC is available at: https://home.apache.org/~iwasakims/hadoop-3.2.4-RC0/ The RC tag is at: https://github.com/apache/hadoop/releases/tag/release-3.2.4-RC0 The Maven artifacts are staged at: https://repository.apache.org/content/repositories/orgapachehadoop-1354 You can find my public key at: https://downloads.apache.org/hadoop/common/KEYS Please evaluate the RC and vote. The vote will be open for (at least) 5 days. Thanks, Masatake Iwasaki - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail:
Re: [VOTE] Release Apache Hadoop 3.2.4 - RC0
TestServiceAM I can see the reported failure of TestServiceAM in some "Apache Hadoop qbt Report: branch-3.2+JDK8 on Linux/x86_64". 3.3.0 and above might be fixed by YARN-8867 which added guard using GenericTestUtils#waitFor for stabilizing the testContainersReleasedWhenPreLaunchFails. YARN 8867 did not modified other code under hadoop-yarn-services. If it is the case, TestServiceAM can be tagged as flaky in branch-3.2. On 2022/07/20 14:21, Masatake Iwasaki wrote: Thanks for testing the RC0, Chris. The following are new test failures for me on 3.2.4: * TestAMRMProxy * TestFsck * TestSLSStreamAMSynth * TestServiceAM I could not reproduce the test failures on my local. For TestFsck, if the failed test case is testFsckListCorruptSnapshotFiles, cherry-picking HDFS-15038 (fixing only test code) could be the fix. The failure of TestSLSStreamAMSynth looks frequently reported by "Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86_64". It could be tagged as known flaky test. On 2022/07/20 9:15, Chris Nauroth wrote: -0 (binding) * Verified all checksums. * Verified all signatures. * Built from source, including native code on Linux. * mvn clean package -Pnative -Psrc -Drequire.openssl -Drequire.snappy -Drequire.zstd -DskipTests * Tests mostly passed, but see below. * mvn --fail-never clean test -Pnative -Dparallel-tests -Drequire.snappy -Drequire.zstd -Drequire.openssl -Dsurefire.rerunFailingTestsCount=3 -DtestsThreadCount=8 The following are new test failures for me on 3.2.4: * TestAMRMProxy * TestFsck * TestSLSStreamAMSynth * TestServiceAM The following tests also failed, but they also fail for me on 3.2.3, so they aren't likely to be related to this release candidate: * TestCapacitySchedulerNodeLabelUpdate * TestFrameworkUploader * TestSLSGenericSynth * TestSLSRunner * test_libhdfs_threaded_hdfspp_test_shim_static I'm not voting a full -1, because I haven't done any root cause analysis on these new test failures. I don't know if it's a quirk to my environment, though I'm using the start-build-env.sh Docker container, so any build dependencies should be consistent. I'd be comfortable moving ahead if others are seeing these tests pass. Chris Nauroth On Thu, Jul 14, 2022 at 7:57 AM Masatake Iwasaki wrote: +1 from myself. * skimmed the contents of site documentation. * built the source tarball on Rocky Linux 8 (x86_64) by OpenJDK 8 with `-Pnative`. * launched pseudo distributed cluster including kms and httpfs with Kerberos and SSL enabled. * created encryption zone, put and read files via httpfs. * ran example MR wordcount over encryption zone. * launched 3-node docker cluster with NN-HA and RM-HA enabled and ran some example MR jobs. * built HBase 2.4.11, Hive 3.1.2 and Spark 3.1.2 against Hadoop 3.2.4 RC0 on CentOS 7 (x86_64) by using Bigtop branch-3.1 and ran smoke-tests. https://github.com/apache/bigtop/pull/942 * Hive needs updating exclusion rule to address HADOOP-18088 (migration to reload4j). * built Spark 3.3.0 against Hadoop 3.2.4 RC0 using the staging repository:: staged staged-releases https://repository.apache.org/content/repositories/orgapachehadoop-1354 true true Thanks, Masatake Iwasaki On 2022/07/13 1:14, Masatake Iwasaki wrote: Hi all, Here's Hadoop 3.2.4 release candidate #0: The RC is available at: https://home.apache.org/~iwasakims/hadoop-3.2.4-RC0/ The RC tag is at: https://github.com/apache/hadoop/releases/tag/release-3.2.4-RC0 The Maven artifacts are staged at: https://repository.apache.org/content/repositories/orgapachehadoop-1354 You can find my public key at: https://downloads.apache.org/hadoop/common/KEYS Please evaluate the RC and vote. The vote will be open for (at least) 5 days. Thanks, Masatake Iwasaki - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
Re: [VOTE] Release Apache Hadoop 3.2.4 - RC0
Thanks for testing the RC0, Chris. The following are new test failures for me on 3.2.4: * TestAMRMProxy * TestFsck * TestSLSStreamAMSynth * TestServiceAM I could not reproduce the test failures on my local. For TestFsck, if the failed test case is testFsckListCorruptSnapshotFiles, cherry-picking HDFS-15038 (fixing only test code) could be the fix. The failure of TestSLSStreamAMSynth looks frequently reported by "Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86_64". It could be tagged as known flaky test. On 2022/07/20 9:15, Chris Nauroth wrote: -0 (binding) * Verified all checksums. * Verified all signatures. * Built from source, including native code on Linux. * mvn clean package -Pnative -Psrc -Drequire.openssl -Drequire.snappy -Drequire.zstd -DskipTests * Tests mostly passed, but see below. * mvn --fail-never clean test -Pnative -Dparallel-tests -Drequire.snappy -Drequire.zstd -Drequire.openssl -Dsurefire.rerunFailingTestsCount=3 -DtestsThreadCount=8 The following are new test failures for me on 3.2.4: * TestAMRMProxy * TestFsck * TestSLSStreamAMSynth * TestServiceAM The following tests also failed, but they also fail for me on 3.2.3, so they aren't likely to be related to this release candidate: * TestCapacitySchedulerNodeLabelUpdate * TestFrameworkUploader * TestSLSGenericSynth * TestSLSRunner * test_libhdfs_threaded_hdfspp_test_shim_static I'm not voting a full -1, because I haven't done any root cause analysis on these new test failures. I don't know if it's a quirk to my environment, though I'm using the start-build-env.sh Docker container, so any build dependencies should be consistent. I'd be comfortable moving ahead if others are seeing these tests pass. Chris Nauroth On Thu, Jul 14, 2022 at 7:57 AM Masatake Iwasaki wrote: +1 from myself. * skimmed the contents of site documentation. * built the source tarball on Rocky Linux 8 (x86_64) by OpenJDK 8 with `-Pnative`. * launched pseudo distributed cluster including kms and httpfs with Kerberos and SSL enabled. * created encryption zone, put and read files via httpfs. * ran example MR wordcount over encryption zone. * launched 3-node docker cluster with NN-HA and RM-HA enabled and ran some example MR jobs. * built HBase 2.4.11, Hive 3.1.2 and Spark 3.1.2 against Hadoop 3.2.4 RC0 on CentOS 7 (x86_64) by using Bigtop branch-3.1 and ran smoke-tests. https://github.com/apache/bigtop/pull/942 * Hive needs updating exclusion rule to address HADOOP-18088 (migration to reload4j). * built Spark 3.3.0 against Hadoop 3.2.4 RC0 using the staging repository:: staged staged-releases https://repository.apache.org/content/repositories/orgapachehadoop-1354 true true Thanks, Masatake Iwasaki On 2022/07/13 1:14, Masatake Iwasaki wrote: Hi all, Here's Hadoop 3.2.4 release candidate #0: The RC is available at: https://home.apache.org/~iwasakims/hadoop-3.2.4-RC0/ The RC tag is at: https://github.com/apache/hadoop/releases/tag/release-3.2.4-RC0 The Maven artifacts are staged at: https://repository.apache.org/content/repositories/orgapachehadoop-1354 You can find my public key at: https://downloads.apache.org/hadoop/common/KEYS Please evaluate the RC and vote. The vote will be open for (at least) 5 days. Thanks, Masatake Iwasaki - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
Re: [VOTE] Release Apache Hadoop 3.2.4 - RC0
-0 (binding) * Verified all checksums. * Verified all signatures. * Built from source, including native code on Linux. * mvn clean package -Pnative -Psrc -Drequire.openssl -Drequire.snappy -Drequire.zstd -DskipTests * Tests mostly passed, but see below. * mvn --fail-never clean test -Pnative -Dparallel-tests -Drequire.snappy -Drequire.zstd -Drequire.openssl -Dsurefire.rerunFailingTestsCount=3 -DtestsThreadCount=8 The following are new test failures for me on 3.2.4: * TestAMRMProxy * TestFsck * TestSLSStreamAMSynth * TestServiceAM The following tests also failed, but they also fail for me on 3.2.3, so they aren't likely to be related to this release candidate: * TestCapacitySchedulerNodeLabelUpdate * TestFrameworkUploader * TestSLSGenericSynth * TestSLSRunner * test_libhdfs_threaded_hdfspp_test_shim_static I'm not voting a full -1, because I haven't done any root cause analysis on these new test failures. I don't know if it's a quirk to my environment, though I'm using the start-build-env.sh Docker container, so any build dependencies should be consistent. I'd be comfortable moving ahead if others are seeing these tests pass. Chris Nauroth On Thu, Jul 14, 2022 at 7:57 AM Masatake Iwasaki wrote: > +1 from myself. > > * skimmed the contents of site documentation. > > * built the source tarball on Rocky Linux 8 (x86_64) by OpenJDK 8 with > `-Pnative`. > > * launched pseudo distributed cluster including kms and httpfs with > Kerberos and SSL enabled. > >* created encryption zone, put and read files via httpfs. >* ran example MR wordcount over encryption zone. > > * launched 3-node docker cluster with NN-HA and RM-HA enabled and ran some > example MR jobs. > > * built HBase 2.4.11, Hive 3.1.2 and Spark 3.1.2 against Hadoop 3.2.4 RC0 >on CentOS 7 (x86_64) by using Bigtop branch-3.1 and ran smoke-tests. >https://github.com/apache/bigtop/pull/942 > >* Hive needs updating exclusion rule to address HADOOP-18088 (migration > to reload4j). > > * built Spark 3.3.0 against Hadoop 3.2.4 RC0 using the staging repository:: > > > staged > staged-releases > > https://repository.apache.org/content/repositories/orgapachehadoop-1354 > > > true > > > true > > > > Thanks, > Masatake Iwasaki > > On 2022/07/13 1:14, Masatake Iwasaki wrote: > > Hi all, > > > > Here's Hadoop 3.2.4 release candidate #0: > > > > The RC is available at: > >https://home.apache.org/~iwasakims/hadoop-3.2.4-RC0/ > > > > The RC tag is at: > >https://github.com/apache/hadoop/releases/tag/release-3.2.4-RC0 > > > > The Maven artifacts are staged at: > > > https://repository.apache.org/content/repositories/orgapachehadoop-1354 > > > > You can find my public key at: > >https://downloads.apache.org/hadoop/common/KEYS > > > > Please evaluate the RC and vote. > > The vote will be open for (at least) 5 days. > > > > Thanks, > > Masatake Iwasaki > > > > - > > To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org > > For additional commands, e-mail: common-dev-h...@hadoop.apache.org > > > > - > To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org > For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org > >
Re: [VOTE] Release Apache Hadoop 3.2.4 - RC0
+1 from myself. * skimmed the contents of site documentation. * built the source tarball on Rocky Linux 8 (x86_64) by OpenJDK 8 with `-Pnative`. * launched pseudo distributed cluster including kms and httpfs with Kerberos and SSL enabled. * created encryption zone, put and read files via httpfs. * ran example MR wordcount over encryption zone. * launched 3-node docker cluster with NN-HA and RM-HA enabled and ran some example MR jobs. * built HBase 2.4.11, Hive 3.1.2 and Spark 3.1.2 against Hadoop 3.2.4 RC0 on CentOS 7 (x86_64) by using Bigtop branch-3.1 and ran smoke-tests. https://github.com/apache/bigtop/pull/942 * Hive needs updating exclusion rule to address HADOOP-18088 (migration to reload4j). * built Spark 3.3.0 against Hadoop 3.2.4 RC0 using the staging repository:: staged staged-releases https://repository.apache.org/content/repositories/orgapachehadoop-1354 true true Thanks, Masatake Iwasaki On 2022/07/13 1:14, Masatake Iwasaki wrote: Hi all, Here's Hadoop 3.2.4 release candidate #0: The RC is available at: https://home.apache.org/~iwasakims/hadoop-3.2.4-RC0/ The RC tag is at: https://github.com/apache/hadoop/releases/tag/release-3.2.4-RC0 The Maven artifacts are staged at: https://repository.apache.org/content/repositories/orgapachehadoop-1354 You can find my public key at: https://downloads.apache.org/hadoop/common/KEYS Please evaluate the RC and vote. The vote will be open for (at least) 5 days. Thanks, Masatake Iwasaki - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org