Re: [VOTE] Release Apache Hadoop 3.1.1 - RC0
Includes these additional votings we got 5 binding votings, and 12 non-binding votings, we have many others like Bibin/Akhil gave offline suggestions and testing results. Thanks everyone for your help! I've done most of staging/pushing works for the 3.1.1 release. Given Apache mirrors will take some time to finish syncing tasks, I plan to send out the release announcement by Thu noon PDT time. If there anything wanna to highlight for the announcement of 3.1.1, please let me know by Thu 10 AM PDT. Thanks again for your help of the release! Best, Wangda On Tue, Aug 7, 2018 at 7:57 PM Chandni Singh wrote: > Thanks Wangda! > > +1 (non-binding) > > Tested the following: > - Built from source and ran a single node cluster > - Ran the example pi job > - Launched yarn service sleep example > - Verified upgrade of yarn service sleep > > Thanks, > Chandni > > > On Tue, Aug 7, 2018 at 7:02 PM Suma Shivaprasad < > sumasai.shivapra...@gmail.com> wrote: > >> Thanks Wangda! >> >> +1 (non-binding) >> >> Tested the following: >> - Built from source >> - Setup single node cluster >> - Tested Dynamic queues >> - Tested MR and DS with default, docker runtime >> - Tested Yarn Services with various restart policies >> >> Thanks >> Suma >> >> >> On Tue, Aug 7, 2018 at 2:45 PM Eric Payne > .invalid> >> wrote: >> >> > Thanks Wangda for creating this release. >> > >> > +1 (binding) >> > Tested: >> > - Built from source >> > - Deployed to 6-node, multi-tennant, unsecured pseudo cluster with >> > hierarchical queue structure (CS) >> > - Refreshed queue (CS) properties >> > - Intra-queue preemption (CS) >> > - inter-queue preemption (CS) >> > - User weights (CS) >> > >> > Issues: >> > - Inter-queue preemption seems to be preempting unnecessarily (flapping) >> > when the queue balancing feature is enabled. This does not seem to be >> > specific to this release. >> > - The preemption-to-balance-queue-after-satisfied.enabled property seems >> > to always be enabled, but again, that is not specific to this release. >> > >> > >> > Eric >> > >> > >> > On Thursday, August 2, 2018, 1:44:22 PM CDT, Wangda Tan < >> > wheele...@gmail.com> wrote: >> > >> > >> > >> > >> > >> > Hi folks, >> > >> > I've created RC0 for Apache Hadoop 3.1.1. The artifacts are available >> here: >> > >> > http://people.apache.org/~wangda/hadoop-3.1.1-RC0/ >> > >> > The RC tag in git is release-3.1.1-RC0: >> > https://github.com/apache/hadoop/commits/release-3.1.1-RC0 >> > >> > The maven artifacts are available via repository.apache.org at >> > >> https://repository.apache.org/content/repositories/orgapachehadoop-1139/ >> > >> > You can find my public key at >> > http://svn.apache.org/repos/asf/hadoop/common/dist/KEYS >> > >> > This vote will run 5 days from now. >> > >> > 3.1.1 contains 435 [1] fixed JIRA issues since 3.1.0. >> > >> > I have done testing with a pseudo cluster and distributed shell job. My >> +1 >> > to start. >> > >> > Best, >> > Wangda Tan >> > >> > [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.1) >> > ORDER BY priority DESC >> > >> > - >> > To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org >> > For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org >> > >> > >> >
Apache Hadoop qbt Report: trunk+JDK8 on Windows/x64
For more details, see https://builds.apache.org/job/hadoop-trunk-win/552/ [Aug 7, 2018 5:15:28 PM] (virajith) HDFS-13796. Allow verbosity of InMemoryLevelDBAliasMapServer to be [Aug 7, 2018 7:36:55 PM] (wangda) YARN-8629. Container cleanup fails while trying to delete Cgroups. (Suma [Aug 7, 2018 7:37:32 PM] (wangda) YARN-7089. Mark the log-aggregation-controller APIs as public. (Zian [Aug 7, 2018 8:01:13 PM] (wangda) YARN-8407. Container launch exception in AM log should be printed in [Aug 7, 2018 10:33:16 PM] (gifuma) YARN-8626. Create HomePolicyManager that sends all the requests to the [Aug 7, 2018 11:13:41 PM] (xiao) HDFS-13799. TestEditLogTailer#testTriggersLogRollsForAllStandbyNN fails [Aug 7, 2018 11:40:33 PM] (aengineer) HDDS-124. Validate all required configs needed for ozone-site.xml and [Aug 8, 2018 2:13:09 AM] (mackrorysd) HADOOP-15400. Improve S3Guard documentation on Authoritative Mode [Aug 8, 2018 2:23:17 AM] (vinayakumarb) HDFS-13786. EC: Display erasure coding policy for sub-directories is not [Aug 8, 2018 5:05:17 AM] (xiao) HDFS-13728. Disk Balancer should not fail if volume usage is greater [Aug 8, 2018 7:12:20 AM] (vinayakumarb) HDFS-13785. EC: 'removePolicy' is not working for built-in/system [Aug 8, 2018 11:50:23 AM] (ewan.higgs) HADOOP-15576. S3A Multipart Uploader to work with S3Guard and encryption -1 overall The following subsystems voted -1: compile mvninstall pathlen unit The following subsystems voted -1 but were configured to be filtered/ignored: cc javac The following subsystems are considered long running: (runtime bigger than 1h 00m 00s) unit Specific tests: Failed junit tests : hadoop.crypto.key.kms.server.TestKMS hadoop.cli.TestAclCLI hadoop.cli.TestAclCLIWithPosixAclInheritance hadoop.cli.TestCacheAdminCLI hadoop.cli.TestCryptoAdminCLI hadoop.cli.TestDeleteCLI hadoop.cli.TestErasureCodingCLI hadoop.cli.TestHDFSCLI hadoop.cli.TestXAttrCLI hadoop.fs.contract.hdfs.TestHDFSContractAppend hadoop.fs.contract.hdfs.TestHDFSContractConcat hadoop.fs.contract.hdfs.TestHDFSContractCreate hadoop.fs.contract.hdfs.TestHDFSContractDelete hadoop.fs.contract.hdfs.TestHDFSContractGetFileStatus hadoop.fs.contract.hdfs.TestHDFSContractMkdir hadoop.fs.contract.hdfs.TestHDFSContractMultipartUploader hadoop.fs.contract.hdfs.TestHDFSContractOpen hadoop.fs.contract.hdfs.TestHDFSContractPathHandle hadoop.fs.contract.hdfs.TestHDFSContractRename hadoop.fs.contract.hdfs.TestHDFSContractRootDirectory hadoop.fs.contract.hdfs.TestHDFSContractSeek hadoop.fs.contract.hdfs.TestHDFSContractSetTimes hadoop.fs.loadGenerator.TestLoadGenerator hadoop.fs.permission.TestStickyBit hadoop.fs.shell.TestHdfsTextCommand hadoop.fs.TestEnhancedByteBufferAccess hadoop.fs.TestFcHdfsCreateMkdir hadoop.fs.TestFcHdfsPermission hadoop.fs.TestFcHdfsSetUMask hadoop.fs.TestGlobPaths hadoop.fs.TestHDFSFileContextMainOperations hadoop.fs.TestHdfsNativeCodeLoader hadoop.fs.TestResolveHdfsSymlink hadoop.fs.TestSWebHdfsFileContextMainOperations hadoop.fs.TestSymlinkHdfsDisable hadoop.fs.TestSymlinkHdfsFileContext hadoop.fs.TestSymlinkHdfsFileSystem hadoop.fs.TestUnbuffer hadoop.fs.TestUrlStreamHandler hadoop.fs.TestWebHdfsFileContextMainOperations hadoop.fs.viewfs.TestViewFileSystemAtHdfsRoot hadoop.fs.viewfs.TestViewFileSystemHdfs hadoop.fs.viewfs.TestViewFileSystemLinkFallback hadoop.fs.viewfs.TestViewFileSystemLinkMergeSlash hadoop.fs.viewfs.TestViewFileSystemWithAcls hadoop.fs.viewfs.TestViewFileSystemWithTruncate hadoop.fs.viewfs.TestViewFileSystemWithXAttrs hadoop.fs.viewfs.TestViewFsAtHdfsRoot hadoop.fs.viewfs.TestViewFsDefaultValue hadoop.fs.viewfs.TestViewFsFileStatusHdfs hadoop.fs.viewfs.TestViewFsHdfs hadoop.fs.viewfs.TestViewFsWithAcls hadoop.fs.viewfs.TestViewFsWithXAttrs hadoop.hdfs.client.impl.TestBlockReaderLocal hadoop.hdfs.client.impl.TestBlockReaderLocalLegacy hadoop.hdfs.client.impl.TestBlockReaderRemote hadoop.hdfs.client.impl.TestClientBlockVerification hadoop.hdfs.crypto.TestHdfsCryptoStreams hadoop.hdfs.protocol.datatransfer.sasl.TestSaslDataTransfer hadoop.hdfs.qjournal.client.TestEpochsAreUnique hadoop.hdfs.qjournal.client.TestQJMWithFaults hadoop.hdfs.qjournal.client.TestQuorumJournalManager hadoop.hdfs.qjournal.server.TestJournal hadoop.hdfs.qjournal.server.TestJournalNode hadoop.hdfs.qjournal.server.TestJournalNodeMXBean hadoop.hdfs.qjournal.server.TestJournalNodeRespectsBindHostKeys hadoop.hdfs.qjournal.server.TestJournalNodeSync
Re: [DISCUSS] Alpha Release of Ozone
Thanks for reporting this issue. I have filed a JIRA to address this issue. https://issues.apache.org/jira/browse/HDDS-341 >So, consider this as a report. IMHO, cutting an Ozone release prior to a >Hadoop release ill-advised given the distribution impact and the requirements >of the merge vote. The Ozone release is being planned to address issues like these; In my mind if we go thru a release exercise, we will be able to identify all ozone and Hadoop related build and release issues. Ozone will tremendously benefit from a release exercise and the community review that comes from that. Thanks Anu On 8/8/18, 1:19 PM, "Allen Wittenauer" wrote: > On Aug 8, 2018, at 12:56 PM, Anu Engineer wrote: > >> Has anyone verified that a Hadoop release doesn't have _any_ of the extra ozone bits that are sprinkled outside the maven modules? > As far as I know that is the state, we have had multiple Hadoop releases after ozone has been merged. So far no one has reported Ozone bits leaking into Hadoop. If we find something like that, it would be a bug. There hasn't been a release from a branch where Ozone has been merged yet. The first one will be 3.2.0. Running create-release off of trunk presently shows bits of Ozone in dev-support, hadoop-dist, and elsewhere in the Hadoop source tar ball. So, consider this as a report. IMHO, cutting an Ozone release prior to a Hadoop release ill-advised given the distribution impact and the requirements of the merge vote.
Re: [DISCUSS] Alpha Release of Ozone
> On Aug 8, 2018, at 12:56 PM, Anu Engineer wrote: > >> Has anyone verified that a Hadoop release doesn't have _any_ of the extra >> ozone bits that are sprinkled outside the maven modules? > As far as I know that is the state, we have had multiple Hadoop releases > after ozone has been merged. So far no one has reported Ozone bits leaking > into Hadoop. If we find something like that, it would be a bug. There hasn't been a release from a branch where Ozone has been merged yet. The first one will be 3.2.0. Running create-release off of trunk presently shows bits of Ozone in dev-support, hadoop-dist, and elsewhere in the Hadoop source tar ball. So, consider this as a report. IMHO, cutting an Ozone release prior to a Hadoop release ill-advised given the distribution impact and the requirements of the merge vote. - To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org
Re: [DISCUSS] Alpha Release of Ozone
> Given that there are some Ozone components spread out past the core maven > modules, is the plan to release a Hadoop Trunk + Ozone tar ball or is more > work going to go into segregating the Ozone components prior to release? The official release will be a source tarball, we intend to release an ozone only binaries to make it easy to for people to deploy. We are still formulating the plans and you are welcome to leave your comments on HDDS-214. > Has anyone verified that a Hadoop release doesn't have _any_ of the extra > ozone bits that are sprinkled outside the maven modules? As far as I know that is the state, we have had multiple Hadoop releases after ozone has been merged. So far no one has reported Ozone bits leaking into Hadoop. If we find something like that, it would be a bug. Thanks Anu On 8/8/18, 12:04 PM, "Allen Wittenauer" wrote: Given that there are some Ozone components spread out past the core maven modules, is the plan to release a Hadoop Trunk + Ozone tar ball or is more work going to go into segregating the Ozone components prior to release? Has anyone verified that a Hadoop release doesn't have _any_ of the extra ozone bits that are sprinkled outside the maven modules? - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org - To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org
Re: [DISCUSS] Alpha Release of Ozone
Given that there are some Ozone components spread out past the core maven modules, is the plan to release a Hadoop Trunk + Ozone tar ball or is more work going to go into segregating the Ozone components prior to release? Has anyone verified that a Hadoop release doesn't have _any_ of the extra ozone bits that are sprinkled outside the maven modules? - To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org
Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86
For more details, see https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/862/ [Aug 7, 2018 9:33:14 AM] (msingh) HDDS-230. ContainerStateMachine should implement readStateMachineData [Aug 7, 2018 10:39:53 AM] (msingh) HDDS-301. ozone command shell does not contain subcommand to run ozoneFS [Aug 7, 2018 5:15:28 PM] (virajith) HDFS-13796. Allow verbosity of InMemoryLevelDBAliasMapServer to be [Aug 7, 2018 7:36:55 PM] (wangda) YARN-8629. Container cleanup fails while trying to delete Cgroups. (Suma [Aug 7, 2018 7:37:32 PM] (wangda) YARN-7089. Mark the log-aggregation-controller APIs as public. (Zian [Aug 7, 2018 8:01:13 PM] (wangda) YARN-8407. Container launch exception in AM log should be printed in [Aug 7, 2018 10:33:16 PM] (gifuma) YARN-8626. Create HomePolicyManager that sends all the requests to the [Aug 7, 2018 11:13:41 PM] (xiao) HDFS-13799. TestEditLogTailer#testTriggersLogRollsForAllStandbyNN fails [Aug 7, 2018 11:40:33 PM] (aengineer) HDDS-124. Validate all required configs needed for ozone-site.xml and -1 overall The following subsystems voted -1: asflicense findbugs pathlen unit xml The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: Failed CTEST tests : test_test_libhdfs_threaded_hdfs_static test_libhdfs_threaded_hdfspp_test_shim_static Failed junit tests : hadoop.hdfs.client.impl.TestBlockReaderLocal hadoop.hdfs.TestDFSInotifyEventInputStreamKerberized hadoop.hdfs.web.TestWebHdfsTimeouts hadoop.yarn.client.TestApplicationMasterServiceProtocolForTimelineV2 hadoop.mapred.TestMRTimelineEventHandling cc: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/862/artifact/out/diff-compile-cc-root.txt [4.0K] javac: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/862/artifact/out/diff-compile-javac-root.txt [332K] checkstyle: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/862/artifact/out/diff-checkstyle-root.txt [4.0K] pathlen: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/862/artifact/out/pathlen.txt [12K] pylint: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/862/artifact/out/diff-patch-pylint.txt [24K] shellcheck: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/862/artifact/out/diff-patch-shellcheck.txt [20K] shelldocs: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/862/artifact/out/diff-patch-shelldocs.txt [16K] whitespace: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/862/artifact/out/whitespace-eol.txt [9.4M] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/862/artifact/out/whitespace-tabs.txt [1.1M] xml: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/862/artifact/out/xml.txt [4.0K] findbugs: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/862/artifact/out/branch-findbugs-hadoop-hdds_client.txt [60K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/862/artifact/out/branch-findbugs-hadoop-hdds_container-service.txt [52K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/862/artifact/out/branch-findbugs-hadoop-hdds_framework.txt [12K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/862/artifact/out/branch-findbugs-hadoop-hdds_server-scm.txt [56K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/862/artifact/out/branch-findbugs-hadoop-hdds_tools.txt [16K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/862/artifact/out/branch-findbugs-hadoop-ozone_client.txt [4.0K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/862/artifact/out/branch-findbugs-hadoop-ozone_common.txt [28K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/862/artifact/out/branch-findbugs-hadoop-ozone_objectstore-service.txt [4.0K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/862/artifact/out/branch-findbugs-hadoop-ozone_ozone-manager.txt [4.0K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/862/artifact/out/branch-findbugs-hadoop-ozone_ozonefs.txt [8.0K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/862/artifact/out/branch-findbugs-hadoop-ozone_tools.txt [4.0K] javadoc: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/862/artifact/out/diff-javadoc-javadoc-root.txt [760K] CTEST:
RE: [DISCUSS] Alpha Release of Ozone
+1. Looking forward to try the fresh Ozone release. Bests, Sammi -Original Message- From: Elek, Marton [mailto:e...@apache.org] Sent: Tuesday, August 7, 2018 1:34 AM To: common-...@hadoop.apache.org; mapreduce-dev@hadoop.apache.org; yarn-...@hadoop.apache.org; hdfs-...@hadoop.apache.org Subject: [DISCUSS] Alpha Release of Ozone Hi All, I would like to discuss creating an Alpha release for Ozone. The core functionality of Ozone is complete but there are two missing features; Security and HA, work on these features are progressing in Branches HDDS-4 and HDDS-151. Right now, Ozone can handle millions of keys and has a Hadoop compatible file system, which allows applications like Hive, Spark, and YARN use Ozone. Having an Alpha release of Ozone will help in getting some early feedback (this release will be marked as an Alpha -- and not production ready). Going through a complete release cycle will help us flush out Ozone release process, update user documentation and nail down deployment models. Please share your thoughts on the Alpha release (over mail or in HDDS-214), as voted on by the community earlier, Ozone release will be independent of Hadoop releases. Thanks a lot, Marton Elek - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org - To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org
