Thanks for putting together the RC, Wangda! Sorry for the late addition. +1 (non-binding)
- Verified checksums and signatures of the artifacts - Deployed a pseudo distributed cluster using the binary tgz, source tgz, and git tag - Ran basic hdfs commands - Ran pi, sleep, distributed shell, and a sleeper service using the default and docker runtimes - Ran dshell using the default and docker runtimes in a secure cluster - Verified docker runtime and privileged containers can be disabled - Verified YARN and HDFS UI's - Validated the change log against the commits, all discrepancies are documented or expected - Basic validation of Registry DNS lookups On Thu, Apr 5, 2018 at 12:26 AM, Naganarasimha Garla < naganarasimha...@apache.org> wrote: > +1 (binding) > > * Verified signatures > * Verified checksums for source and binary artefacts > * Built from source > * Deployed a pseudo Hadoop cluster > * Verified YARN & HDFS web UIs > * Ran sample MapReduce jobs > * Tested NodeLabels > > Regards, > + Naga > > On Thu, Apr 5, 2018 at 2:59 AM, Suma Shivaprasad < > sumasai.shivapra...@gmail.com> wrote: > > > +1 (non binding) > > > > > > *Verified - User Group Queue mapping - Node labels with New UI- Dynamic > > queuesThanksSuma* > > > > > > On Wed, Apr 4, 2018 at 11:48 AM, Steve Loughran <ste...@hortonworks.com> > > wrote: > > > > > that's "dangerously interesting". I think you are right, and I also > think > > > it'll just be the version files which get generated > > > > > > anyway, +1 binding > > > > > > * ran my new Hadoop-3 profile on spark (SPARK-23807), with the > committer > > > binding, then my downstream tests. All is well, provided you also have > a > > > spark hive JAR patched to accept hadoop 3 as a legitimate hadoop > version. > > > That's an ongoing issue in the Spark project. With that JAR on my CP my > > > downstream tests were all happy (yesterday) > > > > > > * today the staging files seem to be missing, at least maven is unable > to > > > find them even when I turn the spark snapshots-and-staging profile on. > > > That'll be the maven dist process at play, nothing else > > > > > > On 4 Apr 2018, at 04:13, Wangda Tan <wheele...@gmail.com<mailto:wh > > > eele...@gmail.com>> wrote: > > > > > > Hi Vinod / Arpit, > > > > > > I checked following versions: > > > - 2.6.5 / 2.7.5 / 2.8.3 / 2.9.0 / 3.0.1: > > > > > > Jars in maven repo [1] are *always* different from jars in the binary > > > tarball [2]: (I only checked hadoop-yarn-api-version.jar) > > > > > > (Following numbers are sizes of the jar) > > > 2.6.5: > > > - Jar in Maven: 1896185 > > > - Jar in tarball: 1891485 > > > > > > 2.7.5: > > > - Jar in Maven: 2039371 (md5: 15e76f7c734b49315ef2bce952509ddf) > > > - Jar in tarball: 2039371 (md5: 0ef9f42f587401f5b49b39f27459f3ef) > > > (Even size is same, md5 is different) > > > > > > 2.8.3: > > > - Jar in Maven: 2451433 > > > - Jar in tarball: 2438975 > > > > > > 2.9.0: > > > - Jar in Maven: 2791477 > > > - Jar in tarball: 2777789 > > > > > > 3.0.1: > > > - Jar in Maven: 2852604 > > > - Jar in tarball: 2851373 > > > > > > I guess the differences come from our release process. > > > > > > Thanks, > > > Wangda > > > > > > [1] Maven jars are downloaded from > > > https://repository.apache.org/service/local/repositories/ > > > releases/content/org/apache/hadoop/hadoop-yarn-api/ > > > <version>/hadoop-yarn-api-<version>.jar > > > [2] Binary tarballs downloaded from http://apache.claz.org/hadoop/ > > common/ > > > > > > > > > On Tue, Apr 3, 2018 at 4:25 PM, Vinod Kumar Vavilapalli < > > > vino...@apache.org<mailto:vino...@apache.org>> > > > wrote: > > > > > > We vote on the source code. The binaries are convenience artifacts. > > > > > > This is what I would do - (a) Just replace both the maven jars as well > as > > > the binaries to be consistent and correct. And then (b) Give a couple > > more > > > days for folks who tested on the binaries to reverify - I count one > such > > > clear vote as of now. > > > > > > Thanks > > > +Vinod > > > > > > > > > On Apr 3, 2018, at 3:30 PM, Wangda Tan <wheele...@gmail.com<mailto:wh > > > eele...@gmail.com>> wrote: > > > > > > HI Arpit, > > > > > > I think it won't match if we do rebuild. It should be fine as far as > > > they're signed, correct? I don't see any policy doesn't allow this. > > > > > > Thanks, > > > Wangda > > > > > > > > > On Tue, Apr 3, 2018 at 9:33 AM, Arpit Agarwal < > aagar...@hortonworks.com< > > > mailto:aagar...@hortonworks.com>> > > > wrote: > > > > > > Thanks Wangda, I see the shaded jars now. > > > > > > Are the repo jars required to be the same as the binary release? They > > > don’t match right now, probably they got rebuilt. > > > > > > +1 (binding), modulo that remaining question. > > > > > > * Verified signatures > > > * Verified checksums for source and binary artefacts > > > * Sanity checked jars on r.a.o. > > > * Built from source > > > * Deployed to 3 node secure cluster with NameNode HA > > > * Verified HDFS web UIs > > > * Tried out HDFS shell commands > > > * Ran sample MapReduce jobs > > > > > > Thanks! > > > > > > > > > ---------------------------------------------------------------------- > > > From: Wangda Tan <wheele...@gmail.com<mailto:wheele...@gmail.com>> > > > Date: Monday, April 2, 2018 at 9:25 PM > > > To: Arpit Agarwal <aagar...@hortonworks.com<mailto: > aagarwal@hortonworks. > > > com>> > > > Cc: Gera Shegalov <ger...@gmail.com<mailto:ger...@gmail.com>>, Sunil > G < > > > sun...@apache.org<mailto:sun...@apache.org>>, " > > > yarn-...@hadoop.apache.org<mailto:yarn-...@hadoop.apache.org>" < > > > yarn-...@hadoop.apache.org<mailto:yarn-...@hadoop.apache.org>>, > > Hdfs-dev < > > > hdfs-...@hadoop.apache.org<mailto:hdfs-...@hadoop.apache.org>>, Hadoop > > > Common <common-dev@hadoop.apache.org<mailto:common-dev@hadoop. > apache.org > > > >>, > > > "mapreduce-...@hadoop.apache.org<mailto:mapreduce-dev@ > hadoop.apache.org > > >" > > > <mapreduce-...@hadoop.apache.org<mailto:mapreduce-dev@ > hadoop.apache.org > > >>, > > > Vinod Kumar Vavilapalli <vino...@apache.org<mailto:vino...@apache.org > >> > > > Subject: Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1) > > > > > > As pointed by Arpit, the previously deployed shared jars are incorrect. > > > Just redeployed jars and staged. @Arpit, could you please check the > > updated > > > Maven repo? https://repository.apache.org/content/repositories/ > > > orgapachehadoop-1092 > > > > > > Since the jars inside binary tarballs are correct ( > > > http://people.apache.org/~wangda/hadoop-3.1.0-RC1/). I think we don't > > > need roll another RC, just update Maven repo should be sufficient. > > > > > > Best, > > > Wangda > > > > > > > > > On Mon, Apr 2, 2018 at 2:39 PM, Wangda Tan <mailto:wheele...@gmail.com > > > > > wrote: > > > Hi Arpit, > > > > > > Thanks for pointing out this. > > > > > > I just removed all .md5 files from artifacts. I found md5 checksums > still > > > exist in .mds files and I didn't remove them from .mds file because it > is > > > generated by create-release script and Apache guidance is "should not" > > > instead of "must not". Please let me know if you think they need to be > > > removed as well. > > > > > > - Wangda > > > > > > > > > > > > On Mon, Apr 2, 2018 at 1:37 PM, Arpit Agarwal <mailto: > > > aagar...@hortonworks.com<mailto:aagar...@hortonworks.com>> wrote: > > > Thanks for putting together this RC, Wangda. > > > > > > The guidance from Apache is to omit MD5s, specifically: > > > SHOULD NOT supply a MD5 checksum file (because MD5 is too broken). > > > > > > https://www.apache.org/dev/release-distribution#sigs-and-sums > > > > > > > > > > > > > > > On Apr 2, 2018, at 7:03 AM, Wangda Tan <mailto:wheele...@gmail.com> > > > wrote: > > > > > > Hi Gera, > > > > > > It's my bad, I thought only src/bin tarball is enough. > > > > > > I just uploaded all other things under artifact/ to > > > http://people.apache.org/~wangda/hadoop-3.1.0-RC1/ > > > > > > Please let me know if you have any other comments. > > > > > > Thanks, > > > Wangda > > > > > > > > > On Mon, Apr 2, 2018 at 12:50 AM, Gera Shegalov <mailto: > ger...@gmail.com> > > > wrote: > > > > > > > > > Thanks, Wangda! > > > > > > There are many more artifacts in previous votes, e.g., see > > > http://home.apache.org/~junping_du/hadoop-2.8.3-RC0/ . Among others > the > > > site tarball is missing. > > > > > > On Sun, Apr 1, 2018 at 11:54 PM Sunil G <mailto:sun...@apache.org> > > wrote: > > > > > > > > > Thanks Wangda for initiating the release. > > > > > > I tested this RC built from source file. > > > > > > > > > - Tested MR apps (sleep, wc) and verified both new YARN UI and old RM > > > UI. > > > - Below feature sanity is done > > > - Application priority > > > - Application timeout > > > - Intra Queue preemption with priority based > > > - DS based affinity tests to verify placement constraints. > > > - Tested basic NodeLabel scenarios. > > > - Added couple of labels to few of nodes and behavior is coming > > > correct. > > > - Verified old UI and new YARN UI for labels. > > > - Submitted apps to labelled cluster and it works fine. > > > - Also performed few cli commands related to nodelabel. > > > - Test basic HA cases and seems correct. > > > - Tested new YARN UI . All pages are getting loaded correctly. > > > > > > > > > - Sunil > > > > > > On Fri, Mar 30, 2018 at 9:45 AM Wangda Tan <mailto:wheele...@gmail.com > > > > > wrote: > > > > > > > > > Hi folks, > > > > > > Thanks to the many who helped with this release since Dec 2017 [1]. > > > We've > > > > > > created RC1 for Apache Hadoop 3.1.0. The artifacts are available here: > > > > > > http://people.apache.org/~wangda/hadoop-3.1.0-RC1 > > > > > > The RC tag in git is release-3.1.0-RC1. Last git commit SHA is > > > 16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d > > > > > > The maven artifacts are available via http://repository.apache.org<h > > > ttp://repository.apache.org/> at > > > https://repository.apache.org/content/repositories/ > > > orgapachehadoop-1090/ > > > > > > This vote will run 5 days, ending on Apr 3 at 11:59 pm Pacific. > > > > > > 3.1.0 contains 766 [2] fixed JIRA issues since 3.0.0. Notable additions > > > include the first class GPU/FPGA support on YARN, Native services, > > > Support > > > > > > rich placement constraints in YARN, S3-related enhancements, allow HDFS > > > block replicas to be provided by an external storage system, etc. > > > > > > For 3.1.0 RC0 vote discussion, please see [3]. > > > > > > We’d like to use this as a starting release for 3.1.x [1], depending on > > > how > > > > > > it goes, get it stabilized and potentially use a 3.1.1 in several weeks > > > as > > > > > > the stable release. > > > > > > We have done testing with a pseudo cluster: > > > - Ran distributed job. > > > - GPU scheduling/isolation. > > > - Placement constraints (intra-application anti-affinity) by using > > > distributed shell. > > > > > > My +1 to start. > > > > > > Best, > > > Wangda/Vinod > > > > > > [1] > > > > > > https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104b > > > c9e19e559aedc5ebedb4ca08c8@%http://3Cyarn-dev.hadoop.apache.org< > > > http://3cyarn-dev.hadoop.apache.org/> > > > <http://3cyarn-dev.hadoop.apache.org/>%3E > > > > > > [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in > (3.1.0) > > > AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status = Resolved ORDER > > > BY > > > > > > fixVersion ASC > > > [3] > > > > > > https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d7 > > > 2d4061f26f83547e41d0983ea6@%http://3Cyarn-dev.hadoop.apache.org< > > > http://3cyarn-dev.hadoop.apache.org/> > > > <http://3cyarn-dev.hadoop.apache.org/>%3E > > > > > > > > >
