Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)

[Wangda Tan]

Thanks @Konstantin for reporting this issue, I will post comments on the
JIRA (HADOOP-15205)

- Wangda

On Tue, Apr 10, 2018 at 12:08 PM, Konstantin Shvachko 
wrote:

> A note to release managers. As discussed in
> https://issues.apache.org/jira/browse/HADOOP-15205
> We are producing release artifacts without sources jars. See e.g.
> https://repository.apache.org/content/repositories/releases/
> org/apache/hadoop/hadoop-common/3.1.0/
> I believe this has something to do with maven deployment stage,
> potentially maven-source-plugin.
> This is similar for all releases now, and I believe it should be fixed.
>
> Thanks,
> --Konstantin
>
> On Fri, Apr 6, 2018 at 2:01 PM, Wangda Tan  wrote:
>
>> Thanks guys for the additional votes! I just sent out announcement email.
>>
>> Best,
>> Wangda
>>
>> On Fri, Apr 6, 2018 at 2:32 AM, 俊平堵  wrote:
>>
>> > Thanks Wangda for the great work! Sorry for my late coming +1 (binding),
>> > based on:
>> >
>> > - Verified signatures
>> >
>> > - Verified checksums for source and binary artifacts
>> >
>> > - Built from source
>> >
>> > - Deployed a single node cluster
>> >
>> > - Verified web UIs, include Namenode, RM, etc.
>> >
>> > * Tried shell commands of HDFS and YARN
>> >
>> > * Ran sample MR jobs, include PI, Sleep, Terasort, etc.
>> >
>> >
>> > Thanks,
>> >
>> >
>> > Junping
>> >
>> >
>> >
>> > Wangda Tan 于2018年3月30日 周五下午12:15写道：
>> >
>> >> Hi folks,
>> >>
>> >> Thanks to the many who helped with this release since Dec 2017 [1].
>> We've
>> >> created RC1 for Apache Hadoop 3.1.0. The artifacts are available here:
>> >>
>> >> http://people.apache.org/~wangda/hadoop-3.1.0-RC1
>> >>
>> >> The RC tag in git is release-3.1.0-RC1. Last git commit SHA is
>> >> 16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d
>> >>
>> >> The maven artifacts are available via repository.apache.org at
>> >> https://repository.apache.org/content/repositories/orgapache
>> hadoop-1090/
>> >> This vote will run 5 days, ending on Apr 3 at 11:59 pm Pacific.
>> >>
>> >> 3.1.0 contains 766 [2] fixed JIRA issues since 3.0.0. Notable additions
>> >> include the first class GPU/FPGA support on YARN, Native services,
>> Support
>> >> rich placement constraints in YARN, S3-related enhancements, allow HDFS
>> >> block replicas to be provided by an external storage system, etc.
>> >>
>> >> For 3.1.0 RC0 vote discussion, please see [3].
>> >>
>> >> We’d like to use this as a starting release for 3.1.x [1], depending on
>> >> how
>> >> it goes, get it stabilized and potentially use a 3.1.1 in several
>> weeks as
>> >> the stable release.
>> >>
>> >> We have done testing with a pseudo cluster:
>> >> - Ran distributed job.
>> >> - GPU scheduling/isolation.
>> >> - Placement constraints (intra-application anti-affinity) by using
>> >> distributed shell.
>> >>
>> >> My +1 to start.
>> >>
>> >> Best,
>> >> Wangda/Vinod
>> >>
>> >> [1]
>> >> https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104b
>> >> c9e19e559aedc5ebedb4ca08c8@%3Cyarn-dev.hadoop.apache.org%3E
>> >> [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in
>> (3.1.0)
>> >> AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status = Resolved ORDER
>> BY
>> >> fixVersion ASC
>> >> [3]
>> >> https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d7
>> >> 2d4061f26f83547e41d0983ea6@%3Cyarn-dev.hadoop.apache.org%3E
>> >>
>> >
>>
>
>

Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)

[Konstantin Shvachko]

A note to release managers. As discussed in
https://issues.apache.org/jira/browse/HADOOP-15205
We are producing release artifacts without sources jars. See e.g.
https://repository.apache.org/content/repositories/releases/org/apache/hadoop/hadoop-common/3.1.0/
I believe this has something to do with maven deployment stage, potentially
maven-source-plugin.
This is similar for all releases now, and I believe it should be fixed.

Thanks,
--Konstantin

On Fri, Apr 6, 2018 at 2:01 PM, Wangda Tan  wrote:

> Thanks guys for the additional votes! I just sent out announcement email.
>
> Best,
> Wangda
>
> On Fri, Apr 6, 2018 at 2:32 AM, 俊平堵  wrote:
>
> > Thanks Wangda for the great work! Sorry for my late coming +1 (binding),
> > based on:
> >
> > - Verified signatures
> >
> > - Verified checksums for source and binary artifacts
> >
> > - Built from source
> >
> > - Deployed a single node cluster
> >
> > - Verified web UIs, include Namenode, RM, etc.
> >
> > * Tried shell commands of HDFS and YARN
> >
> > * Ran sample MR jobs, include PI, Sleep, Terasort, etc.
> >
> >
> > Thanks,
> >
> >
> > Junping
> >
> >
> >
> > Wangda Tan 于2018年3月30日 周五下午12:15写道：
> >
> >> Hi folks,
> >>
> >> Thanks to the many who helped with this release since Dec 2017 [1].
> We've
> >> created RC1 for Apache Hadoop 3.1.0. The artifacts are available here:
> >>
> >> http://people.apache.org/~wangda/hadoop-3.1.0-RC1
> >>
> >> The RC tag in git is release-3.1.0-RC1. Last git commit SHA is
> >> 16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d
> >>
> >> The maven artifacts are available via repository.apache.org at
> >> https://repository.apache.org/content/repositories/
> orgapachehadoop-1090/
> >> This vote will run 5 days, ending on Apr 3 at 11:59 pm Pacific.
> >>
> >> 3.1.0 contains 766 [2] fixed JIRA issues since 3.0.0. Notable additions
> >> include the first class GPU/FPGA support on YARN, Native services,
> Support
> >> rich placement constraints in YARN, S3-related enhancements, allow HDFS
> >> block replicas to be provided by an external storage system, etc.
> >>
> >> For 3.1.0 RC0 vote discussion, please see [3].
> >>
> >> We’d like to use this as a starting release for 3.1.x [1], depending on
> >> how
> >> it goes, get it stabilized and potentially use a 3.1.1 in several weeks
> as
> >> the stable release.
> >>
> >> We have done testing with a pseudo cluster:
> >> - Ran distributed job.
> >> - GPU scheduling/isolation.
> >> - Placement constraints (intra-application anti-affinity) by using
> >> distributed shell.
> >>
> >> My +1 to start.
> >>
> >> Best,
> >> Wangda/Vinod
> >>
> >> [1]
> >> https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104b
> >> c9e19e559aedc5ebedb4ca08c8@%3Cyarn-dev.hadoop.apache.org%3E
> >> [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.0)
> >> AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status = Resolved ORDER
> BY
> >> fixVersion ASC
> >> [3]
> >> https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d7
> >> 2d4061f26f83547e41d0983ea6@%3Cyarn-dev.hadoop.apache.org%3E
> >>
> >
>

Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)

[Wangda Tan]

Thanks guys for the additional votes! I just sent out announcement email.

Best,
Wangda

On Fri, Apr 6, 2018 at 2:32 AM, 俊平堵  wrote:

> Thanks Wangda for the great work! Sorry for my late coming +1 (binding),
> based on:
>
> - Verified signatures
>
> - Verified checksums for source and binary artifacts
>
> - Built from source
>
> - Deployed a single node cluster
>
> - Verified web UIs, include Namenode, RM, etc.
>
> * Tried shell commands of HDFS and YARN
>
> * Ran sample MR jobs, include PI, Sleep, Terasort, etc.
>
>
> Thanks,
>
>
> Junping
>
>
>
> Wangda Tan 于2018年3月30日 周五下午12:15写道：
>
>> Hi folks,
>>
>> Thanks to the many who helped with this release since Dec 2017 [1]. We've
>> created RC1 for Apache Hadoop 3.1.0. The artifacts are available here:
>>
>> http://people.apache.org/~wangda/hadoop-3.1.0-RC1
>>
>> The RC tag in git is release-3.1.0-RC1. Last git commit SHA is
>> 16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d
>>
>> The maven artifacts are available via repository.apache.org at
>> https://repository.apache.org/content/repositories/orgapachehadoop-1090/
>> This vote will run 5 days, ending on Apr 3 at 11:59 pm Pacific.
>>
>> 3.1.0 contains 766 [2] fixed JIRA issues since 3.0.0. Notable additions
>> include the first class GPU/FPGA support on YARN, Native services, Support
>> rich placement constraints in YARN, S3-related enhancements, allow HDFS
>> block replicas to be provided by an external storage system, etc.
>>
>> For 3.1.0 RC0 vote discussion, please see [3].
>>
>> We’d like to use this as a starting release for 3.1.x [1], depending on
>> how
>> it goes, get it stabilized and potentially use a 3.1.1 in several weeks as
>> the stable release.
>>
>> We have done testing with a pseudo cluster:
>> - Ran distributed job.
>> - GPU scheduling/isolation.
>> - Placement constraints (intra-application anti-affinity) by using
>> distributed shell.
>>
>> My +1 to start.
>>
>> Best,
>> Wangda/Vinod
>>
>> [1]
>> https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104b
>> c9e19e559aedc5ebedb4ca08c8@%3Cyarn-dev.hadoop.apache.org%3E
>> [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.0)
>> AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status = Resolved ORDER BY
>> fixVersion ASC
>> [3]
>> https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d7
>> 2d4061f26f83547e41d0983ea6@%3Cyarn-dev.hadoop.apache.org%3E
>>
>

Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)

[俊平堵]

Thanks Wangda for the great work! Sorry for my late coming +1 (binding),
based on:

- Verified signatures

- Verified checksums for source and binary artifacts

- Built from source

- Deployed a single node cluster

- Verified web UIs, include Namenode, RM, etc.

* Tried shell commands of HDFS and YARN

* Ran sample MR jobs, include PI, Sleep, Terasort, etc.


Thanks,


Junping



Wangda Tan 于2018年3月30日 周五下午12:15写道：

> Hi folks,
>
> Thanks to the many who helped with this release since Dec 2017 [1]. We've
> created RC1 for Apache Hadoop 3.1.0. The artifacts are available here:
>
> http://people.apache.org/~wangda/hadoop-3.1.0-RC1
>
> The RC tag in git is release-3.1.0-RC1. Last git commit SHA is
> 16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d
>
> The maven artifacts are available via repository.apache.org at
> https://repository.apache.org/content/repositories/orgapachehadoop-1090/
> This vote will run 5 days, ending on Apr 3 at 11:59 pm Pacific.
>
> 3.1.0 contains 766 [2] fixed JIRA issues since 3.0.0. Notable additions
> include the first class GPU/FPGA support on YARN, Native services, Support
> rich placement constraints in YARN, S3-related enhancements, allow HDFS
> block replicas to be provided by an external storage system, etc.
>
> For 3.1.0 RC0 vote discussion, please see [3].
>
> We’d like to use this as a starting release for 3.1.x [1], depending on how
> it goes, get it stabilized and potentially use a 3.1.1 in several weeks as
> the stable release.
>
> We have done testing with a pseudo cluster:
> - Ran distributed job.
> - GPU scheduling/isolation.
> - Placement constraints (intra-application anti-affinity) by using
> distributed shell.
>
> My +1 to start.
>
> Best,
> Wangda/Vinod
>
> [1]
>
> https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104bc9e19e559aedc5ebedb4ca08c8@%3Cyarn-dev.hadoop.apache.org%3E
> [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.0)
> AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status = Resolved ORDER BY
> fixVersion ASC
> [3]
>
> https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d72d4061f26f83547e41d0983ea6@%3Cyarn-dev.hadoop.apache.org%3E
>

Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)

[Gour Saha]

ures
> * Verified checksums for source and binary artefacts
> * Sanity checked jars on r.a.o.
> * Built from source
> * Deployed to 3 node secure cluster with NameNode HA
> * Verified HDFS web UIs
> * Tried out HDFS shell commands
> * Ran sample MapReduce jobs
>
> Thanks!
>
>
> --
> From: Wangda Tan <wheele...@gmail.com>
> Date: Monday, April 2, 2018 at 9:25 PM
> To: Arpit Agarwal <aagar...@hortonworks.com>
> Cc: Gera Shegalov <ger...@gmail.com>, Sunil G <sun...@apache.org>, "
> yarn-...@hadoop.apache.org" <yarn-...@hadoop.apache.org>, Hdfs-dev <
> hdfs-...@hadoop.apache.org>, Hadoop Common <common-dev@hadoop.apache.org>,
> "mapreduce-...@hadoop.apache.org" <mapreduce-...@hadoop.apache.org>,
> Vinod Kumar Vavilapalli <vino...@apache.org>
> Subject: Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
>
> As pointed by Arpit, the previously deployed shared jars are incorrect.
> Just redeployed jars and staged. @Arpit, could you please check the 
updated
> Maven repo? https://repository.apache.org/content/repositories/
> orgapachehadoop-1092
>
> Since the jars inside binary tarballs are correct (
> http://people.apache.org/~wangda/hadoop-3.1.0-RC1/). I think we don't
> need roll another RC, just update Maven repo should be sufficient.
>
> Best,
> Wangda
>
>
> On Mon, Apr 2, 2018 at 2:39 PM, Wangda Tan <mailto:wheele...@gmail.com>
> wrote:
> Hi Arpit,
>
> Thanks for pointing out this.
>
> I just removed all .md5 files from artifacts. I found md5 checksums still
> exist in .mds files and I didn't remove them from .mds file because it is
> generated by create-release script and Apache guidance is "should not"
> instead of "must not". Please let me know if you think they need to be
> removed as well.
>
> - Wangda
>
>
>
> On Mon, Apr 2, 2018 at 1:37 PM, Arpit Agarwal  aagar...@hortonworks.com> wrote:
> Thanks for putting together this RC, Wangda.
>
> The guidance from Apache is to omit MD5s, specifically:
>
> SHOULD NOT supply a MD5 checksum file (because MD5 is too broken).
>
>
> https://www.apache.org/dev/release-distribution#sigs-and-sums
>
>
>
>
> On Apr 2, 2018, at 7:03 AM, Wangda Tan <mailto:wheele...@gmail.com>
> wrote:
>
> Hi Gera,
>
> It's my bad, I thought only src/bin tarball is enough.
>
> I just uploaded all other things under artifact/ to
> http://people.apache.org/~wangda/hadoop-3.1.0-RC1/
>
> Please let me know if you have any other comments.
>
> Thanks,
> Wangda
>
>
> On Mon, Apr 2, 2018 at 12:50 AM, Gera Shegalov <mailto:ger...@gmail.com>
> wrote:
>
>
> Thanks, Wangda!
>
> There are many more artifacts in previous votes, e.g., see
> http://home.apache.org/~junping_du/hadoop-2.8.3-RC0/ .  Among others the
> site tarball is missing.
>
> On Sun, Apr 1, 2018 at 11:54 PM Sunil G <mailto:sun...@apache.org> wrote:
>
>
> Thanks Wangda for initiating the release.
>
> I tested this RC built from source file.
>
>
>  - Tested MR apps (sleep, wc) and verified both new YARN UI and old RM
> UI.
>  - Below feature sanity is done
> - Application priority
> - Application timeout
> - Intra Queue preemption with priority based
> - DS based affinity tests to verify placement constraints.
>  - Tested basic NodeLabel scenarios.
> - Added couple of labels to few of nodes and behavior is coming
> correct.
> - Verified old UI  and new YARN UI for labels.
> - Submitted apps to labelled cluster and it works fine.
> - Also performed few cli commands related to nodelabel.
>  - Test basic HA cases and seems correct.
>  - Tested new YARN UI . All pages are getting loaded correctly.
>
>
> - Sunil
>
> On Fri, Mar 30, 2018 at 9:45 AM Wangda Tan <mailto:wheele...@gmail.com>
> wrote:
>
>
> Hi folks,
>
> Thanks to the many who helped with this release since Dec 2017 [1].
> We've
>
> created RC1 for Apache Hadoop 3.1.0. The artifacts are available here:
>
> http://people.apache.org/~wangda/hadoop-3.1.0-RC1
>

Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)

[Wangda Tan]

Thanks everybody for voting!
The vote passes successfully with 11 binding +1 votes, 8 non-binding +1
votes and no -1s.

I will work on the staging and releases.

Best,
Wangda Tan


On Thu, Apr 5, 2018 at 10:46 AM, Vinod Kumar Vavilapalli <vino...@apache.org
> wrote:

> That is a great observation. And I missed your previous email about the
> shaded vs unshaded jars already getting fixed.
>
> I guess we are good to go.
>
> 
> --
>
> Looking at the RC. Went through my usual check-list. Here's my summary.
>
> Verification
> - [Check] Successful recompilation from source tar-ball
> - [Check] Signature verification
> -- Note: The format of the mds files changed a bit - not a biggie.
> -- For e.g, in 3.0.0 and 2.x releases, it has lines of the form "
> *hadoop-3.0.0-src.tar.gz*: SHA256 = 8B21AD79 50BD606B 2A7C91FB AE9FC279
> 7BCED50B B2600318 B7E0BE3A 74DFFF71"
> -- But in 3.1.0 RC it is, "
> */build/source/target/artifacts/hadoop-3.1.0.tar.gz*: SHA256 = 670D2CED
> 595FA42D 9FA1A93C 4E39B39F 47002CAD 1553D9DF 163EE828 CA5143E7"
> - [Check] Generating dist tarballs from source tar-ball
> - [Check] Testing
>-- Start NN, DN, RM, NM, JHS, Timeline Service
>-- Ran dist-shell example, MR sleep, wordcount, randomwriter, sort,
> grep, pi
>-- Tested CLIs to print nodes, apps etc and also navigated UIs
>
> +1 binding.
>
> Thanks
> +Vinod
>
> On Apr 3, 2018, at 8:13 PM, Wangda Tan <wheele...@gmail.com> wrote:
>
> Hi Vinod / Arpit,
>
> I checked following versions:
> - 2.6.5 / 2.7.5 / 2.8.3 / 2.9.0 / 3.0.1:
>
> Jars in maven repo [1] are *always* different from jars in the binary
>
> tarball [2]: (I only checked hadoop-yarn-api-version.jar)
>
> (Following numbers are sizes of the jar)
> 2.6.5:
> - Jar in Maven: 1896185
> - Jar in tarball: 1891485
>
> 2.7.5:
> - Jar in Maven: 2039371 (md5: 15e76f7c734b49315ef2bce952509ddf)
> - Jar in tarball: 2039371 (md5: 0ef9f42f587401f5b49b39f27459f3ef)
> (Even size is same, md5 is different)
>
> 2.8.3:
> - Jar in Maven: 2451433
> - Jar in tarball: 2438975
>
> 2.9.0:
> - Jar in Maven: 2791477
> - Jar in tarball: 289
>
> 3.0.1:
> - Jar in Maven: 2852604
> - Jar in tarball: 2851373
>
> I guess the differences come from our release process.
>
> Thanks,
> Wangda
>
> [1] Maven jars are downloaded from
> https://repository.apache.org/service/local/repositories/
> releases/content/org/apache/hadoop/hadoop-yarn-api/
> /hadoop-yarn-api-.jar
> [2] Binary tarballs downloaded from http://apache.claz.org/hadoop/common/
>
>
> On Tue, Apr 3, 2018 at 4:25 PM, Vinod Kumar Vavilapalli <
> vino...@apache.org>
> wrote:
>
> We vote on the source code. The binaries are convenience artifacts.
>
> This is what I would do - (a) Just replace both the maven jars as well as
> the binaries to be consistent and correct. And then (b) Give a couple more
> days for folks who tested on the binaries to reverify - I count one such
> clear vote as of now.
>
> Thanks
> +Vinod
>
>
> On Apr 3, 2018, at 3:30 PM, Wangda Tan <wheele...@gmail.com> wrote:
>
> HI Arpit,
>
> I think it won't match if we do rebuild. It should be fine as far as
> they're signed, correct? I don't see any policy doesn't allow this.
>
> Thanks,
> Wangda
>
>
> On Tue, Apr 3, 2018 at 9:33 AM, Arpit Agarwal <aagar...@hortonworks.com>
> wrote:
>
> Thanks Wangda, I see the shaded jars now.
>
> Are the repo jars required to be the same as the binary release? They
> don’t match right now, probably they got rebuilt.
>
> +1 (binding), modulo that remaining question.
>
> * Verified signatures
> * Verified checksums for source and binary artefacts
> * Sanity checked jars on r.a.o.
> * Built from source
> * Deployed to 3 node secure cluster with NameNode HA
> * Verified HDFS web UIs
> * Tried out HDFS shell commands
> * Ran sample MapReduce jobs
>
> Thanks!
>
>
> --
> From: Wangda Tan <wheele...@gmail.com>
> Date: Monday, April 2, 2018 at 9:25 PM
> To: Arpit Agarwal <aagar...@hortonworks.com>
> Cc: Gera Shegalov <ger...@gmail.com>, Sunil G <sun...@apache.org>, "
> yarn-...@hadoop.apache.org" <yarn-...@hadoop.apache.org>, Hdfs-dev <
> hdfs-...@hadoop.apache.org>, Hadoop Common <common-dev@hadoop.apache.org>,
> "mapreduce-...@hadoop.apache.org" <mapreduce-...@hadoop.apache.org>,
> Vinod Kumar Vavilapalli <vino...@apache.org>
> Subject: 

Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)

[Vinod Kumar Vavilapalli]

That is a great observation. And I missed your previous email about the shaded 
vs unshaded jars already getting fixed.

I guess we are good to go.

--

Looking at the RC. Went through my usual check-list. Here's my summary.

Verification
- [Check] Successful recompilation from source tar-ball
- [Check] Signature verification
-- Note: The format of the mds files changed a bit - not a biggie.
-- For e.g, in 3.0.0 and 2.x releases, it has lines of the form 
"hadoop-3.0.0-src.tar.gz: SHA256 = 8B21AD79 50BD606B 2A7C91FB AE9FC279 7BCED50B 
B2600318 B7E0BE3A 74DFFF71"
-- But in 3.1.0 RC it is, 
"/build/source/target/artifacts/hadoop-3.1.0.tar.gz: SHA256 = 670D2CED 595FA42D 
9FA1A93C 4E39B39F 47002CAD 1553D9DF 163EE828 CA5143E7"
- [Check] Generating dist tarballs from source tar-ball
- [Check] Testing
   -- Start NN, DN, RM, NM, JHS, Timeline Service
   -- Ran dist-shell example, MR sleep, wordcount, randomwriter, sort, grep, pi
   -- Tested CLIs to print nodes, apps etc and also navigated UIs

+1 binding.

Thanks
+Vinod

> On Apr 3, 2018, at 8:13 PM, Wangda Tan <wheele...@gmail.com> wrote:
> 
> Hi Vinod / Arpit,
> 
> I checked following versions:
> - 2.6.5 / 2.7.5 / 2.8.3 / 2.9.0 / 3.0.1:
> 
> Jars in maven repo [1] are *always* different from jars in the binary
> tarball [2]: (I only checked hadoop-yarn-api-version.jar)
> 
> (Following numbers are sizes of the jar)
> 2.6.5:
> - Jar in Maven: 1896185
> - Jar in tarball: 1891485
> 
> 2.7.5:
> - Jar in Maven: 2039371 (md5: 15e76f7c734b49315ef2bce952509ddf)
> - Jar in tarball: 2039371 (md5: 0ef9f42f587401f5b49b39f27459f3ef)
> (Even size is same, md5 is different)
> 
> 2.8.3:
> - Jar in Maven: 2451433
> - Jar in tarball: 2438975
> 
> 2.9.0:
> - Jar in Maven: 2791477
> - Jar in tarball: 289
> 
> 3.0.1:
> - Jar in Maven: 2852604
> - Jar in tarball: 2851373
> 
> I guess the differences come from our release process.
> 
> Thanks,
> Wangda
> 
> [1] Maven jars are downloaded from
> https://repository.apache.org/service/local/repositories/releases/content/org/apache/hadoop/hadoop-yarn-api/
> /hadoop-yarn-api-.jar
> [2] Binary tarballs downloaded from http://apache.claz.org/hadoop/common/
> 
> 
> On Tue, Apr 3, 2018 at 4:25 PM, Vinod Kumar Vavilapalli <vino...@apache.org>
> wrote:
> 
>> We vote on the source code. The binaries are convenience artifacts.
>> 
>> This is what I would do - (a) Just replace both the maven jars as well as
>> the binaries to be consistent and correct. And then (b) Give a couple more
>> days for folks who tested on the binaries to reverify - I count one such
>> clear vote as of now.
>> 
>> Thanks
>> +Vinod
>> 
>> 
>> On Apr 3, 2018, at 3:30 PM, Wangda Tan <wheele...@gmail.com> wrote:
>> 
>> HI Arpit,
>> 
>> I think it won't match if we do rebuild. It should be fine as far as
>> they're signed, correct? I don't see any policy doesn't allow this.
>> 
>> Thanks,
>> Wangda
>> 
>> 
>> On Tue, Apr 3, 2018 at 9:33 AM, Arpit Agarwal <aagar...@hortonworks.com>
>> wrote:
>> 
>>> Thanks Wangda, I see the shaded jars now.
>>> 
>>> Are the repo jars required to be the same as the binary release? They
>>> don’t match right now, probably they got rebuilt.
>>> 
>>> +1 (binding), modulo that remaining question.
>>> 
>>> * Verified signatures
>>> * Verified checksums for source and binary artefacts
>>> * Sanity checked jars on r.a.o.
>>> * Built from source
>>> * Deployed to 3 node secure cluster with NameNode HA
>>> * Verified HDFS web UIs
>>> * Tried out HDFS shell commands
>>> * Ran sample MapReduce jobs
>>> 
>>> Thanks!
>>> 
>>> 
>>> ----------
>>> From: Wangda Tan <wheele...@gmail.com>
>>> Date: Monday, April 2, 2018 at 9:25 PM
>>> To: Arpit Agarwal <aagar...@hortonworks.com>
>>> Cc: Gera Shegalov <ger...@gmail.com>, Sunil G <sun...@apache.org>, "
>>> yarn-...@hadoop.apache.org" <yarn-...@hadoop.apache.org>, Hdfs-dev <
>>> hdfs-...@hadoop.apache.org>, Hadoop Common <common-dev@hadoop.apache.org>,
>>> "mapreduce-...@hadoop.apache.org" <mapreduce-...@hadoop.apache.org>,
>>> Vinod Kumar Vavilapalli <vino...@apache.org>
>>> Subject: Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
>>> 
>>> As point

Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)

[Shane Kumpf]

> wrote:
> > >
> > > HI Arpit,
> > >
> > > I think it won't match if we do rebuild. It should be fine as far as
> > > they're signed, correct? I don't see any policy doesn't allow this.
> > >
> > > Thanks,
> > > Wangda
> > >
> > >
> > > On Tue, Apr 3, 2018 at 9:33 AM, Arpit Agarwal <
> aagar...@hortonworks.com<
> > > mailto:aagar...@hortonworks.com>>
> > > wrote:
> > >
> > > Thanks Wangda, I see the shaded jars now.
> > >
> > > Are the repo jars required to be the same as the binary release? They
> > > don’t match right now, probably they got rebuilt.
> > >
> > > +1 (binding), modulo that remaining question.
> > >
> > > * Verified signatures
> > > * Verified checksums for source and binary artefacts
> > > * Sanity checked jars on r.a.o.
> > > * Built from source
> > > * Deployed to 3 node secure cluster with NameNode HA
> > > * Verified HDFS web UIs
> > > * Tried out HDFS shell commands
> > > * Ran sample MapReduce jobs
> > >
> > > Thanks!
> > >
> > >
> > > --
> > > From: Wangda Tan <wheele...@gmail.com<mailto:wheele...@gmail.com>>
> > > Date: Monday, April 2, 2018 at 9:25 PM
> > > To: Arpit Agarwal <aagar...@hortonworks.com aagarwal@hortonworks.
> > > com>>
> > > Cc: Gera Shegalov <ger...@gmail.com<mailto:ger...@gmail.com>>, Sunil
> G <
> > > sun...@apache.org<mailto:sun...@apache.org>>, "
> > > yarn-...@hadoop.apache.org<mailto:yarn-...@hadoop.apache.org>" <
> > > yarn-...@hadoop.apache.org<mailto:yarn-...@hadoop.apache.org>>,
> > Hdfs-dev <
> > > hdfs-...@hadoop.apache.org<mailto:hdfs-...@hadoop.apache.org>>, Hadoop
> > > Common <common-dev@hadoop.apache.org<mailto:common-dev@hadoop.
> apache.org
> > > >>,
> > > "mapreduce-...@hadoop.apache.org<mailto:mapreduce-dev@
> hadoop.apache.org
> > >"
> > > <mapreduce-...@hadoop.apache.org<mailto:mapreduce-dev@
> hadoop.apache.org
> > >>,
> > > Vinod Kumar Vavilapalli <vino...@apache.org<mailto:vino...@apache.org
> >>
> > > Subject: Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
> > >
> > > As pointed by Arpit, the previously deployed shared jars are incorrect.
> > > Just redeployed jars and staged. @Arpit, could you please check the
> > updated
> > > Maven repo? https://repository.apache.org/content/repositories/
> > > orgapachehadoop-1092
> > >
> > > Since the jars inside binary tarballs are correct (
> > > http://people.apache.org/~wangda/hadoop-3.1.0-RC1/). I think we don't
> > > need roll another RC, just update Maven repo should be sufficient.
> > >
> > > Best,
> > > Wangda
> > >
> > >
> > > On Mon, Apr 2, 2018 at 2:39 PM, Wangda Tan <mailto:wheele...@gmail.com
> >
> > > wrote:
> > > Hi Arpit,
> > >
> > > Thanks for pointing out this.
> > >
> > > I just removed all .md5 files from artifacts. I found md5 checksums
> still
> > > exist in .mds files and I didn't remove them from .mds file because it
> is
> > > generated by create-release script and Apache guidance is "should not"
> > > instead of "must not". Please let me know if you think they need to be
> > > removed as well.
> > >
> > > - Wangda
> > >
> > >
> > >
> > > On Mon, Apr 2, 2018 at 1:37 PM, Arpit Agarwal  > > aagar...@hortonworks.com<mailto:aagar...@hortonworks.com>> wrote:
> > > Thanks for putting together this RC, Wangda.
> > >
> > > The guidance from Apache is to omit MD5s, specifically:
> > > SHOULD NOT supply a MD5 checksum file (because MD5 is too broken).
> > >
> > > https://www.apache.org/dev/release-distribution#sigs-and-sums
> > >
> > >
> > >
> > >
> > > On Apr 2, 2018, at 7:03 AM, Wangda Tan <mailto:wheele...@gmail.com>
> > > wrote:
> > >
> > > Hi Gera,
> > >
> > > It's my bad, I thought only src/bin tarball is enough.
> > >
> > > I just uploaded all other things under artifact/ to
> > > http://people.apache.org/~wangda/hadoop-3.1.0-RC1/
> > >
> > > Please let me know if you

Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)

[Naganarasimha Garla]

8 at 9:25 PM
> > To: Arpit Agarwal <aagar...@hortonworks.com<mailto:aagarwal@hortonworks.
> > com>>
> > Cc: Gera Shegalov <ger...@gmail.com<mailto:ger...@gmail.com>>, Sunil G <
> > sun...@apache.org<mailto:sun...@apache.org>>, "
> > yarn-...@hadoop.apache.org<mailto:yarn-...@hadoop.apache.org>" <
> > yarn-...@hadoop.apache.org<mailto:yarn-...@hadoop.apache.org>>,
> Hdfs-dev <
> > hdfs-...@hadoop.apache.org<mailto:hdfs-...@hadoop.apache.org>>, Hadoop
> > Common <common-dev@hadoop.apache.org<mailto:common-dev@hadoop.apache.org
> > >>,
> > "mapreduce-...@hadoop.apache.org<mailto:mapreduce-...@hadoop.apache.org
> >"
> > <mapreduce-...@hadoop.apache.org<mailto:mapreduce-...@hadoop.apache.org
> >>,
> > Vinod Kumar Vavilapalli <vino...@apache.org<mailto:vino...@apache.org>>
> > Subject: Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
> >
> > As pointed by Arpit, the previously deployed shared jars are incorrect.
> > Just redeployed jars and staged. @Arpit, could you please check the
> updated
> > Maven repo? https://repository.apache.org/content/repositories/
> > orgapachehadoop-1092
> >
> > Since the jars inside binary tarballs are correct (
> > http://people.apache.org/~wangda/hadoop-3.1.0-RC1/). I think we don't
> > need roll another RC, just update Maven repo should be sufficient.
> >
> > Best,
> > Wangda
> >
> >
> > On Mon, Apr 2, 2018 at 2:39 PM, Wangda Tan <mailto:wheele...@gmail.com>
> > wrote:
> > Hi Arpit,
> >
> > Thanks for pointing out this.
> >
> > I just removed all .md5 files from artifacts. I found md5 checksums still
> > exist in .mds files and I didn't remove them from .mds file because it is
> > generated by create-release script and Apache guidance is "should not"
> > instead of "must not". Please let me know if you think they need to be
> > removed as well.
> >
> > - Wangda
> >
> >
> >
> > On Mon, Apr 2, 2018 at 1:37 PM, Arpit Agarwal  > aagar...@hortonworks.com<mailto:aagar...@hortonworks.com>> wrote:
> > Thanks for putting together this RC, Wangda.
> >
> > The guidance from Apache is to omit MD5s, specifically:
> > SHOULD NOT supply a MD5 checksum file (because MD5 is too broken).
> >
> > https://www.apache.org/dev/release-distribution#sigs-and-sums
> >
> >
> >
> >
> > On Apr 2, 2018, at 7:03 AM, Wangda Tan <mailto:wheele...@gmail.com>
> > wrote:
> >
> > Hi Gera,
> >
> > It's my bad, I thought only src/bin tarball is enough.
> >
> > I just uploaded all other things under artifact/ to
> > http://people.apache.org/~wangda/hadoop-3.1.0-RC1/
> >
> > Please let me know if you have any other comments.
> >
> > Thanks,
> > Wangda
> >
> >
> > On Mon, Apr 2, 2018 at 12:50 AM, Gera Shegalov <mailto:ger...@gmail.com>
> > wrote:
> >
> >
> > Thanks, Wangda!
> >
> > There are many more artifacts in previous votes, e.g., see
> > http://home.apache.org/~junping_du/hadoop-2.8.3-RC0/ .  Among others the
> > site tarball is missing.
> >
> > On Sun, Apr 1, 2018 at 11:54 PM Sunil G <mailto:sun...@apache.org>
> wrote:
> >
> >
> > Thanks Wangda for initiating the release.
> >
> > I tested this RC built from source file.
> >
> >
> >  - Tested MR apps (sleep, wc) and verified both new YARN UI and old RM
> > UI.
> >  - Below feature sanity is done
> > - Application priority
> > - Application timeout
> > - Intra Queue preemption with priority based
> > - DS based affinity tests to verify placement constraints.
> >  - Tested basic NodeLabel scenarios.
> > - Added couple of labels to few of nodes and behavior is coming
> > correct.
> > - Verified old UI  and new YARN UI for labels.
> > - Submitted apps to labelled cluster and it works fine.
> > - Also performed few cli commands related to nodelabel.
> >  - Test basic HA cases and seems correct.
> >  - Tested new YARN UI . All pages are getting loaded correctly.
> >
> >
> > - Sunil
> >
> > On Fri, Mar 30, 2018 at 9:45 AM Wangda Tan <mailto:wheele...@gmail.com>
> > wrote:
> >
> >
> > Hi folks,
> >
> > Thanks to the many who helped with this release since Dec 2017 [1].
> > We've
> >
> > created RC1 for Apache Hadoop 3.1.0. The artifacts are availabl

Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)

[Suma Shivaprasad]

+1 (non binding)


*Verified - User Group Queue mapping - Node labels with New UI- Dynamic
queuesThanksSuma*


On Wed, Apr 4, 2018 at 11:48 AM, Steve Loughran <ste...@hortonworks.com>
wrote:

> that's "dangerously interesting". I think you are right, and I also think
> it'll just be the version files which get generated
>
> anyway, +1 binding
>
> * ran my new Hadoop-3 profile on spark (SPARK-23807), with the committer
> binding, then my downstream tests. All is well, provided you also have a
> spark hive JAR patched to accept hadoop 3 as a legitimate hadoop version.
> That's an ongoing issue in the Spark project. With that JAR on my CP my
> downstream tests were all happy (yesterday)
>
> * today the staging files seem to be missing, at least maven is unable to
> find them even when I turn the spark snapshots-and-staging profile on.
> That'll be the maven dist process at play, nothing else
>
> On 4 Apr 2018, at 04:13, Wangda Tan <wheele...@gmail.com<mailto:wh
> eele...@gmail.com>> wrote:
>
> Hi Vinod / Arpit,
>
> I checked following versions:
> - 2.6.5 / 2.7.5 / 2.8.3 / 2.9.0 / 3.0.1:
>
> Jars in maven repo [1] are *always* different from jars in the binary
> tarball [2]: (I only checked hadoop-yarn-api-version.jar)
>
> (Following numbers are sizes of the jar)
> 2.6.5:
> - Jar in Maven: 1896185
> - Jar in tarball: 1891485
>
> 2.7.5:
> - Jar in Maven: 2039371 (md5: 15e76f7c734b49315ef2bce952509ddf)
> - Jar in tarball: 2039371 (md5: 0ef9f42f587401f5b49b39f27459f3ef)
> (Even size is same, md5 is different)
>
> 2.8.3:
> - Jar in Maven: 2451433
> - Jar in tarball: 2438975
>
> 2.9.0:
> - Jar in Maven: 2791477
> - Jar in tarball: 289
>
> 3.0.1:
> - Jar in Maven: 2852604
> - Jar in tarball: 2851373
>
> I guess the differences come from our release process.
>
> Thanks,
> Wangda
>
> [1] Maven jars are downloaded from
> https://repository.apache.org/service/local/repositories/
> releases/content/org/apache/hadoop/hadoop-yarn-api/
> /hadoop-yarn-api-.jar
> [2] Binary tarballs downloaded from http://apache.claz.org/hadoop/common/
>
>
> On Tue, Apr 3, 2018 at 4:25 PM, Vinod Kumar Vavilapalli <
> vino...@apache.org<mailto:vino...@apache.org>>
> wrote:
>
> We vote on the source code. The binaries are convenience artifacts.
>
> This is what I would do - (a) Just replace both the maven jars as well as
> the binaries to be consistent and correct. And then (b) Give a couple more
> days for folks who tested on the binaries to reverify - I count one such
> clear vote as of now.
>
> Thanks
> +Vinod
>
>
> On Apr 3, 2018, at 3:30 PM, Wangda Tan <wheele...@gmail.com<mailto:wh
> eele...@gmail.com>> wrote:
>
> HI Arpit,
>
> I think it won't match if we do rebuild. It should be fine as far as
> they're signed, correct? I don't see any policy doesn't allow this.
>
> Thanks,
> Wangda
>
>
> On Tue, Apr 3, 2018 at 9:33 AM, Arpit Agarwal <aagar...@hortonworks.com<
> mailto:aagar...@hortonworks.com>>
> wrote:
>
> Thanks Wangda, I see the shaded jars now.
>
> Are the repo jars required to be the same as the binary release? They
> don’t match right now, probably they got rebuilt.
>
> +1 (binding), modulo that remaining question.
>
> * Verified signatures
> * Verified checksums for source and binary artefacts
> * Sanity checked jars on r.a.o.
> * Built from source
> * Deployed to 3 node secure cluster with NameNode HA
> * Verified HDFS web UIs
> * Tried out HDFS shell commands
> * Ran sample MapReduce jobs
>
> Thanks!
>
>
> --
> From: Wangda Tan <wheele...@gmail.com<mailto:wheele...@gmail.com>>
> Date: Monday, April 2, 2018 at 9:25 PM
> To: Arpit Agarwal <aagar...@hortonworks.com<mailto:aagarwal@hortonworks.
> com>>
> Cc: Gera Shegalov <ger...@gmail.com<mailto:ger...@gmail.com>>, Sunil G <
> sun...@apache.org<mailto:sun...@apache.org>>, "
> yarn-...@hadoop.apache.org<mailto:yarn-...@hadoop.apache.org>" <
> yarn-...@hadoop.apache.org<mailto:yarn-...@hadoop.apache.org>>, Hdfs-dev <
> hdfs-...@hadoop.apache.org<mailto:hdfs-...@hadoop.apache.org>>, Hadoop
> Common <common-dev@hadoop.apache.org<mailto:common-dev@hadoop.apache.org
> >>,
> "mapreduce-...@hadoop.apache.org<mailto:mapreduce-...@hadoop.apache.org>"
> <mapreduce-...@hadoop.apache.org<mailto:mapreduce-...@hadoop.apache.org>>,
> Vinod Kumar Vavilapalli <vino...@apache.org<mailto:vino...@apache.org>>
> Subject: Re: [VOTE] Release Apache Hadoo

Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)

[Steve Loughran]

that's "dangerously interesting". I think you are right, and I also think it'll 
just be the version files which get generated

anyway, +1 binding

* ran my new Hadoop-3 profile on spark (SPARK-23807), with the committer 
binding, then my downstream tests. All is well, provided you also have a spark 
hive JAR patched to accept hadoop 3 as a legitimate hadoop version. That's an 
ongoing issue in the Spark project. With that JAR on my CP my downstream tests 
were all happy (yesterday)

* today the staging files seem to be missing, at least maven is unable to find 
them even when I turn the spark snapshots-and-staging profile on. That'll be 
the maven dist process at play, nothing else

On 4 Apr 2018, at 04:13, Wangda Tan 
<wheele...@gmail.com<mailto:wheele...@gmail.com>> wrote:

Hi Vinod / Arpit,

I checked following versions:
- 2.6.5 / 2.7.5 / 2.8.3 / 2.9.0 / 3.0.1:

Jars in maven repo [1] are *always* different from jars in the binary
tarball [2]: (I only checked hadoop-yarn-api-version.jar)

(Following numbers are sizes of the jar)
2.6.5:
- Jar in Maven: 1896185
- Jar in tarball: 1891485

2.7.5:
- Jar in Maven: 2039371 (md5: 15e76f7c734b49315ef2bce952509ddf)
- Jar in tarball: 2039371 (md5: 0ef9f42f587401f5b49b39f27459f3ef)
(Even size is same, md5 is different)

2.8.3:
- Jar in Maven: 2451433
- Jar in tarball: 2438975

2.9.0:
- Jar in Maven: 2791477
- Jar in tarball: 289

3.0.1:
- Jar in Maven: 2852604
- Jar in tarball: 2851373

I guess the differences come from our release process.

Thanks,
Wangda

[1] Maven jars are downloaded from
https://repository.apache.org/service/local/repositories/releases/content/org/apache/hadoop/hadoop-yarn-api/
/hadoop-yarn-api-.jar
[2] Binary tarballs downloaded from http://apache.claz.org/hadoop/common/


On Tue, Apr 3, 2018 at 4:25 PM, Vinod Kumar Vavilapalli 
<vino...@apache.org<mailto:vino...@apache.org>>
wrote:

We vote on the source code. The binaries are convenience artifacts.

This is what I would do - (a) Just replace both the maven jars as well as
the binaries to be consistent and correct. And then (b) Give a couple more
days for folks who tested on the binaries to reverify - I count one such
clear vote as of now.

Thanks
+Vinod


On Apr 3, 2018, at 3:30 PM, Wangda Tan 
<wheele...@gmail.com<mailto:wheele...@gmail.com>> wrote:

HI Arpit,

I think it won't match if we do rebuild. It should be fine as far as
they're signed, correct? I don't see any policy doesn't allow this.

Thanks,
Wangda


On Tue, Apr 3, 2018 at 9:33 AM, Arpit Agarwal 
<aagar...@hortonworks.com<mailto:aagar...@hortonworks.com>>
wrote:

Thanks Wangda, I see the shaded jars now.

Are the repo jars required to be the same as the binary release? They
don’t match right now, probably they got rebuilt.

+1 (binding), modulo that remaining question.

* Verified signatures
* Verified checksums for source and binary artefacts
* Sanity checked jars on r.a.o.
* Built from source
* Deployed to 3 node secure cluster with NameNode HA
* Verified HDFS web UIs
* Tried out HDFS shell commands
* Ran sample MapReduce jobs

Thanks!


--
From: Wangda Tan <wheele...@gmail.com<mailto:wheele...@gmail.com>>
Date: Monday, April 2, 2018 at 9:25 PM
To: Arpit Agarwal <aagar...@hortonworks.com<mailto:aagar...@hortonworks.com>>
Cc: Gera Shegalov <ger...@gmail.com<mailto:ger...@gmail.com>>, Sunil G 
<sun...@apache.org<mailto:sun...@apache.org>>, "
yarn-...@hadoop.apache.org<mailto:yarn-...@hadoop.apache.org>" 
<yarn-...@hadoop.apache.org<mailto:yarn-...@hadoop.apache.org>>, Hdfs-dev <
hdfs-...@hadoop.apache.org<mailto:hdfs-...@hadoop.apache.org>>, Hadoop Common 
<common-dev@hadoop.apache.org<mailto:common-dev@hadoop.apache.org>>,
"mapreduce-...@hadoop.apache.org<mailto:mapreduce-...@hadoop.apache.org>" 
<mapreduce-...@hadoop.apache.org<mailto:mapreduce-...@hadoop.apache.org>>,
Vinod Kumar Vavilapalli <vino...@apache.org<mailto:vino...@apache.org>>
Subject: Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)

As pointed by Arpit, the previously deployed shared jars are incorrect.
Just redeployed jars and staged. @Arpit, could you please check the updated
Maven repo? https://repository.apache.org/content/repositories/
orgapachehadoop-1092

Since the jars inside binary tarballs are correct (
http://people.apache.org/~wangda/hadoop-3.1.0-RC1/). I think we don't
need roll another RC, just update Maven repo should be sufficient.

Best,
Wangda


On Mon, Apr 2, 2018 at 2:39 PM, Wangda Tan <mailto:wheele...@gmail.com>
wrote:
Hi Arpit,

Thanks for pointing out this.

I just removed all .md5 files from artifacts. I found md5 checksums still
exist in .mds files and I didn't remove them from .mds file because it is
generated by create-release script and Apache guidance is &qu

Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)

[Wangda Tan]

Hi Vinod / Arpit,

I checked following versions:
- 2.6.5 / 2.7.5 / 2.8.3 / 2.9.0 / 3.0.1:

Jars in maven repo [1] are *always* different from jars in the binary
tarball [2]: (I only checked hadoop-yarn-api-version.jar)

(Following numbers are sizes of the jar)
2.6.5:
- Jar in Maven: 1896185
- Jar in tarball: 1891485

2.7.5:
- Jar in Maven: 2039371 (md5: 15e76f7c734b49315ef2bce952509ddf)
- Jar in tarball: 2039371 (md5: 0ef9f42f587401f5b49b39f27459f3ef)
(Even size is same, md5 is different)

2.8.3:
- Jar in Maven: 2451433
- Jar in tarball: 2438975

2.9.0:
- Jar in Maven: 2791477
- Jar in tarball: 289

3.0.1:
- Jar in Maven: 2852604
- Jar in tarball: 2851373

I guess the differences come from our release process.

Thanks,
Wangda

[1] Maven jars are downloaded from
https://repository.apache.org/service/local/repositories/releases/content/org/apache/hadoop/hadoop-yarn-api/
/hadoop-yarn-api-.jar
[2] Binary tarballs downloaded from http://apache.claz.org/hadoop/common/


On Tue, Apr 3, 2018 at 4:25 PM, Vinod Kumar Vavilapalli <vino...@apache.org>
wrote:

> We vote on the source code. The binaries are convenience artifacts.
>
> This is what I would do - (a) Just replace both the maven jars as well as
> the binaries to be consistent and correct. And then (b) Give a couple more
> days for folks who tested on the binaries to reverify - I count one such
> clear vote as of now.
>
> Thanks
> +Vinod
>
>
> On Apr 3, 2018, at 3:30 PM, Wangda Tan <wheele...@gmail.com> wrote:
>
> HI Arpit,
>
> I think it won't match if we do rebuild. It should be fine as far as
> they're signed, correct? I don't see any policy doesn't allow this.
>
> Thanks,
> Wangda
>
>
> On Tue, Apr 3, 2018 at 9:33 AM, Arpit Agarwal <aagar...@hortonworks.com>
> wrote:
>
>> Thanks Wangda, I see the shaded jars now.
>>
>> Are the repo jars required to be the same as the binary release? They
>> don’t match right now, probably they got rebuilt.
>>
>> +1 (binding), modulo that remaining question.
>>
>> * Verified signatures
>> * Verified checksums for source and binary artefacts
>> * Sanity checked jars on r.a.o.
>> * Built from source
>> * Deployed to 3 node secure cluster with NameNode HA
>> * Verified HDFS web UIs
>> * Tried out HDFS shell commands
>> * Ran sample MapReduce jobs
>>
>> Thanks!
>>
>>
>> --
>> From: Wangda Tan <wheele...@gmail.com>
>> Date: Monday, April 2, 2018 at 9:25 PM
>> To: Arpit Agarwal <aagar...@hortonworks.com>
>> Cc: Gera Shegalov <ger...@gmail.com>, Sunil G <sun...@apache.org>, "
>> yarn-...@hadoop.apache.org" <yarn-...@hadoop.apache.org>, Hdfs-dev <
>> hdfs-...@hadoop.apache.org>, Hadoop Common <common-dev@hadoop.apache.org>,
>> "mapreduce-...@hadoop.apache.org" <mapreduce-...@hadoop.apache.org>,
>> Vinod Kumar Vavilapalli <vino...@apache.org>
>> Subject: Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
>>
>> As pointed by Arpit, the previously deployed shared jars are incorrect.
>> Just redeployed jars and staged. @Arpit, could you please check the updated
>> Maven repo? https://repository.apache.org/content/repositories/
>> orgapachehadoop-1092
>>
>> Since the jars inside binary tarballs are correct (
>> http://people.apache.org/~wangda/hadoop-3.1.0-RC1/). I think we don't
>> need roll another RC, just update Maven repo should be sufficient.
>>
>> Best,
>> Wangda
>>
>>
>> On Mon, Apr 2, 2018 at 2:39 PM, Wangda Tan <mailto:wheele...@gmail.com>
>> wrote:
>> Hi Arpit,
>>
>> Thanks for pointing out this.
>>
>> I just removed all .md5 files from artifacts. I found md5 checksums still
>> exist in .mds files and I didn't remove them from .mds file because it is
>> generated by create-release script and Apache guidance is "should not"
>> instead of "must not". Please let me know if you think they need to be
>> removed as well.
>>
>> - Wangda
>>
>>
>>
>> On Mon, Apr 2, 2018 at 1:37 PM, Arpit Agarwal > aagar...@hortonworks.com> wrote:
>> Thanks for putting together this RC, Wangda.
>>
>> The guidance from Apache is to omit MD5s, specifically:
>>   > SHOULD NOT supply a MD5 checksum file (because MD5 is too broken).
>>
>> https://www.apache.org/dev/release-distribution#sigs-and-sums
>>
>>
>>
>>
>> On Apr 2, 2018, at 7:03 AM, Wangda Tan <mailto:wheele...@gmail.com>
>> wrote:
>>
>> Hi Gera,
>>
>> It's my bad, I thought 

Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)

[Iñigo Goiri]

+1 (non binding)

* Deployed with 4 subclusters with HDFS Router-based federation.
* Executed DistCp across subclusters through the Router
* Checked documentation and tgz

On Tue, Apr 3, 2018 at 4:25 PM, Vinod Kumar Vavilapalli <vino...@apache.org>
wrote:

> We vote on the source code. The binaries are convenience artifacts.
>
> This is what I would do - (a) Just replace both the maven jars as well as
> the binaries to be consistent and correct. And then (b) Give a couple more
> days for folks who tested on the binaries to reverify - I count one such
> clear vote as of now.
>
> Thanks
> +Vinod
>
> > On Apr 3, 2018, at 3:30 PM, Wangda Tan <wheele...@gmail.com> wrote:
> >
> > HI Arpit,
> >
> > I think it won't match if we do rebuild. It should be fine as far as
> they're signed, correct? I don't see any policy doesn't allow this.
> >
> > Thanks,
> > Wangda
> >
> >
> > On Tue, Apr 3, 2018 at 9:33 AM, Arpit Agarwal <aagar...@hortonworks.com
> <mailto:aagar...@hortonworks.com>> wrote:
> > Thanks Wangda, I see the shaded jars now.
> >
> > Are the repo jars required to be the same as the binary release? They
> don’t match right now, probably they got rebuilt.
> >
> > +1 (binding), modulo that remaining question.
> >
> > * Verified signatures
> > * Verified checksums for source and binary artefacts
> > * Sanity checked jars on r.a.o.
> > * Built from source
> > * Deployed to 3 node secure cluster with NameNode HA
> > * Verified HDFS web UIs
> > * Tried out HDFS shell commands
> > * Ran sample MapReduce jobs
> >
> > Thanks!
> >
> >
> > --
> > From: Wangda Tan <wheele...@gmail.com <mailto:wheele...@gmail.com>>
> > Date: Monday, April 2, 2018 at 9:25 PM
> > To: Arpit Agarwal <aagar...@hortonworks.com  aagar...@hortonworks.com>>
> > Cc: Gera Shegalov <ger...@gmail.com <mailto:ger...@gmail.com>>, Sunil G
> <sun...@apache.org <mailto:sun...@apache.org>>, "
> yarn-...@hadoop.apache.org <mailto:yarn-...@hadoop.apache.org>" <
> yarn-...@hadoop.apache.org <mailto:yarn-...@hadoop.apache.org>>, Hdfs-dev
> <hdfs-...@hadoop.apache.org <mailto:hdfs-...@hadoop.apache.org>>, Hadoop
> Common <common-dev@hadoop.apache.org <mailto:common-dev@hadoop.apache.org>>,
> "mapreduce-...@hadoop.apache.org <mailto:mapreduce-...@hadoop.apache.org>"
> <mapreduce-...@hadoop.apache.org <mailto:mapreduce-...@hadoop.apache.org>>,
> Vinod Kumar Vavilapalli <vino...@apache.org <mailto:vino...@apache.org>>
> > Subject: Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
> >
> > As pointed by Arpit, the previously deployed shared jars are incorrect.
> Just redeployed jars and staged. @Arpit, could you please check the updated
> Maven repo? https://repository.apache.org/content/repositories/
> orgapachehadoop-1092 <https://repository.apache.org/content/repositories/
> orgapachehadoop-1092>
> >
> > Since the jars inside binary tarballs are correct (
> http://people.apache.org/~wangda/hadoop-3.1.0-RC1/ <
> http://people.apache.org/~wangda/hadoop-3.1.0-RC1/>). I think we don't
> need roll another RC, just update Maven repo should be sufficient.
> >
> > Best,
> > Wangda
> >
> >
> > On Mon, Apr 2, 2018 at 2:39 PM, Wangda Tan <mailto:wheele...@gmail.com
> <mailto:wheele...@gmail.com>> wrote:
> > Hi Arpit,
> >
> > Thanks for pointing out this.
> >
> > I just removed all .md5 files from artifacts. I found md5 checksums
> still exist in .mds files and I didn't remove them from .mds file because
> it is generated by create-release script and Apache guidance is "should
> not" instead of "must not". Please let me know if you think they need to be
> removed as well.
> >
> > - Wangda
> >
> >
> >
> > On Mon, Apr 2, 2018 at 1:37 PM, Arpit Agarwal  aagar...@hortonworks.com <mailto:aagar...@hortonworks.com>> wrote:
> > Thanks for putting together this RC, Wangda.
> >
> > The guidance from Apache is to omit MD5s, specifically:
> >   > SHOULD NOT supply a MD5 checksum file (because MD5 is too broken).
> >
> > https://www.apache.org/dev/release-distribution#sigs-and-sums <
> https://www.apache.org/dev/release-distribution#sigs-and-sums>
> >
> >
> >
> >
> > On Apr 2, 2018, at 7:03 AM, Wangda Tan <mailto:wheele...@gmail.com
> <mailto:wheele...@gmail.com>> wrote:
> >
> > H

Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)

[Vinod Kumar Vavilapalli]

We vote on the source code. The binaries are convenience artifacts.

This is what I would do - (a) Just replace both the maven jars as well as the 
binaries to be consistent and correct. And then (b) Give a couple more days for 
folks who tested on the binaries to reverify - I count one such clear vote as 
of now.

Thanks
+Vinod

> On Apr 3, 2018, at 3:30 PM, Wangda Tan <wheele...@gmail.com> wrote:
> 
> HI Arpit,
> 
> I think it won't match if we do rebuild. It should be fine as far as they're 
> signed, correct? I don't see any policy doesn't allow this. 
> 
> Thanks,
> Wangda
> 
> 
> On Tue, Apr 3, 2018 at 9:33 AM, Arpit Agarwal <aagar...@hortonworks.com 
> <mailto:aagar...@hortonworks.com>> wrote:
> Thanks Wangda, I see the shaded jars now.
> 
> Are the repo jars required to be the same as the binary release? They don’t 
> match right now, probably they got rebuilt.
> 
> +1 (binding), modulo that remaining question.
> 
> * Verified signatures
> * Verified checksums for source and binary artefacts
> * Sanity checked jars on r.a.o.
> * Built from source
> * Deployed to 3 node secure cluster with NameNode HA
> * Verified HDFS web UIs
> * Tried out HDFS shell commands
> * Ran sample MapReduce jobs
> 
> Thanks!
> 
> 
> --
> From: Wangda Tan <wheele...@gmail.com <mailto:wheele...@gmail.com>>
> Date: Monday, April 2, 2018 at 9:25 PM
> To: Arpit Agarwal <aagar...@hortonworks.com <mailto:aagar...@hortonworks.com>>
> Cc: Gera Shegalov <ger...@gmail.com <mailto:ger...@gmail.com>>, Sunil G 
> <sun...@apache.org <mailto:sun...@apache.org>>, "yarn-...@hadoop.apache.org 
> <mailto:yarn-...@hadoop.apache.org>" <yarn-...@hadoop.apache.org 
> <mailto:yarn-...@hadoop.apache.org>>, Hdfs-dev <hdfs-...@hadoop.apache.org 
> <mailto:hdfs-...@hadoop.apache.org>>, Hadoop Common 
> <common-dev@hadoop.apache.org <mailto:common-dev@hadoop.apache.org>>, 
> "mapreduce-...@hadoop.apache.org <mailto:mapreduce-...@hadoop.apache.org>" 
> <mapreduce-...@hadoop.apache.org <mailto:mapreduce-...@hadoop.apache.org>>, 
> Vinod Kumar Vavilapalli <vino...@apache.org <mailto:vino...@apache.org>>
> Subject: Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
> 
> As pointed by Arpit, the previously deployed shared jars are incorrect. Just 
> redeployed jars and staged. @Arpit, could you please check the updated Maven 
> repo? https://repository.apache.org/content/repositories/orgapachehadoop-1092 
> <https://repository.apache.org/content/repositories/orgapachehadoop-1092>
> 
> Since the jars inside binary tarballs are correct 
> (http://people.apache.org/~wangda/hadoop-3.1.0-RC1/ 
> <http://people.apache.org/~wangda/hadoop-3.1.0-RC1/>). I think we don't need 
> roll another RC, just update Maven repo should be sufficient. 
> 
> Best,
> Wangda
> 
> 
> On Mon, Apr 2, 2018 at 2:39 PM, Wangda Tan <mailto:wheele...@gmail.com 
> <mailto:wheele...@gmail.com>> wrote:
> Hi Arpit,
> 
> Thanks for pointing out this.
> 
> I just removed all .md5 files from artifacts. I found md5 checksums still 
> exist in .mds files and I didn't remove them from .mds file because it is 
> generated by create-release script and Apache guidance is "should not" 
> instead of "must not". Please let me know if you think they need to be 
> removed as well. 
> 
> - Wangda
> 
> 
> 
> On Mon, Apr 2, 2018 at 1:37 PM, Arpit Agarwal 
> <mailto:aagar...@hortonworks.com <mailto:aagar...@hortonworks.com>> wrote:
> Thanks for putting together this RC, Wangda.
> 
> The guidance from Apache is to omit MD5s, specifically:
>   > SHOULD NOT supply a MD5 checksum file (because MD5 is too broken).
> 
> https://www.apache.org/dev/release-distribution#sigs-and-sums 
> <https://www.apache.org/dev/release-distribution#sigs-and-sums>
> 
>  
> 
> 
> On Apr 2, 2018, at 7:03 AM, Wangda Tan <mailto:wheele...@gmail.com 
> <mailto:wheele...@gmail.com>> wrote:
> 
> Hi Gera,
> 
> It's my bad, I thought only src/bin tarball is enough.
> 
> I just uploaded all other things under artifact/ to
> http://people.apache.org/~wangda/hadoop-3.1.0-RC1/ 
> <http://people.apache.org/~wangda/hadoop-3.1.0-RC1/>
> 
> Please let me know if you have any other comments.
> 
> Thanks,
> Wangda
> 
> 
> On Mon, Apr 2, 2018 at 12:50 AM, Gera Shegalov <mailto:ger...@gmail.com 
> <mailto:ger...@gmail.com>> wrote:
> 
> 
> Thanks, Wangda!
> 
> There are many more artifacts in previou

Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)

[Wangda Tan]

HI Arpit,

I think it won't match if we do rebuild. It should be fine as far as
they're signed, correct? I don't see any policy doesn't allow this.

Thanks,
Wangda


On Tue, Apr 3, 2018 at 9:33 AM, Arpit Agarwal <aagar...@hortonworks.com>
wrote:

> Thanks Wangda, I see the shaded jars now.
>
> Are the repo jars required to be the same as the binary release? They
> don’t match right now, probably they got rebuilt.
>
> +1 (binding), modulo that remaining question.
>
> * Verified signatures
> * Verified checksums for source and binary artefacts
> * Sanity checked jars on r.a.o.
> * Built from source
> * Deployed to 3 node secure cluster with NameNode HA
> * Verified HDFS web UIs
> * Tried out HDFS shell commands
> * Ran sample MapReduce jobs
>
> Thanks!
>
>
> --
> From: Wangda Tan <wheele...@gmail.com>
> Date: Monday, April 2, 2018 at 9:25 PM
> To: Arpit Agarwal <aagar...@hortonworks.com>
> Cc: Gera Shegalov <ger...@gmail.com>, Sunil G <sun...@apache.org>, "
> yarn-...@hadoop.apache.org" <yarn-...@hadoop.apache.org>, Hdfs-dev <
> hdfs-...@hadoop.apache.org>, Hadoop Common <common-dev@hadoop.apache.org>,
> "mapreduce-...@hadoop.apache.org" <mapreduce-...@hadoop.apache.org>,
> Vinod Kumar Vavilapalli <vino...@apache.org>
> Subject: Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
>
> As pointed by Arpit, the previously deployed shared jars are incorrect.
> Just redeployed jars and staged. @Arpit, could you please check the updated
> Maven repo? https://repository.apache.org/content/
> repositories/orgapachehadoop-1092
>
> Since the jars inside binary tarballs are correct (
> http://people.apache.org/~wangda/hadoop-3.1.0-RC1/). I think we don't
> need roll another RC, just update Maven repo should be sufficient.
>
> Best,
> Wangda
>
>
> On Mon, Apr 2, 2018 at 2:39 PM, Wangda Tan <mailto:wheele...@gmail.com>
> wrote:
> Hi Arpit,
>
> Thanks for pointing out this.
>
> I just removed all .md5 files from artifacts. I found md5 checksums still
> exist in .mds files and I didn't remove them from .mds file because it is
> generated by create-release script and Apache guidance is "should not"
> instead of "must not". Please let me know if you think they need to be
> removed as well.
>
> - Wangda
>
>
>
> On Mon, Apr 2, 2018 at 1:37 PM, Arpit Agarwal  aagar...@hortonworks.com> wrote:
> Thanks for putting together this RC, Wangda.
>
> The guidance from Apache is to omit MD5s, specifically:
>   > SHOULD NOT supply a MD5 checksum file (because MD5 is too broken).
>
> https://www.apache.org/dev/release-distribution#sigs-and-sums
>
>
>
>
> On Apr 2, 2018, at 7:03 AM, Wangda Tan <mailto:wheele...@gmail.com> wrote:
>
> Hi Gera,
>
> It's my bad, I thought only src/bin tarball is enough.
>
> I just uploaded all other things under artifact/ to
> http://people.apache.org/~wangda/hadoop-3.1.0-RC1/
>
> Please let me know if you have any other comments.
>
> Thanks,
> Wangda
>
>
> On Mon, Apr 2, 2018 at 12:50 AM, Gera Shegalov <mailto:ger...@gmail.com>
> wrote:
>
>
> Thanks, Wangda!
>
> There are many more artifacts in previous votes, e.g., see
> http://home.apache.org/~junping_du/hadoop-2.8.3-RC0/ .  Among others the
> site tarball is missing.
>
> On Sun, Apr 1, 2018 at 11:54 PM Sunil G <mailto:sun...@apache.org> wrote:
>
>
> Thanks Wangda for initiating the release.
>
> I tested this RC built from source file.
>
>
>   - Tested MR apps (sleep, wc) and verified both new YARN UI and old RM
> UI.
>   - Below feature sanity is done
>  - Application priority
>  - Application timeout
>  - Intra Queue preemption with priority based
>  - DS based affinity tests to verify placement constraints.
>   - Tested basic NodeLabel scenarios.
>  - Added couple of labels to few of nodes and behavior is coming
>  correct.
>  - Verified old UI  and new YARN UI for labels.
>  - Submitted apps to labelled cluster and it works fine.
>  - Also performed few cli commands related to nodelabel.
>   - Test basic HA cases and seems correct.
>   - Tested new YARN UI . All pages are getting loaded correctly.
>
>
> - Sunil
>
> On Fri, Mar 30, 2018 at 9:45 AM Wangda Tan <mailto:wheele...@gmail.com>
> wrote:
>
>
> Hi folks,
>
> Thanks to the many who helped with this release since Dec 2017 [1].
> We've
>
> created RC1 for Apache Hadoop 3.1.0. The artifacts are available here:
>
> http://people.apache.org/~wangda/hadoop-3.1.0-RC1
>
> 

Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)

[Eric Payne]

 +1 (binding)
Thanks Wangda for doing the work to produce this release.
I did the following to test the release:
- Built from source
- Installed on 6-node pseudo cluster
- Interacted with RM CLI and GUI
- Tested streaming jobs
- Tested yarn distributed shell jobs
- Tested Max AM Resource Percent

- Tested simple inter-queue preemption
- Tested priority first intra-queue preemption

- Tested userlimit first intra-queue preemption

Thanks,Eric Payne


===
On Thursday, March 29, 2018, 11:15:51 PM CDT, Wangda Tan 
 wrote:  
 
 Hi folks,

Thanks to the many who helped with this release since Dec 2017 [1]. We've
created RC1 for Apache Hadoop 3.1.0. The artifacts are available here:

http://people.apache.org/~wangda/hadoop-3.1.0-RC1

The RC tag in git is release-3.1.0-RC1. Last git commit SHA is
16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d

The maven artifacts are available via repository.apache.org at
https://repository.apache.org/content/repositories/orgapachehadoop-1090/
This vote will run 5 days, ending on Apr 3 at 11:59 pm Pacific.

3.1.0 contains 766 [2] fixed JIRA issues since 3.0.0. Notable additions
include the first class GPU/FPGA support on YARN, Native services, Support
rich placement constraints in YARN, S3-related enhancements, allow HDFS
block replicas to be provided by an external storage system, etc.

For 3.1.0 RC0 vote discussion, please see [3].

We’d like to use this as a starting release for 3.1.x [1], depending on how
it goes, get it stabilized and potentially use a 3.1.1 in several weeks as
the stable release.

We have done testing with a pseudo cluster:
- Ran distributed job.
- GPU scheduling/isolation.
- Placement constraints (intra-application anti-affinity) by using
distributed shell.

My +1 to start.

Best,
Wangda/Vinod

[1]
https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104bc9e19e559aedc5ebedb4ca08c8@%3Cyarn-dev.hadoop.apache.org%3E
[2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.0)
AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status = Resolved ORDER BY
fixVersion ASC
[3]
https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d72d4061f26f83547e41d0983ea6@%3Cyarn-dev.hadoop.apache.org%3E
  

Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)

[Gera Shegalov]

+1 (non-binding)

- built from source
- tested SparkPi on minicluster (modulo YARN-7747
<https://issues.apache.org/jira/browse/YARN-7747>)
- tested SparkPi on pseudo-distributed cluster
- browsed HDFS doc in the site tarball


On Tue, Apr 3, 2018 at 11:55 AM Hanisha Koneru <hkon...@hortonworks.com>
wrote:

> Correction: My vote is NON-BINDING. Sorry for the confusion.
>
>
> Thanks,
> Hanisha
>
>
>
>
>
>
>
>
> On 4/3/18, 11:40 AM, "Hanisha Koneru" <hkon...@hortonworks.com> wrote:
>
> >Thanks Wangda for putting up the RC for 3.1.0.
> >
> >+1 (binding).
> >
> >Verified the following:
> >- Built from source
> >- Deployed binary to a 3-node docker cluster
> >- Sanity checks
> >   - Basic dfs operations
> >   - MapReduce Wordcount & Grep
> >
> >
> >Thanks,
> >Hanisha
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >On 4/3/18, 9:33 AM, "Arpit Agarwal" <aagar...@hortonworks.com> wrote:
> >
> >>Thanks Wangda, I see the shaded jars now.
> >>
> >>Are the repo jars required to be the same as the binary release? They
> don’t match right now, probably they got rebuilt.
> >>
> >>+1 (binding), modulo that remaining question.
> >>
> >>* Verified signatures
> >>* Verified checksums for source and binary artefacts
> >>* Sanity checked jars on r.a.o.
> >>* Built from source
> >>* Deployed to 3 node secure cluster with NameNode HA
> >>* Verified HDFS web UIs
> >>* Tried out HDFS shell commands
> >>* Ran sample MapReduce jobs
> >>
> >>Thanks!
> >>
> >>
> >>--
> >>From: Wangda Tan <wheele...@gmail.com>
> >>Date: Monday, April 2, 2018 at 9:25 PM
> >>To: Arpit Agarwal <aagar...@hortonworks.com>
> >>Cc: Gera Shegalov <ger...@gmail.com>, Sunil G <sun...@apache.org>, "
> yarn-...@hadoop.apache.org" <yarn-...@hadoop.apache.org>, Hdfs-dev <
> hdfs-...@hadoop.apache.org>, Hadoop Common <common-dev@hadoop.apache.org>,
> "mapreduce-...@hadoop.apache.org" <mapreduce-...@hadoop.apache.org>,
> Vinod Kumar Vavilapalli <vino...@apache.org>
> >>Subject: Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
> >>
> >>As pointed by Arpit, the previously deployed shared jars are incorrect.
> Just redeployed jars and staged. @Arpit, could you please check the updated
> Maven repo?
> https://repository.apache.org/content/repositories/orgapachehadoop-1092
> >>
> >>Since the jars inside binary tarballs are correct (
> http://people.apache.org/~wangda/hadoop-3.1.0-RC1/). I think we don't
> need roll another RC, just update Maven repo should be sufficient.
> >>
> >>Best,
> >>Wangda
> >>
> >>
> >>On Mon, Apr 2, 2018 at 2:39 PM, Wangda Tan <mailto:wheele...@gmail.com>
> wrote:
> >>Hi Arpit,
> >>
> >>Thanks for pointing out this.
> >>
> >>I just removed all .md5 files from artifacts. I found md5 checksums
> still exist in .mds files and I didn't remove them from .mds file because
> it is generated by create-release script and Apache guidance is "should
> not" instead of "must not". Please let me know if you think they need to be
> removed as well.
> >>
> >>- Wangda
> >>
> >>
> >>
> >>On Mon, Apr 2, 2018 at 1:37 PM, Arpit Agarwal  aagar...@hortonworks.com> wrote:
> >>Thanks for putting together this RC, Wangda.
> >>
> >>The guidance from Apache is to omit MD5s, specifically:
> >>  > SHOULD NOT supply a MD5 checksum file (because MD5 is too broken).
> >>
> >>https://www.apache.org/dev/release-distribution#sigs-and-sums
> >>
> >>
> >>
> >>
> >>On Apr 2, 2018, at 7:03 AM, Wangda Tan <mailto:wheele...@gmail.com>
> wrote:
> >>
> >>Hi Gera,
> >>
> >>It's my bad, I thought only src/bin tarball is enough.
> >>
> >>I just uploaded all other things under artifact/ to
> >>http://people.apache.org/~wangda/hadoop-3.1.0-RC1/
> >>
> >>Please let me know if you have any other comments.
> >>
> >>Thanks,
> >>Wangda
> >>
> >>
> >>On Mon, Apr 2, 2018 at 12:50 AM, Gera Shegalov <mailto:ger...@gmail.com>
> wrote:
> >>
> >>
> >>Thanks, Wangda!
> >>
> 

Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)

[Hanisha Koneru]

Correction: My vote is NON-BINDING. Sorry for the confusion.


Thanks,
Hanisha








On 4/3/18, 11:40 AM, "Hanisha Koneru" <hkon...@hortonworks.com> wrote:

>Thanks Wangda for putting up the RC for 3.1.0.
>
>+1 (binding).
>
>Verified the following:
>- Built from source
>- Deployed binary to a 3-node docker cluster
>- Sanity checks
>   - Basic dfs operations
>   - MapReduce Wordcount & Grep
>
>
>Thanks,
>Hanisha
>
>
>
>
>
>
>
>
>
>On 4/3/18, 9:33 AM, "Arpit Agarwal" <aagar...@hortonworks.com> wrote:
>
>>Thanks Wangda, I see the shaded jars now.
>>
>>Are the repo jars required to be the same as the binary release? They don’t 
>>match right now, probably they got rebuilt.
>>
>>+1 (binding), modulo that remaining question.
>>
>>* Verified signatures
>>* Verified checksums for source and binary artefacts
>>* Sanity checked jars on r.a.o. 
>>* Built from source
>>* Deployed to 3 node secure cluster with NameNode HA
>>* Verified HDFS web UIs
>>* Tried out HDFS shell commands
>>* Ran sample MapReduce jobs
>>
>>Thanks!
>>
>>
>>--
>>From: Wangda Tan <wheele...@gmail.com>
>>Date: Monday, April 2, 2018 at 9:25 PM
>>To: Arpit Agarwal <aagar...@hortonworks.com>
>>Cc: Gera Shegalov <ger...@gmail.com>, Sunil G <sun...@apache.org>, 
>>"yarn-...@hadoop.apache.org" <yarn-...@hadoop.apache.org>, Hdfs-dev 
>><hdfs-...@hadoop.apache.org>, Hadoop Common <common-dev@hadoop.apache.org>, 
>>"mapreduce-...@hadoop.apache.org" <mapreduce-...@hadoop.apache.org>, Vinod 
>>Kumar Vavilapalli <vino...@apache.org>
>>Subject: Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
>>
>>As pointed by Arpit, the previously deployed shared jars are incorrect. Just 
>>redeployed jars and staged. @Arpit, could you please check the updated Maven 
>>repo? https://repository.apache.org/content/repositories/orgapachehadoop-1092 
>>
>>Since the jars inside binary tarballs are correct 
>>(http://people.apache.org/~wangda/hadoop-3.1.0-RC1/). I think we don't need 
>>roll another RC, just update Maven repo should be sufficient. 
>>
>>Best,
>>Wangda
>>
>>
>>On Mon, Apr 2, 2018 at 2:39 PM, Wangda Tan <mailto:wheele...@gmail.com> wrote:
>>Hi Arpit, 
>>
>>Thanks for pointing out this.
>>
>>I just removed all .md5 files from artifacts. I found md5 checksums still 
>>exist in .mds files and I didn't remove them from .mds file because it is 
>>generated by create-release script and Apache guidance is "should not" 
>>instead of "must not". Please let me know if you think they need to be 
>>removed as well. 
>>
>>- Wangda
>>
>>
>>
>>On Mon, Apr 2, 2018 at 1:37 PM, Arpit Agarwal 
>><mailto:aagar...@hortonworks.com> wrote:
>>Thanks for putting together this RC, Wangda. 
>>
>>The guidance from Apache is to omit MD5s, specifically:
>>  > SHOULD NOT supply a MD5 checksum file (because MD5 is too broken).
>>
>>https://www.apache.org/dev/release-distribution#sigs-and-sums
>>
>> 
>>
>>
>>On Apr 2, 2018, at 7:03 AM, Wangda Tan <mailto:wheele...@gmail.com> wrote:
>>
>>Hi Gera,
>>
>>It's my bad, I thought only src/bin tarball is enough.
>>
>>I just uploaded all other things under artifact/ to
>>http://people.apache.org/~wangda/hadoop-3.1.0-RC1/
>>
>>Please let me know if you have any other comments.
>>
>>Thanks,
>>Wangda
>>
>>
>>On Mon, Apr 2, 2018 at 12:50 AM, Gera Shegalov <mailto:ger...@gmail.com> 
>>wrote:
>>
>>
>>Thanks, Wangda!
>>
>>There are many more artifacts in previous votes, e.g., see
>>http://home.apache.org/~junping_du/hadoop-2.8.3-RC0/ .  Among others the
>>site tarball is missing.
>>
>>On Sun, Apr 1, 2018 at 11:54 PM Sunil G <mailto:sun...@apache.org> wrote:
>>
>>
>>Thanks Wangda for initiating the release.
>>
>>I tested this RC built from source file.
>>
>>
>>  - Tested MR apps (sleep, wc) and verified both new YARN UI and old RM
>>UI.
>>  - Below feature sanity is done
>> - Application priority
>> - Application timeout
>> - Intra Queue preemption with priority based
>> - DS based affinity tests to verify placement constraints.
>>  - Tested basic NodeLabel scenarios.
>

Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)

[Hanisha Koneru]

Thanks Wangda for putting up the RC for 3.1.0.

+1 (binding).

Verified the following:
- Built from source
- Deployed binary to a 3-node docker cluster
- Sanity checks
- Basic dfs operations
- MapReduce Wordcount & Grep


Thanks,
Hanisha









On 4/3/18, 9:33 AM, "Arpit Agarwal" <aagar...@hortonworks.com> wrote:

>Thanks Wangda, I see the shaded jars now.
>
>Are the repo jars required to be the same as the binary release? They don’t 
>match right now, probably they got rebuilt.
>
>+1 (binding), modulo that remaining question.
>
>* Verified signatures
>* Verified checksums for source and binary artefacts
>* Sanity checked jars on r.a.o. 
>* Built from source
>* Deployed to 3 node secure cluster with NameNode HA
>* Verified HDFS web UIs
>* Tried out HDFS shell commands
>* Ran sample MapReduce jobs
>
>Thanks!
>
>
>--
>From: Wangda Tan <wheele...@gmail.com>
>Date: Monday, April 2, 2018 at 9:25 PM
>To: Arpit Agarwal <aagar...@hortonworks.com>
>Cc: Gera Shegalov <ger...@gmail.com>, Sunil G <sun...@apache.org>, 
>"yarn-...@hadoop.apache.org" <yarn-...@hadoop.apache.org>, Hdfs-dev 
><hdfs-...@hadoop.apache.org>, Hadoop Common <common-dev@hadoop.apache.org>, 
>"mapreduce-...@hadoop.apache.org" <mapreduce-...@hadoop.apache.org>, Vinod 
>Kumar Vavilapalli <vino...@apache.org>
>Subject: Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
>
>As pointed by Arpit, the previously deployed shared jars are incorrect. Just 
>redeployed jars and staged. @Arpit, could you please check the updated Maven 
>repo? https://repository.apache.org/content/repositories/orgapachehadoop-1092 
>
>Since the jars inside binary tarballs are correct 
>(http://people.apache.org/~wangda/hadoop-3.1.0-RC1/). I think we don't need 
>roll another RC, just update Maven repo should be sufficient. 
>
>Best,
>Wangda
>
>
>On Mon, Apr 2, 2018 at 2:39 PM, Wangda Tan <mailto:wheele...@gmail.com> wrote:
>Hi Arpit, 
>
>Thanks for pointing out this.
>
>I just removed all .md5 files from artifacts. I found md5 checksums still 
>exist in .mds files and I didn't remove them from .mds file because it is 
>generated by create-release script and Apache guidance is "should not" instead 
>of "must not". Please let me know if you think they need to be removed as 
>well. 
>
>- Wangda
>
>
>
>On Mon, Apr 2, 2018 at 1:37 PM, Arpit Agarwal 
><mailto:aagar...@hortonworks.com> wrote:
>Thanks for putting together this RC, Wangda. 
>
>The guidance from Apache is to omit MD5s, specifically:
>  > SHOULD NOT supply a MD5 checksum file (because MD5 is too broken).
>
>https://www.apache.org/dev/release-distribution#sigs-and-sums
>
> 
>
>
>On Apr 2, 2018, at 7:03 AM, Wangda Tan <mailto:wheele...@gmail.com> wrote:
>
>Hi Gera,
>
>It's my bad, I thought only src/bin tarball is enough.
>
>I just uploaded all other things under artifact/ to
>http://people.apache.org/~wangda/hadoop-3.1.0-RC1/
>
>Please let me know if you have any other comments.
>
>Thanks,
>Wangda
>
>
>On Mon, Apr 2, 2018 at 12:50 AM, Gera Shegalov <mailto:ger...@gmail.com> wrote:
>
>
>Thanks, Wangda!
>
>There are many more artifacts in previous votes, e.g., see
>http://home.apache.org/~junping_du/hadoop-2.8.3-RC0/ .  Among others the
>site tarball is missing.
>
>On Sun, Apr 1, 2018 at 11:54 PM Sunil G <mailto:sun...@apache.org> wrote:
>
>
>Thanks Wangda for initiating the release.
>
>I tested this RC built from source file.
>
>
>  - Tested MR apps (sleep, wc) and verified both new YARN UI and old RM
>UI.
>  - Below feature sanity is done
> - Application priority
> - Application timeout
> - Intra Queue preemption with priority based
> - DS based affinity tests to verify placement constraints.
>  - Tested basic NodeLabel scenarios.
> - Added couple of labels to few of nodes and behavior is coming
> correct.
> - Verified old UI  and new YARN UI for labels.
> - Submitted apps to labelled cluster and it works fine.
> - Also performed few cli commands related to nodelabel.
>  - Test basic HA cases and seems correct.
>  - Tested new YARN UI . All pages are getting loaded correctly.
>
>
>- Sunil
>
>On Fri, Mar 30, 2018 at 9:45 AM Wangda Tan <mailto:wheele...@gmail.com> wrote:
>
>
>Hi folks,
>
>Thanks to the many who helped with this release since Dec 2017 [1].
>We've
>
>created RC1 for Apache Hadoop 3.1.0. The artifacts are available here:
>
>http://people.apache.org/~wan

Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)

[Arpit Agarwal]

Thanks Wangda, I see the shaded jars now.

Are the repo jars required to be the same as the binary release? They don’t 
match right now, probably they got rebuilt.

+1 (binding), modulo that remaining question.

* Verified signatures
* Verified checksums for source and binary artefacts
* Sanity checked jars on r.a.o. 
* Built from source
* Deployed to 3 node secure cluster with NameNode HA
* Verified HDFS web UIs
* Tried out HDFS shell commands
* Ran sample MapReduce jobs

Thanks!


--
From: Wangda Tan <wheele...@gmail.com>
Date: Monday, April 2, 2018 at 9:25 PM
To: Arpit Agarwal <aagar...@hortonworks.com>
Cc: Gera Shegalov <ger...@gmail.com>, Sunil G <sun...@apache.org>, 
"yarn-...@hadoop.apache.org" <yarn-...@hadoop.apache.org>, Hdfs-dev 
<hdfs-...@hadoop.apache.org>, Hadoop Common <common-dev@hadoop.apache.org>, 
"mapreduce-...@hadoop.apache.org" <mapreduce-...@hadoop.apache.org>, Vinod 
Kumar Vavilapalli <vino...@apache.org>
Subject: Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)

As pointed by Arpit, the previously deployed shared jars are incorrect. Just 
redeployed jars and staged. @Arpit, could you please check the updated Maven 
repo? https://repository.apache.org/content/repositories/orgapachehadoop-1092 

Since the jars inside binary tarballs are correct 
(http://people.apache.org/~wangda/hadoop-3.1.0-RC1/). I think we don't need 
roll another RC, just update Maven repo should be sufficient. 

Best,
Wangda


On Mon, Apr 2, 2018 at 2:39 PM, Wangda Tan <mailto:wheele...@gmail.com> wrote:
Hi Arpit, 

Thanks for pointing out this.

I just removed all .md5 files from artifacts. I found md5 checksums still exist 
in .mds files and I didn't remove them from .mds file because it is generated 
by create-release script and Apache guidance is "should not" instead of "must 
not". Please let me know if you think they need to be removed as well. 

- Wangda



On Mon, Apr 2, 2018 at 1:37 PM, Arpit Agarwal <mailto:aagar...@hortonworks.com> 
wrote:
Thanks for putting together this RC, Wangda. 

The guidance from Apache is to omit MD5s, specifically:
  > SHOULD NOT supply a MD5 checksum file (because MD5 is too broken).

https://www.apache.org/dev/release-distribution#sigs-and-sums

 


On Apr 2, 2018, at 7:03 AM, Wangda Tan <mailto:wheele...@gmail.com> wrote:

Hi Gera,

It's my bad, I thought only src/bin tarball is enough.

I just uploaded all other things under artifact/ to
http://people.apache.org/~wangda/hadoop-3.1.0-RC1/

Please let me know if you have any other comments.

Thanks,
Wangda


On Mon, Apr 2, 2018 at 12:50 AM, Gera Shegalov <mailto:ger...@gmail.com> wrote:


Thanks, Wangda!

There are many more artifacts in previous votes, e.g., see
http://home.apache.org/~junping_du/hadoop-2.8.3-RC0/ .  Among others the
site tarball is missing.

On Sun, Apr 1, 2018 at 11:54 PM Sunil G <mailto:sun...@apache.org> wrote:


Thanks Wangda for initiating the release.

I tested this RC built from source file.


  - Tested MR apps (sleep, wc) and verified both new YARN UI and old RM
UI.
  - Below feature sanity is done
 - Application priority
 - Application timeout
 - Intra Queue preemption with priority based
 - DS based affinity tests to verify placement constraints.
  - Tested basic NodeLabel scenarios.
 - Added couple of labels to few of nodes and behavior is coming
 correct.
 - Verified old UI  and new YARN UI for labels.
 - Submitted apps to labelled cluster and it works fine.
 - Also performed few cli commands related to nodelabel.
  - Test basic HA cases and seems correct.
  - Tested new YARN UI . All pages are getting loaded correctly.


- Sunil

On Fri, Mar 30, 2018 at 9:45 AM Wangda Tan <mailto:wheele...@gmail.com> wrote:


Hi folks,

Thanks to the many who helped with this release since Dec 2017 [1].
We've

created RC1 for Apache Hadoop 3.1.0. The artifacts are available here:

http://people.apache.org/~wangda/hadoop-3.1.0-RC1

The RC tag in git is release-3.1.0-RC1. Last git commit SHA is
16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d

The maven artifacts are available via http://repository.apache.org at
https://repository.apache.org/content/repositories/
orgapachehadoop-1090/

This vote will run 5 days, ending on Apr 3 at 11:59 pm Pacific.

3.1.0 contains 766 [2] fixed JIRA issues since 3.0.0. Notable additions
include the first class GPU/FPGA support on YARN, Native services,
Support

rich placement constraints in YARN, S3-related enhancements, allow HDFS
block replicas to be provided by an external storage system, etc.

For 3.1.0 RC0 vote discussion, please see [3].

We’d like to use this as a starting release for 3.1.x [1], depending on
how

it goes, get it stabilized and potentially use a 3.1.1 in several weeks
as

the stable release.

We have done testing with a pseudo clust

RE: [VOTE] Release Apache Hadoop 3.1.0 (RC1)

[Takanobu Asanuma]

Thanks for your efforts, Wangda. Sorry for participating late.

+1 (non-binding)
   - Verified the checksum of the tarball
   - Succeeded "mvn clean package -Pdist,native -Dtar -DskipTests"
   - Started docker hadoop cluster with 1 master and 5 slaves
   - Verified TeraGen/TeraSort
   - Verified some erasure coding operations
   - Seems the shaded jars (hadoop-client-runtime, hadoop-client-api, 
hadoop-client-minicluster) have correct sizes in orgapachehadoop-1092.

-Takanobu

-Original Message-
From: Wangda Tan [mailto:wheele...@gmail.com] 
Sent: Tuesday, April 03, 2018 1:25 PM
To: Arpit Agarwal <aagar...@hortonworks.com>
Cc: Gera Shegalov <ger...@gmail.com>; Sunil G <sun...@apache.org>; 
yarn-...@hadoop.apache.org; Hdfs-dev <hdfs-...@hadoop.apache.org>; Hadoop 
Common <common-dev@hadoop.apache.org>; mapreduce-...@hadoop.apache.org; Vinod 
Kumar Vavilapalli <vino...@apache.org>
Subject: Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)

As pointed by Arpit, the previously deployed shared jars are incorrect.
Just redeployed jars and staged. @Arpit, could you please check the updated 
Maven repo?
https://repository.apache.org/content/repositories/orgapachehadoop-1092

Since the jars inside binary tarballs are correct ( 
http://people.apache.org/~wangda/hadoop-3.1.0-RC1/). I think we don't need roll 
another RC, just update Maven repo should be sufficient.

Best,
Wangda


On Mon, Apr 2, 2018 at 2:39 PM, Wangda Tan <wheele...@gmail.com> wrote:

> Hi Arpit,
>
> Thanks for pointing out this.
>
> I just removed all .md5 files from artifacts. I found md5 checksums 
> still exist in .mds files and I didn't remove them from .mds file 
> because it is generated by create-release script and Apache guidance is 
> "should not"
> instead of "must not". Please let me know if you think they need to be 
> removed as well.
>
> - Wangda
>
>
>
> On Mon, Apr 2, 2018 at 1:37 PM, Arpit Agarwal 
> <aagar...@hortonworks.com>
> wrote:
>
>> Thanks for putting together this RC, Wangda.
>>
>> The guidance from Apache is to omit MD5s, specifically:
>>   > SHOULD NOT supply a MD5 checksum file (because MD5 is too broken).
>>
>> https://www.apache.org/dev/release-distribution#sigs-and-sums
>>
>>
>>
>> On Apr 2, 2018, at 7:03 AM, Wangda Tan <wheele...@gmail.com> wrote:
>>
>> Hi Gera,
>>
>> It's my bad, I thought only src/bin tarball is enough.
>>
>> I just uploaded all other things under artifact/ to 
>> http://people.apache.org/~wangda/hadoop-3.1.0-RC1/
>>
>> Please let me know if you have any other comments.
>>
>> Thanks,
>> Wangda
>>
>>
>> On Mon, Apr 2, 2018 at 12:50 AM, Gera Shegalov <ger...@gmail.com> wrote:
>>
>> Thanks, Wangda!
>>
>> There are many more artifacts in previous votes, e.g., see 
>> http://home.apache.org/~junping_du/hadoop-2.8.3-RC0/ .  Among others 
>> the site tarball is missing.
>>
>> On Sun, Apr 1, 2018 at 11:54 PM Sunil G <sun...@apache.org> wrote:
>>
>> Thanks Wangda for initiating the release.
>>
>> I tested this RC built from source file.
>>
>>
>>   - Tested MR apps (sleep, wc) and verified both new YARN UI and old 
>> RM UI.
>>   - Below feature sanity is done
>>  - Application priority
>>  - Application timeout
>>  - Intra Queue preemption with priority based
>>  - DS based affinity tests to verify placement constraints.
>>   - Tested basic NodeLabel scenarios.
>>  - Added couple of labels to few of nodes and behavior is coming
>>  correct.
>>  - Verified old UI  and new YARN UI for labels.
>>  - Submitted apps to labelled cluster and it works fine.
>>  - Also performed few cli commands related to nodelabel.
>>   - Test basic HA cases and seems correct.
>>   - Tested new YARN UI . All pages are getting loaded correctly.
>>
>>
>> - Sunil
>>
>> On Fri, Mar 30, 2018 at 9:45 AM Wangda Tan <wheele...@gmail.com> wrote:
>>
>> Hi folks,
>>
>> Thanks to the many who helped with this release since Dec 2017 [1].
>>
>> We've
>>
>> created RC1 for Apache Hadoop 3.1.0. The artifacts are available here:
>>
>> http://people.apache.org/~wangda/hadoop-3.1.0-RC1
>>
>> The RC tag in git is release-3.1.0-RC1. Last git commit SHA is 
>> 16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d
>>
>> The maven artifacts are available via repository.apache.org at 
>> https://repository.apache.org/content/repositories/
>>
>> orgapachehadoop-1090/
>>
>> This vote will run 5 days, endi

Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)

[Konstantinos Karanasos]

+1 (binding)

Thanks for putting this together, Wangda.

I deployed a 7-node hadoop cluster and:
- ran various MR jobs
- ran the same jobs with a mix of opportunistic containers via centralized
scheduling
- ran the jobs with opportunistic containers and distributed scheduling
- ran some jobs with various placement constraints.

All worked as expected.

Thanks,
Konstantinos

On Thu, Mar 29, 2018 at 9:15 PM, Wangda Tan  wrote:

> Hi folks,
>
> Thanks to the many who helped with this release since Dec 2017 [1]. We've
> created RC1 for Apache Hadoop 3.1.0. The artifacts are available here:
>
> http://people.apache.org/~wangda/hadoop-3.1.0-RC1
>
> The RC tag in git is release-3.1.0-RC1. Last git commit SHA is
> 16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d
>
> The maven artifacts are available via repository.apache.org at
> https://repository.apache.org/content/repositories/orgapachehadoop-1090/
> This vote will run 5 days, ending on Apr 3 at 11:59 pm Pacific.
>
> 3.1.0 contains 766 [2] fixed JIRA issues since 3.0.0. Notable additions
> include the first class GPU/FPGA support on YARN, Native services, Support
> rich placement constraints in YARN, S3-related enhancements, allow HDFS
> block replicas to be provided by an external storage system, etc.
>
> For 3.1.0 RC0 vote discussion, please see [3].
>
> We’d like to use this as a starting release for 3.1.x [1], depending on how
> it goes, get it stabilized and potentially use a 3.1.1 in several weeks as
> the stable release.
>
> We have done testing with a pseudo cluster:
> - Ran distributed job.
> - GPU scheduling/isolation.
> - Placement constraints (intra-application anti-affinity) by using
> distributed shell.
>
> My +1 to start.
>
> Best,
> Wangda/Vinod
>
> [1]
> https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104b
> c9e19e559aedc5ebedb4ca08c8@%3Cyarn-dev.hadoop.apache.org%3E
> [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.0)
> AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status = Resolved ORDER BY
> fixVersion ASC
> [3]
> https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d7
> 2d4061f26f83547e41d0983ea6@%3Cyarn-dev.hadoop.apache.org%3E
>

Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)

[Wangda Tan]

As pointed by Arpit, the previously deployed shared jars are incorrect.
Just redeployed jars and staged. @Arpit, could you please check the updated
Maven repo?
https://repository.apache.org/content/repositories/orgapachehadoop-1092

Since the jars inside binary tarballs are correct (
http://people.apache.org/~wangda/hadoop-3.1.0-RC1/). I think we don't need
roll another RC, just update Maven repo should be sufficient.

Best,
Wangda


On Mon, Apr 2, 2018 at 2:39 PM, Wangda Tan  wrote:

> Hi Arpit,
>
> Thanks for pointing out this.
>
> I just removed all .md5 files from artifacts. I found md5 checksums still
> exist in .mds files and I didn't remove them from .mds file because it is
> generated by create-release script and Apache guidance is "should not"
> instead of "must not". Please let me know if you think they need to be
> removed as well.
>
> - Wangda
>
>
>
> On Mon, Apr 2, 2018 at 1:37 PM, Arpit Agarwal 
> wrote:
>
>> Thanks for putting together this RC, Wangda.
>>
>> The guidance from Apache is to omit MD5s, specifically:
>>   > SHOULD NOT supply a MD5 checksum file (because MD5 is too broken).
>>
>> https://www.apache.org/dev/release-distribution#sigs-and-sums
>>
>>
>>
>> On Apr 2, 2018, at 7:03 AM, Wangda Tan  wrote:
>>
>> Hi Gera,
>>
>> It's my bad, I thought only src/bin tarball is enough.
>>
>> I just uploaded all other things under artifact/ to
>> http://people.apache.org/~wangda/hadoop-3.1.0-RC1/
>>
>> Please let me know if you have any other comments.
>>
>> Thanks,
>> Wangda
>>
>>
>> On Mon, Apr 2, 2018 at 12:50 AM, Gera Shegalov  wrote:
>>
>> Thanks, Wangda!
>>
>> There are many more artifacts in previous votes, e.g., see
>> http://home.apache.org/~junping_du/hadoop-2.8.3-RC0/ .  Among others the
>> site tarball is missing.
>>
>> On Sun, Apr 1, 2018 at 11:54 PM Sunil G  wrote:
>>
>> Thanks Wangda for initiating the release.
>>
>> I tested this RC built from source file.
>>
>>
>>   - Tested MR apps (sleep, wc) and verified both new YARN UI and old RM
>> UI.
>>   - Below feature sanity is done
>>  - Application priority
>>  - Application timeout
>>  - Intra Queue preemption with priority based
>>  - DS based affinity tests to verify placement constraints.
>>   - Tested basic NodeLabel scenarios.
>>  - Added couple of labels to few of nodes and behavior is coming
>>  correct.
>>  - Verified old UI  and new YARN UI for labels.
>>  - Submitted apps to labelled cluster and it works fine.
>>  - Also performed few cli commands related to nodelabel.
>>   - Test basic HA cases and seems correct.
>>   - Tested new YARN UI . All pages are getting loaded correctly.
>>
>>
>> - Sunil
>>
>> On Fri, Mar 30, 2018 at 9:45 AM Wangda Tan  wrote:
>>
>> Hi folks,
>>
>> Thanks to the many who helped with this release since Dec 2017 [1].
>>
>> We've
>>
>> created RC1 for Apache Hadoop 3.1.0. The artifacts are available here:
>>
>> http://people.apache.org/~wangda/hadoop-3.1.0-RC1
>>
>> The RC tag in git is release-3.1.0-RC1. Last git commit SHA is
>> 16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d
>>
>> The maven artifacts are available via repository.apache.org at
>> https://repository.apache.org/content/repositories/
>>
>> orgapachehadoop-1090/
>>
>> This vote will run 5 days, ending on Apr 3 at 11:59 pm Pacific.
>>
>> 3.1.0 contains 766 [2] fixed JIRA issues since 3.0.0. Notable additions
>> include the first class GPU/FPGA support on YARN, Native services,
>>
>> Support
>>
>> rich placement constraints in YARN, S3-related enhancements, allow HDFS
>> block replicas to be provided by an external storage system, etc.
>>
>> For 3.1.0 RC0 vote discussion, please see [3].
>>
>> We’d like to use this as a starting release for 3.1.x [1], depending on
>>
>> how
>>
>> it goes, get it stabilized and potentially use a 3.1.1 in several weeks
>>
>> as
>>
>> the stable release.
>>
>> We have done testing with a pseudo cluster:
>> - Ran distributed job.
>> - GPU scheduling/isolation.
>> - Placement constraints (intra-application anti-affinity) by using
>> distributed shell.
>>
>> My +1 to start.
>>
>> Best,
>> Wangda/Vinod
>>
>> [1]
>>
>> https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104b
>>
>> c9e19e559aedc5ebedb4ca08c8@%3Cyarn-dev.hadoop.apache.org%3E
>>
>> [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.0)
>> AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status = Resolved ORDER
>>
>> BY
>>
>> fixVersion ASC
>> [3]
>>
>> https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d7
>>
>> 2d4061f26f83547e41d0983ea6@%3Cyarn-dev.hadoop.apache.org%3E
>>
>>
>>
>>
>>
>>
>

Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)

[Wangda Tan]

Hi Arpit,

Thanks for pointing out this.

I just removed all .md5 files from artifacts. I found md5 checksums still
exist in .mds files and I didn't remove them from .mds file because it is
generated by create-release script and Apache guidance is "should not"
instead of "must not". Please let me know if you think they need to be
removed as well.

- Wangda



On Mon, Apr 2, 2018 at 1:37 PM, Arpit Agarwal 
wrote:

> Thanks for putting together this RC, Wangda.
>
> The guidance from Apache is to omit MD5s, specifically:
>   > SHOULD NOT supply a MD5 checksum file (because MD5 is too broken).
>
> https://www.apache.org/dev/release-distribution#sigs-and-sums
>
>
>
> On Apr 2, 2018, at 7:03 AM, Wangda Tan  wrote:
>
> Hi Gera,
>
> It's my bad, I thought only src/bin tarball is enough.
>
> I just uploaded all other things under artifact/ to
> http://people.apache.org/~wangda/hadoop-3.1.0-RC1/
>
> Please let me know if you have any other comments.
>
> Thanks,
> Wangda
>
>
> On Mon, Apr 2, 2018 at 12:50 AM, Gera Shegalov  wrote:
>
> Thanks, Wangda!
>
> There are many more artifacts in previous votes, e.g., see
> http://home.apache.org/~junping_du/hadoop-2.8.3-RC0/ .  Among others the
> site tarball is missing.
>
> On Sun, Apr 1, 2018 at 11:54 PM Sunil G  wrote:
>
> Thanks Wangda for initiating the release.
>
> I tested this RC built from source file.
>
>
>   - Tested MR apps (sleep, wc) and verified both new YARN UI and old RM
> UI.
>   - Below feature sanity is done
>  - Application priority
>  - Application timeout
>  - Intra Queue preemption with priority based
>  - DS based affinity tests to verify placement constraints.
>   - Tested basic NodeLabel scenarios.
>  - Added couple of labels to few of nodes and behavior is coming
>  correct.
>  - Verified old UI  and new YARN UI for labels.
>  - Submitted apps to labelled cluster and it works fine.
>  - Also performed few cli commands related to nodelabel.
>   - Test basic HA cases and seems correct.
>   - Tested new YARN UI . All pages are getting loaded correctly.
>
>
> - Sunil
>
> On Fri, Mar 30, 2018 at 9:45 AM Wangda Tan  wrote:
>
> Hi folks,
>
> Thanks to the many who helped with this release since Dec 2017 [1].
>
> We've
>
> created RC1 for Apache Hadoop 3.1.0. The artifacts are available here:
>
> http://people.apache.org/~wangda/hadoop-3.1.0-RC1
>
> The RC tag in git is release-3.1.0-RC1. Last git commit SHA is
> 16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d
>
> The maven artifacts are available via repository.apache.org at
> https://repository.apache.org/content/repositories/
>
> orgapachehadoop-1090/
>
> This vote will run 5 days, ending on Apr 3 at 11:59 pm Pacific.
>
> 3.1.0 contains 766 [2] fixed JIRA issues since 3.0.0. Notable additions
> include the first class GPU/FPGA support on YARN, Native services,
>
> Support
>
> rich placement constraints in YARN, S3-related enhancements, allow HDFS
> block replicas to be provided by an external storage system, etc.
>
> For 3.1.0 RC0 vote discussion, please see [3].
>
> We’d like to use this as a starting release for 3.1.x [1], depending on
>
> how
>
> it goes, get it stabilized and potentially use a 3.1.1 in several weeks
>
> as
>
> the stable release.
>
> We have done testing with a pseudo cluster:
> - Ran distributed job.
> - GPU scheduling/isolation.
> - Placement constraints (intra-application anti-affinity) by using
> distributed shell.
>
> My +1 to start.
>
> Best,
> Wangda/Vinod
>
> [1]
>
> https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104b
>
> c9e19e559aedc5ebedb4ca08c8@%3Cyarn-dev.hadoop.apache.org%3E
>
> [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.0)
> AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status = Resolved ORDER
>
> BY
>
> fixVersion ASC
> [3]
>
> https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d7
>
> 2d4061f26f83547e41d0983ea6@%3Cyarn-dev.hadoop.apache.org%3E
>
>
>
>
>
>

Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)

[Arpit Agarwal]

Thanks for putting together this RC, Wangda.

The guidance from Apache is to omit MD5s, specifically:
  > SHOULD NOT supply a MD5 checksum file (because MD5 is too broken).

https://www.apache.org/dev/release-distribution#sigs-and-sums



On Apr 2, 2018, at 7:03 AM, Wangda Tan 
> wrote:

Hi Gera,

It's my bad, I thought only src/bin tarball is enough.

I just uploaded all other things under artifact/ to
http://people.apache.org/~wangda/hadoop-3.1.0-RC1/

Please let me know if you have any other comments.

Thanks,
Wangda


On Mon, Apr 2, 2018 at 12:50 AM, Gera Shegalov  wrote:

Thanks, Wangda!

There are many more artifacts in previous votes, e.g., see
http://home.apache.org/~junping_du/hadoop-2.8.3-RC0/ .  Among others the
site tarball is missing.

On Sun, Apr 1, 2018 at 11:54 PM Sunil G  wrote:

Thanks Wangda for initiating the release.

I tested this RC built from source file.


  - Tested MR apps (sleep, wc) and verified both new YARN UI and old RM
UI.
  - Below feature sanity is done
 - Application priority
 - Application timeout
 - Intra Queue preemption with priority based
 - DS based affinity tests to verify placement constraints.
  - Tested basic NodeLabel scenarios.
 - Added couple of labels to few of nodes and behavior is coming
 correct.
 - Verified old UI  and new YARN UI for labels.
 - Submitted apps to labelled cluster and it works fine.
 - Also performed few cli commands related to nodelabel.
  - Test basic HA cases and seems correct.
  - Tested new YARN UI . All pages are getting loaded correctly.


- Sunil

On Fri, Mar 30, 2018 at 9:45 AM Wangda Tan  wrote:

Hi folks,

Thanks to the many who helped with this release since Dec 2017 [1].
We've
created RC1 for Apache Hadoop 3.1.0. The artifacts are available here:

http://people.apache.org/~wangda/hadoop-3.1.0-RC1

The RC tag in git is release-3.1.0-RC1. Last git commit SHA is
16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d

The maven artifacts are available via repository.apache.org at
https://repository.apache.org/content/repositories/
orgapachehadoop-1090/
This vote will run 5 days, ending on Apr 3 at 11:59 pm Pacific.

3.1.0 contains 766 [2] fixed JIRA issues since 3.0.0. Notable additions
include the first class GPU/FPGA support on YARN, Native services,
Support
rich placement constraints in YARN, S3-related enhancements, allow HDFS
block replicas to be provided by an external storage system, etc.

For 3.1.0 RC0 vote discussion, please see [3].

We’d like to use this as a starting release for 3.1.x [1], depending on
how
it goes, get it stabilized and potentially use a 3.1.1 in several weeks
as
the stable release.

We have done testing with a pseudo cluster:
- Ran distributed job.
- GPU scheduling/isolation.
- Placement constraints (intra-application anti-affinity) by using
distributed shell.

My +1 to start.

Best,
Wangda/Vinod

[1]

https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104b
c9e19e559aedc5ebedb4ca08c8@%3Cyarn-dev.hadoop.apache.org%3E
[2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.0)
AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status = Resolved ORDER
BY
fixVersion ASC
[3]

https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d7
2d4061f26f83547e41d0983ea6@%3Cyarn-dev.hadoop.apache.org%3E

Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)

[Wangda Tan]

Hi Gera,

It's my bad, I thought only src/bin tarball is enough.

I just uploaded all other things under artifact/ to
http://people.apache.org/~wangda/hadoop-3.1.0-RC1/

Please let me know if you have any other comments.

Thanks,
Wangda


On Mon, Apr 2, 2018 at 12:50 AM, Gera Shegalov  wrote:

> Thanks, Wangda!
>
> There are many more artifacts in previous votes, e.g., see
> http://home.apache.org/~junping_du/hadoop-2.8.3-RC0/ .  Among others the
> site tarball is missing.
>
> On Sun, Apr 1, 2018 at 11:54 PM Sunil G  wrote:
>
>> Thanks Wangda for initiating the release.
>>
>> I tested this RC built from source file.
>>
>>
>>- Tested MR apps (sleep, wc) and verified both new YARN UI and old RM
>> UI.
>>- Below feature sanity is done
>>   - Application priority
>>   - Application timeout
>>   - Intra Queue preemption with priority based
>>   - DS based affinity tests to verify placement constraints.
>>- Tested basic NodeLabel scenarios.
>>   - Added couple of labels to few of nodes and behavior is coming
>>   correct.
>>   - Verified old UI  and new YARN UI for labels.
>>   - Submitted apps to labelled cluster and it works fine.
>>   - Also performed few cli commands related to nodelabel.
>>- Test basic HA cases and seems correct.
>>- Tested new YARN UI . All pages are getting loaded correctly.
>>
>>
>> - Sunil
>>
>> On Fri, Mar 30, 2018 at 9:45 AM Wangda Tan  wrote:
>>
>> > Hi folks,
>> >
>> > Thanks to the many who helped with this release since Dec 2017 [1].
>> We've
>> > created RC1 for Apache Hadoop 3.1.0. The artifacts are available here:
>> >
>> > http://people.apache.org/~wangda/hadoop-3.1.0-RC1
>> >
>> > The RC tag in git is release-3.1.0-RC1. Last git commit SHA is
>> > 16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d
>> >
>> > The maven artifacts are available via repository.apache.org at
>> > https://repository.apache.org/content/repositories/
>> orgapachehadoop-1090/
>> > This vote will run 5 days, ending on Apr 3 at 11:59 pm Pacific.
>> >
>> > 3.1.0 contains 766 [2] fixed JIRA issues since 3.0.0. Notable additions
>> > include the first class GPU/FPGA support on YARN, Native services,
>> Support
>> > rich placement constraints in YARN, S3-related enhancements, allow HDFS
>> > block replicas to be provided by an external storage system, etc.
>> >
>> > For 3.1.0 RC0 vote discussion, please see [3].
>> >
>> > We’d like to use this as a starting release for 3.1.x [1], depending on
>> how
>> > it goes, get it stabilized and potentially use a 3.1.1 in several weeks
>> as
>> > the stable release.
>> >
>> > We have done testing with a pseudo cluster:
>> > - Ran distributed job.
>> > - GPU scheduling/isolation.
>> > - Placement constraints (intra-application anti-affinity) by using
>> > distributed shell.
>> >
>> > My +1 to start.
>> >
>> > Best,
>> > Wangda/Vinod
>> >
>> > [1]
>> >
>> > https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104b
>> c9e19e559aedc5ebedb4ca08c8@%3Cyarn-dev.hadoop.apache.org%3E
>> > [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.0)
>> > AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status = Resolved ORDER
>> BY
>> > fixVersion ASC
>> > [3]
>> >
>> > https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d7
>> 2d4061f26f83547e41d0983ea6@%3Cyarn-dev.hadoop.apache.org%3E
>> >
>>
>

Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)

[Sunil G]

+1 (binding)

On Mon 2 Apr, 2018, 12:24 Sunil G,  wrote:

> Thanks Wangda for initiating the release.
>
> I tested this RC built from source file.
>
>
>- Tested MR apps (sleep, wc) and verified both new YARN UI and old RM
>UI.
>- Below feature sanity is done
>   - Application priority
>   - Application timeout
>   - Intra Queue preemption with priority based
>   - DS based affinity tests to verify placement constraints.
>- Tested basic NodeLabel scenarios.
>   - Added couple of labels to few of nodes and behavior is coming
>   correct.
>   - Verified old UI  and new YARN UI for labels.
>   - Submitted apps to labelled cluster and it works fine.
>   - Also performed few cli commands related to nodelabel.
>- Test basic HA cases and seems correct.
>- Tested new YARN UI . All pages are getting loaded correctly.
>
>
> - Sunil
>
>
> On Fri, Mar 30, 2018 at 9:45 AM Wangda Tan  wrote:
>
>> Hi folks,
>>
>> Thanks to the many who helped with this release since Dec 2017 [1]. We've
>> created RC1 for Apache Hadoop 3.1.0. The artifacts are available here:
>>
>> http://people.apache.org/~wangda/hadoop-3.1.0-RC1
>>
>> The RC tag in git is release-3.1.0-RC1. Last git commit SHA is
>> 16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d
>>
>> The maven artifacts are available via repository.apache.org at
>> https://repository.apache.org/content/repositories/orgapachehadoop-1090/
>> This vote will run 5 days, ending on Apr 3 at 11:59 pm Pacific.
>>
>> 3.1.0 contains 766 [2] fixed JIRA issues since 3.0.0. Notable additions
>> include the first class GPU/FPGA support on YARN, Native services, Support
>> rich placement constraints in YARN, S3-related enhancements, allow HDFS
>> block replicas to be provided by an external storage system, etc.
>>
>> For 3.1.0 RC0 vote discussion, please see [3].
>>
>> We’d like to use this as a starting release for 3.1.x [1], depending on
>> how
>> it goes, get it stabilized and potentially use a 3.1.1 in several weeks as
>> the stable release.
>>
>> We have done testing with a pseudo cluster:
>> - Ran distributed job.
>> - GPU scheduling/isolation.
>> - Placement constraints (intra-application anti-affinity) by using
>> distributed shell.
>>
>> My +1 to start.
>>
>> Best,
>> Wangda/Vinod
>>
>> [1]
>>
>> https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104bc9e19e559aedc5ebedb4ca08c8@%3Cyarn-dev.hadoop.apache.org%3E
>> [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.0)
>> AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status = Resolved ORDER BY
>> fixVersion ASC
>> [3]
>>
>> https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d72d4061f26f83547e41d0983ea6@%3Cyarn-dev.hadoop.apache.org%3E
>>
>

Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)

[Gera Shegalov]

Thanks, Wangda!

There are many more artifacts in previous votes, e.g., see
http://home.apache.org/~junping_du/hadoop-2.8.3-RC0/ .  Among others the
site tarball is missing.

On Sun, Apr 1, 2018 at 11:54 PM Sunil G  wrote:

> Thanks Wangda for initiating the release.
>
> I tested this RC built from source file.
>
>
>- Tested MR apps (sleep, wc) and verified both new YARN UI and old RM
> UI.
>- Below feature sanity is done
>   - Application priority
>   - Application timeout
>   - Intra Queue preemption with priority based
>   - DS based affinity tests to verify placement constraints.
>- Tested basic NodeLabel scenarios.
>   - Added couple of labels to few of nodes and behavior is coming
>   correct.
>   - Verified old UI  and new YARN UI for labels.
>   - Submitted apps to labelled cluster and it works fine.
>   - Also performed few cli commands related to nodelabel.
>- Test basic HA cases and seems correct.
>- Tested new YARN UI . All pages are getting loaded correctly.
>
>
> - Sunil
>
> On Fri, Mar 30, 2018 at 9:45 AM Wangda Tan  wrote:
>
> > Hi folks,
> >
> > Thanks to the many who helped with this release since Dec 2017 [1]. We've
> > created RC1 for Apache Hadoop 3.1.0. The artifacts are available here:
> >
> > http://people.apache.org/~wangda/hadoop-3.1.0-RC1
> >
> > The RC tag in git is release-3.1.0-RC1. Last git commit SHA is
> > 16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d
> >
> > The maven artifacts are available via repository.apache.org at
> > https://repository.apache.org/content/repositories/orgapachehadoop-1090/
> > This vote will run 5 days, ending on Apr 3 at 11:59 pm Pacific.
> >
> > 3.1.0 contains 766 [2] fixed JIRA issues since 3.0.0. Notable additions
> > include the first class GPU/FPGA support on YARN, Native services,
> Support
> > rich placement constraints in YARN, S3-related enhancements, allow HDFS
> > block replicas to be provided by an external storage system, etc.
> >
> > For 3.1.0 RC0 vote discussion, please see [3].
> >
> > We’d like to use this as a starting release for 3.1.x [1], depending on
> how
> > it goes, get it stabilized and potentially use a 3.1.1 in several weeks
> as
> > the stable release.
> >
> > We have done testing with a pseudo cluster:
> > - Ran distributed job.
> > - GPU scheduling/isolation.
> > - Placement constraints (intra-application anti-affinity) by using
> > distributed shell.
> >
> > My +1 to start.
> >
> > Best,
> > Wangda/Vinod
> >
> > [1]
> >
> >
> https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104bc9e19e559aedc5ebedb4ca08c8@%3Cyarn-dev.hadoop.apache.org%3E
> > [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.0)
> > AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status = Resolved ORDER BY
> > fixVersion ASC
> > [3]
> >
> >
> https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d72d4061f26f83547e41d0983ea6@%3Cyarn-dev.hadoop.apache.org%3E
> >
>

Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)

[Sunil G]

Thanks Wangda for initiating the release.

I tested this RC built from source file.


   - Tested MR apps (sleep, wc) and verified both new YARN UI and old RM UI.
   - Below feature sanity is done
  - Application priority
  - Application timeout
  - Intra Queue preemption with priority based
  - DS based affinity tests to verify placement constraints.
   - Tested basic NodeLabel scenarios.
  - Added couple of labels to few of nodes and behavior is coming
  correct.
  - Verified old UI  and new YARN UI for labels.
  - Submitted apps to labelled cluster and it works fine.
  - Also performed few cli commands related to nodelabel.
   - Test basic HA cases and seems correct.
   - Tested new YARN UI . All pages are getting loaded correctly.


- Sunil

On Fri, Mar 30, 2018 at 9:45 AM Wangda Tan  wrote:

> Hi folks,
>
> Thanks to the many who helped with this release since Dec 2017 [1]. We've
> created RC1 for Apache Hadoop 3.1.0. The artifacts are available here:
>
> http://people.apache.org/~wangda/hadoop-3.1.0-RC1
>
> The RC tag in git is release-3.1.0-RC1. Last git commit SHA is
> 16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d
>
> The maven artifacts are available via repository.apache.org at
> https://repository.apache.org/content/repositories/orgapachehadoop-1090/
> This vote will run 5 days, ending on Apr 3 at 11:59 pm Pacific.
>
> 3.1.0 contains 766 [2] fixed JIRA issues since 3.0.0. Notable additions
> include the first class GPU/FPGA support on YARN, Native services, Support
> rich placement constraints in YARN, S3-related enhancements, allow HDFS
> block replicas to be provided by an external storage system, etc.
>
> For 3.1.0 RC0 vote discussion, please see [3].
>
> We’d like to use this as a starting release for 3.1.x [1], depending on how
> it goes, get it stabilized and potentially use a 3.1.1 in several weeks as
> the stable release.
>
> We have done testing with a pseudo cluster:
> - Ran distributed job.
> - GPU scheduling/isolation.
> - Placement constraints (intra-application anti-affinity) by using
> distributed shell.
>
> My +1 to start.
>
> Best,
> Wangda/Vinod
>
> [1]
>
> https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104bc9e19e559aedc5ebedb4ca08c8@%3Cyarn-dev.hadoop.apache.org%3E
> [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.0)
> AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status = Resolved ORDER BY
> fixVersion ASC
> [3]
>
> https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d72d4061f26f83547e41d0983ea6@%3Cyarn-dev.hadoop.apache.org%3E
>

Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)

[Weiwei Yang]

Hi Wangda

+1 (non-binding)

- Smoke tests with teragen/terasort/DS jobs
- Various of metrics, UI displays validation, compatible tests
- Tested GPU resource-type
- Verified RM fail-over, app-recovery
- Verified 2 threads async-scheduling
- Enabled placement constraints, tested affinity/anti-affinity allocations
- SLS tests

--
Weiwei

On 2 Apr 2018, 1:13 PM +0800, Brahma Reddy Battula , wrote:
Wangda thanks for driving this.

+1(binding)

--Built from source
--Installed HA cluster
--Verified Basic Shell commands
--Ran Sample Jobs
--Browsed the UI's.


On Fri, Mar 30, 2018 at 9:45 AM, Wangda Tan  wrote:

Hi folks,

Thanks to the many who helped with this release since Dec 2017 [1]. We've
created RC1 for Apache Hadoop 3.1.0. The artifacts are available here:

http://people.apache.org/~wangda/hadoop-3.1.0-RC1

The RC tag in git is release-3.1.0-RC1. Last git commit SHA is
16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d

The maven artifacts are available via repository.apache.org at
https://repository.apache.org/content/repositories/orgapachehadoop-1090/
This vote will run 5 days, ending on Apr 3 at 11:59 pm Pacific.

3.1.0 contains 766 [2] fixed JIRA issues since 3.0.0. Notable additions
include the first class GPU/FPGA support on YARN, Native services, Support
rich placement constraints in YARN, S3-related enhancements, allow HDFS
block replicas to be provided by an external storage system, etc.

For 3.1.0 RC0 vote discussion, please see [3].

We’d like to use this as a starting release for 3.1.x [1], depending on how
it goes, get it stabilized and potentially use a 3.1.1 in several weeks as
the stable release.

We have done testing with a pseudo cluster:
- Ran distributed job.
- GPU scheduling/isolation.
- Placement constraints (intra-application anti-affinity) by using
distributed shell.

My +1 to start.

Best,
Wangda/Vinod

[1]
https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104b
c9e19e559aedc5ebedb4ca08c8@%3Cyarn-dev.hadoop.apache.org%3E
[2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.0)
AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status = Resolved ORDER BY
fixVersion ASC
[3]
https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d7
2d4061f26f83547e41d0983ea6@%3Cyarn-dev.hadoop.apache.org%3E

Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)

[Brahma Reddy Battula]

Wangda thanks for driving this.

+1(binding)

--Built from source
--Installed HA cluster
--Verified Basic Shell commands
--Ran Sample Jobs
--Browsed the UI's.


On Fri, Mar 30, 2018 at 9:45 AM, Wangda Tan  wrote:

> Hi folks,
>
> Thanks to the many who helped with this release since Dec 2017 [1]. We've
> created RC1 for Apache Hadoop 3.1.0. The artifacts are available here:
>
> http://people.apache.org/~wangda/hadoop-3.1.0-RC1
>
> The RC tag in git is release-3.1.0-RC1. Last git commit SHA is
> 16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d
>
> The maven artifacts are available via repository.apache.org at
> https://repository.apache.org/content/repositories/orgapachehadoop-1090/
> This vote will run 5 days, ending on Apr 3 at 11:59 pm Pacific.
>
> 3.1.0 contains 766 [2] fixed JIRA issues since 3.0.0. Notable additions
> include the first class GPU/FPGA support on YARN, Native services, Support
> rich placement constraints in YARN, S3-related enhancements, allow HDFS
> block replicas to be provided by an external storage system, etc.
>
> For 3.1.0 RC0 vote discussion, please see [3].
>
> We’d like to use this as a starting release for 3.1.x [1], depending on how
> it goes, get it stabilized and potentially use a 3.1.1 in several weeks as
> the stable release.
>
> We have done testing with a pseudo cluster:
> - Ran distributed job.
> - GPU scheduling/isolation.
> - Placement constraints (intra-application anti-affinity) by using
> distributed shell.
>
> My +1 to start.
>
> Best,
> Wangda/Vinod
>
> [1]
> https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104b
> c9e19e559aedc5ebedb4ca08c8@%3Cyarn-dev.hadoop.apache.org%3E
> [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.0)
> AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status = Resolved ORDER BY
> fixVersion ASC
> [3]
> https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d7
> 2d4061f26f83547e41d0983ea6@%3Cyarn-dev.hadoop.apache.org%3E
>

Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)

[Rohith Sharma K S]

+1 (binding)

* Downloaded source, built from source with -Dhbase.profile=2.0.
* Installed RM HA cluster integrated with ATSv2. Installed HBase-2.0-beta1
for ATSv2 back end. Scheduler is configured with 2 level queue hierarchy.
* Ran sample jobs such as MR/Distributed shell  and verified for
** ATSv2 Timeline Reader REST API's. Validated for data published in ATSv2.
** YARN UI2 accessed for Flow Activity pages. Navigated inside flow
activity page for all other info.
** YARN old UI also verified for all the pages
** RM HA/Restart/work-preserving-restart are tested while running a job.
** NM restart scenarios are verified.
** Application timeout and application priority feature is tested.

Thanks & Regards
Rohith Sharma K S

On 30 March 2018 at 09:45, Wangda Tan  wrote:

> Hi folks,
>
> Thanks to the many who helped with this release since Dec 2017 [1]. We've
> created RC1 for Apache Hadoop 3.1.0. The artifacts are available here:
>
> http://people.apache.org/~wangda/hadoop-3.1.0-RC1
>
> The RC tag in git is release-3.1.0-RC1. Last git commit SHA is
> 16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d
>
> The maven artifacts are available via repository.apache.org at
> https://repository.apache.org/content/repositories/orgapachehadoop-1090/
> This vote will run 5 days, ending on Apr 3 at 11:59 pm Pacific.
>
> 3.1.0 contains 766 [2] fixed JIRA issues since 3.0.0. Notable additions
> include the first class GPU/FPGA support on YARN, Native services, Support
> rich placement constraints in YARN, S3-related enhancements, allow HDFS
> block replicas to be provided by an external storage system, etc.
>
> For 3.1.0 RC0 vote discussion, please see [3].
>
> We’d like to use this as a starting release for 3.1.x [1], depending on how
> it goes, get it stabilized and potentially use a 3.1.1 in several weeks as
> the stable release.
>
> We have done testing with a pseudo cluster:
> - Ran distributed job.
> - GPU scheduling/isolation.
> - Placement constraints (intra-application anti-affinity) by using
> distributed shell.
>
> My +1 to start.
>
> Best,
> Wangda/Vinod
>
> [1]
> https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104b
> c9e19e559aedc5ebedb4ca08c8@%3Cyarn-dev.hadoop.apache.org%3E
> [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.0)
> AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status = Resolved ORDER BY
> fixVersion ASC
> [3]
> https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d7
> 2d4061f26f83547e41d0983ea6@%3Cyarn-dev.hadoop.apache.org%3E
>

Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)

[Yufei Gu]

 Thanks Wangda for working on this!

+1 (non-binding)

   - Downloaded the binary tarball and verified the checksum.
   - Started a pseudo cluster inside one docker container
   - Run Resource Manager with Fair Scheduler
   - Verified distributed shell
   - Verified mapreduce pi job
   - Sanity checked RM WebUI

Best,

Yufei

On Thu, Mar 29, 2018 at 9:15 PM, Wangda Tan  wrote:

> Hi folks,
>
> Thanks to the many who helped with this release since Dec 2017 [1]. We've
> created RC1 for Apache Hadoop 3.1.0. The artifacts are available here:
>
> http://people.apache.org/~wangda/hadoop-3.1.0-RC1
>
> The RC tag in git is release-3.1.0-RC1. Last git commit SHA is
> 16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d
>
> The maven artifacts are available via repository.apache.org at
> https://repository.apache.org/content/repositories/orgapachehadoop-1090/
> This vote will run 5 days, ending on Apr 3 at 11:59 pm Pacific.
>
> 3.1.0 contains 766 [2] fixed JIRA issues since 3.0.0. Notable additions
> include the first class GPU/FPGA support on YARN, Native services, Support
> rich placement constraints in YARN, S3-related enhancements, allow HDFS
> block replicas to be provided by an external storage system, etc.
>
> For 3.1.0 RC0 vote discussion, please see [3].
>
> We’d like to use this as a starting release for 3.1.x [1], depending on how
> it goes, get it stabilized and potentially use a 3.1.1 in several weeks as
> the stable release.
>
> We have done testing with a pseudo cluster:
> - Ran distributed job.
> - GPU scheduling/isolation.
> - Placement constraints (intra-application anti-affinity) by using
> distributed shell.
>
> My +1 to start.
>
> Best,
> Wangda/Vinod
>
> [1]
> https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104b
> c9e19e559aedc5ebedb4ca08c8@%3Cyarn-dev.hadoop.apache.org%3E
> [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.0)
> AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status = Resolved ORDER BY
> fixVersion ASC
> [3]
> https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d7
> 2d4061f26f83547e41d0983ea6@%3Cyarn-dev.hadoop.apache.org%3E
>