Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)

[Rohith Sharma K S]

+1 (binding)

* Downloaded source, built from source with -Dhbase.profile=2.0.
* Installed RM HA cluster integrated with ATSv2. Installed HBase-2.0-beta1
for ATSv2 back end. Scheduler is configured with 2 level queue hierarchy.
* Ran sample jobs such as MR/Distributed shell  and verified for
** ATSv2 Timeline Reader REST API's. Validated for data published in ATSv2.
** YARN UI2 accessed for Flow Activity pages. Navigated inside flow
activity page for all other info.
** YARN old UI also verified for all the pages
** RM HA/Restart/work-preserving-restart are tested while running a job.
** NM restart scenarios are verified.
** Application timeout and application priority feature is tested.

Thanks & Regards
Rohith Sharma K S

On 30 March 2018 at 09:45, Wangda Tan  wrote:

> Hi folks,
>
> Thanks to the many who helped with this release since Dec 2017 [1]. We've
> created RC1 for Apache Hadoop 3.1.0. The artifacts are available here:
>
> http://people.apache.org/~wangda/hadoop-3.1.0-RC1
>
> The RC tag in git is release-3.1.0-RC1. Last git commit SHA is
> 16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d
>
> The maven artifacts are available via repository.apache.org at
> https://repository.apache.org/content/repositories/orgapachehadoop-1090/
> This vote will run 5 days, ending on Apr 3 at 11:59 pm Pacific.
>
> 3.1.0 contains 766 [2] fixed JIRA issues since 3.0.0. Notable additions
> include the first class GPU/FPGA support on YARN, Native services, Support
> rich placement constraints in YARN, S3-related enhancements, allow HDFS
> block replicas to be provided by an external storage system, etc.
>
> For 3.1.0 RC0 vote discussion, please see [3].
>
> We’d like to use this as a starting release for 3.1.x [1], depending on how
> it goes, get it stabilized and potentially use a 3.1.1 in several weeks as
> the stable release.
>
> We have done testing with a pseudo cluster:
> - Ran distributed job.
> - GPU scheduling/isolation.
> - Placement constraints (intra-application anti-affinity) by using
> distributed shell.
>
> My +1 to start.
>
> Best,
> Wangda/Vinod
>
> [1]
> https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104b
> c9e19e559aedc5ebedb4ca08c8@%3Cyarn-dev.hadoop.apache.org%3E
> [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.0)
> AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status = Resolved ORDER BY
> fixVersion ASC
> [3]
> https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d7
> 2d4061f26f83547e41d0983ea6@%3Cyarn-dev.hadoop.apache.org%3E
>

Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)

[Yufei Gu]

 Thanks Wangda for working on this!

+1 (non-binding)

   - Downloaded the binary tarball and verified the checksum.
   - Started a pseudo cluster inside one docker container
   - Run Resource Manager with Fair Scheduler
   - Verified distributed shell
   - Verified mapreduce pi job
   - Sanity checked RM WebUI

Best,

Yufei

On Thu, Mar 29, 2018 at 9:15 PM, Wangda Tan  wrote:

> Hi folks,
>
> Thanks to the many who helped with this release since Dec 2017 [1]. We've
> created RC1 for Apache Hadoop 3.1.0. The artifacts are available here:
>
> http://people.apache.org/~wangda/hadoop-3.1.0-RC1
>
> The RC tag in git is release-3.1.0-RC1. Last git commit SHA is
> 16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d
>
> The maven artifacts are available via repository.apache.org at
> https://repository.apache.org/content/repositories/orgapachehadoop-1090/
> This vote will run 5 days, ending on Apr 3 at 11:59 pm Pacific.
>
> 3.1.0 contains 766 [2] fixed JIRA issues since 3.0.0. Notable additions
> include the first class GPU/FPGA support on YARN, Native services, Support
> rich placement constraints in YARN, S3-related enhancements, allow HDFS
> block replicas to be provided by an external storage system, etc.
>
> For 3.1.0 RC0 vote discussion, please see [3].
>
> We’d like to use this as a starting release for 3.1.x [1], depending on how
> it goes, get it stabilized and potentially use a 3.1.1 in several weeks as
> the stable release.
>
> We have done testing with a pseudo cluster:
> - Ran distributed job.
> - GPU scheduling/isolation.
> - Placement constraints (intra-application anti-affinity) by using
> distributed shell.
>
> My +1 to start.
>
> Best,
> Wangda/Vinod
>
> [1]
> https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104b
> c9e19e559aedc5ebedb4ca08c8@%3Cyarn-dev.hadoop.apache.org%3E
> [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.0)
> AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status = Resolved ORDER BY
> fixVersion ASC
> [3]
> https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d7
> 2d4061f26f83547e41d0983ea6@%3Cyarn-dev.hadoop.apache.org%3E
>

[VOTE] Release Apache Hadoop 3.1.0 (RC1)

[Wangda Tan]

Hi folks,

Thanks to the many who helped with this release since Dec 2017 [1]. We've
created RC1 for Apache Hadoop 3.1.0. The artifacts are available here:

http://people.apache.org/~wangda/hadoop-3.1.0-RC1

The RC tag in git is release-3.1.0-RC1. Last git commit SHA is
16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d

The maven artifacts are available via repository.apache.org at
https://repository.apache.org/content/repositories/orgapachehadoop-1090/
This vote will run 5 days, ending on Apr 3 at 11:59 pm Pacific.

3.1.0 contains 766 [2] fixed JIRA issues since 3.0.0. Notable additions
include the first class GPU/FPGA support on YARN, Native services, Support
rich placement constraints in YARN, S3-related enhancements, allow HDFS
block replicas to be provided by an external storage system, etc.

For 3.1.0 RC0 vote discussion, please see [3].

We’d like to use this as a starting release for 3.1.x [1], depending on how
it goes, get it stabilized and potentially use a 3.1.1 in several weeks as
the stable release.

We have done testing with a pseudo cluster:
- Ran distributed job.
- GPU scheduling/isolation.
- Placement constraints (intra-application anti-affinity) by using
distributed shell.

My +1 to start.

Best,
Wangda/Vinod

[1]
https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104bc9e19e559aedc5ebedb4ca08c8@%3Cyarn-dev.hadoop.apache.org%3E
[2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.0)
AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status = Resolved ORDER BY
fixVersion ASC
[3]
https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d72d4061f26f83547e41d0983ea6@%3Cyarn-dev.hadoop.apache.org%3E

Apache Hadoop qbt Report: branch2+JDK7 on Linux/x86

[Apache Jenkins Server]

For more details, see 
https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/179/

[Mar 30, 2018 1:03:32 AM] (shv) HADOOP-12862. LDAP Group Mapping over SSL can 
not specify trust store.




-1 overall


The following subsystems voted -1:
docker


Powered by Apache Yetus 0.8.0-SNAPSHOT   http://yetus.apache.org

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Resolved] (HADOOP-15325) Make Configuration#getPasswordFromCredentialsProvider() a public API

[Konstantin Shvachko (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-15325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Konstantin Shvachko resolved HADOOP-15325.
--
Resolution: Implemented

This has been incorporated into HADOOP-12862.

> Make Configuration#getPasswordFromCredentialsProvider() a public API
> 
>
> Key: HADOOP-15325
> URL: https://issues.apache.org/jira/browse/HADOOP-15325
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: conf
>Affects Versions: 2.6.0
>Reporter: Wei-Chiu Chuang
>Assignee: Zsolt Venczel
>Priority: Major
>
> HADOOP-10607 added a public API Configuration.getPassword() which reads 
> passwords from credential provider and then falls back to reading from 
> configuration if one is not available.
> This API has been used throughout Hadoop codebase and downstream 
> applications. It is understandable for old password configuration keys to 
> fallback to configuration to maintain backward compatibility. But for new 
> configuration passwords that don't have legacy, there should be an option to 
> _not_ fallback, because storing passwords in configuration is considered a 
> bad security practice.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.1.0 (RC0)

[Wangda Tan]

Arun helped me to figure this out. It was pushed to a wrong location. The
correct one is: https://dist.apache.org/repos/dist/release/hadoop/common/

.

Thanks,
Wangda

On Thu, Mar 29, 2018 at 11:16 AM, Wangda Tan  wrote:

> Thanks Junping for reporting this,
>
> I did append my keys to https://svn.apache.org/
> repos/asf/hadoop/common/dist/KEYS, I'm not sure how to propagate it to
> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS, I couldn't
> find any guide about this. Could you share how you did this?
>
> - Wangda
>
>
> On Thu, Mar 29, 2018 at 10:28 AM, 俊平堵  wrote:
>
>> Hi Wanda,
>>  Thanks for the release effort on 3.1.0. Another reminder is I tried
>> to verify the signature of RC0 bits but found there is no known key get
>> uploaded to
>> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS. Would you
>> please upload it there then we can go ahead to verify the signature of
>> following RCs.Thanks!
>>
>> Thanks,
>>
>> Junping
>>
>> Wangda Tan 于2018年3月30日 周五上午12:01写道：
>>
>>> Thanks all for doing testing and reporting issues.
>>>
>>> YARN-8085 discovered by Weiwei/Tao is a blocker, we will roll an RC1
>>> today.
>>>
>>> - Wangda
>>>
>>> On Thu, Mar 29, 2018 at 7:26 AM, Weiwei Yang 
>>> wrote:
>>>
>>> > Hi Wangda
>>> >
>>> > While testing the build, we found a bug https://issues.apache.org/
>>> > jira/browse/YARN-8085, it might cause NPE during RM fail-over. I think
>>> > that needs to be included in 3.1.0.
>>> >
>>> > We have run some other tests, and they look good so far. This includes
>>> > - Some basic teragen/terasort/DS jobs
>>> > - Various of metrics, UI displays, compatible tests
>>> > - Creating queues, different queue configurations
>>> > - Verified RM fail-over, app-recovery
>>> > - Enabled async scheduling
>>> > - Enabled placement constraints
>>> > - SLS tests
>>> >
>>> > Hope it helps.
>>> > Thanks
>>> >
>>> > --
>>> > Weiwei
>>> >
>>> > On 29 Mar 2018, 12:34 PM +0800, Ajay Kumar >> >,
>>> > wrote:
>>> >
>>> > +1 (non-binding)
>>> >
>>> > - verified binary checksum
>>> > - built from source and setup 4 node cluster
>>> > - run basic hdfs command
>>> > - run wordcount, pi & TestDFSIO (read/write)
>>> >
>>> > Ajay
>>> >
>>> > On 3/28/18, 5:45 PM, "Jonathan Hung"  wrote:
>>> >
>>> > Hi Wangda, thanks for handling this release.
>>> >
>>> > +1 (non-binding)
>>> >
>>> > - verified binary checksum
>>> > - launched single node RM
>>> > - verified refreshQueues functionality
>>> > - verified capacity scheduler conf mutation disabled in this case
>>> > - verified capacity scheduler conf mutation with leveldb storage
>>> > - verified refreshQueues mutation is disabled in this case
>>> >
>>> >
>>> > Jonathan Hung
>>> >
>>> > On Thu, Mar 22, 2018 at 9:10 AM, Wangda Tan 
>>> wrote:
>>> >
>>> > Thanks @Bharat for the quick check, the previously staged repository
>>> has
>>> > some issues. I re-deployed jars to nexus.
>>> >
>>> > Here's the new repo (1087)
>>> >
>>> > https://repository.apache.org/content/repositories/orgapache
>>> hadoop-1087/
>>> >
>>> > Other artifacts remain same, no additional code changes.
>>> >
>>> > On Wed, Mar 21, 2018 at 11:54 PM, Bharat Viswanadham <
>>> > bviswanad...@hortonworks.com> wrote:
>>> >
>>> > Hi Wangda,
>>> > Maven Artifact repositories is not having all Hadoop jars. (It is
>>> missing
>>> > many like hadoop-hdfs, hadoop-client etc.,)
>>> > https://repository.apache.org/content/repositories/orgapache
>>> hadoop-1086/
>>> >
>>> >
>>> > Thanks,
>>> > Bharat
>>> >
>>> >
>>> > On 3/21/18, 11:44 PM, "Wangda Tan"  wrote:
>>> >
>>> > Hi folks,
>>> >
>>> > Thanks to the many who helped with this release since Dec 2017 [1].
>>> > We've
>>> > created RC0 for Apache Hadoop 3.1.0. The artifacts are available
>>> >
>>> > here:
>>> >
>>> >
>>> > http://people.apache.org/~wangda/hadoop-3.1.0-RC0/
>>> >
>>> > The RC tag in git is release-3.1.0-RC0.
>>> >
>>> > The maven artifacts are available via repository.apache.org at
>>> > https://repository.apache.org/content/repositories/
>>> > orgapachehadoop-1086/
>>> >
>>> > This vote will run 7 days (5 weekdays), ending on Mar 28 at 11:59 pm
>>> > Pacific.
>>> >
>>> > 3.1.0 contains 727 [2] fixed JIRA issues since 3.0.0. Notable
>>> >
>>> > additions
>>> >
>>> > include the first class GPU/FPGA support on YARN, Native services,
>>> > Support
>>> > rich placement constraints in YARN, S3-related enhancements, allow
>>> >
>>> > HDFS
>>> >
>>> > block replicas to be provided by an external storage system, etc.
>>> >
>>> > We’d like to use this as a starting release for 3.1.x [1], depending
>>> > on how
>>> > it goes, get it stabilized and potentially use a 3.1.1 in several
>>> > weeks as

Re: [VOTE] Release Apache Hadoop 3.1.0 (RC0)

[Wangda Tan]

Thanks Junping for reporting this,

I did append my keys to
https://svn.apache.org/repos/asf/hadoop/common/dist/KEYS, I'm not sure how
to propagate it to https://dist.apache.org/repos/dist/release/hadoop/common/
KEYS, I couldn't find any guide about this. Could you share how you did
this?

- Wangda


On Thu, Mar 29, 2018 at 10:28 AM, 俊平堵  wrote:

> Hi Wanda,
>  Thanks for the release effort on 3.1.0. Another reminder is I tried
> to verify the signature of RC0 bits but found there is no known key get
> uploaded to
> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS. Would you
> please upload it there then we can go ahead to verify the signature of
> following RCs.Thanks!
>
> Thanks,
>
> Junping
>
> Wangda Tan 于2018年3月30日 周五上午12:01写道：
>
>> Thanks all for doing testing and reporting issues.
>>
>> YARN-8085 discovered by Weiwei/Tao is a blocker, we will roll an RC1
>> today.
>>
>> - Wangda
>>
>> On Thu, Mar 29, 2018 at 7:26 AM, Weiwei Yang 
>> wrote:
>>
>> > Hi Wangda
>> >
>> > While testing the build, we found a bug https://issues.apache.org/
>> > jira/browse/YARN-8085, it might cause NPE during RM fail-over. I think
>> > that needs to be included in 3.1.0.
>> >
>> > We have run some other tests, and they look good so far. This includes
>> > - Some basic teragen/terasort/DS jobs
>> > - Various of metrics, UI displays, compatible tests
>> > - Creating queues, different queue configurations
>> > - Verified RM fail-over, app-recovery
>> > - Enabled async scheduling
>> > - Enabled placement constraints
>> > - SLS tests
>> >
>> > Hope it helps.
>> > Thanks
>> >
>> > --
>> > Weiwei
>> >
>> > On 29 Mar 2018, 12:34 PM +0800, Ajay Kumar > >,
>> > wrote:
>> >
>> > +1 (non-binding)
>> >
>> > - verified binary checksum
>> > - built from source and setup 4 node cluster
>> > - run basic hdfs command
>> > - run wordcount, pi & TestDFSIO (read/write)
>> >
>> > Ajay
>> >
>> > On 3/28/18, 5:45 PM, "Jonathan Hung"  wrote:
>> >
>> > Hi Wangda, thanks for handling this release.
>> >
>> > +1 (non-binding)
>> >
>> > - verified binary checksum
>> > - launched single node RM
>> > - verified refreshQueues functionality
>> > - verified capacity scheduler conf mutation disabled in this case
>> > - verified capacity scheduler conf mutation with leveldb storage
>> > - verified refreshQueues mutation is disabled in this case
>> >
>> >
>> > Jonathan Hung
>> >
>> > On Thu, Mar 22, 2018 at 9:10 AM, Wangda Tan 
>> wrote:
>> >
>> > Thanks @Bharat for the quick check, the previously staged repository has
>> > some issues. I re-deployed jars to nexus.
>> >
>> > Here's the new repo (1087)
>> >
>> > https://repository.apache.org/content/repositories/
>> orgapachehadoop-1087/
>> >
>> > Other artifacts remain same, no additional code changes.
>> >
>> > On Wed, Mar 21, 2018 at 11:54 PM, Bharat Viswanadham <
>> > bviswanad...@hortonworks.com> wrote:
>> >
>> > Hi Wangda,
>> > Maven Artifact repositories is not having all Hadoop jars. (It is
>> missing
>> > many like hadoop-hdfs, hadoop-client etc.,)
>> > https://repository.apache.org/content/repositories/
>> orgapachehadoop-1086/
>> >
>> >
>> > Thanks,
>> > Bharat
>> >
>> >
>> > On 3/21/18, 11:44 PM, "Wangda Tan"  wrote:
>> >
>> > Hi folks,
>> >
>> > Thanks to the many who helped with this release since Dec 2017 [1].
>> > We've
>> > created RC0 for Apache Hadoop 3.1.0. The artifacts are available
>> >
>> > here:
>> >
>> >
>> > http://people.apache.org/~wangda/hadoop-3.1.0-RC0/
>> >
>> > The RC tag in git is release-3.1.0-RC0.
>> >
>> > The maven artifacts are available via repository.apache.org at
>> > https://repository.apache.org/content/repositories/
>> > orgapachehadoop-1086/
>> >
>> > This vote will run 7 days (5 weekdays), ending on Mar 28 at 11:59 pm
>> > Pacific.
>> >
>> > 3.1.0 contains 727 [2] fixed JIRA issues since 3.0.0. Notable
>> >
>> > additions
>> >
>> > include the first class GPU/FPGA support on YARN, Native services,
>> > Support
>> > rich placement constraints in YARN, S3-related enhancements, allow
>> >
>> > HDFS
>> >
>> > block replicas to be provided by an external storage system, etc.
>> >
>> > We’d like to use this as a starting release for 3.1.x [1], depending
>> > on how
>> > it goes, get it stabilized and potentially use a 3.1.1 in several
>> > weeks as
>> > the stable release.
>> >
>> > We have done testing with a pseudo cluster and distributed shell job.
>> > My +1
>> > to start.
>> >
>> > Best,
>> > Wangda/Vinod
>> >
>> > [1]
>> > https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104b
>> > c9e19e559aedc5ebedb4ca08c8@%3Cyarn-dev.hadoop.apache.org%3E
>> > [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in
>> > (3.1.0)
>> > AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status = Resolved
>> >
>> > ORDER
>> >
>> > BY
>> > fixVersion ASC
>> >
>> >
>> >
>> >
>> >
>> >
>> > 

Re: [VOTE] Release Apache Hadoop 3.1.0 (RC0)

[俊平堵]

Hi Wanda,
 Thanks for the release effort on 3.1.0. Another reminder is I tried to
verify the signature of RC0 bits but found there is no known key get
uploaded to
https://dist.apache.org/repos/dist/release/hadoop/common/KEYS. Would you
please upload it there then we can go ahead to verify the signature of
following RCs.Thanks!

Thanks,

Junping

Wangda Tan 于2018年3月30日 周五上午12:01写道：

> Thanks all for doing testing and reporting issues.
>
> YARN-8085 discovered by Weiwei/Tao is a blocker, we will roll an RC1 today.
>
> - Wangda
>
> On Thu, Mar 29, 2018 at 7:26 AM, Weiwei Yang 
> wrote:
>
> > Hi Wangda
> >
> > While testing the build, we found a bug https://issues.apache.org/
> > jira/browse/YARN-8085, it might cause NPE during RM fail-over. I think
> > that needs to be included in 3.1.0.
> >
> > We have run some other tests, and they look good so far. This includes
> > - Some basic teragen/terasort/DS jobs
> > - Various of metrics, UI displays, compatible tests
> > - Creating queues, different queue configurations
> > - Verified RM fail-over, app-recovery
> > - Enabled async scheduling
> > - Enabled placement constraints
> > - SLS tests
> >
> > Hope it helps.
> > Thanks
> >
> > --
> > Weiwei
> >
> > On 29 Mar 2018, 12:34 PM +0800, Ajay Kumar ,
> > wrote:
> >
> > +1 (non-binding)
> >
> > - verified binary checksum
> > - built from source and setup 4 node cluster
> > - run basic hdfs command
> > - run wordcount, pi & TestDFSIO (read/write)
> >
> > Ajay
> >
> > On 3/28/18, 5:45 PM, "Jonathan Hung"  wrote:
> >
> > Hi Wangda, thanks for handling this release.
> >
> > +1 (non-binding)
> >
> > - verified binary checksum
> > - launched single node RM
> > - verified refreshQueues functionality
> > - verified capacity scheduler conf mutation disabled in this case
> > - verified capacity scheduler conf mutation with leveldb storage
> > - verified refreshQueues mutation is disabled in this case
> >
> >
> > Jonathan Hung
> >
> > On Thu, Mar 22, 2018 at 9:10 AM, Wangda Tan  wrote:
> >
> > Thanks @Bharat for the quick check, the previously staged repository has
> > some issues. I re-deployed jars to nexus.
> >
> > Here's the new repo (1087)
> >
> > https://repository.apache.org/content/repositories/orgapachehadoop-1087/
> >
> > Other artifacts remain same, no additional code changes.
> >
> > On Wed, Mar 21, 2018 at 11:54 PM, Bharat Viswanadham <
> > bviswanad...@hortonworks.com> wrote:
> >
> > Hi Wangda,
> > Maven Artifact repositories is not having all Hadoop jars. (It is missing
> > many like hadoop-hdfs, hadoop-client etc.,)
> > https://repository.apache.org/content/repositories/orgapachehadoop-1086/
> >
> >
> > Thanks,
> > Bharat
> >
> >
> > On 3/21/18, 11:44 PM, "Wangda Tan"  wrote:
> >
> > Hi folks,
> >
> > Thanks to the many who helped with this release since Dec 2017 [1].
> > We've
> > created RC0 for Apache Hadoop 3.1.0. The artifacts are available
> >
> > here:
> >
> >
> > http://people.apache.org/~wangda/hadoop-3.1.0-RC0/
> >
> > The RC tag in git is release-3.1.0-RC0.
> >
> > The maven artifacts are available via repository.apache.org at
> > https://repository.apache.org/content/repositories/
> > orgapachehadoop-1086/
> >
> > This vote will run 7 days (5 weekdays), ending on Mar 28 at 11:59 pm
> > Pacific.
> >
> > 3.1.0 contains 727 [2] fixed JIRA issues since 3.0.0. Notable
> >
> > additions
> >
> > include the first class GPU/FPGA support on YARN, Native services,
> > Support
> > rich placement constraints in YARN, S3-related enhancements, allow
> >
> > HDFS
> >
> > block replicas to be provided by an external storage system, etc.
> >
> > We’d like to use this as a starting release for 3.1.x [1], depending
> > on how
> > it goes, get it stabilized and potentially use a 3.1.1 in several
> > weeks as
> > the stable release.
> >
> > We have done testing with a pseudo cluster and distributed shell job.
> > My +1
> > to start.
> >
> > Best,
> > Wangda/Vinod
> >
> > [1]
> > https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104b
> > c9e19e559aedc5ebedb4ca08c8@%3Cyarn-dev.hadoop.apache.org%3E
> > [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in
> > (3.1.0)
> > AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status = Resolved
> >
> > ORDER
> >
> > BY
> > fixVersion ASC
> >
> >
> >
> >
> >
> >
> > B�CB�
> > �[��X��ܚX�KK[XZ[���Y]�][��X��ܚX�PY
> ���\X�K�ܙ�B��܈Y][ۘ[��[X[��K[XZ[���Y]�Z[Y
> > ���\X�K�ܙ�B�
> >
> >
>

Re: [VOTE] Release Apache Hadoop 3.1.0 (RC0)

[Wangda Tan]

Thanks all for doing testing and reporting issues.

YARN-8085 discovered by Weiwei/Tao is a blocker, we will roll an RC1 today.

- Wangda

On Thu, Mar 29, 2018 at 7:26 AM, Weiwei Yang  wrote:

> Hi Wangda
>
> While testing the build, we found a bug https://issues.apache.org/
> jira/browse/YARN-8085, it might cause NPE during RM fail-over. I think
> that needs to be included in 3.1.0.
>
> We have run some other tests, and they look good so far. This includes
> - Some basic teragen/terasort/DS jobs
> - Various of metrics, UI displays, compatible tests
> - Creating queues, different queue configurations
> - Verified RM fail-over, app-recovery
> - Enabled async scheduling
> - Enabled placement constraints
> - SLS tests
>
> Hope it helps.
> Thanks
>
> --
> Weiwei
>
> On 29 Mar 2018, 12:34 PM +0800, Ajay Kumar ,
> wrote:
>
> +1 (non-binding)
>
> - verified binary checksum
> - built from source and setup 4 node cluster
> - run basic hdfs command
> - run wordcount, pi & TestDFSIO (read/write)
>
> Ajay
>
> On 3/28/18, 5:45 PM, "Jonathan Hung"  wrote:
>
> Hi Wangda, thanks for handling this release.
>
> +1 (non-binding)
>
> - verified binary checksum
> - launched single node RM
> - verified refreshQueues functionality
> - verified capacity scheduler conf mutation disabled in this case
> - verified capacity scheduler conf mutation with leveldb storage
> - verified refreshQueues mutation is disabled in this case
>
>
> Jonathan Hung
>
> On Thu, Mar 22, 2018 at 9:10 AM, Wangda Tan  wrote:
>
> Thanks @Bharat for the quick check, the previously staged repository has
> some issues. I re-deployed jars to nexus.
>
> Here's the new repo (1087)
>
> https://repository.apache.org/content/repositories/orgapachehadoop-1087/
>
> Other artifacts remain same, no additional code changes.
>
> On Wed, Mar 21, 2018 at 11:54 PM, Bharat Viswanadham <
> bviswanad...@hortonworks.com> wrote:
>
> Hi Wangda,
> Maven Artifact repositories is not having all Hadoop jars. (It is missing
> many like hadoop-hdfs, hadoop-client etc.,)
> https://repository.apache.org/content/repositories/orgapachehadoop-1086/
>
>
> Thanks,
> Bharat
>
>
> On 3/21/18, 11:44 PM, "Wangda Tan"  wrote:
>
> Hi folks,
>
> Thanks to the many who helped with this release since Dec 2017 [1].
> We've
> created RC0 for Apache Hadoop 3.1.0. The artifacts are available
>
> here:
>
>
> http://people.apache.org/~wangda/hadoop-3.1.0-RC0/
>
> The RC tag in git is release-3.1.0-RC0.
>
> The maven artifacts are available via repository.apache.org at
> https://repository.apache.org/content/repositories/
> orgapachehadoop-1086/
>
> This vote will run 7 days (5 weekdays), ending on Mar 28 at 11:59 pm
> Pacific.
>
> 3.1.0 contains 727 [2] fixed JIRA issues since 3.0.0. Notable
>
> additions
>
> include the first class GPU/FPGA support on YARN, Native services,
> Support
> rich placement constraints in YARN, S3-related enhancements, allow
>
> HDFS
>
> block replicas to be provided by an external storage system, etc.
>
> We’d like to use this as a starting release for 3.1.x [1], depending
> on how
> it goes, get it stabilized and potentially use a 3.1.1 in several
> weeks as
> the stable release.
>
> We have done testing with a pseudo cluster and distributed shell job.
> My +1
> to start.
>
> Best,
> Wangda/Vinod
>
> [1]
> https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104b
> c9e19e559aedc5ebedb4ca08c8@%3Cyarn-dev.hadoop.apache.org%3E
> [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in
> (3.1.0)
> AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status = Resolved
>
> ORDER
>
> BY
> fixVersion ASC
>
>
>
>
>
>
> B�CB�
> �[��X��ܚX�KK[XZ[���Y]�][��X��ܚX�PY ���\X�K�ܙ�B��܈Y][ۘ[��[X[��K[XZ[���Y]�Z[Y
> ���\X�K�ܙ�B�
>
>

Re: [VOTE] Release Apache Hadoop 3.1.0 (RC0)

[Weiwei Yang]

Hi Wangda

While testing the build, we found a bug 
https://issues.apache.org/jira/browse/YARN-8085, it might cause NPE during RM 
fail-over. I think that needs to be included in 3.1.0.

We have run some other tests, and they look good so far. This includes
- Some basic teragen/terasort/DS jobs
- Various of metrics, UI displays, compatible tests
- Creating queues, different queue configurations
- Verified RM fail-over, app-recovery
- Enabled async scheduling
- Enabled placement constraints
- SLS tests

Hope it helps.
Thanks

--
Weiwei

On 29 Mar 2018, 12:34 PM +0800, Ajay Kumar , wrote:
+1 (non-binding)

- verified binary checksum
- built from source and setup 4 node cluster
- run basic hdfs command
- run wordcount, pi & TestDFSIO (read/write)

Ajay

On 3/28/18, 5:45 PM, "Jonathan Hung"  wrote:

Hi Wangda, thanks for handling this release.

+1 (non-binding)

- verified binary checksum
- launched single node RM
- verified refreshQueues functionality
- verified capacity scheduler conf mutation disabled in this case
- verified capacity scheduler conf mutation with leveldb storage
- verified refreshQueues mutation is disabled in this case


Jonathan Hung

On Thu, Mar 22, 2018 at 9:10 AM, Wangda Tan  wrote:

Thanks @Bharat for the quick check, the previously staged repository has
some issues. I re-deployed jars to nexus.

Here's the new repo (1087)

https://repository.apache.org/content/repositories/orgapachehadoop-1087/

Other artifacts remain same, no additional code changes.

On Wed, Mar 21, 2018 at 11:54 PM, Bharat Viswanadham <
bviswanad...@hortonworks.com> wrote:

Hi Wangda,
Maven Artifact repositories is not having all Hadoop jars. (It is missing
many like hadoop-hdfs, hadoop-client etc.,)
https://repository.apache.org/content/repositories/orgapachehadoop-1086/


Thanks,
Bharat


On 3/21/18, 11:44 PM, "Wangda Tan"  wrote:

Hi folks,

Thanks to the many who helped with this release since Dec 2017 [1].
We've
created RC0 for Apache Hadoop 3.1.0. The artifacts are available
here:

http://people.apache.org/~wangda/hadoop-3.1.0-RC0/

The RC tag in git is release-3.1.0-RC0.

The maven artifacts are available via repository.apache.org at
https://repository.apache.org/content/repositories/
orgapachehadoop-1086/

This vote will run 7 days (5 weekdays), ending on Mar 28 at 11:59 pm
Pacific.

3.1.0 contains 727 [2] fixed JIRA issues since 3.0.0. Notable
additions
include the first class GPU/FPGA support on YARN, Native services,
Support
rich placement constraints in YARN, S3-related enhancements, allow
HDFS
block replicas to be provided by an external storage system, etc.

We’d like to use this as a starting release for 3.1.x [1], depending
on how
it goes, get it stabilized and potentially use a 3.1.1 in several
weeks as
the stable release.

We have done testing with a pseudo cluster and distributed shell job.
My +1
to start.

Best,
Wangda/Vinod

[1]
https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104b
c9e19e559aedc5ebedb4ca08c8@%3Cyarn-dev.hadoop.apache.org%3E
[2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in
(3.1.0)
AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status = Resolved
ORDER
BY
fixVersion ASC






B�CB��[��X��ܚX�KK[XZ[���Y]�][��X��ܚX�PY���\X�K�ܙ�B��܈Y][ۘ[��[X[��K[XZ[���Y]�Z[Y���\X�K�ܙ�B�

[jira] [Created] (HADOOP-15354) hadoop-allyun module to mark hadoop-common as provided

[Steve Loughran (JIRA)]

Steve Loughran created HADOOP-15354:
---

 Summary: hadoop-allyun module to mark hadoop-common as provided
 Key: HADOOP-15354
 URL: https://issues.apache.org/jira/browse/HADOOP-15354
 Project: Hadoop Common
  Issue Type: Improvement
  Components: build, fs/oss
Reporter: Steve Loughran
Assignee: Steve Loughran


Although the azure/aws/openstack and adl modules now declare hadopo-common as 
"provided" the hadoop-allyun module doesn't, so it gets into the set of 
dependencies passed on through hadoop-cloud-storage. It should be switched to 
provided too.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86

[Apache Jenkins Server]

For more details, see 
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/735/

[Mar 28, 2018 2:33:18 AM] (sunilg) YARN-8075. DShell does not fail when we ask 
more GPUs than available
[Mar 28, 2018 3:00:08 AM] (yqlin) HDFS-13347. RBF: Cache datanode reports. 
Contributed by Inigo Goiri.
[Mar 28, 2018 9:35:38 AM] (wwei) YARN-7734. Fix UT failure
[Mar 28, 2018 4:05:25 PM] (szegedim) YARN-8077. The vmemLimit parameter in
[Mar 28, 2018 6:07:45 PM] (wangda) YARN-6629. NPE occurred when container 
allocation proposal is applied
[Mar 28, 2018 6:26:50 PM] (subru) Revert "YARN-8010. Add config in 
FederationRMFailoverProxy to not bypass
[Mar 28, 2018 6:33:19 PM] (subru) YARN-8010. Add config in 
FederationRMFailoverProxy to not bypass facade
[Mar 28, 2018 6:41:20 PM] (zezhang) YARN-7623. Fix the CapacityScheduler Queue 
configuration documentation.
[Mar 28, 2018 6:58:59 PM] (cdouglas) HADOOP-15320. Remove customized 
getFileBlockLocations for hadoop-azure
[Mar 28, 2018 7:49:17 PM] (arp) HDFS-13314. NameNode should optionally exit if 
it detects FsImage
[Mar 29, 2018 12:17:37 AM] (eyang) YARN-8069.  Clean up example hostnames for 
RegistryDNS.




-1 overall


The following subsystems voted -1:
findbugs unit xml


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
unit


Specific tests:

FindBugs :

   module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api 
   org.apache.hadoop.yarn.api.records.Resource.getResources() may expose 
internal representation by returning Resource.resources At Resource.java:by 
returning Resource.resources At Resource.java:[line 234] 

Failed junit tests :

   hadoop.conf.TestCommonConfigurationFields 
   hadoop.hdfs.server.namenode.TestReencryptionWithKMS 
   hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes 
   hadoop.hdfs.web.TestWebHdfsTimeouts 
   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure 
   
hadoop.yarn.server.nodemanager.containermanager.scheduler.TestContainerSchedulerQueuing
 
   hadoop.yarn.applications.distributedshell.TestDistributedShell 
  

   cc:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/735/artifact/out/diff-compile-cc-root.txt
  [4.0K]

   javac:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/735/artifact/out/diff-compile-javac-root.txt
  [288K]

   checkstyle:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/735/artifact/out/diff-checkstyle-root.txt
  [17M]

   pylint:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/735/artifact/out/diff-patch-pylint.txt
  [24K]

   shellcheck:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/735/artifact/out/diff-patch-shellcheck.txt
  [20K]

   shelldocs:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/735/artifact/out/diff-patch-shelldocs.txt
  [12K]

   whitespace:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/735/artifact/out/whitespace-eol.txt
  [9.2M]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/735/artifact/out/whitespace-tabs.txt
  [1.1M]

   xml:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/735/artifact/out/xml.txt
  [4.0K]

   findbugs:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/735/artifact/out/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api-warnings.html
  [8.0K]

   javadoc:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/735/artifact/out/diff-javadoc-javadoc-root.txt
  [760K]

   unit:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/735/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt
  [160K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/735/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
  [300K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/735/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
  [48K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/735/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-applications_hadoop-yarn-applications-distributedshell.txt
  [12K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/735/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt
  [84K]

Powered by Apache Yetus 0.8.0-SNAPSHOT   http://yetus.apache.org

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Reopened] (HADOOP-14651) Update okhttp version to 2.7.5

[Steve Loughran (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-14651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Steve Loughran reopened HADOOP-14651:
-

reopening to cherry pick back


> Update okhttp version to 2.7.5
> --
>
> Key: HADOOP-14651
> URL: https://issues.apache.org/jira/browse/HADOOP-14651
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/adl
>Affects Versions: 3.0.0-beta1
>Reporter: Ray Chiang
>Assignee: Ray Chiang
>Priority: Major
> Fix For: 3.1.0
>
> Attachments: HADOOP-14651-branch-3.0.004.patch, 
> HADOOP-14651.001.patch, HADOOP-14651.002.patch, HADOOP-14651.003.patch, 
> HADOOP-14651.004.patch
>
>
> The current artifact is:
> com.squareup.okhttp:okhttp:2.4.0
> That version could either be bumped to 2.7.5 (the latest of that line), or 
> use the latest artifact:
> com.squareup.okhttp3:okhttp:3.8.1



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-15353) Bump default yetus version in the yetus-wrapper

[Elek, Marton (JIRA)]

Elek, Marton created HADOOP-15353:
-

 Summary: Bump default yetus version in the yetus-wrapper
 Key: HADOOP-15353
 URL: https://issues.apache.org/jira/browse/HADOOP-15353
 Project: Hadoop Common
  Issue Type: Improvement
Affects Versions: 3.1.0
Reporter: Elek, Marton
Assignee: Elek, Marton
 Attachments: HADOOP-15353.001.patch

The current precommit hook uses yetus 0.8.0-SNAPSHOT. The default version in 
the yetus-wrapper script is 0.4.0. It could be adjusted HADOOP_YETUS_VERSION 
but I suggest to set the default version to 0.7.0 to get results similar to the 
jenkins results locally without adjustments.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-15352) FIx default local maven repository path in create-release script

[Elek, Marton (JIRA)]

Elek, Marton created HADOOP-15352:
-

 Summary: FIx default local maven repository path in create-release 
script 
 Key: HADOOP-15352
 URL: https://issues.apache.org/jira/browse/HADOOP-15352
 Project: Hadoop Common
  Issue Type: Bug
  Components: scripts
Affects Versions: 3.1.0
Reporter: Elek, Marton
Assignee: Elek, Marton


I am testing the create-release script locally. In case the MVNCACHE is not set 
the local ~/.m2 is used. Which is not good as the packages are downloaded to 
~/.m2/org/.../... instead of ~/.m2/repository/org/.../.../...





--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org