[jira] [Created] (HDFS-12423) Ozone: TopN container choosing policy should ignore unnecessary containers

2017-09-11 Thread Yiqun Lin (JIRA) 
Yiqun Lin created HDFS-12423:


 Summary: Ozone: TopN container choosing policy should ignore 
unnecessary containers
 Key: HDFS-12423
 URL: https://issues.apache.org/jira/browse/HDFS-12423
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: ozone
Affects Versions: HDFS-7240
Reporter: Yiqun Lin
Assignee: Yiqun Lin


TopN container choosing policy should ignore unnecessary containers. Currently 
TopN policy selects specified count of containers but not check the number of 
pending deletion blocks. So there is a chance there will be some unnecessary 
containers being chosen. That is say we will choose some containers that don't 
include any pending deletion blocks. The related output:
{noformat}
17/09/11 02:58:30 INFO impl.TopNOrderedContainerDeletionChoosingPolicy: Select 
container c7a85e6e-3528-45d1-8063-cd7fec114545 for block deletion, pending 
deletion blocks num: 1.
17/09/11 02:58:30 INFO impl.TopNOrderedContainerDeletionChoosingPolicy: Select 
container 1d163265-8d47-4ed3-845f-f7d3eb569b83 for block deletion, pending 
deletion blocks num: 0.
17/09/11 02:58:30 INFO impl.TopNOrderedContainerDeletionChoosingPolicy: Select 
container 21017018-be32-44e6-9f62-e7fc9f6a6021 for block deletion, pending 
deletion blocks num: 0.
17/09/11 02:58:30 INFO impl.TopNOrderedContainerDeletionChoosingPolicy: Select 
container 4cccadc8-ef5e-466d-bd9e-5b9705f8748c for block deletion, pending 
deletion blocks num: 0.
17/09/11 02:58:30 INFO impl.TopNOrderedContainerDeletionChoosingPolicy: Select 
container 55b11be9-f16f-4620-a310-07c9be3bbfee for block deletion, pending 
deletion blocks num: 0.
{noformat}
We should ignore these containers which pending deletion blocks num is 0, this 
can reduce some unnecessary iterations.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Resolved] (HDFS-11772) Ozone: KSM: set creationTime for volume/bucket/key

2017-09-11 Thread Anu Engineer (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HDFS-11772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anu Engineer resolved HDFS-11772.
-
Resolution: Duplicate

> Ozone: KSM: set creationTime for volume/bucket/key
> --
>
> Key: HDFS-11772
> URL: https://issues.apache.org/jira/browse/HDFS-11772
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Anu Engineer
>Assignee: Mukul Kumar Singh
>  Labels: ozoneMerge
>
> Returns the volume information from KSM.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 2.8.2 (RC0)

2017-09-11 Thread Junping Du 
Miklos just ping me offline that a security fix should get landed to resolve a 
docker runtime issue. I will retrieval RC0 for security fixes landing.
In the mean while, if people here really think a document here is necessary 
(although not enough verification to work as an alpha feature) and can work out 
a patch soon, I am open to accept it. 

Thanks,

Junping

From: Junping Du 
Sent: Monday, September 11, 2017 5:32 PM
To: Daniel Templeton; Chris Douglas
Cc: Miklos Szegedi; Mingliang Liu; Hadoop Common; Hdfs-dev; 
mapreduce-...@hadoop.apache.org; yarn-...@hadoop.apache.org; junping_du
Subject: Re: [VOTE] Release Apache Hadoop 2.8.2 (RC0)

Thanks Daniel for volunteering for documentation effort.
I suspect the problem we are facing here is not a simply documentation truncate 
effort, but indeed a incomplete feature problem. The umbrella JIRA YARN-3611 
has 31 issues are marked as resolved but only 9 patches get backport to 
branch-2.8 no matter intentioned or not. So far, I haven't heard anyone claimed 
that they are deploying/testing 2.8 release without cherry pick additional 
patches for enabling new docker executor runtime.
IMO, it is better to treat this feature in 2.8 as incomplete feature instead of 
an alpha/experimental feature, especially it is out of our previous 2.8 scope. 
If so, instead of documenting something misleading, we should keep new 
settings/configurations private (as it is now) to get rid of getting enabled by 
users occasionally.
Thoughts?


Thanks,

Junping

From: Daniel Templeton 
Sent: Monday, September 11, 2017 4:45 PM
To: Chris Douglas; Junping Du
Cc: Miklos Szegedi; Mingliang Liu; Hadoop Common; Hdfs-dev; 
mapreduce-...@hadoop.apache.org; yarn-...@hadoop.apache.org; junping_du
Subject: Re: [VOTE] Release Apache Hadoop 2.8.2 (RC0)

YARN-6622 is now committed to 2.9.  We could backport YARN-5258 and
YARN-6622 for 2.8, but it'll take some editing.  We'll have to check to
see what features are unsupported in 2.8 and remove those from the
docs.  Not a huge effort overall, though.  Probably a hour's work.  I
may have time to try do it later this week.  Anyone else want to volunteer?

Daniel

On 9/11/17 3:01 PM, Chris Douglas wrote:
> On Mon, Sep 11, 2017 at 2:52 PM, Junping Du  wrote:
>> I don't think this -1 is reasonable, because:
>> - If you look at YARN-6622 closely, it targets to fix a problematic 
>> documentation work on YARN-5258 which get checked into 2.9 and 3.0 branch 
>> only. It means it targets to fix a problem that 2.8.2 never exists.
> ...we're not going to document security implications- which include
> escalations to root- because we don't have _any_ documentation? Why
> don't we backport the documentation?
>
>> - New docker container support (replace of old DockerContainerExectutor) is 
>> still an alpha feature now which doesn't highlight in 2.8 major 
>> features/improvement (http://hadoop.apache.org/docs/r2.8.0/index.html). So 
>> adding documentation here is also not a blocker.
> YARN-6622 is *documenting* the fact that this is an alpha feature and
> that it shouldn't be enabled in secure environments. How are users
> supposed to make this determination without it?
>
>> Vote still continue until a real blocker comes.
> Soright. I remain -1. -C
>
>> 
>> From: Chris Douglas 
>> Sent: Monday, September 11, 2017 12:00 PM
>> To: Junping Du
>> Cc: Miklos Szegedi; Mingliang Liu; Hadoop Common; Hdfs-dev; 
>> mapreduce-...@hadoop.apache.org; yarn-...@hadoop.apache.org; junping_du
>> Subject: Re: [VOTE] Release Apache Hadoop 2.8.2 (RC0)
>>
>> -1 (binding)
>>
>> I don't think we should release this without YARN-6622.
>>
>> Since this doesn't happen often: a -1 in this case is NOT a veto.
>> Releases are approved by majority vote of the PMC. -C
>>
>> On Mon, Sep 11, 2017 at 11:45 AM, Junping Du  wrote:
>>> Thanks Mikols for notifying on this. I think docker support is general 
>>> known as alpha feature so document it as experimental is nice to have but 
>>> not a blocker for 2.8.2. I also noticed that our 2.7.x document 
>>> (https://hadoop.apache.org/docs/r2.7.4/hadoop-yarn/hadoop-yarn-site/DockerContainerExecutor.html)
>>>  without mentioning docker support is experimental. We may need to fix that 
>>> as well in following releases.
>>>
>>> I can also add it (mentioning docker container support feature is 
>>> experimental) to release message in public website just like previous 
>>> release we call 2.7.0/2.8.0 as non-production release.
>>>
>>> I think vote should continue until we could find a real blocker.
>>>
>>>
>>> Thanks,
>>>
>>>
>>> Junping
>>>
>>>
>>> 
>>> From: Miklos Szegedi 
>>> Sent: Monday, September 11, 2017 10:07 AM
>>> To: Mingliang Liu
>>> Cc: Hadoop Common;

Re: [VOTE] Release Apache Hadoop 2.8.2 (RC0)

2017-09-11 Thread Junping Du 
Thanks Daniel for volunteering for documentation effort. 
I suspect the problem we are facing here is not a simply documentation truncate 
effort, but indeed a incomplete feature problem. The umbrella JIRA YARN-3611 
has 31 issues are marked as resolved but only 9 patches get backport to 
branch-2.8 no matter intentioned or not. So far, I haven't heard anyone claimed 
that they are deploying/testing 2.8 release without cherry pick additional 
patches for enabling new docker executor runtime.
IMO, it is better to treat this feature in 2.8 as incomplete feature instead of 
an alpha/experimental feature, especially it is out of our previous 2.8 scope. 
If so, instead of documenting something misleading, we should keep new 
settings/configurations private (as it is now) to get rid of getting enabled by 
users occasionally.
Thoughts?


Thanks,

Junping

From: Daniel Templeton 
Sent: Monday, September 11, 2017 4:45 PM
To: Chris Douglas; Junping Du
Cc: Miklos Szegedi; Mingliang Liu; Hadoop Common; Hdfs-dev; 
mapreduce-...@hadoop.apache.org; yarn-...@hadoop.apache.org; junping_du
Subject: Re: [VOTE] Release Apache Hadoop 2.8.2 (RC0)

YARN-6622 is now committed to 2.9.  We could backport YARN-5258 and
YARN-6622 for 2.8, but it'll take some editing.  We'll have to check to
see what features are unsupported in 2.8 and remove those from the
docs.  Not a huge effort overall, though.  Probably a hour's work.  I
may have time to try do it later this week.  Anyone else want to volunteer?

Daniel

On 9/11/17 3:01 PM, Chris Douglas wrote:
> On Mon, Sep 11, 2017 at 2:52 PM, Junping Du  wrote:
>> I don't think this -1 is reasonable, because:
>> - If you look at YARN-6622 closely, it targets to fix a problematic 
>> documentation work on YARN-5258 which get checked into 2.9 and 3.0 branch 
>> only. It means it targets to fix a problem that 2.8.2 never exists.
> ...we're not going to document security implications- which include
> escalations to root- because we don't have _any_ documentation? Why
> don't we backport the documentation?
>
>> - New docker container support (replace of old DockerContainerExectutor) is 
>> still an alpha feature now which doesn't highlight in 2.8 major 
>> features/improvement (http://hadoop.apache.org/docs/r2.8.0/index.html). So 
>> adding documentation here is also not a blocker.
> YARN-6622 is *documenting* the fact that this is an alpha feature and
> that it shouldn't be enabled in secure environments. How are users
> supposed to make this determination without it?
>
>> Vote still continue until a real blocker comes.
> Soright. I remain -1. -C
>
>> 
>> From: Chris Douglas 
>> Sent: Monday, September 11, 2017 12:00 PM
>> To: Junping Du
>> Cc: Miklos Szegedi; Mingliang Liu; Hadoop Common; Hdfs-dev; 
>> mapreduce-...@hadoop.apache.org; yarn-...@hadoop.apache.org; junping_du
>> Subject: Re: [VOTE] Release Apache Hadoop 2.8.2 (RC0)
>>
>> -1 (binding)
>>
>> I don't think we should release this without YARN-6622.
>>
>> Since this doesn't happen often: a -1 in this case is NOT a veto.
>> Releases are approved by majority vote of the PMC. -C
>>
>> On Mon, Sep 11, 2017 at 11:45 AM, Junping Du  wrote:
>>> Thanks Mikols for notifying on this. I think docker support is general 
>>> known as alpha feature so document it as experimental is nice to have but 
>>> not a blocker for 2.8.2. I also noticed that our 2.7.x document 
>>> (https://hadoop.apache.org/docs/r2.7.4/hadoop-yarn/hadoop-yarn-site/DockerContainerExecutor.html)
>>>  without mentioning docker support is experimental. We may need to fix that 
>>> as well in following releases.
>>>
>>> I can also add it (mentioning docker container support feature is 
>>> experimental) to release message in public website just like previous 
>>> release we call 2.7.0/2.8.0 as non-production release.
>>>
>>> I think vote should continue until we could find a real blocker.
>>>
>>>
>>> Thanks,
>>>
>>>
>>> Junping
>>>
>>>
>>> 
>>> From: Miklos Szegedi 
>>> Sent: Monday, September 11, 2017 10:07 AM
>>> To: Mingliang Liu
>>> Cc: Hadoop Common; Hdfs-dev; mapreduce-...@hadoop.apache.org; 
>>> yarn-...@hadoop.apache.org; junping_du; Junping Du
>>> Subject: Re: [VOTE] Release Apache Hadoop 2.8.2 (RC0)
>>>
>>> Hello Junping,
>>>
>>> Thank you for working on this. Should not YARN-6622 be addressed first? 
>>> "Summary: Document Docker work as experimental".
>>>
>>> Thank you,
>>> Miklos
>>>
>>>
>>> On Sun, Sep 10, 2017 at 6:39 PM, Mingliang Liu 
>>> > wrote:
>>> Thanks Junping for doing this!
>>>
>>> +1 (non-binding)
>>>
>>> - Download the hadoop-2.8.2-src.tar.gz file and checked the md5 value
>>> - Build package using maven (skipping tests) with Java 8
>>> - Spin up a test cluster in Docker containers having

Re: Release 2.9.0 update

2017-09-11 Thread Wangda Tan 
Thanks Arun Suresh for creating the page.

+1 to have a page to track all planned releases, I also updated 3.1 release
plans.

Several suggestions:
1) Only put large features / changes which needed to be included by the
release, otherwise we will have too long list of JIRAs.
2) For release updates, detailed informations. Keep them in a separated
page. (Like
https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+3+release+status+updates
)
3) Regarding to order of releases, I suggest to sort them by planned
release date in ascending order.

- Wangda


On Mon, Sep 11, 2017 at 3:10 PM, Arun Suresh  wrote:

> Hello Folks
>
> Thanks to everyone who chimed in in the discussion thread:
> https://www.mail-archive.com/yarn-dev@hadoop.apache.org/msg27104.html
>
> We have updated the roadmap wiki based on the feedback:
> https://cwiki.apache.org/confluence/display/HADOOP/
> Roadmap#Roadmap-Version2.9
>
> Given the explosion of YARN features currently in the roadmap and based on
> offline discussions with Vinod et. al, We have decided to add a production
> sponsor criteria for the features listed. Essentially, what it means is
> that we will favor inclusion of a feature in the 2.9.0 release if it is
> associated with an immediate production deployment requirement, failing
> which we would recommend moving it to the next major release. For eg., as
> decided by the community for HDFS Erasure coding. This is also in
> consideration with advanced stages of 3.0.0 and 3.1.0 planning.
>
> In summary. We would like to invite sponsors for each of the features in
> the roadmap wiki. Kindly do so by filling in the "Production Sponsor"
> column on the table:
> https://cwiki.apache.org/confluence/display/HADOOP/Roadmap#Roadmap-
> Plannedfeatures:.2
>
>
> Thanks
> Arun / Subru
>

Re: [VOTE] Release Apache Hadoop 2.8.2 (RC0)

2017-09-11 Thread Daniel Templeton 
YARN-6622 is now committed to 2.9.  We could backport YARN-5258 and 
YARN-6622 for 2.8, but it'll take some editing.  We'll have to check to 
see what features are unsupported in 2.8 and remove those from the 
docs.  Not a huge effort overall, though.  Probably a hour's work.  I 
may have time to try do it later this week.  Anyone else want to volunteer?


Daniel

On 9/11/17 3:01 PM, Chris Douglas wrote:

On Mon, Sep 11, 2017 at 2:52 PM, Junping Du  wrote:

I don't think this -1 is reasonable, because:
- If you look at YARN-6622 closely, it targets to fix a problematic 
documentation work on YARN-5258 which get checked into 2.9 and 3.0 branch only. 
It means it targets to fix a problem that 2.8.2 never exists.

...we're not going to document security implications- which include
escalations to root- because we don't have _any_ documentation? Why
don't we backport the documentation?


- New docker container support (replace of old DockerContainerExectutor) is 
still an alpha feature now which doesn't highlight in 2.8 major 
features/improvement (http://hadoop.apache.org/docs/r2.8.0/index.html). So 
adding documentation here is also not a blocker.

YARN-6622 is *documenting* the fact that this is an alpha feature and
that it shouldn't be enabled in secure environments. How are users
supposed to make this determination without it?


Vote still continue until a real blocker comes.

Soright. I remain -1. -C



From: Chris Douglas 
Sent: Monday, September 11, 2017 12:00 PM
To: Junping Du
Cc: Miklos Szegedi; Mingliang Liu; Hadoop Common; Hdfs-dev; 
mapreduce-...@hadoop.apache.org; yarn-...@hadoop.apache.org; junping_du
Subject: Re: [VOTE] Release Apache Hadoop 2.8.2 (RC0)

-1 (binding)

I don't think we should release this without YARN-6622.

Since this doesn't happen often: a -1 in this case is NOT a veto.
Releases are approved by majority vote of the PMC. -C

On Mon, Sep 11, 2017 at 11:45 AM, Junping Du  wrote:

Thanks Mikols for notifying on this. I think docker support is general known as 
alpha feature so document it as experimental is nice to have but not a blocker 
for 2.8.2. I also noticed that our 2.7.x document 
(https://hadoop.apache.org/docs/r2.7.4/hadoop-yarn/hadoop-yarn-site/DockerContainerExecutor.html)
 without mentioning docker support is experimental. We may need to fix that as 
well in following releases.

I can also add it (mentioning docker container support feature is experimental) 
to release message in public website just like previous release we call 
2.7.0/2.8.0 as non-production release.

I think vote should continue until we could find a real blocker.


Thanks,


Junping



From: Miklos Szegedi 
Sent: Monday, September 11, 2017 10:07 AM
To: Mingliang Liu
Cc: Hadoop Common; Hdfs-dev; mapreduce-...@hadoop.apache.org; 
yarn-...@hadoop.apache.org; junping_du; Junping Du
Subject: Re: [VOTE] Release Apache Hadoop 2.8.2 (RC0)

Hello Junping,

Thank you for working on this. Should not YARN-6622 be addressed first? "Summary: 
Document Docker work as experimental".

Thank you,
Miklos


On Sun, Sep 10, 2017 at 6:39 PM, Mingliang Liu 
> wrote:
Thanks Junping for doing this!

+1 (non-binding)

- Download the hadoop-2.8.2-src.tar.gz file and checked the md5 value
- Build package using maven (skipping tests) with Java 8
- Spin up a test cluster in Docker containers having 1 master node (NN/RM) and 
3 slave nodes (DN/NM)
- Operate the basic HDFS/YARN operations from command line, both client and 
admin
- Check NN/RM Web UI
- Run distcp to copy files from/to local and HDFS
- Run hadoop mapreduce examples: grep and wordcount
- Check the HDFS service logs

All looked good to me.

Mingliang


On Sep 10, 2017, at 5:00 PM, Junping Du 
> wrote:

Hi folks,
 With fix of HADOOP-14842 get in, I've created our first release candidate 
(RC0) for Apache Hadoop 2.8.2.

 Apache Hadoop 2.8.2 is the first stable release of Hadoop 2.8 line and 
will be the latest stable/production release for Apache Hadoop - it includes 
305 new fixed issues since 2.8.1 and 63 fixes are marked as blocker/critical 
issues.

  More information about the 2.8.2 release plan can be found here: 
https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.8+Release

  New RC is available at: 
http://home.apache.org/~junping_du/hadoop-2.8.2-RC0

  The RC tag in git is: release-2.8.2-RC0, and the latest commit id is: 
e6597fe3000b06847d2bf55f2bab81770f4b2505

  The maven artifacts are available via 
repository.apache.org at: 
https://repository.apache.org/content/repositories/orgapachehadoop-1062

  Please try the release and vote; the vote will run for the usual 5 days, 
ending on 09/15/2017 5pm PST time.

Thanks,

Junping

Release 2.9.0 update

2017-09-11 Thread Arun Suresh 
Hello Folks

Thanks to everyone who chimed in in the discussion thread:
https://www.mail-archive.com/yarn-dev@hadoop.apache.org/msg27104.html

We have updated the roadmap wiki based on the feedback:
https://cwiki.apache.org/confluence/display/HADOOP/Roadmap#Roadmap-Version2.9

Given the explosion of YARN features currently in the roadmap and based on
offline discussions with Vinod et. al, We have decided to add a production
sponsor criteria for the features listed. Essentially, what it means is
that we will favor inclusion of a feature in the 2.9.0 release if it is
associated with an immediate production deployment requirement, failing
which we would recommend moving it to the next major release. For eg., as
decided by the community for HDFS Erasure coding. This is also in
consideration with advanced stages of 3.0.0 and 3.1.0 planning.

In summary. We would like to invite sponsors for each of the features in
the roadmap wiki. Kindly do so by filling in the "Production Sponsor"
column on the table:
https://cwiki.apache.org/confluence/display/HADOOP/Roadmap#Roadmap-Plannedfeatures:.2


Thanks
Arun / Subru

Re: [VOTE] Release Apache Hadoop 2.8.2 (RC0)

2017-09-11 Thread Chris Douglas 
On Mon, Sep 11, 2017 at 2:52 PM, Junping Du  wrote:
> I don't think this -1 is reasonable, because:
> - If you look at YARN-6622 closely, it targets to fix a problematic 
> documentation work on YARN-5258 which get checked into 2.9 and 3.0 branch 
> only. It means it targets to fix a problem that 2.8.2 never exists.

...we're not going to document security implications- which include
escalations to root- because we don't have _any_ documentation? Why
don't we backport the documentation?

> - New docker container support (replace of old DockerContainerExectutor) is 
> still an alpha feature now which doesn't highlight in 2.8 major 
> features/improvement (http://hadoop.apache.org/docs/r2.8.0/index.html). So 
> adding documentation here is also not a blocker.

YARN-6622 is *documenting* the fact that this is an alpha feature and
that it shouldn't be enabled in secure environments. How are users
supposed to make this determination without it?

> Vote still continue until a real blocker comes.

Soright. I remain -1. -C

> 
> From: Chris Douglas 
> Sent: Monday, September 11, 2017 12:00 PM
> To: Junping Du
> Cc: Miklos Szegedi; Mingliang Liu; Hadoop Common; Hdfs-dev; 
> mapreduce-...@hadoop.apache.org; yarn-...@hadoop.apache.org; junping_du
> Subject: Re: [VOTE] Release Apache Hadoop 2.8.2 (RC0)
>
> -1 (binding)
>
> I don't think we should release this without YARN-6622.
>
> Since this doesn't happen often: a -1 in this case is NOT a veto.
> Releases are approved by majority vote of the PMC. -C
>
> On Mon, Sep 11, 2017 at 11:45 AM, Junping Du  wrote:
>> Thanks Mikols for notifying on this. I think docker support is general known 
>> as alpha feature so document it as experimental is nice to have but not a 
>> blocker for 2.8.2. I also noticed that our 2.7.x document 
>> (https://hadoop.apache.org/docs/r2.7.4/hadoop-yarn/hadoop-yarn-site/DockerContainerExecutor.html)
>>  without mentioning docker support is experimental. We may need to fix that 
>> as well in following releases.
>>
>> I can also add it (mentioning docker container support feature is 
>> experimental) to release message in public website just like previous 
>> release we call 2.7.0/2.8.0 as non-production release.
>>
>> I think vote should continue until we could find a real blocker.
>>
>>
>> Thanks,
>>
>>
>> Junping
>>
>>
>> 
>> From: Miklos Szegedi 
>> Sent: Monday, September 11, 2017 10:07 AM
>> To: Mingliang Liu
>> Cc: Hadoop Common; Hdfs-dev; mapreduce-...@hadoop.apache.org; 
>> yarn-...@hadoop.apache.org; junping_du; Junping Du
>> Subject: Re: [VOTE] Release Apache Hadoop 2.8.2 (RC0)
>>
>> Hello Junping,
>>
>> Thank you for working on this. Should not YARN-6622 be addressed first? 
>> "Summary: Document Docker work as experimental".
>>
>> Thank you,
>> Miklos
>>
>>
>> On Sun, Sep 10, 2017 at 6:39 PM, Mingliang Liu 
>> > wrote:
>> Thanks Junping for doing this!
>>
>> +1 (non-binding)
>>
>> - Download the hadoop-2.8.2-src.tar.gz file and checked the md5 value
>> - Build package using maven (skipping tests) with Java 8
>> - Spin up a test cluster in Docker containers having 1 master node (NN/RM) 
>> and 3 slave nodes (DN/NM)
>> - Operate the basic HDFS/YARN operations from command line, both client and 
>> admin
>> - Check NN/RM Web UI
>> - Run distcp to copy files from/to local and HDFS
>> - Run hadoop mapreduce examples: grep and wordcount
>> - Check the HDFS service logs
>>
>> All looked good to me.
>>
>> Mingliang
>>
>>> On Sep 10, 2017, at 5:00 PM, Junping Du 
>>> > wrote:
>>>
>>> Hi folks,
>>> With fix of HADOOP-14842 get in, I've created our first release 
>>> candidate (RC0) for Apache Hadoop 2.8.2.
>>>
>>> Apache Hadoop 2.8.2 is the first stable release of Hadoop 2.8 line and 
>>> will be the latest stable/production release for Apache Hadoop - it 
>>> includes 305 new fixed issues since 2.8.1 and 63 fixes are marked as 
>>> blocker/critical issues.
>>>
>>>  More information about the 2.8.2 release plan can be found here: 
>>> https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.8+Release
>>>
>>>  New RC is available at: 
>>> http://home.apache.org/~junping_du/hadoop-2.8.2-RC0
>>>
>>>  The RC tag in git is: release-2.8.2-RC0, and the latest commit id is: 
>>> e6597fe3000b06847d2bf55f2bab81770f4b2505
>>>
>>>  The maven artifacts are available via 
>>> repository.apache.org at: 
>>> https://repository.apache.org/content/repositories/orgapachehadoop-1062
>>>
>>>  Please try the release and vote; the vote will run for the usual 5 
>>> days, ending on 09/15/2017 5pm PST time.
>>>
>>> Thanks,
>>>
>>> Junping
>>>
>>
>>
>> -
>> To unsubscribe, e-mail: 
>>

Re: [VOTE] Release Apache Hadoop 2.8.2 (RC0)

2017-09-11 Thread Junping Du 
I don't think this -1 is reasonable, because:
- If you look at YARN-6622 closely, it targets to fix a problematic 
documentation work on YARN-5258 which get checked into 2.9 and 3.0 branch only. 
It means it targets to fix a problem that 2.8.2 never exists.
- New docker container support (replace of old DockerContainerExectutor) is 
still an alpha feature now which doesn't highlight in 2.8 major 
features/improvement (http://hadoop.apache.org/docs/r2.8.0/index.html). So 
adding documentation here is also not a blocker.

Vote still continue until a real blocker comes.

Thanks,

Junping


From: Chris Douglas 
Sent: Monday, September 11, 2017 12:00 PM
To: Junping Du
Cc: Miklos Szegedi; Mingliang Liu; Hadoop Common; Hdfs-dev; 
mapreduce-...@hadoop.apache.org; yarn-...@hadoop.apache.org; junping_du
Subject: Re: [VOTE] Release Apache Hadoop 2.8.2 (RC0)

-1 (binding)

I don't think we should release this without YARN-6622.

Since this doesn't happen often: a -1 in this case is NOT a veto.
Releases are approved by majority vote of the PMC. -C

On Mon, Sep 11, 2017 at 11:45 AM, Junping Du  wrote:
> Thanks Mikols for notifying on this. I think docker support is general known 
> as alpha feature so document it as experimental is nice to have but not a 
> blocker for 2.8.2. I also noticed that our 2.7.x document 
> (https://hadoop.apache.org/docs/r2.7.4/hadoop-yarn/hadoop-yarn-site/DockerContainerExecutor.html)
>  without mentioning docker support is experimental. We may need to fix that 
> as well in following releases.
>
> I can also add it (mentioning docker container support feature is 
> experimental) to release message in public website just like previous release 
> we call 2.7.0/2.8.0 as non-production release.
>
> I think vote should continue until we could find a real blocker.
>
>
> Thanks,
>
>
> Junping
>
>
> 
> From: Miklos Szegedi 
> Sent: Monday, September 11, 2017 10:07 AM
> To: Mingliang Liu
> Cc: Hadoop Common; Hdfs-dev; mapreduce-...@hadoop.apache.org; 
> yarn-...@hadoop.apache.org; junping_du; Junping Du
> Subject: Re: [VOTE] Release Apache Hadoop 2.8.2 (RC0)
>
> Hello Junping,
>
> Thank you for working on this. Should not YARN-6622 be addressed first? 
> "Summary: Document Docker work as experimental".
>
> Thank you,
> Miklos
>
>
> On Sun, Sep 10, 2017 at 6:39 PM, Mingliang Liu 
> > wrote:
> Thanks Junping for doing this!
>
> +1 (non-binding)
>
> - Download the hadoop-2.8.2-src.tar.gz file and checked the md5 value
> - Build package using maven (skipping tests) with Java 8
> - Spin up a test cluster in Docker containers having 1 master node (NN/RM) 
> and 3 slave nodes (DN/NM)
> - Operate the basic HDFS/YARN operations from command line, both client and 
> admin
> - Check NN/RM Web UI
> - Run distcp to copy files from/to local and HDFS
> - Run hadoop mapreduce examples: grep and wordcount
> - Check the HDFS service logs
>
> All looked good to me.
>
> Mingliang
>
>> On Sep 10, 2017, at 5:00 PM, Junping Du 
>> > wrote:
>>
>> Hi folks,
>> With fix of HADOOP-14842 get in, I've created our first release 
>> candidate (RC0) for Apache Hadoop 2.8.2.
>>
>> Apache Hadoop 2.8.2 is the first stable release of Hadoop 2.8 line and 
>> will be the latest stable/production release for Apache Hadoop - it includes 
>> 305 new fixed issues since 2.8.1 and 63 fixes are marked as blocker/critical 
>> issues.
>>
>>  More information about the 2.8.2 release plan can be found here: 
>> https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.8+Release
>>
>>  New RC is available at: 
>> http://home.apache.org/~junping_du/hadoop-2.8.2-RC0
>>
>>  The RC tag in git is: release-2.8.2-RC0, and the latest commit id is: 
>> e6597fe3000b06847d2bf55f2bab81770f4b2505
>>
>>  The maven artifacts are available via 
>> repository.apache.org at: 
>> https://repository.apache.org/content/repositories/orgapachehadoop-1062
>>
>>  Please try the release and vote; the vote will run for the usual 5 
>> days, ending on 09/15/2017 5pm PST time.
>>
>> Thanks,
>>
>> Junping
>>
>
>
> -
> To unsubscribe, e-mail: 
> mapreduce-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: 
> mapreduce-dev-h...@hadoop.apache.org
>
>

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org



-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands,

[jira] [Created] (HDFS-12422) Replace DataNode in Pipeline for Last Packet

2017-09-11 Thread Lukas Majercak (JIRA) 
Lukas Majercak created HDFS-12422:
-

 Summary: Replace DataNode in Pipeline for Last Packet
 Key: HDFS-12422
 URL: https://issues.apache.org/jira/browse/HDFS-12422
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: hdfs, hdfs-client
Reporter: Lukas Majercak
Assignee: Lukas Majercak


# Create a file with replicationFactor = 4, minReplicas = 2
# Fail waiting for the last packet, followed by 2 exceptions when
recovering the leftover pipeline
# The leftover pipeline will only have one DN and NN will never
close such block, resulting in failure to write



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Resolved] (HDFS-11982) Add markdown documentation about Ozone

2017-09-11 Thread Anu Engineer (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HDFS-11982?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anu Engineer resolved HDFS-11982.
-
Resolution: Later

> Add markdown documentation about Ozone
> --
>
> Key: HDFS-11982
> URL: https://issues.apache.org/jira/browse/HDFS-11982
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Mukul Kumar Singh
>Assignee: Mukul Kumar Singh
>  Labels: documentation, ozoneMerge, tocheck
> Fix For: HDFS-7240
>
>
> Ozone should be documented in detail using markdown.
> This documentation should include information about setting up various Ozone 
> components, usage and configuration.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Resolved] (HDFS-12239) Ozone: OzoneClient : Remove createContainer handling from client

2017-09-11 Thread Anu Engineer (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HDFS-12239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anu Engineer resolved HDFS-12239.
-
Resolution: Duplicate

> Ozone: OzoneClient : Remove createContainer handling from client
> 
>
> Key: HDFS-12239
> URL: https://issues.apache.org/jira/browse/HDFS-12239
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Anu Engineer
>Assignee: Yuanbo Liu
>  Labels: ozoneMerge
> Fix For: HDFS-7240
>
>
> In HDFS-12178, we have committed some special handling of creating 
> containers. This is not needed in the long run. This JIRA tracks removing 
> that.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Resolved] (HDFS-12252) Ozone: SCM: Container State Machine -1- Track container creation state in SCM

2017-09-11 Thread Anu Engineer (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HDFS-12252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anu Engineer resolved HDFS-12252.
-
Resolution: Duplicate

> Ozone: SCM: Container State Machine -1- Track container creation state in SCM
> -
>
> Key: HDFS-12252
> URL: https://issues.apache.org/jira/browse/HDFS-12252
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>Reporter: Xiaoyu Yao
>Assignee: Xiaoyu Yao
>  Labels: ozoneMerge
>
> Currently, SCM only persist the pipeline info of a container in container.db 
> upon allocation. This provides better scalability which offloads the query of 
> container info to datanodes. 
> However, this is not enough enough information for SCM to provide shared 
> container service as other clients may not know the container creation state 
> on datanodes by the creator of the container. 
> This is the first patch that introduces container states on SCM and ACK 
> message from creator of the container to update/persist the container state 
> on SCM container store. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: [VOTE] Merge yarn-native-services branch into trunk

2017-09-11 Thread Andrew Wang 
Thanks for your consideration Jian, let's track this for GA then.

Best,
Andrew

On Fri, Sep 8, 2017 at 3:02 PM, Jian He  wrote:

> Hi Andrew,
>
> At this point, there are no more release blockers including documentations
> from our side - all work done.
> But I agree it is too close to the release, after talking with other team
> members, we are fine to drop  this from beta,
>
> And we want to target this for GA.
> I’m withdrawing this vote and will start afresh vote later for GA.
> Thanks all who voted this effort !
>
> Thanks,
> Jian
>
>
> > On Sep 7, 2017, at 3:59 PM, Andrew Wang 
> wrote:
> >
> > Hi folks,
> >
> > This vote closes today. I see a -1 from Allen on inclusion in beta1. I
> see
> > there's active fixing going on, but given that we're one week out from
> RC0,
> > I think we should drop this from beta1.
> >
> > Allen, Jian, others, is this reasonable? What release should we retarget
> > this for? I don't have a sense for how much work there is left to do, but
> > as a reminder, we're planning GA for Nov 1st, and 3.1.0 for January.
> >
> > Best,
> > Andrew
> >
> > On Wed, Sep 6, 2017 at 10:19 AM, Jian He  wrote:
> >
> >>>  Please correct me if I’m wrong, but the current summary of the
> >> branch, post these changes, looks like:
> >> Sorry for confusion, I was actively writing the formal documentation for
> >> how to use/how it works etc. and will post soon in a few hours.
> >>
> >>
> >>> On Sep 6, 2017, at 10:15 AM, Allen Wittenauer <
> a...@effectivemachines.com>
> >> wrote:
> >>>
> >>>
>  On Sep 5, 2017, at 6:23 PM, Jian He  wrote:
> 
> >If it doesn’t have all the bells and whistles, then it shouldn’t
> >> be on port 53 by default.
>  Sure, I’ll change the default port to not use 53 and document it.
> >*how* is it getting launched on a privileged port? It sounds like
> >> the expectation is to run “command” as root.   *ALL* of the previous
> >> daemons in Hadoop that needed a privileged port used jsvc.  Why isn’t
> this
> >> one? These questions matter from a security standpoint.
>  Yes, it is running as “root” to be able to use the privileged port.
> The
> >> DNS server is not yet integrated with the hadoop script.
> 
> > Check the output.  It’s pretty obviously borked:
>  Thanks for pointing out. Missed this when rebasing onto trunk.
> >>>
> >>>
> >>>  Please correct me if I’m wrong, but the current summary of the
> >> branch, post these changes, looks like:
> >>>
> >>>  * A bunch of mostly new Java code that may or may not have
> >> javadocs (post-revert YARN-6877, still working out HADOOP-14835)
> >>>  * ~1/3 of the docs are roadmap/TBD
> >>>  * ~1/3 of the docs are for an optional DNS daemon that has
> >> no end user hook to start it
> >>>  * ~1/3 of the docs are for a REST API that comes from some
> >> undefined daemon (apiserver?)
> >>>  * Two new, but undocumented, subcommands to yarn
> >>>  * There are no docs for admins or users on how to actually
> >> start or use this completely new/separate/optional feature
> >>>
> >>>  How are outside people (e.g., non-branch committers) supposed to
> >> test this new feature under these conditions?
> >>>
> >>
> >>
> >> -
> >> To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
> >> For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
> >>
> >>
>
>

[jira] [Created] (HDFS-12421) Balancer to emit standard metrics

2017-09-11 Thread Erik Krogen (JIRA) 
Erik Krogen created HDFS-12421:
--

 Summary: Balancer to emit standard metrics
 Key: HDFS-12421
 URL: https://issues.apache.org/jira/browse/HDFS-12421
 Project: Hadoop HDFS
  Issue Type: Improvement
  Components: balancer & mover
Reporter: Erik Krogen
Assignee: Erik Krogen
Priority: Minor


The Balancer currently prints some statistics about its operation to stdout 
while it is running. This is fine if the balancer is manually run via CLI by an 
operator, but for the more common case of it being a scheduled execution, it is 
cumbersome to have to track down the logs to be able to monitor its progress.

We already have a standard metrics system in place; I propose that we have the 
Balancer emit metrics while it is running so that they can be tracked via 
standard metrics infrastructure. We can start with just the things that the 
balancer already prints to stdout: bytes already moved, bytes left to move, 
bytes currently being moved, and iteration number.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: If primary replica is unresponsive, hsync() hangs

2017-09-11 Thread Yongjun Zhang 
Thanks for finding the issue Wei-Chiu.

I agree hsync should be handling DN failure similarly as write-pipeline
recovery, as you stated. If it's not doing that, it should be fixed.

--Yongjun

On Mon, Sep 11, 2017 at 10:53 AM, Wei-Chiu Chuang 
wrote:

> Hello my dear HDFS dev colleagues,
>
> It appears that when a dfs client writes and hsync(), and if the primary
> replica (that is, the first DataNode in the write pipeline) is unresponsive
> to the hsync() request, the hsync() would wait at
> DataStreamer#waitForAckedSeqno().
>
> In one scenario, we saw this behavior when the primary DataNode has a flaky
> disk drive controller, and DataNode was thus unable to write back ack to
> client because it was unable to write to the disk successfully. The client
> is a Flume agent and it finally bailed out after 180 seconds.
>
> My question is: why doesn't hsync() replace bad DataNodes in the pipeline
> just like the typical write pipeline failure recovery?
>
> I would like to understand if this is intended before I file a jira and
> post a patch.
>
> Thanks,
> Wei-Chiu
> --
> A very happy Hadoop contributor
>

Re: [VOTE] Release Apache Hadoop 2.8.2 (RC0)

2017-09-11 Thread Chris Douglas 
-1 (binding)

I don't think we should release this without YARN-6622.

Since this doesn't happen often: a -1 in this case is NOT a veto.
Releases are approved by majority vote of the PMC. -C

On Mon, Sep 11, 2017 at 11:45 AM, Junping Du  wrote:
> Thanks Mikols for notifying on this. I think docker support is general known 
> as alpha feature so document it as experimental is nice to have but not a 
> blocker for 2.8.2. I also noticed that our 2.7.x document 
> (https://hadoop.apache.org/docs/r2.7.4/hadoop-yarn/hadoop-yarn-site/DockerContainerExecutor.html)
>  without mentioning docker support is experimental. We may need to fix that 
> as well in following releases.
>
> I can also add it (mentioning docker container support feature is 
> experimental) to release message in public website just like previous release 
> we call 2.7.0/2.8.0 as non-production release.
>
> I think vote should continue until we could find a real blocker.
>
>
> Thanks,
>
>
> Junping
>
>
> 
> From: Miklos Szegedi 
> Sent: Monday, September 11, 2017 10:07 AM
> To: Mingliang Liu
> Cc: Hadoop Common; Hdfs-dev; mapreduce-...@hadoop.apache.org; 
> yarn-...@hadoop.apache.org; junping_du; Junping Du
> Subject: Re: [VOTE] Release Apache Hadoop 2.8.2 (RC0)
>
> Hello Junping,
>
> Thank you for working on this. Should not YARN-6622 be addressed first? 
> "Summary: Document Docker work as experimental".
>
> Thank you,
> Miklos
>
>
> On Sun, Sep 10, 2017 at 6:39 PM, Mingliang Liu 
> > wrote:
> Thanks Junping for doing this!
>
> +1 (non-binding)
>
> - Download the hadoop-2.8.2-src.tar.gz file and checked the md5 value
> - Build package using maven (skipping tests) with Java 8
> - Spin up a test cluster in Docker containers having 1 master node (NN/RM) 
> and 3 slave nodes (DN/NM)
> - Operate the basic HDFS/YARN operations from command line, both client and 
> admin
> - Check NN/RM Web UI
> - Run distcp to copy files from/to local and HDFS
> - Run hadoop mapreduce examples: grep and wordcount
> - Check the HDFS service logs
>
> All looked good to me.
>
> Mingliang
>
>> On Sep 10, 2017, at 5:00 PM, Junping Du 
>> > wrote:
>>
>> Hi folks,
>> With fix of HADOOP-14842 get in, I've created our first release 
>> candidate (RC0) for Apache Hadoop 2.8.2.
>>
>> Apache Hadoop 2.8.2 is the first stable release of Hadoop 2.8 line and 
>> will be the latest stable/production release for Apache Hadoop - it includes 
>> 305 new fixed issues since 2.8.1 and 63 fixes are marked as blocker/critical 
>> issues.
>>
>>  More information about the 2.8.2 release plan can be found here: 
>> https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.8+Release
>>
>>  New RC is available at: 
>> http://home.apache.org/~junping_du/hadoop-2.8.2-RC0
>>
>>  The RC tag in git is: release-2.8.2-RC0, and the latest commit id is: 
>> e6597fe3000b06847d2bf55f2bab81770f4b2505
>>
>>  The maven artifacts are available via 
>> repository.apache.org at: 
>> https://repository.apache.org/content/repositories/orgapachehadoop-1062
>>
>>  Please try the release and vote; the vote will run for the usual 5 
>> days, ending on 09/15/2017 5pm PST time.
>>
>> Thanks,
>>
>> Junping
>>
>
>
> -
> To unsubscribe, e-mail: 
> mapreduce-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: 
> mapreduce-dev-h...@hadoop.apache.org
>
>

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 2.8.2 (RC0)

2017-09-11 Thread Junping Du 
Thanks Mikols for notifying on this. I think docker support is general known as 
alpha feature so document it as experimental is nice to have but not a blocker 
for 2.8.2. I also noticed that our 2.7.x document 
(https://hadoop.apache.org/docs/r2.7.4/hadoop-yarn/hadoop-yarn-site/DockerContainerExecutor.html)
 without mentioning docker support is experimental. We may need to fix that as 
well in following releases.

I can also add it (mentioning docker container support feature is experimental) 
to release message in public website just like previous release we call 
2.7.0/2.8.0 as non-production release.

I think vote should continue until we could find a real blocker.


Thanks,


Junping



From: Miklos Szegedi 
Sent: Monday, September 11, 2017 10:07 AM
To: Mingliang Liu
Cc: Hadoop Common; Hdfs-dev; mapreduce-...@hadoop.apache.org; 
yarn-...@hadoop.apache.org; junping_du; Junping Du
Subject: Re: [VOTE] Release Apache Hadoop 2.8.2 (RC0)

Hello Junping,

Thank you for working on this. Should not YARN-6622 be addressed first? 
"Summary: Document Docker work as experimental".

Thank you,
Miklos


On Sun, Sep 10, 2017 at 6:39 PM, Mingliang Liu 
> wrote:
Thanks Junping for doing this!

+1 (non-binding)

- Download the hadoop-2.8.2-src.tar.gz file and checked the md5 value
- Build package using maven (skipping tests) with Java 8
- Spin up a test cluster in Docker containers having 1 master node (NN/RM) and 
3 slave nodes (DN/NM)
- Operate the basic HDFS/YARN operations from command line, both client and 
admin
- Check NN/RM Web UI
- Run distcp to copy files from/to local and HDFS
- Run hadoop mapreduce examples: grep and wordcount
- Check the HDFS service logs

All looked good to me.

Mingliang

> On Sep 10, 2017, at 5:00 PM, Junping Du 
> > wrote:
>
> Hi folks,
> With fix of HADOOP-14842 get in, I've created our first release candidate 
> (RC0) for Apache Hadoop 2.8.2.
>
> Apache Hadoop 2.8.2 is the first stable release of Hadoop 2.8 line and 
> will be the latest stable/production release for Apache Hadoop - it includes 
> 305 new fixed issues since 2.8.1 and 63 fixes are marked as blocker/critical 
> issues.
>
>  More information about the 2.8.2 release plan can be found here: 
> https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.8+Release
>
>  New RC is available at: 
> http://home.apache.org/~junping_du/hadoop-2.8.2-RC0
>
>  The RC tag in git is: release-2.8.2-RC0, and the latest commit id is: 
> e6597fe3000b06847d2bf55f2bab81770f4b2505
>
>  The maven artifacts are available via 
> repository.apache.org at: 
> https://repository.apache.org/content/repositories/orgapachehadoop-1062
>
>  Please try the release and vote; the vote will run for the usual 5 days, 
> ending on 09/15/2017 5pm PST time.
>
> Thanks,
>
> Junping
>


-
To unsubscribe, e-mail: 
mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: 
mapreduce-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-12420) Improve error message on Namenode format

2017-09-11 Thread Ajay Kumar (JIRA) 
Ajay Kumar created HDFS-12420:
-

 Summary: Improve error message on Namenode format
 Key: HDFS-12420
 URL: https://issues.apache.org/jira/browse/HDFS-12420
 Project: Hadoop HDFS
  Issue Type: Improvement
Reporter: Ajay Kumar


While Formatting NameNode user should be informed that this will result in 
complete loss of data. In addition to current message we should display:
"This will result in complete loss of all cluster data."



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-12419) Fix TestDistCpViewFs failures on branch-2

2017-09-11 Thread Xiaoyu Yao (JIRA) 
Xiaoyu Yao created HDFS-12419:
-

 Summary: Fix TestDistCpViewFs failures on branch-2
 Key: HDFS-12419
 URL: https://issues.apache.org/jira/browse/HDFS-12419
 Project: Hadoop HDFS
  Issue Type: Bug
Reporter: Xiaoyu Yao


testUpdateGlobTargetMissingSingleLevel, testGlobTargetMissingMultiLevel, 
testGlobTargetMissingSingleLevel and testUpdateGlobTargetMissingMultiLevel
are failing on branch-2 with mismatched file count like below.


{code}
java.lang.AssertionError: 
Expected :4
Actual   :5
 


at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at org.junit.Assert.assertEquals(Assert.java:542)
at 
org.apache.hadoop.tools.TestDistCpViewFs.checkResult(TestDistCpViewFs.java:428)
at 
org.apache.hadoop.tools.TestDistCpViewFs.testUpdateGlobTargetMissingSingleLevel(TestDistCpViewFs.java:326)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
at 
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
at org.junit.runner.JUnitCore.run(JUnitCore.java:160)
at 
com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
at 
com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:51)
at 
com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:242)
at 
com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70)


{code}






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 2.8.2 (RC0)

2017-09-11 Thread Brahma Reddy Battula 
Thanks Junping Du.

+1 (non-binding)

-Built from source
-Installed HA cluster
-Executed basic shell commands
-Ran sample jobs
-Verified HDFS/YARN UI.



--Brahma Reddy Battula


On Mon, 11 Sep 2017 at 5:30 AM, Junping Du  wrote:

> Hi folks,
>  With fix of HADOOP-14842 get in, I've created our first release
> candidate (RC0) for Apache Hadoop 2.8.2.
>
>  Apache Hadoop 2.8.2 is the first stable release of Hadoop 2.8 line
> and will be the latest stable/production release for Apache Hadoop - it
> includes 305 new fixed issues since 2.8.1 and 63 fixes are marked as
> blocker/critical issues.
>
>   More information about the 2.8.2 release plan can be found here:
> https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.8+Release
>
>   New RC is available at:
> http://home.apache.org/~junping_du/hadoop-2.8.2-RC0
>
>   The RC tag in git is: release-2.8.2-RC0, and the latest commit id
> is: e6597fe3000b06847d2bf55f2bab81770f4b2505
>
>   The maven artifacts are available via repository.apache.org at:
> https://repository.apache.org/content/repositories/orgapachehadoop-1062
>
>   Please try the release and vote; the vote will run for the usual 5
> days, ending on 09/15/2017 5pm PST time.
>
> Thanks,
>
> Junping
>
> --



--Brahma Reddy Battula

If primary replica is unresponsive, hsync() hangs

2017-09-11 Thread Wei-Chiu Chuang 
Hello my dear HDFS dev colleagues,

It appears that when a dfs client writes and hsync(), and if the primary
replica (that is, the first DataNode in the write pipeline) is unresponsive
to the hsync() request, the hsync() would wait at
DataStreamer#waitForAckedSeqno().

In one scenario, we saw this behavior when the primary DataNode has a flaky
disk drive controller, and DataNode was thus unable to write back ack to
client because it was unable to write to the disk successfully. The client
is a Flume agent and it finally bailed out after 180 seconds.

My question is: why doesn't hsync() replace bad DataNodes in the pipeline
just like the typical write pipeline failure recovery?

I would like to understand if this is intended before I file a jira and
post a patch.

Thanks,
Wei-Chiu
-- 
A very happy Hadoop contributor

[jira] [Created] (HDFS-12418) Fix BackupNode UT broken by HDFS-10391

2017-09-11 Thread Arpit Agarwal (JIRA) 
Arpit Agarwal created HDFS-12418:


 Summary: Fix BackupNode UT broken by HDFS-10391
 Key: HDFS-12418
 URL: https://issues.apache.org/jira/browse/HDFS-12418
 Project: Hadoop HDFS
  Issue Type: Bug
Affects Versions: 3.0.0-beta1
Reporter: Arpit Agarwal


TestBackupNode was disabled as part of HDFS-10391.

This Jira is to fix and re-enable it.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-12417) Disable flaky TestDFSStripedOutputStreamWithFailure

2017-09-11 Thread Chris Douglas (JIRA) 
Chris Douglas created HDFS-12417:


 Summary: Disable flaky TestDFSStripedOutputStreamWithFailure
 Key: HDFS-12417
 URL: https://issues.apache.org/jira/browse/HDFS-12417
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: test
Reporter: Chris Douglas


Some subset of TestDFSStripedOutputStreamWithFailure tests almost always fail 
in test-patch runs. Since its failure is no longer seen as a blocker for 
commit, it should be disabled until it is more reliable.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Reopened] (HDFS-12349) Improve log message when it could not alloc enough blocks for EC

2017-09-11 Thread Chris Douglas (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HDFS-12349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chris Douglas reopened HDFS-12349:
--

> Improve log message when it could not alloc enough blocks for EC 
> -
>
> Key: HDFS-12349
> URL: https://issues.apache.org/jira/browse/HDFS-12349
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: erasure-coding
>Affects Versions: 3.0.0-alpha3
>Reporter: Lei (Eddy) Xu
>Assignee: Lei (Eddy) Xu
>Priority: Minor
> Fix For: 3.0.0-beta1
>
> Attachments: HDFS-12349.00.patch, HDFS-12349.01.patch, 
> HDFS-12349.02.patch
>
>
> When an EC output stream could not alloc enough blocks for parity blocks, it 
> sets the warning.
> {code}
> if (blocks[i] == null) {
> LOG.warn("Failed to get block location for parity block, index=" + i);
> {code}
> We should clarify the cause of this warning message.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86

2017-09-11 Thread Apache Jenkins Server 
For more details, see 
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/520/

[Sep 11, 2017 4:58:10 AM] (sunilg) YARN-7163. RMContext need not to be injected 
to webapp and other Always
[Sep 11, 2017 6:17:59 AM] (yufei) YARN-6799. Remove the duplicated code in 
CGroupsHandlerImp.java.




-1 overall


The following subsystems voted -1:
findbugs unit


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
unit


Specific tests:

FindBugs :

   module:hadoop-hdfs-project/hadoop-hdfs 
   Format-string method String.format(String, Object[]) called with format 
string "File %s could only be written to %d of the %d %s. There are %d 
datanode(s) running and %s node(s) are excluded in this operation." wants 6 
arguments but is given 7 in 
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4NewBlock(String,
 int, Node, Set, long, List, byte, BlockType, ErasureCodingPolicy, EnumSet) At 
BlockManager.java:with format string "File %s could only be written to %d of 
the %d %s. There are %d datanode(s) running and %s node(s) are excluded in this 
operation." wants 6 arguments but is given 7 in 
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4NewBlock(String,
 int, Node, Set, long, List, byte, BlockType, ErasureCodingPolicy, EnumSet) At 
BlockManager.java:[line 2076] 

FindBugs :

   
module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
 
   Hard coded reference to an absolute pathname in 
org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.DockerLinuxContainerRuntime.launchContainer(ContainerRuntimeContext)
 At DockerLinuxContainerRuntime.java:absolute pathname in 
org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.DockerLinuxContainerRuntime.launchContainer(ContainerRuntimeContext)
 At DockerLinuxContainerRuntime.java:[line 490] 

Failed junit tests :

   hadoop.ha.TestZKFailoverController 
   hadoop.hdfs.TestReconstructStripedFile 
   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure150 
   hadoop.hdfs.TestLeaseRecoveryStriped 
   hadoop.hdfs.TestClientProtocolForPipelineRecovery 
   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure050 
   hadoop.hdfs.server.blockmanagement.TestReplicationPolicyWithNodeGroup 
   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure 
   hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean 
   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure180 
   hadoop.hdfs.TestFileAppendRestart 
   hadoop.hdfs.server.namenode.TestNamenodeCapacityReport 
   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure030 
   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure160 
   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080 
   hadoop.hdfs.TestReadStripedFileWithMissingBlocks 
   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure140 
   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure090 
   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure040 
   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting 
   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure010 
   hadoop.hdfs.server.blockmanagement.TestBlockManager 
   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure120 
   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure020 
   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure200 
   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure110 
   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure100 
   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure070 
   hadoop.hdfs.protocol.datatransfer.sasl.TestSaslDataTransfer 
   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure190 
   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure 
   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure000 
   hadoop.fs.http.client.TestHttpFSFWithWebhdfsFileSystem 
   
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation 
   hadoop.yarn.server.resourcemanager.scheduler.TestAbstractYarnScheduler 
   hadoop.yarn.server.router.webapp.TestRouterWebServiceUtil 
   hadoop.mapreduce.v2.hs.webapp.TestHSWebApp 
   hadoop.yarn.sls.TestReservationSystemInvariants 
   hadoop.yarn.sls.TestSLSRunner 

Timed out junit tests :

   org.apache.hadoop.hdfs.TestWriteReadStripedFile 
  

   cc:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/520/artifact/out/diff-compile-cc-root.txt
  [4.0K]

   javac:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/520/artifact/out/diff-compile-javac-root.txt
  [292K]

   checkstyle:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/520/artifact/out/diff-checkstyle-root.txt
  [17M]

   pylint:

[jira] [Created] (HDFS-12416) BlockPlacementPolicyDefault will cause NN shutdown if log level is changed

2017-09-11 Thread Suhan Mao (JIRA) 
Suhan Mao created HDFS-12416:


 Summary: BlockPlacementPolicyDefault will cause NN shutdown if log 
level is changed
 Key: HDFS-12416
 URL: https://issues.apache.org/jira/browse/HDFS-12416
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: block placement
Affects Versions: 3.0.0-alpha3, 2.7.4
Reporter: Suhan Mao


In BlockPlacementPolicyDefault.chooseRandom method.
The code are in below structure:
{code:java}
StringBuilder builder = null;
if (LOG.isDebugEnabled()) {
  builder = debugLoggingBuilder.get();
  builder.setLength(0);
  builder.append("[");
}
while(numOfReplicas > 0){
chooseDataNode(scope, excludedNodes)
if (LOG.isDebugEnabled()) {
builder.append("\nNode ").append(NodeBase.getPath(chosenNode))
.append(" [");
  }
}
{code}

There's a possibility that the loglevel is INFO before entering while loop, but 
the loglevel is changed to DEBUG inside the loop through web UI.
In that case, builder is not initialized in the beginning and 
NullPointerException will throw and this will cause NN exiting.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-12415) Ozone: TestXceiverClientManager occasionally fails

2017-09-11 Thread Weiwei Yang (JIRA) 
Weiwei Yang created HDFS-12415:
--

 Summary: Ozone: TestXceiverClientManager occasionally fails
 Key: HDFS-12415
 URL: https://issues.apache.org/jira/browse/HDFS-12415
 Project: Hadoop HDFS
  Issue Type: Sub-task
Affects Versions: HDFS-7240
Reporter: Weiwei Yang
Assignee: Weiwei Yang
Priority: Minor


TestXceiverClientManager seems to be occasionally failing in some jenkins jobs,

{noformat}
java.lang.NullPointerException
 at 
org.apache.hadoop.ozone.scm.node.SCMNodeManager.getNodeStat(SCMNodeManager.java:828)
 at 
org.apache.hadoop.ozone.scm.container.placement.algorithms.SCMCommonPolicy.hasEnoughSpace(SCMCommonPolicy.java:147)
 at 
org.apache.hadoop.ozone.scm.container.placement.algorithms.SCMCommonPolicy.lambda$chooseDatanodes$0(SCMCommonPolicy.java:125)
{noformat}

see more from [this 
report|https://builds.apache.org/job/PreCommit-HDFS-Build/21065/testReport/]



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org