[VOTE] Release Apache Hadoop 2.8.2 (RC1)

2017-10-19 Thread Junping Du 
Hi folks,
 I've created our new release candidate (RC1) for Apache Hadoop 2.8.2.

 Apache Hadoop 2.8.2 is the first stable release of Hadoop 2.8 line and 
will be the latest stable/production release for Apache Hadoop - it includes 
315 new fixed issues since 2.8.1 and 69 fixes are marked as blocker/critical 
issues.

  More information about the 2.8.2 release plan can be found here: 
https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.8+Release

  New RC is available at: 
http://home.apache.org/~junping_du/hadoop-2.8.2-RC1

  The RC tag in git is: release-2.8.2-RC1, and the latest commit id is: 
66c47f2a01ad9637879e95f80c41f798373828fb

  The maven artifacts are available via 
repository.apache.org at: 
https://repository.apache.org/content/repositories/orgapachehadoop-1064

  Please try the release and vote; the vote will run for the usual 5 days, 
ending on 10/24/2017 6pm PST time.

Thanks,

Junping

Re: [Update] Apache Hadoop 2.8.2 Release Status

2017-10-19 Thread Junping Du 
A quick update: the last patch (YARN-7230) for docker container support in 2.8 
just get committed yesterday. Now there is no left blocker/critical issues for 
2.8.2 and I checked all landed commits are matching with JIRA's fix version. 
With kicking off a new RC build, I will publish RC bits for vote once the build 
process get finished. In the mean time, please hold on any commits to 
branch-2.8.2 unless it really belongs to a blocker and please ping me ahead. 

Thanks all for your patience!

Thanks,

Junping

From: Junping Du 
Sent: Friday, September 22, 2017 5:57 PM
To: common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-dev@hadoop.apache.org; yarn-...@hadoop.apache.org
Cc: Shane Kumpf; Miklos Szegedi; Varun Vasudev
Subject: [Update] Apache Hadoop 2.8.2 Release Status

Hi folks,
 I would like to give you a quick update on 2.8.2 release status:

- First release candidate (RC0) is published over the last weekend, but several 
docker container blockers (bugs, documents, etc.)
 are reported so we decided to cancel the RC0 for vote.

- New coming release blockers (for docker container support) are YARN-7034 
(just committed), YARN-6623, YARN-6930 and YARN-7230.
Shane, Miklos and Varun are actively working on this. Appreciate the effort 
here!

- I will kick off new release candidate (RC1) once these blockers are resolved.

To all committers, branch-2.8.2 is still open for blocker/critical issues 
landing, but for major/minor/trivial issues, please commit to branch-2.8 and 
marked the fixed version as 2.8.3.

Thanks all for heads up. Have a good weekend!


Thanks,

Junping


From: Junping Du 
Sent: Tuesday, September 5, 2017 2:57 PM
To: larry mccay; Steve Loughran
Cc: common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-dev@hadoop.apache.org; yarn-...@hadoop.apache.org
Subject: Re: Apache Hadoop 2.8.2 Release Plan

I assume the quiet over the holiday means we agreed to move forward without 
taking HADOOP-14439 into 2.8.2.
There is a new release building (docker based) issue could be related to 
HADOOP-14474 where we removed oracle java 7 installer due to recent download 
address/contract change by Oracle. The build refuse to work - report as 
JAVA_HOME issue, but hard coded my local java home in create-release or 
Dockerfile doesn't help so we may need to add java 7 installation back (no 
matter Oracle JDK 7 or openJDK 7).
Filed HADOOP-14842 with more details to track as blocker for 2.8.2.

Thanks,

Junping

From: Junping Du 
Sent: Friday, September 1, 2017 12:37 PM
To: larry mccay; Steve Loughran
Cc: common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-dev@hadoop.apache.org; yarn-...@hadoop.apache.org
Subject: Re: Apache Hadoop 2.8.2 Release Plan

This issue (HADOOP-14439) is out of my radar given it is marked as Minor 
priority. If my understanding is correct, here is a trade-off between security 
and backward compatibility. IMO, priority of security is generally higher than 
backward compatibility especially 2.8.0 is still non-production release.
I think we should skip this for 2.8.2 in case it doesn't break compatibility 
from 2.7.x. Thoughts?

Thanks,

Junping

From: larry mccay 
Sent: Friday, September 1, 2017 10:55 AM
To: Steve Loughran
Cc: common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-dev@hadoop.apache.org; yarn-...@hadoop.apache.org
Subject: Re: Apache Hadoop 2.8.2 Release Plan

If we do "fix" this in 2.8.2 we should seriously consider not doing so in
3.0.
This is a very poor practice.

I can see an argument for backward compatibility in 2.8.x line though.

On Fri, Sep 1, 2017 at 1:41 PM, Steve Loughran 
wrote:

> One thing we need to consider is
>
> HADOOP-14439: regression: secret stripping from S3x URIs breaks some
> downstream code
>
> Hadoop 2.8 has a best-effort attempt to strip out secrets from the
> toString() value of an s3a or s3n path where someone has embedded them in
> the URI; this has caused problems in some uses, specifically: when people
> use secrets this way (bad) and assume that you can round trip paths to
> string and back
>
> Should we fix this? If so, Hadoop 2.8.2 is the time to do it
>
>
> > On 1 Sep 2017, at 11:14, Junping Du  wrote:
> >
> > HADOOP-14814 get committed and HADOOP-9747 get push out to 2.8.3, so we
> are clean on blocker/critical issues now.
> > I finish practice of going through JACC report and no more incompatible
> public API changes get found between 2.8.2 and 2.7.4. Also I check commit
> history and fixed 10+ commits which are missing from branch-2.8.2 for some
> reason. So, the current branch-2.8.2 should be good to go for RC stage, and
> I will kick off our first RC tomorrow.
> > In the meanwhile, please don't land any commits to branch-2.8.2 since
> now. If some issues really belong to blocker, please ping me on the JIRA

Re: Do we still have nightly (or even weekly) unit test run for Hadoop projects?

2017-10-19 Thread Wangda Tan 
Gotcha, thanks!

- Wangda

On Thu, Oct 19, 2017 at 7:25 AM, Sean Busbey  wrote:

> Here's the email from last night to common-dev@hadoop:
>
> https://s.apache.org/ARe1
>
> On Wed, Oct 18, 2017 at 10:42 PM, Akira Ajisaka 
> wrote:
>
>> Yes, qbt runs nightly and it sends e-mail to dev lists.
>> https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/
>>
>> Regards,
>> Akira
>>
>>
>> On 2017/10/19 7:54, Wangda Tan wrote:
>>
>>> Hi,
>>>
>>> Do we still have nightly (or even weekly) unit test run for Hadoop
>>> projects? I couldn't find it on Jenkins dashboard and I haven't seen
>>> reports set to dev lists for a while.
>>>
>>> Thanks,
>>> Wangda
>>>
>>>
>> -
>> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
>> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
>>
>>
>
>
> --
> busbey
>

Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86

2017-10-19 Thread Apache Jenkins Server 
For more details, see 
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/562/

[Oct 18, 2017 10:06:30 PM] (junping_du) HADOOP-14958. Fix source-level 
compatibility after HADOOP-11252.




-1 overall


The following subsystems voted -1:
unit


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
unit


Specific tests:

Failed junit tests :

   hadoop.net.TestDNS 
   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting 
   hadoop.hdfs.TestReadStripedFileWithMissingBlocks 
   hadoop.yarn.server.nodemanager.scheduler.TestDistributedScheduler 

Timed out junit tests :

   
org.apache.hadoop.yarn.server.resourcemanager.TestSubmitApplicationWithRMHA 
  

   cc:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/562/artifact/out/diff-compile-cc-root.txt
  [4.0K]

   javac:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/562/artifact/out/diff-compile-javac-root.txt
  [284K]

   checkstyle:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/562/artifact/out/diff-checkstyle-root.txt
  [17M]

   pylint:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/562/artifact/out/diff-patch-pylint.txt
  [20K]

   shellcheck:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/562/artifact/out/diff-patch-shellcheck.txt
  [20K]

   shelldocs:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/562/artifact/out/diff-patch-shelldocs.txt
  [12K]

   whitespace:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/562/artifact/out/whitespace-eol.txt
  [11M]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/562/artifact/out/whitespace-tabs.txt
  [1.3M]

   javadoc:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/562/artifact/out/diff-javadoc-javadoc-root.txt
  [1.9M]

   unit:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/562/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt
  [148K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/562/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
  [380K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/562/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
  [40K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/562/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
  [64K]

Powered by Apache Yetus 0.6.0-SNAPSHOT   http://yetus.apache.org

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: Do we still have nightly (or even weekly) unit test run for Hadoop projects?

2017-10-19 Thread Sean Busbey 
Here's the email from last night to common-dev@hadoop:

https://s.apache.org/ARe1

On Wed, Oct 18, 2017 at 10:42 PM, Akira Ajisaka  wrote:

> Yes, qbt runs nightly and it sends e-mail to dev lists.
> https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/
>
> Regards,
> Akira
>
>
> On 2017/10/19 7:54, Wangda Tan wrote:
>
>> Hi,
>>
>> Do we still have nightly (or even weekly) unit test run for Hadoop
>> projects? I couldn't find it on Jenkins dashboard and I haven't seen
>> reports set to dev lists for a while.
>>
>> Thanks,
>> Wangda
>>
>>
> -
> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
>
>


-- 
busbey

[jira] [Created] (MAPREDUCE-6987) JHS Log Scanner and Cleaner blocked

2017-10-19 Thread Bibin A Chundatt (JIRA) 
Bibin A Chundatt created MAPREDUCE-6987:
---

 Summary: JHS Log Scanner and Cleaner blocked
 Key: MAPREDUCE-6987
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6987
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Reporter: Bibin A Chundatt
Priority: Critical






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

[jira] [Created] (MAPREDUCE-6986) Fail to set job configuration by -D option in example WordMedian

2017-10-19 Thread Tao Jie (JIRA) 
Tao Jie created MAPREDUCE-6986:
--

 Summary: Fail to set job configuration by -D option in example 
WordMedian 
 Key: MAPREDUCE-6986
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6986
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Reporter: Tao Jie
Assignee: Tao Jie
Priority: Trivial


I tried to submit mr example job wordmedian and specify queue by -D option:
{{bin/hadoop jar ~/hadoop-mapreduce-examples-2.8.2-bc1.4.0.jar wordmedian 
-Dmapreduce.job.queuename=root.user1 /input /output6}}. This option did not 
work in wordmedian while it works in other example jobs such Terasort, 
Wordcount.
The cause is there is unnecessary {{ setConf(new Configuration())}} in 
{{run()}} method, which would override Configuration with properties by -D 
options.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org