Re: [VOTE] Release Apache Hadoop 3.0.0 RC0

[Andrew Wang]

Hi Arpit,

I agree the timing is not great here, but extending it to meaningfully
avoid the holidays would mean extending it an extra week (e.g. to the
29th). We've been coordinating with ASF PR for that Tuesday, so I'd really,
really like to get the RC out before then.

In terms of downstream testing, we've done extensive integration testing
with downstreams via the alphas and betas, and we have continuous
integration running at Cloudera against branch-3.0. Because of this, I have
more confidence in our integration for 3.0.0 than most Hadoop releases.

Is it meaningful to extend to say, the 21st, which provides for a full week
of voting?

Best,
Andrew

On Fri, Nov 17, 2017 at 1:27 PM, Arpit Agarwal 
wrote:

> Hi Andrew,
>
> Thank you for your hard work in getting us to this step. This is our first
> major GA release in many years.
>
> I feel a 5-day vote window ending over the weekend before thanksgiving may
> not provide sufficient time to evaluate this RC especially for downstream
> components.
>
> Would you please consider extending the voting deadline until a few days
> after the thanksgiving holiday? It would be a courtesy to our broader
> community and I see no harm in giving everyone a few days to evaluate it
> more thoroughly.
>
> On a lighter note, your deadline is also 4 minutes short of the required 5
> days. :)
>
> Regards,
> Arpit
>
>
>
> On 11/14/17, 1:34 PM, "Andrew Wang"  wrote:
>
> Hi folks,
>
> Thanks as always to the many, many contributors who helped with this
> release. I've created RC0 for Apache Hadoop 3.0.0. The artifacts are
> available here:
>
> http://people.apache.org/~wang/3.0.0-RC0/
>
> This vote will run 5 days, ending on Nov 19th at 1:30pm Pacific.
>
> 3.0.0 GA contains 291 fixed JIRA issues since 3.0.0-beta1. Notable
> additions include the merge of YARN resource types, API-based
> configuration
> of the CapacityScheduler, and HDFS router-based federation.
>
> I've done my traditional testing with a pseudo cluster and a Pi job.
> My +1
> to start.
>
> Best,
> Andrew
>
>
>

Apache Hadoop qbt Report: branch2+JDK7 on Linux/x86

[Apache Jenkins Server]

For more details, see 
https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/44/

[Nov 16, 2017 5:12:52 PM] (sunilg) YARN-7469. Capacity Scheduler Intra-queue 
preemption: User can starve if
[Nov 16, 2017 7:12:01 PM] (rkanter) HADOOP-14982. Clients using 
FailoverOnNetworkExceptionRetry can go into
[Nov 17, 2017 6:35:35 AM] (vvasudev) YARN-7430. Enable user re-mapping for 
Docker containers by default.


[Error replacing 'FILE' - Workspace is not accessible]

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.0.0 RC0

[Eric Payne]

Thanks for the all of the work it took to finally get htere.
+1 (binding)
Built from source and stood up a pseudo cluster with 4 NMs
Tested the following:
o Cross queue preemption
o Restarting the RM preserves work
o User limits are honored during In-queue preemption 

o Priorities are honored during In-queue preemption
o Users with different weights are assigned resources proportional to their 
weights.
o User weights are refreshable, and in-queue preemption works to honor the 
post-refresh weights 

Thanks,
-Eric Payne

  From: Andrew Wang 
 To: "common-...@hadoop.apache.org" ; 
"yarn-...@hadoop.apache.org" ; 
"mapreduce-dev@hadoop.apache.org" ; 
"hdfs-...@hadoop.apache.org"  
 Sent: Tuesday, November 14, 2017 3:34 PM
 Subject: [VOTE] Release Apache Hadoop 3.0.0 RC0
   
Hi folks,

Thanks as always to the many, many contributors who helped with this
release. I've created RC0 for Apache Hadoop 3.0.0. The artifacts are
available here:

http://people.apache.org/~wang/3.0.0-RC0/

This vote will run 5 days, ending on Nov 19th at 1:30pm Pacific.

3.0.0 GA contains 291 fixed JIRA issues since 3.0.0-beta1. Notable
additions include the merge of YARN resource types, API-based configuration
of the CapacityScheduler, and HDFS router-based federation.

I've done my traditional testing with a pseudo cluster and a Pi job. My +1
to start.

Best,
Andrew


   

Re: [VOTE] Release Apache Hadoop 2.9.0 (RC3)

[Subru Krishnan]

Wrapping up the vote with my +1.

Deployed RC3 on a federated YARN cluster with 6 sub-clusters:
- ran multiple sample jobs
- enabled opportunistic containers and submitted more samples
- configured HDFS federation and reran jobs.


With 13 binding +1s and 7 non-binding +1s and no +/-1s, pleased to announce
the vote is passed successfully.

Thanks to the many of you who contributed to the release and made this
possible and to everyone in this thread who took the time/effort to
validate and vote!

We’ll push the release bits and send out an announcement for 2.9.0 soon.

Cheers,
Subru

On Fri, Nov 17, 2017 at 12:41 PM Eric Payne 
wrote:

> Thanks Arun and Subru for the hard work on this release.
>
> +1 (binding)
>
> Built from source and stood up a pseudo cluster with 4 NMs
>
> Tested the following:
>
> o User limits are honored during In-queue preemption
>
> o Priorities are honored during In-queue preemption
>
> o Can kill applications from the command line
>
> o Users with different weights are assigned resources proportional to
> their weights.
>
> Thanks,
> -Eric Payne
>
>
> --
> *From:* Arun Suresh 
> *To:* yarn-...@hadoop.apache.org; mapreduce-dev@hadoop.apache.org; Hadoop
> Common ; Hdfs-dev <
> hdfs-...@hadoop.apache.org>
> *Cc:* Subramaniam Krishnan 
> *Sent:* Monday, November 13, 2017 6:10 PM
>
> *Subject:* [VOTE] Release Apache Hadoop 2.9.0 (RC3)
>
> Hi Folks,
>
> Apache Hadoop 2.9.0 is the first release of Hadoop 2.9 line and will be the
> starting release for Apache Hadoop 2.9.x line - it includes 30 New Features
> with 500+ subtasks, 407 Improvements, 790 Bug fixes new fixed issues since
> 2.8.2.
>
> More information about the 2.9.0 release plan can be found here:
> *
> https://cwiki.apache.org/confluence/display/HADOOP/Roadmap#Roadmap-Version2.9
> <
> https://cwiki.apache.org/confluence/display/HADOOP/Roadmap#Roadmap-Version2.9
> >*
>
> New RC is available at: *
> https://home.apache.org/~asuresh/hadoop-2.9.0-RC3/
> *
>
> The RC tag in git is: release-2.9.0-RC3, and the latest commit id is:
> 756ebc8394e473ac25feac05fa493f6d612e6c50.
>
> The maven artifacts are available via repository.apache.org at:
> <
> https://www.google.com/url?q=https%3A%2F%2Frepository.apache.org%2Fcontent%2Frepositories%2Forgapachehadoop-1066=D=1=AFQjCNFcern4uingMV_sEreko_zeLlgdlg
> >*https://repository.apache.org/content/repositories/orgapachehadoop-1068/
>  >*
>
> We are carrying over the votes from the previous RC given that the delta is
> the license fix.
>
> Given the above - we are also going to stick with the original deadline for
> the vote : ending on Friday 17th November 2017 2pm PT time.
>
> Thanks,
> -Arun/Subru
>
>
>

Re: [VOTE] Release Apache Hadoop 3.0.0 RC0

[Arpit Agarwal]

Hi Andrew,

Thank you for your hard work in getting us to this step. This is our first 
major GA release in many years.

I feel a 5-day vote window ending over the weekend before thanksgiving may not 
provide sufficient time to evaluate this RC especially for downstream 
components.

Would you please consider extending the voting deadline until a few days after 
the thanksgiving holiday? It would be a courtesy to our broader community and I 
see no harm in giving everyone a few days to evaluate it more thoroughly.

On a lighter note, your deadline is also 4 minutes short of the required 5 
days. :)

Regards,
Arpit



On 11/14/17, 1:34 PM, "Andrew Wang"  wrote:

Hi folks,

Thanks as always to the many, many contributors who helped with this
release. I've created RC0 for Apache Hadoop 3.0.0. The artifacts are
available here:

http://people.apache.org/~wang/3.0.0-RC0/

This vote will run 5 days, ending on Nov 19th at 1:30pm Pacific.

3.0.0 GA contains 291 fixed JIRA issues since 3.0.0-beta1. Notable
additions include the merge of YARN resource types, API-based configuration
of the CapacityScheduler, and HDFS router-based federation.

I've done my traditional testing with a pseudo cluster and a Pi job. My +1
to start.

Best,
Andrew

Re: [VOTE] Release Apache Hadoop 2.9.0 (RC3)

[Eric Payne]

Thanks Arun and Subru for the hard work on this release.

+1 (binding)

Built from source and stood up a pseudo cluster with 4 NMs
Tested the following:
o User limits are honored during In-queue preemption 

o Priorities are honored during In-queue preemption
o Can kill applications from the command line
o Users with different weights are assigned resources proportional to their 
weights.
Thanks,-Eric Payne


  From: Arun Suresh 
 To: yarn-...@hadoop.apache.org; mapreduce-dev@hadoop.apache.org; Hadoop Common 
; Hdfs-dev  
Cc: Subramaniam Krishnan 
 Sent: Monday, November 13, 2017 6:10 PM
 Subject: [VOTE] Release Apache Hadoop 2.9.0 (RC3)
   
Hi Folks,

Apache Hadoop 2.9.0 is the first release of Hadoop 2.9 line and will be the
starting release for Apache Hadoop 2.9.x line - it includes 30 New Features
with 500+ subtasks, 407 Improvements, 790 Bug fixes new fixed issues since
2.8.2.

More information about the 2.9.0 release plan can be found here:
*https://cwiki.apache.org/confluence/display/HADOOP/Roadmap#Roadmap-Version2.9
*

New RC is available at: *https://home.apache.org/~asuresh/hadoop-2.9.0-RC3/
*

The RC tag in git is: release-2.9.0-RC3, and the latest commit id is:
756ebc8394e473ac25feac05fa493f6d612e6c50.

The maven artifacts are available via repository.apache.org at:
*https://repository.apache.org/content/repositories/orgapachehadoop-1068/
*

We are carrying over the votes from the previous RC given that the delta is
the license fix.

Given the above - we are also going to stick with the original deadline for
the vote : ending on Friday 17th November 2017 2pm PT time.

Thanks,
-Arun/Subru


   

Re: [VOTE] Release Apache Hadoop 3.0.0 RC0

[Andrew Wang]

Thanks for the spot, normally create-release spits those out. I uploaded
asc and mds for the release artifacts.

Best,
Andrew

On Thu, Nov 16, 2017 at 11:33 PM, Akira Ajisaka  wrote:

> Hi Andrew,
>
> Signatures are missing. Would you upload them?
>
> Thanks,
> Akira
>
>
> On 2017/11/15 6:34, Andrew Wang wrote:
>
>> Hi folks,
>>
>> Thanks as always to the many, many contributors who helped with this
>> release. I've created RC0 for Apache Hadoop 3.0.0. The artifacts are
>> available here:
>>
>> http://people.apache.org/~wang/3.0.0-RC0/
>>
>> This vote will run 5 days, ending on Nov 19th at 1:30pm Pacific.
>>
>> 3.0.0 GA contains 291 fixed JIRA issues since 3.0.0-beta1. Notable
>> additions include the merge of YARN resource types, API-based
>> configuration
>> of the CapacityScheduler, and HDFS router-based federation.
>>
>> I've done my traditional testing with a pseudo cluster and a Pi job. My +1
>> to start.
>>
>> Best,
>> Andrew
>>
>>

Re: [VOTE] Release Apache Hadoop 2.9.0 (RC3)

[Jason Lowe]

Thanks for putting this release together!

+1 (binding)

- Verified signatures and digests
- Successfully built from source including native
- Deployed to single-node cluster and ran some test jobs

Jason


On Mon, Nov 13, 2017 at 6:10 PM, Arun Suresh  wrote:

> Hi Folks,
>
> Apache Hadoop 2.9.0 is the first release of Hadoop 2.9 line and will be the
> starting release for Apache Hadoop 2.9.x line - it includes 30 New Features
> with 500+ subtasks, 407 Improvements, 790 Bug fixes new fixed issues since
> 2.8.2.
>
> More information about the 2.9.0 release plan can be found here:
> *https://cwiki.apache.org/confluence/display/HADOOP/
> Roadmap#Roadmap-Version2.9
>  Roadmap#Roadmap-Version2.9>*
>
> New RC is available at: *https://home.apache.org/~
> asuresh/hadoop-2.9.0-RC3/
> *
>
> The RC tag in git is: release-2.9.0-RC3, and the latest commit id is:
> 756ebc8394e473ac25feac05fa493f6d612e6c50.
>
> The maven artifacts are available via repository.apache.org at:
>  apache.org%2Fcontent%2Frepositories%2Forgapachehadoop-1066=D&
> sntz=1=AFQjCNFcern4uingMV_sEreko_zeLlgdlg>*https://
> repository.apache.org/content/repositories/orgapachehadoop-1068/
>  >*
>
> We are carrying over the votes from the previous RC given that the delta is
> the license fix.
>
> Given the above - we are also going to stick with the original deadline for
> the vote : ending on Friday 17th November 2017 2pm PT time.
>
> Thanks,
> -Arun/Subru
>

Re: [DISCUSS] Apache Hadoop 2.7.5 Release Plan

[Wei-Chiu Chuang]

Hi Konstantin,
Thanks for initiating the release effort.

I am marking HDFS-12641  as
a blocker for Hadoop 2.7.5 because during our internal testing for CDH we
found a regression in HDFS-11445 that was fixed by HDFS-11755 (technically
not a real regression since HDFS-11755 was committed before HDFS-11445).
The regression results in bogus corrupt block reports. It is not clear to
me if the same behavior is in Apache Hadoop, but since the later
(HDFS-11755) is currently Hadoop 2.8.x and above, I would want to be more
cautious about it.

On Thu, Nov 16, 2017 at 5:20 PM, Konstantin Shvachko 
wrote:

> Hi developers,
>
> We have accumulated about 30 commits on branch-2.7. Those are mostly
> valuable bug fixes, minor optimizations and test corrections. I would like
> to propose to make a quick maintenance release 2.7.5.
>
> If there are no objections I'll start preparations.
>
> Thanks,
> --Konstantin
>



-- 
A very happy Clouderan

Re: [VOTE] Release Apache Hadoop 2.9.0 (RC3)

[Arun Suresh]

+1 (binding)

* Built from source
* Set up a 4 node cluster with RM HA enabled
* Performed some basic HDFS commands
* Ran a bunch of pi & sleep jobs - with and without opportunistic containers
* Performed some basic RM failover testing.

Cheers
-Arun

On Thu, Nov 16, 2017 at 11:12 PM, Akira Ajisaka  wrote:

> +1
>
> * Downloaded source tarball and verified checksum and signature
> * Compiled the source with OpenJDK 1.8.0_151 and CentOS 7.4
> * Deployed a pseudo cluster and ran some simple MR jobs
>
> I noticed ISA-L build options are documented in BUILDING.txt
> but the options do not exist in 2.x releases.
> Filed HADOOP-15045 to fix this issue.
> I think this issue doesn't block the release.
>
> Thanks and regards,
> Akira
>
> On 2017/11/14 9:10, Arun Suresh wrote:
>
>> Hi Folks,
>>
>> Apache Hadoop 2.9.0 is the first release of Hadoop 2.9 line and will be
>> the
>> starting release for Apache Hadoop 2.9.x line - it includes 30 New
>> Features
>> with 500+ subtasks, 407 Improvements, 790 Bug fixes new fixed issues since
>> 2.8.2.
>>
>> More information about the 2.9.0 release plan can be found here:
>> *https://cwiki.apache.org/confluence/display/HADOOP/Roadmap#
>> Roadmap-Version2.9
>> > Roadmap-Version2.9>*
>>
>> New RC is available at: *https://home.apache.org/~asur
>> esh/hadoop-2.9.0-RC3/
>> *
>>
>> The RC tag in git is: release-2.9.0-RC3, and the latest commit id is:
>> 756ebc8394e473ac25feac05fa493f6d612e6c50.
>>
>> The maven artifacts are available via repository.apache.org at:
>> > e.org%2Fcontent%2Frepositories%2Forgapachehadoop-1066=D
>> ntz=1=AFQjCNFcern4uingMV_sEreko_zeLlgdlg>*https://reposi
>> tory.apache.org/content/repositories/orgapachehadoop-1068/
>> > >*
>>
>> We are carrying over the votes from the previous RC given that the delta
>> is
>> the license fix.
>>
>> Given the above - we are also going to stick with the original deadline
>> for
>> the vote : ending on Friday 17th November 2017 2pm PT time.
>>
>> Thanks,
>> -Arun/Subru
>>
>>

Apache Hadoop qbt Report: branch2+JDK7 on Linux/x86

[Apache Jenkins Server]

For more details, see 
https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/43/

[Nov 15, 2017 6:33:48 PM] (jlowe) YARN-7361. Improve the docker container 
runtime documentation.
[Nov 16, 2017 12:45:06 AM] (xiao) HADOOP-15023. ValueQueue should also validate 
(int) (lowWatermark *




-1 overall


The following subsystems voted -1:
asflicense unit xml


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
unit


Specific tests:

Unreaped Processes :

   hadoop-hdfs:17 
   bkjournal:7 
   hadoop-mapreduce-client-jobclient:9 
   hadoop-archive-logs:1 
   hadoop-archives:1 
   hadoop-distcp:5 
   hadoop-extras:1 
   hadoop-sls:1 
   hadoop-yarn-applications-distributedshell:1 
   hadoop-yarn-client:5 
   hadoop-yarn-server-tests:2 
   hadoop-yarn-server-timelineservice:1 

Failed junit tests :

   hadoop.net.TestDNS 
   hadoop.hdfs.TestAppendSnapshotTruncate 
   hadoop.hdfs.TestDFSAddressConfig 
   hadoop.mapred.TestSpecialCharactersInOutputPath 
   hadoop.tools.TestIntegration 
   hadoop.tools.TestDistCpViewFs 
   hadoop.tools.TestDistCpSystem 
   hadoop.resourceestimator.solver.impl.TestLpSolver 
   hadoop.resourceestimator.service.TestResourceEstimatorService 
   hadoop.yarn.sls.nodemanager.TestNMSimulator 
   
hadoop.yarn.applications.distributedshell.TestDistributedShellWithNodeLabels 

Timed out junit tests :

   org.apache.hadoop.hdfs.TestReservedRawPaths 
   org.apache.hadoop.hdfs.TestAclsEndToEnd 
   org.apache.hadoop.hdfs.TestFileCreation 
   org.apache.hadoop.hdfs.TestDatanodeDeath 
   org.apache.hadoop.hdfs.TestDFSClientRetries 
   org.apache.hadoop.hdfs.TestFileAppend2 
   org.apache.hadoop.hdfs.TestFileCorruption 
   org.apache.hadoop.hdfs.TestFileCreationDelete 
   org.apache.hadoop.hdfs.TestSeekBug 
   org.apache.hadoop.hdfs.TestRestartDFS 
   org.apache.hadoop.hdfs.TestDFSClientSocketSize 
   org.apache.hadoop.hdfs.shortcircuit.TestShortCircuitLocalRead 
   org.apache.hadoop.hdfs.TestDFSRollback 
   org.apache.hadoop.hdfs.TestDFSClientExcludedNodes 
   org.apache.hadoop.hdfs.TestAbandonBlock 
   org.apache.hadoop.contrib.bkjournal.TestBootstrapStandbyWithBKJM 
   org.apache.hadoop.contrib.bkjournal.TestBookKeeperJournalManager 
   org.apache.hadoop.contrib.bkjournal.TestBookKeeperHACheckpoints 
   org.apache.hadoop.contrib.bkjournal.TestBookKeeperAsHASharedDir 
   org.apache.hadoop.contrib.bkjournal.TestBookKeeperEditLogStreams 
   org.apache.hadoop.contrib.bkjournal.TestBookKeeperSpeculativeRead 
   org.apache.hadoop.contrib.bkjournal.TestCurrentInprogress 
   org.apache.hadoop.mapred.TestClusterMapReduceTestCase 
   org.apache.hadoop.mapred.TestMRIntermediateDataEncryption 
   org.apache.hadoop.mapred.TestJobSysDirWithDFS 
   org.apache.hadoop.mapred.TestMRTimelineEventHandling 
   org.apache.hadoop.mapred.join.TestDatamerge 
   org.apache.hadoop.mapred.TestMiniMRWithDFSWithDistinctUsers 
   org.apache.hadoop.mapred.TestMiniMRClientCluster 
   org.apache.hadoop.mapred.TestReduceFetchFromPartialMem 
   org.apache.hadoop.mapred.TestMROpportunisticMaps 
   org.apache.hadoop.tools.TestHadoopArchiveLogsRunner 
   org.apache.hadoop.tools.TestHadoopArchives 
   org.apache.hadoop.tools.TestDistCpWithAcls 
   org.apache.hadoop.tools.TestDistCpSync 
   org.apache.hadoop.tools.TestDistCpWithXAttrs 
   org.apache.hadoop.tools.TestDistCpSyncReverseFromTarget 
   org.apache.hadoop.tools.TestDistCpSyncReverseFromSource 
   org.apache.hadoop.tools.TestCopyFiles 
   org.apache.hadoop.yarn.sls.TestSLSRunner 
   
org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell 
   org.apache.hadoop.yarn.client.api.impl.TestAMRMProxy 
   org.apache.hadoop.yarn.client.cli.TestYarnCLI 
   org.apache.hadoop.yarn.client.TestApplicationClientProtocolOnHA 
   org.apache.hadoop.yarn.client.api.impl.TestAMRMClient 
   org.apache.hadoop.yarn.client.api.impl.TestYarnClient 
   org.apache.hadoop.yarn.server.TestMiniYarnCluster 
   org.apache.hadoop.yarn.server.TestContainerManagerSecurity 
   
org.apache.hadoop.yarn.server.timelineservice.reader.TestTimelineReaderWebServices
 
  

   cc:

   
https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/43/artifact/out/diff-compile-cc-root.txt
  [4.0K]

   javac:

   
https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/43/artifact/out/diff-compile-javac-root.txt
  [324K]

   checkstyle:

   
https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/43/artifact/out/diff-checkstyle-root.txt
  [16M]

   pylint: