Re: [VOTE] Release Apache Hadoop 3.3.2 - RC2

[Eric Payne]

+1 (binding)

- Built from source

- Brought up a non-secure virtual cluster w/ NN, 1 DN, RM, AHS, JHS, and 3 NMs

- Validated inter- and intra-queue preemption

- Validated exclusive node labels

Thanks a lot Chao for your diligence and hard work on this release.

Eric















On Wednesday, January 19, 2022, 11:50:34 AM CST, Chao Sun  
wrote: 





Hi all,

I've put together Hadoop 3.3.2 RC2 below:

The RC is available at: http://people.apache.org/~sunchao/hadoop-3.3.2-RC2/
The RC tag is at:
https://github.com/apache/hadoop/releases/tag/release-3.3.2-RC2
The Maven artifacts are staged at:
https://repository.apache.org/content/repositories/orgapachehadoop-1332

You can find my public key at:
https://downloads.apache.org/hadoop/common/KEYS

I've done the following tests and they look good:
- Ran all the unit tests
- Started a single node HDFS cluster and tested a few simple commands
- Ran all the tests in Spark using the RC2 artifacts

Please evaluate the RC and vote, thanks!

Best,
Chao

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

[jira] [Reopened] (MAPREDUCE-7203) TestRuntimeEstimators fails intermittent

[Eric Payne (Jira)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-7203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eric Payne reopened MAPREDUCE-7203:
---

Sorry! I was in the wrong window!

> TestRuntimeEstimators fails intermittent
> 
>
> Key: MAPREDUCE-7203
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7203
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: test
>Affects Versions: 3.3.0
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Minor
> Fix For: 3.4.0, 2.10.2, 3.2.3, 3.3.2
>
>
> TestRuntimeEstimators fails intermittent.
> {code}
> [ERROR] 
> testExponentialEstimator(org.apache.hadoop.mapreduce.v2.app.TestRuntimeEstimators)
>   Time elapsed: 9.637 s  <<< FAILURE!
> java.lang.AssertionError: We got the wrong number of successful speculations. 
> expected:<3> but was:<1>
>   at org.junit.Assert.fail(Assert.java:88)
>   at org.junit.Assert.failNotEquals(Assert.java:834)
>   at org.junit.Assert.assertEquals(Assert.java:645)
>   at 
> org.apache.hadoop.mapreduce.v2.app.TestRuntimeEstimators.coreTestEstimator(TestRuntimeEstimators.java:243)
>   at 
> org.apache.hadoop.mapreduce.v2.app.TestRuntimeEstimators.testExponentialEstimator(TestRuntimeEstimators.java:257)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
>   at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>   at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
>   at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
>   at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
>   at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
>   at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
>   at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
>   at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
>   at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
>   at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
>   at 
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
>   at 
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
>   at 
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
>   at 
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
>   at 
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
>   at 
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
>   at 
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
>   at 
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

[jira] [Resolved] (MAPREDUCE-7203) TestRuntimeEstimators fails intermittent

[Eric Payne (Jira)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-7203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eric Payne resolved MAPREDUCE-7203.
---
Fix Version/s: 3.3.2
   3.2.3
   2.10.2
   3.4.0
   Resolution: Fixed

> TestRuntimeEstimators fails intermittent
> 
>
> Key: MAPREDUCE-7203
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7203
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: test
>Affects Versions: 3.3.0
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Minor
> Fix For: 3.4.0, 2.10.2, 3.2.3, 3.3.2
>
>
> TestRuntimeEstimators fails intermittent.
> {code}
> [ERROR] 
> testExponentialEstimator(org.apache.hadoop.mapreduce.v2.app.TestRuntimeEstimators)
>   Time elapsed: 9.637 s  <<< FAILURE!
> java.lang.AssertionError: We got the wrong number of successful speculations. 
> expected:<3> but was:<1>
>   at org.junit.Assert.fail(Assert.java:88)
>   at org.junit.Assert.failNotEquals(Assert.java:834)
>   at org.junit.Assert.assertEquals(Assert.java:645)
>   at 
> org.apache.hadoop.mapreduce.v2.app.TestRuntimeEstimators.coreTestEstimator(TestRuntimeEstimators.java:243)
>   at 
> org.apache.hadoop.mapreduce.v2.app.TestRuntimeEstimators.testExponentialEstimator(TestRuntimeEstimators.java:257)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
>   at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>   at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
>   at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
>   at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
>   at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
>   at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
>   at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
>   at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
>   at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
>   at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
>   at 
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
>   at 
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
>   at 
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
>   at 
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
>   at 
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
>   at 
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
>   at 
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
>   at 
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.2.2 - RC4

[Eric Payne]

+1 (binding)

- Built from source
- Brought up cluster
- Tested streaming and sleep jobs

-Eric Payne


On Wednesday, December 9, 2020, 11:01:38 AM CST, Xiaoqiao He 
 wrote: 

Hi folks,

The release candidate (RC4) for Hadoop-3.2.2 is available now.
There are 10 commits[1] differences between RC4 and RC3[2].

The RC4 is available at:
http://people.apache.org/~hexiaoqiao/hadoop-3.2.2-RC4
The RC4 tag in github is here:
https://github.com/apache/hadoop/tree/release-3.2.2-RC4
The maven artifacts are staged at:
https://repository.apache.org/content/repositories/orgapachehadoop-1296

You can find my public key at:
https://dist.apache.org/repos/dist/release/hadoop/common/KEYS or
https://people.apache.org/keys/committer/hexiaoqiao.asc directly.

Please try the release and vote.

I have done a simple testing with my pseudo cluster.
* Verify gpg sign and md5sum.
* Setup pseudo cluster with HDFS and YARN.
* Run simple FsShell - mkdir/put/get/rename/mv.
* Submit example mr job and check the result - Pi/wordcount.
* Check the web UI of NameNode/DataNode/Resourcemanager/NodeManager.
My +1 to start.

Thanks,
He Xiaoqiao

[1]
https://github.com/apache/hadoop/compare/release-3.2.2-RC3...release-3.2.2-RC4
[2]
https://lists.apache.org/thread.html/rfb74c3a5d4f223c5804d8ee622829263740cd8701c8f3fc8b6f970af%40%3Chdfs-dev.hadoop.apache.org%3E
[3]
https://issues.apache.org/jira/secure/Dashboard.jspa?selectPageId=12335948

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 2.10.1 (RC0)

[Eric Payne]

Masatake,

Thank you for the good work on creating this release!

+1

I downloaded and built the source. I ran a one-node cluster with 6 NMs.
I manually ran apps in the Capacity Scheduler to test labels and capacity 
assignments.

-Eric


On Monday, September 14, 2020, 12:59:17 PM CDT, Masatake Iwasaki 
 wrote: 

Hi folks,

This is the first release candidate for the second release of Apache Hadoop 
2.10.
It contains 218 fixes/improvements since 2.10.0 [1].

The RC0 artifacts are at:
http://home.apache.org/~iwasakims/hadoop-2.10.1-RC0/

RC tag is release-2.10.1-RC0:
https://github.com/apache/hadoop/tree/release-2.10.1-RC0

The maven artifacts are hosted here:
https://repository.apache.org/content/repositories/orgapachehadoop-1279/

My public key is available here:
https://dist.apache.org/repos/dist/release/hadoop/common/KEYS

The vote will run for 5 days, until Saturday, September 19 at 10:00 am PDT.

[1] 
https://issues.apache.org/jira/issues/?jql=project%20in%20(HDFS%2C%20YARN%2C%20HADOOP%2C%20MAPREDUCE)%20AND%20resolution%20%3D%20Fixed%20AND%20fixVersion%20%3D%202.10.1

Thanks,
Masatake Iwasaki

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org


-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: [ANNOUNCE] New Apache Hadoop Committer - Tao Yang

[Eric Payne]

 Congratulations Tao! Well deserved!

On Monday, July 15, 2019, 4:54:10 AM CDT, Weiwei Yang  
wrote:  
 
 Hi Dear Apache Hadoop Community

It's my pleasure to announce that Tao Yang has been elected as an Apache
Hadoop committer, this is to recognize his contributions to Apache Hadoop
YARN project.

Congratulations and welcome on board!

Weiwei
(On behalf of the Apache Hadoop PMC)
  

[ANNOUNCE] Eric Badger is now a committer!

[Eric Payne]

It is my pleasure to announce that Eric Badger has accepted an invitation to 
become a Hadoop Core committer.

Congratulations, Eric! This is well-deserved!

-Eric Payne

Re: [VOTE] Release Apache Hadoop 3.2.0 - RC1

[Eric Payne]

+1 (binding)

- RM refresh updates values as expected
- Streaming jobs complete successfully
- Moving apps between queues succeeds
- Inter-queue preemption works as expected
- Successfully ran selected yarn unit tests.

===
Eric Payne
===


On Tuesday, January 8, 2019, 5:42:46 AM CST, Sunil G  wrote: 





Hi folks,


Thanks to all of you who helped in this release [1] and for helping to vote
for RC0. I have created second release candidate (RC1) for Apache Hadoop
3.2.0.


Artifacts for this RC are available here:

http://home.apache.org/~sunilg/hadoop-3.2.0-RC1/


RC tag in git is release-3.2.0-RC1.



The maven artifacts are available via repository.apache.org at
https://repository.apache.org/content/repositories/orgapachehadoop-1178/


This vote will run 7 days (5 weekdays), ending on 14th Jan at 11:59 pm PST.



3.2.0 contains 1092 [2] fixed JIRA issues since 3.1.0. Below feature
additions

are the highlights of this release.

1. Node Attributes Support in YARN

2. Hadoop Submarine project for running Deep Learning workloads on YARN

3. Support service upgrade via YARN Service API and CLI

4. HDFS Storage Policy Satisfier

5. Support Windows Azure Storage - Blob file system in Hadoop

6. Phase 3 improvements for S3Guard and Phase 5 improvements S3a

7. Improvements in Router-based HDFS federation



Thanks to Wangda, Vinod, Marton for helping me in preparing the release.

I have done few testing with my pseudo cluster. My +1 to start.



Regards,

Sunil



[1]

https://lists.apache.org/thread.html/68c1745dcb65602aecce6f7e6b7f0af3d974b1bf0048e7823e58b06f@%3Cyarn-dev.hadoop.apache.org%3E

[2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.2.0)
AND fixVersion not in (3.1.0, 3.0.0, 3.0.0-beta1) AND status = Resolved
ORDER BY fixVersion ASC

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.2.0 - RC0

[Eric Payne]

The problem is not with preemption. The yarn-site.xml that I use for my 
pseudo-cluster includes a second xml:
xi:include href=".../yarn-scheduler.xml"

The property for yarn.resourcemanager.scheduler.monitor.enable = true is in 
this yarn-scheduler.xml.

This value IS READ when then RM starts.

However, when the refreshQueues command is run, this value IS NOT READ.

So, it looks like xml include files are not read on refresh. This will affect 
any property. I just happened to notice it on the preemption properties.

I would like input from all of you to determine if this is a blocker for 
release. I'm on the fence.

Thanks,
-Eric






On Wednesday, November 28, 2018, 4:58:50 PM CST, Eric Payne 
 wrote: 





Sunil,

So, the basic symptoms are that if preemption is enabled on any queue, the 
preemption is disabled after a 'yarn rm -refreshQueues'. In addition, all of 
the preemption-specific properties are set back to the default values.

This was introduced in branch-3.1, so it is NOT new behavior for release 3.2.0. 
I am still tracking down the cause. I will open a JIRA once I do further 
investigation if there is not one already.

This will be a problem for installations which use preemption and which use the 
refreshQueues feature.

Thanks,
-Eric


On Wednesday, November 28, 2018, 11:47:06 AM CST, Eric Payne 
 wrote: 





Sunil, thanks for all of the hard work on this release.

I have discovered that queue refresh doesn't work in some cases. For example, 
when I change yarn.scheduler.capacity.root.default.disable_preemption, it 
doesn't take effect unless I restart the RM.

I am still investigating, but I thought I should bring this up asap.

Thanks,
-Eric




On Friday, November 23, 2018, 6:07:04 AM CST, Sunil G  
wrote: 





Hi folks,



Thanks to all contributors who helped in this release [1]. I have created

first release candidate (RC0) for Apache Hadoop 3.2.0.


Artifacts for this RC are available here:

http://home.apache.org/~sunilg/hadoop-3.2.0-RC0/



RC tag in git is release-3.2.0-RC0.



The maven artifacts are available via repository.apache.org at

https://repository.apache.org/content/repositories/orgapachehadoop-1174/


This vote will run 7 days (5 weekdays), ending on Nov 30 at 11:59 pm PST.



3.2.0 contains 1079 [2] fixed JIRA issues since 3.1.0. Below feature
additions

are the highlights of this release.

1. Node Attributes Support in YARN

2. Hadoop Submarine project for running Deep Learning workloads on YARN

3. Support service upgrade via YARN Service API and CLI

4. HDFS Storage Policy Satisfier

5. Support Windows Azure Storage - Blob file system in Hadoop

6. Phase 3 improvements for S3Guard and Phase 5 improvements S3a

7. Improvements in Router-based HDFS federation



Thanks to Wangda, Vinod, Marton for helping me in preparing the release.

I have done few testing with my pseudo cluster. My +1 to start.



Regards,

Sunil



[1]

https://lists.apache.org/thread.html/68c1745dcb65602aecce6f7e6b7f0af3d974b1bf0048e7823e58b06f@%3Cyarn-dev.hadoop.apache.org%3E

[2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.2.0)
AND fixVersion not in (3.1.0, 3.0.0, 3.0.0-beta1) AND status = Resolved
ORDER BY fixVersion ASC

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org


-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.2.0 - RC0

[Eric Payne]

Sunil,

So, the basic symptoms are that if preemption is enabled on any queue, the 
preemption is disabled after a 'yarn rm -refreshQueues'. In addition, all of 
the preemption-specific properties are set back to the default values.

This was introduced in branch-3.1, so it is NOT new behavior for release 3.2.0. 
I am still tracking down the cause. I will open a JIRA once I do further 
investigation if there is not one already.

This will be a problem for installations which use preemption and which use the 
refreshQueues feature.

Thanks,
-Eric


On Wednesday, November 28, 2018, 11:47:06 AM CST, Eric Payne 
 wrote: 





Sunil, thanks for all of the hard work on this release.

I have discovered that queue refresh doesn't work in some cases. For example, 
when I change yarn.scheduler.capacity.root.default.disable_preemption, it 
doesn't take effect unless I restart the RM.

I am still investigating, but I thought I should bring this up asap.

Thanks,
-Eric




On Friday, November 23, 2018, 6:07:04 AM CST, Sunil G  
wrote: 





Hi folks,



Thanks to all contributors who helped in this release [1]. I have created

first release candidate (RC0) for Apache Hadoop 3.2.0.


Artifacts for this RC are available here:

http://home.apache.org/~sunilg/hadoop-3.2.0-RC0/



RC tag in git is release-3.2.0-RC0.



The maven artifacts are available via repository.apache.org at

https://repository.apache.org/content/repositories/orgapachehadoop-1174/


This vote will run 7 days (5 weekdays), ending on Nov 30 at 11:59 pm PST.



3.2.0 contains 1079 [2] fixed JIRA issues since 3.1.0. Below feature
additions

are the highlights of this release.

1. Node Attributes Support in YARN

2. Hadoop Submarine project for running Deep Learning workloads on YARN

3. Support service upgrade via YARN Service API and CLI

4. HDFS Storage Policy Satisfier

5. Support Windows Azure Storage - Blob file system in Hadoop

6. Phase 3 improvements for S3Guard and Phase 5 improvements S3a

7. Improvements in Router-based HDFS federation



Thanks to Wangda, Vinod, Marton for helping me in preparing the release.

I have done few testing with my pseudo cluster. My +1 to start.



Regards,

Sunil



[1]

https://lists.apache.org/thread.html/68c1745dcb65602aecce6f7e6b7f0af3d974b1bf0048e7823e58b06f@%3Cyarn-dev.hadoop.apache.org%3E

[2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.2.0)
AND fixVersion not in (3.1.0, 3.0.0, 3.0.0-beta1) AND status = Resolved
ORDER BY fixVersion ASC

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.2.0 - RC0

[Eric Payne]

Sunil, thanks for all of the hard work on this release.

I have discovered that queue refresh doesn't work in some cases. For example, 
when I change yarn.scheduler.capacity.root.default.disable_preemption, it 
doesn't take effect unless I restart the RM.

I am still investigating, but I thought I should bring this up asap.

Thanks,
-Eric




On Friday, November 23, 2018, 6:07:04 AM CST, Sunil G  
wrote: 





Hi folks,



Thanks to all contributors who helped in this release [1]. I have created

first release candidate (RC0) for Apache Hadoop 3.2.0.


Artifacts for this RC are available here:

http://home.apache.org/~sunilg/hadoop-3.2.0-RC0/



RC tag in git is release-3.2.0-RC0.



The maven artifacts are available via repository.apache.org at

https://repository.apache.org/content/repositories/orgapachehadoop-1174/


This vote will run 7 days (5 weekdays), ending on Nov 30 at 11:59 pm PST.



3.2.0 contains 1079 [2] fixed JIRA issues since 3.1.0. Below feature
additions

are the highlights of this release.

1. Node Attributes Support in YARN

2. Hadoop Submarine project for running Deep Learning workloads on YARN

3. Support service upgrade via YARN Service API and CLI

4. HDFS Storage Policy Satisfier

5. Support Windows Azure Storage - Blob file system in Hadoop

6. Phase 3 improvements for S3Guard and Phase 5 improvements S3a

7. Improvements in Router-based HDFS federation



Thanks to Wangda, Vinod, Marton for helping me in preparing the release.

I have done few testing with my pseudo cluster. My +1 to start.



Regards,

Sunil



[1]

https://lists.apache.org/thread.html/68c1745dcb65602aecce6f7e6b7f0af3d974b1bf0048e7823e58b06f@%3Cyarn-dev.hadoop.apache.org%3E

[2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.2.0)
AND fixVersion not in (3.1.0, 3.0.0, 3.0.0-beta1) AND status = Resolved
ORDER BY fixVersion ASC

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 2.9.2 (RC0)

[Eric Payne]

  +1 (binding)
-- Built from source-- Installed on 6-node pseudo cluster-- Tested intra- 
inter-queue preemption, user weights-- Ran streaming jobs, word count, and tara 
gen/sort tests
Thanks Akira for all of the hard work.-Eric Payne



 

On Tuesday, November 13, 2018, 7:02:51 PM CST, Akira Ajisaka 
 wrote:  
 
 Hi folks,

I have put together a release candidate (RC0) for Hadoop 2.9.2. It
includes 204 bug fixes and improvements since 2.9.1. [1]

The RC is available at http://home.apache.org/~aajisaka/hadoop-2.9.2-RC0/
Git signed tag is release-2.9.2-RC0 and the checksum is
826afbeae31ca687bc2f8471dc841b66ed2c6704
The maven artifacts are staged at
https://repository.apache.org/content/repositories/orgapachehadoop-1166/

You can find my public key at:
https://dist.apache.org/repos/dist/release/hadoop/common/KEYS

Please try the release and vote. The vote will run for 5 days.

[1] https://s.apache.org/2.9.2-fixed-jiras

Thanks,
Akira

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

  

Re: [VOTE] Release Apache Hadoop 2.8.5 (RC0)

[Eric Payne]

Thanks a lot Junping!

+1 (binding)

Tested the following:
- Built from source
- Installed on a 7 node, multi-tenant, insecure pseudo cluster, running YARN 
capacity scheduler
- Added a queue via refresh
- Verified various GUI pages
- Streaming jobs
- Cross-queue (Inter) preemption
- In-queue (Intra) preemption
- Teragen / terasort


-Eric




On Monday, September 10, 2018, 7:01:46 AM CDT, 俊平堵  
wrote: 





Hi all,

    I've created the first release candidate (RC0) for Apache
Hadoop 2.8.5. This is our next point release to follow up 2.8.4. It
includes 33 important fixes and improvements.


    The RC artifacts are available at:
http://home.apache.org/~junping_du/hadoop-2.8.5-RC0


    The RC tag in git is: release-2.8.5-RC0



    The maven artifacts are available via repository.apache.org<
http://repository.apache.org> at:

https://repository.apache.org/content/repositories/orgapachehadoop-1140


    Please try the release and vote; the vote will run for the usual 5 working
days, ending on 9/15/2018 PST time.


Thanks,


Junping

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.1.1 - RC0

[Eric Payne]

Thanks Wangda for creating this release.

+1 (binding)
Tested:
- Built from source
- Deployed to 6-node, multi-tennant, unsecured pseudo cluster with hierarchical 
queue structure (CS)
- Refreshed queue (CS) properties
- Intra-queue preemption (CS)
- inter-queue preemption (CS)
- User weights (CS)

Issues:
- Inter-queue preemption seems to be preempting unnecessarily (flapping) when 
the queue balancing feature is enabled. This does not seem to be specific to 
this release.
- The preemption-to-balance-queue-after-satisfied.enabled property seems to 
always be enabled, but again, that is not specific to this release.


Eric


On Thursday, August 2, 2018, 1:44:22 PM CDT, Wangda Tan  
wrote: 





Hi folks,

I've created RC0 for Apache Hadoop 3.1.1. The artifacts are available here:

http://people.apache.org/~wangda/hadoop-3.1.1-RC0/

The RC tag in git is release-3.1.1-RC0:
https://github.com/apache/hadoop/commits/release-3.1.1-RC0

The maven artifacts are available via repository.apache.org at
https://repository.apache.org/content/repositories/orgapachehadoop-1139/

You can find my public key at
http://svn.apache.org/repos/asf/hadoop/common/dist/KEYS

This vote will run 5 days from now.

3.1.1 contains 435 [1] fixed JIRA issues since 3.1.0.

I have done testing with a pseudo cluster and distributed shell job. My +1
to start.

Best,
Wangda Tan

[1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.1)
ORDER BY priority DESC

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.0.3 (RC0)

[Eric Payne]

 Sorry, Yongjun. My +1 is also binding+1 (binding)-Eric Payne

On Friday, June 1, 2018, 12:25:36 PM CDT, Eric Payne 
 wrote:  
 
 

Thanks a lot, Yongjun, for your hard work on this release.

+1
- Built from source
- Installed on 6 node pseudo cluster


Tested the following in the Capacity Scheduler:
- Verified that running apps in labelled queues restricts tasks to the labelled 
nodes.
- Verified that various queue config properties for CS are refreshable
- Verified streaming jobs work as expected
- Verified that user weights work as expected
- Verified that FairOrderingPolicy in a CS queue will evenly assign resources
- Verified running yarn shell application runs as expected







On Friday, June 1, 2018, 12:48:26 AM CDT, Yongjun Zhang  
wrote: 





Greetings all,

I've created the first release candidate (RC0) for Apache Hadoop
3.0.3. This is our next maintenance release to follow up 3.0.2. It includes
about 249
important fixes and improvements, among which there are 8 blockers. See
https://issues.apache.org/jira/issues/?filter=12343997

The RC artifacts are available at:
https://dist.apache.org/repos/dist/dev/hadoop/3.0.3-RC0/

The maven artifacts are available via
https://repository.apache.org/content/repositories/orgapachehadoop-1126

Please try the release and vote; the vote will run for the usual 5 working
days, ending on 06/07/2018 PST time. Would really appreciate your
participation here.

I bumped into quite some issues along the way, many thanks to quite a few
people who helped, especially Sammi Chen, Andrew Wang, Junping Du, Eddy Xu.

Thanks,

--Yongjun
  

Re: [VOTE] Release Apache Hadoop 3.0.3 (RC0)

[Eric Payne]

Thanks a lot, Yongjun, for your hard work on this release.

+1
- Built from source
- Installed on 6 node pseudo cluster


Tested the following in the Capacity Scheduler:
- Verified that running apps in labelled queues restricts tasks to the labelled 
nodes.
- Verified that various queue config properties for CS are refreshable
- Verified streaming jobs work as expected
- Verified that user weights work as expected
- Verified that FairOrderingPolicy in a CS queue will evenly assign resources
- Verified running yarn shell application runs as expected







On Friday, June 1, 2018, 12:48:26 AM CDT, Yongjun Zhang  
wrote: 





Greetings all,

I've created the first release candidate (RC0) for Apache Hadoop
3.0.3. This is our next maintenance release to follow up 3.0.2. It includes
about 249
important fixes and improvements, among which there are 8 blockers. See
https://issues.apache.org/jira/issues/?filter=12343997

The RC artifacts are available at:
https://dist.apache.org/repos/dist/dev/hadoop/3.0.3-RC0/

The maven artifacts are available via
https://repository.apache.org/content/repositories/orgapachehadoop-1126

Please try the release and vote; the vote will run for the usual 5 working
days, ending on 06/07/2018 PST time. Would really appreciate your
participation here.

I bumped into quite some issues along the way, many thanks to quite a few
people who helped, especially Sammi Chen, Andrew Wang, Junping Du, Eddy Xu.

Thanks,

--Yongjun

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: Apache Hadoop 3.0.3 Release plan

[Eric Payne]

>  We plan to cut branch-3.0.3 by the coming Wednesday (May 9th) and vote for 
>RC on May 30th
I much prefer to wait to cut the branch until just before the production of the 
release and the vote. With so many branches, we sometimes miss putting critical 
bug fixes in unreleased branches if the branch is cut too early.

My 2 cents...
Thanks,
-Eric Payne





On Monday, May 7, 2018, 12:09:00 AM CDT, Yongjun Zhang <yjzhan...@apache.org> 
wrote: 





Hi All,

>
We have released Apache Hadoop 3.0.2 in April of this year [1]. Since then,
there are quite some commits done to branch-3.0. To further improve the
quality of release, we plan to do 3.0.3 release now. The focus of 3.0.3
will be fixing blockers (3), critical bugs (17) and bug fixes (~130), see
[2].

Usually no new feature should be included for maintenance releases, I
noticed we have https://issues.apache.org/jira/browse/HADOOP-13055 in the
branch classified as new feature. I will talk with the developers to see if
we should include it in 3.0.3.

I also noticed that there are more commits in the branch than can be found
by query [2], also some commits committed to 3.0.3 do not have their jira
target release field filled in accordingly. I will go through them to
update the jira.

>
We plan to cut branch-3.0.3 by the coming Wednesday (May 9th) and vote for
RC on May 30th, targeting for Jun 8th release.

>
Your insights are welcome.

>
[1] https://www.mail-archive.com/general@hadoop.apache.org/msg07790.html

> [2] https://issues.apache.org/jira/issues/?filter=12343874  See Note below
Note: seems I need some admin change so that I can make the filter in [2]
public, I'm working on that. For now, you can use jquery
(project = hadoop OR project = "Hadoop HDFS" OR project = "Hadoop YARN" OR
project = "Hadoop Map/Reduce") AND fixVersion in (3.0.3) ORDER BY priority
DESC

Thanks and best regards,

--Yongjun

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: RE: [VOTE] Release Apache Hadoop 2.9.1 (RC0)

[Eric Payne]

Thanks Sammi for all of the hard work!

+1 (binding)
Tested the following:
- Build from source
- Deploy to pseudo cluster
- Various error cases where the AM fails to start. The failure reasons were 
prop ogated to the GUI and to the client.
- Streaming mapred job
- killing apps from the command line
- intra-queue preemption
- inter-queue preemption- verified preemption properties are refreshable
Thanks,
Eric Payne








On Wednesday, April 25, 2018, 12:12:24 AM CDT, Chen, Sammi 
<sammi.c...@intel.com> wrote: 






Paste the links here,

The artifacts are available here:  
https://dist.apache.org/repos/dist/dev/hadoop/2.9.1-RC0/  

The RC tag in git is release-2.9.1-RC0. Last git commit SHA is 
e30710aea4e6e55e69372929106cf119af06fd0e.

The maven artifacts are available at:
https://repository.apache.org/content/repositories/orgapachehadoop-1115/ 

My public key is available from:
https://dist.apache.org/repos/dist/release/hadoop/common/KEYS 


Bests,
Sammi

-Original Message-
From: Chen, Sammi [mailto:sammi.c...@intel.com] 
Sent: Wednesday, April 25, 2018 12:02 PM
To: junping...@apache.org
Cc: Hadoop Common <common-...@hadoop.apache.org>; Rushabh Shah 
<rusha...@oath.com>; hdfs-dev <hdfs-...@hadoop.apache.org>; 
mapreduce-dev@hadoop.apache.org; yarn-...@hadoop.apache.org
Subject: RE: [VOTE] Release Apache Hadoop 2.9.1 (RC0)


Thanks Jason Lowe for the quick investigation to find out that the test 
failures belong to the test only.

Based on the current facts, I would like to continue calling the VOTE for 2.9.1 
RC0,  and extend the vote deadline to end of this week 4/27.


I will add following note to the final release notes,

HADOOP-15385<https://issues.apache.org/jira/browse/HADOOP-15385>  Test case 
failures in Hadoop-distcp project doesn’t impact the distcp function in 2.9.1


Bests,
Sammi
From: 俊平堵 [mailto:junping...@apache.org]
Sent: Tuesday, April 24, 2018 11:50 PM
To: Chen, Sammi <sammi.c...@intel.com>
Cc: Hadoop Common <common-...@hadoop.apache.org>; Rushabh Shah 
<rusha...@oath.com>; hdfs-dev <hdfs-...@hadoop.apache.org>; 
mapreduce-dev@hadoop.apache.org; yarn-...@hadoop.apache.org
Subject: Re: [VOTE] Release Apache Hadoop 2.9.1 (RC0)

Thanks for reporting the issue, Rushabh! Actually, we found that these test 
failures belong to test issues but not production issue, so not really a solid 
blocker for release. Anyway, I will let RM of 2.9.1 to decide if to cancel RC 
or not for this test issue.

Thanks,

Junping


Chen, Sammi <sammi.c...@intel.com<mailto:sammi.c...@intel.com>>于2018年4月24日 
周二下午7:50写道：
Hi Rushabh,

Thanks for reporting the issue.  I will upload a new RC candidate soon after 
the test failing issue is resolved.


Bests,
Sammi Chen
From: Rushabh Shah [mailto:rusha...@oath.com<mailto:rusha...@oath.com>]
Sent: Friday, April 20, 2018 5:12 AM
To: Chen, Sammi <sammi.c...@intel.com<mailto:sammi.c...@intel.com>>
Cc: Hadoop Common 
<common-...@hadoop.apache.org<mailto:common-...@hadoop.apache.org>>; hdfs-dev 
<hdfs-...@hadoop.apache.org<mailto:hdfs-...@hadoop.apache.org>>; 
mapreduce-dev@hadoop.apache.org<mailto:mapreduce-dev@hadoop.apache.org>; 
yarn-...@hadoop.apache.org<mailto:yarn-...@hadoop.apache.org>
Subject: Re: [VOTE] Release Apache Hadoop 2.9.1 (RC0)

Hi Chen,
I am so sorry to bring this up now but there are 16 tests failing in 
hadoop-distcp project.
I have opened a ticket and cc'ed Junping since he is branch-2.8 committer but I 
missed to ping you.

IMHO we should fix the unit tests before we release but I would leave upto 
other members to give their opinion.


-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org


-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.0.2 (RC1)

[Eric Payne]

 Thanks for all of your hard work to produce this release!
+1 (binding)
I tested the following, and all seems well:
- built from source
- brought up on pseudo-cluster with 4 NMs
- tested yarn shell with and without container-preserving restert
- tested Capacity Scheduler FairOrderingPolicy, with and without size-based 
weight
- tested user weights

Thanks!
Eric Payne



On Monday, April 16, 2018, 7:00:03 PM CDT, Lei Xu <l...@apache.org> wrote:  
 
 Hi, All

I've created release candidate RC-1 for Apache Hadoop 3.0.2, to
address missing source jars in the maven repository in RC-0.

Thanks Ajay Kumar for spotting the error.

Please note: this is an amendment for Apache Hadoop 3.0.1 release to
fix shaded jars in apache maven repository. The codebase of 3.0.2
release is the same as 3.0.1.  New bug fixes will be included in
Apache Hadoop 3.0.3 instead.

The release page is:
https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+3.0+Release

New RC is available at: http://home.apache.org/~lei/hadoop-3.0.2-RC1/

The git tag is release-3.0.2-RC1, and the latest commit is
5c141f7c0f24c12cb8704a6ccc1ff8ec991f41ee, which is the same as RC-0.

The maven artifacts are available at:
https://repository.apache.org/content/repositories/orgapachehadoop-1102/

Please try the release, especially, *verify the maven artifacts*, and vote.

The vote will run 5 days, ending 4/21/2018.

Here is my +1.

Best,

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

  

Re: [VOTE] Release Apache Hadoop 3.0.2 (RC0)

[Eric Payne]

Thanks a lot for working to produce this release.

+1 (binding)
Tested the following:
- built from source and installed on 6-node pseudo-cluster
- tested Capacity Scheduler FairOrderingPolicy and FifoOrderingPolicy to 
determine that capacity was assigned as expected in each case
- tested user weights with FifoOrderingPolicy to ensure that weights were 
assigned to users as expected.

Eric Payne






On Friday, April 6, 2018, 1:17:10 PM CDT, Lei Xu <l...@apache.org> wrote: 





Hi, All

I've created release candidate RC-0 for Apache Hadoop 3.0.2.

Please note: this is an amendment for Apache Hadoop 3.0.1 release to
fix shaded jars in apache maven repository. The codebase of 3.0.2
release is the same as 3.0.1.  New bug fixes will be included in
Apache Hadoop 3.0.3 instead.

The release page is:
https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+3.0+Release

New RC is available at: http://home.apache.org/~lei/hadoop-3.0.2-RC0/

The git tag is release-3.0.2-RC0, and the latest commit is
5c141f7c0f24c12cb8704a6ccc1ff8ec991f41ee

The maven artifacts are available at
https://repository.apache.org/content/repositories/orgapachehadoop-1096/

Please try the release, especially, *verify the maven artifacts*, and vote.

The vote will run 5 days, ending 4/11/2018.

Thanks for everyone who helped to spot the error and proposed fixes!

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org


-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)

[Eric Payne]

 +1 (binding)
Thanks Wangda for doing the work to produce this release.
I did the following to test the release:
- Built from source
- Installed on 6-node pseudo cluster
- Interacted with RM CLI and GUI
- Tested streaming jobs
- Tested yarn distributed shell jobs
- Tested Max AM Resource Percent

- Tested simple inter-queue preemption
- Tested priority first intra-queue preemption

- Tested userlimit first intra-queue preemption

Thanks,Eric Payne


===
On Thursday, March 29, 2018, 11:15:51 PM CDT, Wangda Tan 
<wheele...@gmail.com> wrote:  
 
 Hi folks,

Thanks to the many who helped with this release since Dec 2017 [1]. We've
created RC1 for Apache Hadoop 3.1.0. The artifacts are available here:

http://people.apache.org/~wangda/hadoop-3.1.0-RC1

The RC tag in git is release-3.1.0-RC1. Last git commit SHA is
16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d

The maven artifacts are available via repository.apache.org at
https://repository.apache.org/content/repositories/orgapachehadoop-1090/
This vote will run 5 days, ending on Apr 3 at 11:59 pm Pacific.

3.1.0 contains 766 [2] fixed JIRA issues since 3.0.0. Notable additions
include the first class GPU/FPGA support on YARN, Native services, Support
rich placement constraints in YARN, S3-related enhancements, allow HDFS
block replicas to be provided by an external storage system, etc.

For 3.1.0 RC0 vote discussion, please see [3].

We’d like to use this as a starting release for 3.1.x [1], depending on how
it goes, get it stabilized and potentially use a 3.1.1 in several weeks as
the stable release.

We have done testing with a pseudo cluster:
- Ran distributed job.
- GPU scheduling/isolation.
- Placement constraints (intra-application anti-affinity) by using
distributed shell.

My +1 to start.

Best,
Wangda/Vinod

[1]
https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104bc9e19e559aedc5ebedb4ca08c8@%3Cyarn-dev.hadoop.apache.org%3E
[2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.0)
AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status = Resolved ORDER BY
fixVersion ASC
[3]
https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d72d4061f26f83547e41d0983ea6@%3Cyarn-dev.hadoop.apache.org%3E
  

Re: [VOTE] Release Apache Hadoop 3.0.1 (RC1)

[Eric Payne]

 Thanks for working on this release!
+1 (binding)
I tested the following:
- yarn distributed shell job

- yarn streaming job

- inter-queue preemption

- compared behavior of fair and fifo ordering policy

- both userlimit_first mode and priority_first mode of intra-queue preemption

Eric Payne



On Saturday, March 17, 2018, 11:11:32 PM CDT, Lei Xu <l...@apache.org> 
wrote:  
 
 Hi, all

I've created release candidate RC-1 for Apache Hadoop 3.0.1

Apache Hadoop 3.0.1 will be the first bug fix release for Apache
Hadoop 3.0 release. It includes 49 bug fixes and security fixes, which
include 12
blockers and 17 are critical.

Please note:
* HDFS-12990. Change default NameNode RPC port back to 8020. It makes
incompatible changes to Hadoop 3.0.0.  After 3.0.1 releases, Apache
Hadoop 3.0.0 will be deprecated due to this change.

The release page is:
https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+3.0+Release

New RC is available at: http://home.apache.org/~lei/hadoop-3.0.1-RC1/

The git tag is release-3.0.1-RC1, and the latest commit is
496dc57cc2e4f4da117f7a8e3840aaeac0c1d2d0

The maven artifacts are available at:
https://repository.apache.org/content/repositories/orgapachehadoop-1081/

Please try the release and vote; the vote will run for the usual 5
days, ending on 3/22/2017 6pm PST time.

Thanks!

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

  

Re: [VOTE] Release Apache Hadoop 2.7.5 (RC1)

[Eric Payne]

Thanks for the hard work on this release, Konstantin.
+1 (binding)
- Built from source
- Verified that refreshing of queues works as expected.

- Verified can run multiple users in a single queue
- Ran terasort test
- Verified that cross-queue preemption works as expected
Thanks. Eric Payne

  From: Konstantin Shvachko <shv.had...@gmail.com>
 To: "common-...@hadoop.apache.org" <common-...@hadoop.apache.org>; 
"hdfs-...@hadoop.apache.org" <hdfs-...@hadoop.apache.org>; 
"mapreduce-dev@hadoop.apache.org" <mapreduce-dev@hadoop.apache.org>; 
"yarn-...@hadoop.apache.org" <yarn-...@hadoop.apache.org> 
 Sent: Thursday, December 7, 2017 9:22 PM
 Subject: [VOTE] Release Apache Hadoop 2.7.5 (RC1)
   
Hi everybody,

I updated CHANGES.txt and fixed documentation links.
Also committed  MAPREDUCE-6165, which fixes a consistently failing test.

This is RC1 for the next dot release of Apache Hadoop 2.7 line. The
previous one 2.7.4 was release August 4, 2017.
Release 2.7.5 includes critical bug fixes and optimizations. See more
details in Release Note:
http://home.apache.org/~shv/hadoop-2.7.5-RC1/releasenotes.html

The RC0 is available at: http://home.apache.org/~shv/hadoop-2.7.5-RC1/

Please give it a try and vote on this thread. The vote will run for 5 days
ending 12/13/2017.

My up to date public key is available from:
https://dist.apache.org/repos/dist/release/hadoop/common/KEYS

Thanks,
--Konstantin


   

Re: [VOTE] Release Apache Hadoop 2.8.3 (RC0)

[Eric Payne]

Thanks Junping for the hard work on this release.
+1 (binding)
On a 6 node pseudo cluster (4 NMs), I performed the following manual tests:
- Built and installed from source
- Successfully ran a stream job
- Verified that user weights are honored by assigning the appropriate amount of 
resources to the weighted users.
- Ensured that FiarOrderingPolicy and FifoOrderingPolicy worked in the Capacity 
Scheduler as expected
- Applications with higher priorities are assigned containers as expected in 
the FifoOrderingPolicy of the Capacity Scheduler until the user reaches its 
user resource limit.
Eric Payne


  From: Junping Du <j...@hortonworks.com>
 To: "common-...@hadoop.apache.org" <common-...@hadoop.apache.org>; 
"hdfs-...@hadoop.apache.org" <hdfs-...@hadoop.apache.org>; 
"mapreduce-dev@hadoop.apache.org" <mapreduce-dev@hadoop.apache.org>; 
"yarn-...@hadoop.apache.org" <yarn-...@hadoop.apache.org> 
 Sent: Tuesday, December 5, 2017 3:58 AM
 Subject: [VOTE] Release Apache Hadoop 2.8.3 (RC0)
   
Hi all,
    I've created the first release candidate (RC0) for Apache Hadoop 2.8.3. 
This is our next maint release to follow up 2.8.2. It includes 79 important 
fixes and improvements.

      The RC artifacts are available at: 
http://home.apache.org/~junping_du/hadoop-2.8.3-RC0

      The RC tag in git is: release-2.8.3-RC0

      The maven artifacts are available via repository.apache.org at: 
https://repository.apache.org/content/repositories/orgapachehadoop-1072

      Please try the release and vote; the vote will run for the usual 5 
working days, ending on 12/12/2017 PST time.

Thanks,

Junping

   

Re: [VOTE] Merge Absolute resource configuration support in Capacity Scheduler (YARN-5881) to trunk

[Eric Payne]

+1. Thanks Sunil for the work on this branch.
Eric

  From: Sunil G <sun...@apache.org>
 To: "yarn-...@hadoop.apache.org" <yarn-...@hadoop.apache.org>; Hdfs-dev 
<hdfs-...@hadoop.apache.org>; Hadoop Common <common-...@hadoop.apache.org>; 
"mapreduce-dev@hadoop.apache.org" <mapreduce-dev@hadoop.apache.org> 
 Sent: Wednesday, November 29, 2017 7:56 PM
 Subject: [VOTE] Merge Absolute resource configuration support in Capacity 
Scheduler (YARN-5881) to trunk
   
Hi All,


Based on the discussion at [1], I'd like to start a vote to merge feature
branch

YARN-5881 to trunk. Vote will run for 7 days, ending Wednesday Dec 6 at
6:00PM PDT.


This branch adds support to configure queue capacity as absolute resource in

capacity scheduler. This will help admins who want fine control of
resources of queues.


Feature development is done at YARN-5881 [2], jenkins build is here
(YARN-7510 [3]).

All required tasks for this feature are committed. This feature changes
RM’s Capacity Scheduler only,

and we did extensive tests for the feature in the last couple of months
including performance tests.


Key points:

- The feature is turned off by default, and have to configure absolute
resource to enable same.

- Detailed documentation about how to use this feature is done as part of
[4].

- No major performance degradation is observed with this branch work. SLS
and UT performance

tests are done.


There were 11 subtasks completed for this feature.


Huge thanks to everyone who helped with reviews, commits, guidance, and

technical discussion/design, including Wangda Tan, Vinod Vavilapalli,
Rohith Sharma K S, Eric Payne .


[1] :
http://mail-archives.apache.org/mod_mbox/hadoop-yarn-dev/201711.mbox/%3CCACYiTuhKhF1JCtR7ZFuZSEKQ4sBvN_n_tV5GHsbJ3YeyJP%2BP4Q%40mail.gmail.com%3E

[2] : https://issues.apache.org/jira/browse/YARN-5881

[3] : https://issues.apache.org/jira/browse/YARN-7510

[4] : https://issues.apache.org/jira/browse/YARN-7533


Regards

Sunil and Wangda

   

Re: [DISCUSS] Merge Absolute resource configuration support in Capacity Scheduler (YARN-5881) to trunk

[Eric Payne]

Thanks Sunil for the great work on this feature.
I looked through the design document, reviewed the code, and tested out branch 
YARN-5881. The design makes sense and the code looks like it is implementing 
the desing in a sensible way. However, I have encountered a couple of bugs. I 
opened https://issues.apache.org/jira/browse/YARN-7575 to track my findings. 
Basically, here's a summary:

The design document from YARN-5881 says that for max-capacity:
3)  For each queue, we require: a) if max-resource not set, it automatically 
set to parent.max-resource
 
When I try not setting 
anyyarn.scheduler.capacity..maximum-capacity, the RMUI scheduler 
page refuses to render. It looks like it's in 
CapacitySchedulerPage$LeafQueueInfoBlock.

Also... A job will run in the leaf queue with no max capacity set and it will 
grow to the max capacity of the cluster, but if I add resources to the node, 
the job won't grow any more even though it has pending resources.

Thanks,Eric


  From: Sunil G 
 To: "yarn-...@hadoop.apache.org" ; Hadoop Common 
; Hdfs-dev ; 
"mapreduce-dev@hadoop.apache.org"  
 Sent: Friday, November 24, 2017 11:49 AM
 Subject: [DISCUSS] Merge Absolute resource configuration support in Capacity 
Scheduler (YARN-5881) to trunk
   
Hi All,

We would like to bring up the discussion of merging “absolute min/max
resources support in capacity scheduler” branch (YARN-5881) [2] into trunk
in a few weeks. The goal is to get it in for Hadoop 3.1.

*Major work happened in this branch*

  - YARN-6471. Support to add min/max resource configuration for a queue
  - YARN-7332. Compute effectiveCapacity per each resource vector
  - YARN-7411. Inter-Queue preemption's computeFixpointAllocation need to
  handle absolute resources.

*Regarding design details*

Please refer [1] for detailed design document.

*Regarding to testing:*

We did extensive tests for the feature in the last couple of months.
Comparing to latest trunk.

- For SLS benchmark: We didn't see observable performance gap from
simulated test based on 8K nodes SLS traces (1 PB memory). We got 3k+
containers allocated per second.

- For microbenchmark: We use performance test cases added by YARN 6775, it
did not show much performance regression comparing to trunk.

*YARN-5881* 

#ResourceTypes = 2. Avg of fastest 20: 55294.52
#ResourceTypes = 2. Avg of fastest 20: 55401.66

*trunk*
#ResourceTypes = 2. Avg of fastest 20: 55865.92
#ResourceTypes = 2. Avg of fastest 20: 55096.418

*Regarding to API stability:*

All newly added @Public APIs are @Unstable.

Documentation jira [3] could help to provide detailed configuration
details. This feature works from end-to-end and we are running this in our
development cluster for last couple of months and undergone good amount of
testing. Branch code is run against trunk and tracked via [4].

We would love to get your thoughts before opening a voting thread.

Special thanks to a team of folks who worked hard and contributed towards
this efforts including design discussion / patch / reviews, etc.: Wangda
Tan, Vinod Kumar Vavilappali, Rohith Sharma K S.

[1] :
https://issues.apache.org/jira/secure/attachment/12855984/YARN-5881.Support.Absolute.Min.Max.Resource.In.Capacity.Scheduler.design-doc.v1.pdf
[2] : https://issues.apache.org/jira/browse/YARN-5881

[3] : https://issues.apache.org/jira/browse/YARN-7533

[4] : https://issues.apache.org/jira/browse/YARN-7510

Thanks,

Sunil G and Wangda Tan

   

Re: [VOTE] Release Apache Hadoop 3.0.0 RC0

[Eric Payne]

Thanks for the all of the work it took to finally get htere.
+1 (binding)
Built from source and stood up a pseudo cluster with 4 NMs
Tested the following:
o Cross queue preemption
o Restarting the RM preserves work
o User limits are honored during In-queue preemption 

o Priorities are honored during In-queue preemption
o Users with different weights are assigned resources proportional to their 
weights.
o User weights are refreshable, and in-queue preemption works to honor the 
post-refresh weights 

Thanks,
-Eric Payne

  From: Andrew Wang <andrew.w...@cloudera.com>
 To: "common-...@hadoop.apache.org" <common-...@hadoop.apache.org>; 
"yarn-...@hadoop.apache.org" <yarn-...@hadoop.apache.org>; 
"mapreduce-dev@hadoop.apache.org" <mapreduce-dev@hadoop.apache.org>; 
"hdfs-...@hadoop.apache.org" <hdfs-...@hadoop.apache.org> 
 Sent: Tuesday, November 14, 2017 3:34 PM
 Subject: [VOTE] Release Apache Hadoop 3.0.0 RC0
   
Hi folks,

Thanks as always to the many, many contributors who helped with this
release. I've created RC0 for Apache Hadoop 3.0.0. The artifacts are
available here:

http://people.apache.org/~wang/3.0.0-RC0/

This vote will run 5 days, ending on Nov 19th at 1:30pm Pacific.

3.0.0 GA contains 291 fixed JIRA issues since 3.0.0-beta1. Notable
additions include the merge of YARN resource types, API-based configuration
of the CapacityScheduler, and HDFS router-based federation.

I've done my traditional testing with a pseudo cluster and a Pi job. My +1
to start.

Best,
Andrew


   

Re: [VOTE] Release Apache Hadoop 2.9.0 (RC3)

[Eric Payne]

Thanks Arun and Subru for the hard work on this release.

+1 (binding)

Built from source and stood up a pseudo cluster with 4 NMs
Tested the following:
o User limits are honored during In-queue preemption 

o Priorities are honored during In-queue preemption
o Can kill applications from the command line
o Users with different weights are assigned resources proportional to their 
weights.
Thanks,-Eric Payne


  From: Arun Suresh <asur...@apache.org>
 To: yarn-...@hadoop.apache.org; mapreduce-dev@hadoop.apache.org; Hadoop Common 
<common-...@hadoop.apache.org>; Hdfs-dev <hdfs-...@hadoop.apache.org> 
Cc: Subramaniam Krishnan <su...@apache.org>
 Sent: Monday, November 13, 2017 6:10 PM
 Subject: [VOTE] Release Apache Hadoop 2.9.0 (RC3)
   
Hi Folks,

Apache Hadoop 2.9.0 is the first release of Hadoop 2.9 line and will be the
starting release for Apache Hadoop 2.9.x line - it includes 30 New Features
with 500+ subtasks, 407 Improvements, 790 Bug fixes new fixed issues since
2.8.2.

More information about the 2.9.0 release plan can be found here:
*https://cwiki.apache.org/confluence/display/HADOOP/Roadmap#Roadmap-Version2.9
<https://cwiki.apache.org/confluence/display/HADOOP/Roadmap#Roadmap-Version2.9>*

New RC is available at: *https://home.apache.org/~asuresh/hadoop-2.9.0-RC3/
<https://home.apache.org/~asuresh/hadoop-2.9.0-RC3/>*

The RC tag in git is: release-2.9.0-RC3, and the latest commit id is:
756ebc8394e473ac25feac05fa493f6d612e6c50.

The maven artifacts are available via repository.apache.org at:
<https://www.google.com/url?q=https%3A%2F%2Frepository.apache.org%2Fcontent%2Frepositories%2Forgapachehadoop-1066=D=1=AFQjCNFcern4uingMV_sEreko_zeLlgdlg>*https://repository.apache.org/content/repositories/orgapachehadoop-1068/
<https://repository.apache.org/content/repositories/orgapachehadoop-1068/>*

We are carrying over the votes from the previous RC given that the delta is
the license fix.

Given the above - we are also going to stick with the original deadline for
the vote : ending on Friday 17th November 2017 2pm PT time.

Thanks,
-Arun/Subru


   

Re: [VOTE] Release Apache Hadoop 3.0.0-beta1 RC0

[Eric Payne]

+1 (binding)
Thanks Andrew for all of your very diligent efforts.
Build from source, installed on a 6-node pseudo cluster, and successfully 
tested the following manual use cases:
o MapReduce sleep job
o MapReduce streaming jobœ
o Cross-queue (inter-queue) preemption 
o In-queue (intra-queue) preemption, both USERFIRST and PRIORITYFIRST
o User weights not equals to 1
o User weights in conjunction with in-queue preemption.
-Eric Payne

  From: Andrew Wang <andrew.w...@cloudera.com>
 To: "common-...@hadoop.apache.org" <common-...@hadoop.apache.org>; 
"hdfs-...@hadoop.apache.org" <hdfs-...@hadoop.apache.org>; 
"mapreduce-dev@hadoop.apache.org" <mapreduce-dev@hadoop.apache.org>; 
"yarn-...@hadoop.apache.org" <yarn-...@hadoop.apache.org> 
 Sent: Thursday, September 28, 2017 7:04 PM
 Subject: [VOTE] Release Apache Hadoop 3.0.0-beta1 RC0
   
Hi all,

Let me start, as always, by thanking the many, many contributors who helped
with this release! I've prepared an RC0 for 3.0.0-beta1:

http://home.apache.org/~wang/3.0.0-beta1-RC0/

This vote will run five days, ending on Nov 3rd at 5PM Pacific.

beta1 contains 576 fixed JIRA issues comprising a number of bug fixes,
improvements, and feature enhancements. Notable additions include the
addition of YARN Timeline Service v2 alpha2, S3Guard, completion of the
shaded client, and HDFS erasure coding pluggable policy support.

I've done the traditional testing of running a Pi job on a pseudo cluster.
My +1 to start.

We're working internally on getting this run through our integration test
rig. I'm hoping Vijay or Ray can ring in with a +1 once that's complete.

Best,
Andrew


   

Re: [DISCUSS] Branches and versions for Hadoop 3

[Eric Payne]

+1 for this branching proposal.-Eric


  From: Andrew Wang 
 To: "common-...@hadoop.apache.org" ; 
"mapreduce-dev@hadoop.apache.org" ; 
"hdfs-...@hadoop.apache.org" ; 
"yarn-...@hadoop.apache.org"  
 Sent: Friday, August 25, 2017 12:36 PM
 Subject: [DISCUSS] Branches and versions for Hadoop 3
   
Hi folks,

With 3.0.0-beta1 fast approaching, I wanted to go over the proposed
branching strategy.

In the early 2.x days, moving trunk immediately to 3.0.0 was a mistake.
branch-2 and trunk were virtually identical, and increased backport
complexity. Until we need to make incompatible changes, there's no need for
a Hadoop 4.0 version.

Thus, here's a proposal of branches and versions:

trunk: 3.1.0-SNAPSHOT
branch-3.0: 3.0.0-beta1-SNAPSHOT
branch-2 and etc: remain as is

LMK questions/comments/etc. Appreciate your attentiveness; I'm hoping to
build consensus quickly since we have a number of open VOTEs for branch
merges.

Thanks,
Andrew


   

Re: [VOTE] Release Apache Hadoop 2.7.4 (RC0)

[Eric Payne]

+1 (binding)
Tested the following:




- Application History Server

-- Apps can be observed from UI
-- App and container metadata can be retrieved via REST APIs



- RM UI

-- Can kill an app from the RM UI


- Apps run in different frameworks. Frameworks tested: MR and yarn shell

-- In yarn shell framework, containers are preserved across AM restart.


- Cross-queue preemption (Inter-queue):

-- Inter-queue preemption will preempt the correct number of containers from a p
reemptable queue.

-- Inter-queue preemption will not preempt from queues with preemption disabled.


- Labeled queues work as expected where apps assigned to a queue that has a spec
ific label will run only on labeled nodes.





From: Konstantin Shvachko 
To: Chris Douglas  
Cc: Andrew Wang ; Allen Wittenauer 
; "common-...@hadoop.apache.org" 
; "hdfs-...@hadoop.apache.org" 
; "mapreduce-dev@hadoop.apache.org" 
; "yarn-...@hadoop.apache.org" 

Sent: Monday, July 31, 2017 8:57 PM
Subject: Re: [VOTE] Release Apache Hadoop 2.7.4 (RC0)



Uploaded new binaries hadoop-2.7.4-RC0.tar.gz, which adds lib/native/.
Same place: http://home.apache.org/~shv/hadoop-2.7.4-RC0/

Thanks,
--Konstantin


On Mon, Jul 31, 2017 at 3:56 PM, Chris Douglas  wrote:

> On Mon, Jul 31, 2017 at 3:02 PM, Konstantin Shvachko
>  wrote:
> > For the packaging, here is the exact phrasing from the sited
> release-policy
> > document relevant to binaries:
> > "As a convenience to users that might not have the appropriate tools to
> > build a compiled version of the source, binary/bytecode packages MAY be
> > distributed alongside official Apache releases. In all such cases, the
> > binary/bytecode package MUST have the same version number as the source
> > release and MUST only add binary/bytecode files that are the result of
> > compiling that version of the source code release and its dependencies."
> > I don't think my binary package violates any of these.
>
> +1 The PMC VOTE applies to source code, only. If someone wants to
> rebuild the binary tarball with native libs and replace this one,
> that's fine.
>
> My reading of the above is that source code must be distributed with
> binaries, not that we omit the source code from binary releases... -C
>
> > But I'll upload an additional tar.gz with native bits and no src, as you
> > guys requested.
> > Will keep it as RC0 as there is no source code change and it comes from
> the
> > same build.
> > Hope this is satisfactory.
> >
> > Thanks,
> > --Konstantin
> >
> > On Mon, Jul 31, 2017 at 1:53 PM, Andrew Wang 
> > wrote:
> >
> >> I agree with Brahma on the two issues flagged (having src in the binary
> >> tarball, missing native libs). These are regressions from prior
> releases.
> >>
> >> As an aside, "we release binaries as a convenience" doesn't relax the
> >> quality bar. The binaries are linked on our website and distributed
> through
> >> official Apache channels. They have to adhere to Apache release
> >> requirements. And, most users consume our work via Maven dependencies,
> >> which are binary artifacts.
> >>
> >> http://www.apache.org/legal/release-policy.html goes into this in more
> >> detail. A release must minimally include source packages, and can also
> >> include binary artifacts.
> >>
> >> Best,
> >> Andrew
> >>
> >> On Mon, Jul 31, 2017 at 12:30 PM, Konstantin Shvachko <
> >> shv.had...@gmail.com> wrote:
> >>
> >>> To avoid any confusion in this regard. I built RC0 manually in
> compliance
> >>> with Apache release policy
> >>> http://www.apache.org/legal/release-policy.html
> >>> I edited the HowToReleasePreDSBCR page to make sure people don't use
> >>> Jenkins option for building.
> >>>
> >>> A side note. This particular build is broken anyways, so no worries
> there.
> >>> I think though it would be useful to have it working for testing and
> as a
> >>> packaging standard.
> >>>
> >>> Thanks,
> >>> --Konstantin
> >>>
> >>> On Mon, Jul 31, 2017 at 11:40 AM, Allen Wittenauer <
> >>> a...@effectivemachines.com
> >>> > wrote:
> >>>
> >>> >
> >>> > > On Jul 31, 2017, at 11:20 AM, Konstantin Shvachko <
> >>> shv.had...@gmail.com>
> >>> > wrote:
> >>> > >
> >>> > > https://wiki.apache.org/hadoop/HowToReleasePreDSBCR
> >>> >
> >>> > FYI:
> >>> >
> >>> > If you are using ASF Jenkins to create an ASF release
> >>> > artifact, it's pretty much an automatic vote failure as any such
> >>> release is
> >>> > in violation of ASF policy.
> >>> >
> >>> >
> >>>
> >>
> >>
>

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.0.0-alpha4-RC0

[Eric Payne]

Thanks Andrew.
I downloaded the source, built it, and installed it onto a pseudo distributed 
4-node cluster. 

I ran mapred and streaming test cases, including sleep and wordcount.
+1 (non-binding)
-Eric

  From: Andrew Wang 
 To: "common-...@hadoop.apache.org" ; 
"hdfs-...@hadoop.apache.org" ; 
"mapreduce-dev@hadoop.apache.org" ; 
"yarn-...@hadoop.apache.org"  
 Sent: Thursday, June 29, 2017 9:41 PM
 Subject: [VOTE] Release Apache Hadoop 3.0.0-alpha4-RC0
   
Hi all,

As always, thanks to the many, many contributors who helped with this
release! I've prepared an RC0 for 3.0.0-alpha4:

http://home.apache.org/~wang/3.0.0-alpha4-RC0/

The standard 5-day vote would run until midnight on Tuesday, July 4th.
Given that July 4th is a holiday in the US, I expect this vote might have
to be extended, but I'd like to close the vote relatively soon after.

I've done my traditional testing of a pseudo-distributed cluster with a
single task pi job, which was successful.

Normally my testing would end there, but I'm slightly more confident this
time. At Cloudera, we've successfully packaged and deployed a snapshot from
a few days ago, and run basic smoke tests. Some bugs found from this
include HDFS-11956, which fixes backwards compat with Hadoop 2 clients, and
the revert of HDFS-11696, which broke NN QJM HA setup.

Vijay is working on a test run with a fuller test suite (the results of
which we can hopefully post soon).

My +1 to start,

Best,
Andrew


   

Pre commit builds seem broken

[Eric Payne]

Hey does anyone know why the precommit builds seem broken? I try to kick one 
and it hangs for a long time and then produces the following:

#1​62​88 (pending—H14 doesn’t have label Hadoop&&!H9&&!H5&&!H6; H15 doesn’t 
have label Hadoop&&!H9&&!H5&&!H6; H16 doesn’t have label Hadoop&&!H9&&!H5&&!H6; 
H17 doesn’t have label Hadoop&&!H9&&!H5&&!H6; H18 doesn’t have label 
Hadoop&&!H9&&!H5&&!H6; H19 doesn’t have label Hadoop&&!H9&&!H5&&!H6; H20 
doesn’t have label Hadoop&&!H9&&!H5&&!H6; H21 doesn’t have label 
Hadoop&&!H9&&!H5&&!H6; H22 doesn’t have label Hadoop&&!H9&&!H5&&!H6; H24 
doesn’t have label Hadoop&&!H9&&!H5&&!H6; H5 doesn’t have label 
Hadoop&&!H9&&!H5&&!H6; H6 doesn’t have label Hadoop&&!H9&&!H5&&!H6; H9 doesn’t 
have label Hadoop&&!H9&&!H5&&!H6; Waiting for pending items to get a node 
assigned; beam1 doesn’t have label Hadoop&&!H9&&!H5&&!H6; beam2 doesn’t have 
label Hadoop&&!H9&&!H5&&!H6; beam3 doesn’t have label Hadoop&&!H9&&!H5&&!H6; 
beam4 doesn’t have label Hadoop&&!H9&&!H5&&!H6; beam5 doesn’t have label 
Hadoop&&!H9&&!H5&&!H6; beam6 doesn’t have label Hadoop&&!H9&&!H5&&!H6; beam7 
doesn’t have label Hadoop&&!H9&&!H5&&!H6; beam8 doesn’t have label 
Hadoop&&!H9&&!H5&&!H6; cassandra1 doesn’t have label Hadoop&&!H9&&!H5&&!H6; 
cassandra10 doesn’t have label Hadoop&&!H9&&!H5&&!H6; cassan

[jira] [Created] (MAPREDUCE-6812) Capacity Scheduler: Support user-specific minimum user limit percent

[Eric Payne (JIRA)]

Eric Payne created MAPREDUCE-6812:
-

 Summary: Capacity Scheduler: Support user-specific minimum user 
limit percent
 Key: MAPREDUCE-6812
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6812
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
  Components: yarn, capacity-sched
Reporter: Eric Payne


Currently, in the capacity scheduler, the {{minimum-user-limit-percent}} 
property is per queue. A cluster admin should be able to set the minimum user 
limit percent on a per-user basis within the queue.

This functionality is needed so that when intra-queue preemption is enabled 
(YARN-4945 / YARN-2113), some users can be deemed as more important than other 
users, and resources from VIP users won't be as likely to be preempted.

For example, if the {{getstuffdone}} queue has a MULP of 25 percent, but user 
{{jane}} is a power user of queue {{getstuffdone}} and needs to be guaranteed 
75 percent, the properties for {{getstuffdone}} and {{jane}} would look like 
this:

{code}
  

yarn.scheduler.capacity.root.getstuffdone.minimum-user-limit-percent
25
  

  

yarn.scheduler.capacity.root.getstuffdone.jane.minimum-user-limit-percent
75
  
{code}

NOTE: This should be implemented in a way that user-limit-percent-intra-queue 
preemption (YARN-2113) should not be affected.




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: Updated 2.8.0-SNAPSHOT artifact

[Eric Payne]

How do we come to a resolution regarding whether or not re-cut branch-2.8 or 
release it as it is (after fixing blockers)?
There are some things in branch-2 that I would like to pull back into 
branch-2.8, so a resolution to this question will affect how I proceed.
Thanks,-Eric
  From: Karthik Kambatla 
 To: Ming Ma  
Cc: Sangjin Lee ; Jason Lowe ; Akira 
Ajisaka ; Brahma Reddy Battula 
; Vinod Kumar Vavilapalli ; 
"common-...@hadoop.apache.org" ; 
"hdfs-...@hadoop.apache.org" ; 
"mapreduce-dev@hadoop.apache.org" ; 
"yarn-...@hadoop.apache.org" 
 Sent: Thursday, November 10, 2016 1:56 AM
 Subject: Re: Updated 2.8.0-SNAPSHOT artifact
   
If there is interest in releasing off of branch-2.8, we should definitely
do that. As Sangjin mentioned, there might be value in doing 2.9 off
branch-2 too.

How do we go about maintenance releases along those minor lines, and when
would we discontinue 2.6.x/2.7.x releases?

On Wed, Nov 9, 2016 at 12:06 PM, Ming Ma  wrote:

> I would also prefer releasing current 2.8 branch sooner. There are several
> incomplete features in branch-2 such as YARN-914 and HDFS-7877 that are
> better served if we can complete them in the next major release. Letting
> them span across multiple releases might not be desirable as there could be
> some potential compatibility issues involved. Therefore if we recut 2.8 it
> means we have to work on those items before the new 2.8 is released which
> could cause major delay on the schedule.
>
> On Mon, Nov 7, 2016 at 10:37 AM, Sangjin Lee  wrote:
>
>> +1. Resetting the 2.8 effort and the branch at this point may be
>> counter-productive. IMO we should focus on resolving the remaining
>> blockers
>> and getting it out the door. I also think that we should seriously
>> consider
>> 2.9 as well, as a fairly large number of changes have accumulated in
>> branch-2 (over branch-2.8).
>>
>>
>> Sangjin
>>
>> On Fri, Nov 4, 2016 at 3:38 PM, Jason Lowe 
>> wrote:
>>
>> > At this point my preference would be to do the most expeditious thing to
>> > release 2.8, whether that's sticking with the branch-2.8 we have today
>> or
>> > re-cutting it on branch-2.  Doing a quick JIRA query, there's been
>> almost
>> > 2,400 JIRAs resolved in 2.8.0 (1).  For many of them, it's well-past
>> time
>> > they saw a release vehicle.  If re-cutting the branch means we have to
>> wrap
>> > up a few extra things that are still in-progress on branch-2 or add a
>> few
>> > more blockers to the list before we release then I'd rather stay where
>> > we're at and ship it ASAP.
>> >
>> > Jason
>> > (1) https://issues.apache.org/jira/issues/?jql=project%20in%
>> > 20%28hadoop%2C%20yarn%2C%20mapreduce%2C%20hdfs%29%
>> > 20and%20resolution%20%3D%20Fixed%20and%20fixVersion%20%3D%202.8.0
>> >
>> >
>> >
>> >
>> >
>> >    On Tuesday, October 25, 2016 5:31 PM, Karthik Kambatla <
>> > ka...@cloudera.com> wrote:
>> >
>> >
>> >  Is there value in releasing current branch-2.8? Aren't we better off
>> > re-cutting the branch off of branch-2?
>> >
>> > On Tue, Oct 25, 2016 at 12:20 AM, Akira Ajisaka <
>> > ajisa...@oss.nttdata.co.jp>
>> > wrote:
>> >
>> > > It's almost a year since branch-2.8 has cut.
>> > > I'm thinking we need to release 2.8.0 ASAP.
>> > >
>> > > According to the following list, there are 5 blocker and 6 critical
>> > issues.
>> > > https://issues.apache.org/jira/issues/?filter=12334985
>> > >
>> > > Regards,
>> > > Akira
>> > >
>> > >
>> > > On 10/18/16 10:47, Brahma Reddy Battula wrote:
>> > >
>> > >> Hi Vinod,
>> > >>
>> > >> Any plan on first RC for branch-2.8 ? I think, it has been long time.
>> > >>
>> > >>
>> > >>
>> > >>
>> > >> --Brahma Reddy Battula
>> > >>
>> > >> -Original Message-
>> > >> From: Vinod Kumar Vavilapalli [mailto:vino...@apache.org]
>> > >> Sent: 20 August 2016 00:56
>> > >> To: Jonathan Eagles
>> > >> Cc: common-...@hadoop.apache.org
>> > >> Subject: Re: Updated 2.8.0-SNAPSHOT artifact
>> > >>
>> > >> Jon,
>> > >>
>> > >> That is around the time when I branched 2.8, so I guess you were
>> getting
>> > >> SNAPSHOT artifacts till then from the branch-2 nightly builds.
>> > >>
>> > >> If you need it, we can set up SNAPSHOT builds. Or just wait for the
>> > first
>> > >> RC, which is around the corner.
>> > >>
>> > >> +Vinod
>> > >>
>> > >> On Jul 28, 2016, at 4:27 PM, Jonathan Eagles 
>> wrote:
>> > >>>
>> > >>> Latest snapshot is uploaded in Nov 2015, but checkins are still
>> coming
>> > >>> in quite frequently.
>> > >>> https://repository.apache.org/content/repositories/snapshots
>> /org/apach
>> > >>> e/hadoop/hadoop-yarn-api/
>> > >>>
>> > >>> Are there any plans to start producing 

Re: [VOTE] Release Apache Hadoop 3.0.0-alpha1 RC0

[Eric Payne]

Thank you very much, Andrew.

+1 (non-binding)

- Downloaded source and built native
- Installed on 3-node, non-secure cluster
- Ran sleep jobs
- Ensured preemption works as expected

-Eric Payne


- Original Message -
From: Andrew Wang <andrew.w...@cloudera.com>
To: "common-...@hadoop.apache.org" <common-...@hadoop.apache.org>; 
"hdfs-...@hadoop.apache.org" <hdfs-...@hadoop.apache.org>; 
"mapreduce-dev@hadoop.apache.org" <mapreduce-dev@hadoop.apache.org>; 
"yarn-...@hadoop.apache.org" <yarn-...@hadoop.apache.org>
Sent: Tuesday, August 30, 2016 10:51 AM
Subject: [VOTE] Release Apache Hadoop 3.0.0-alpha1 RC0

Hi all,

Thanks to the combined work of many, many contributors, here's an RC0 for
3.0.0-alpha1:

http://home.apache.org/~wang/3.0.0-alpha1-RC0/

alpha1 is the first in a series of planned alpha releases leading up to GA.
The objective is to get an artifact out to downstreams for testing and to
iterate quickly based on their feedback. So, please keep that in mind when
voting; hopefully most issues can be addressed by future alphas rather than
future RCs.

Sorry for getting this out on a Tuesday, but I'd still like this vote to
run the normal 5 days, thus ending Saturday (9/3) at 9AM PDT. I'll extend
if we lack the votes.

Please try it out and let me know what you think.

Best,
Andrew

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 2.7.3 RC2

[Eric Payne]

Thanks, Vinod, for working so hard on each 2.7 release.

+1 (non-binding)

Here's what I did:

- Built native 
- Installed on 3-node unsecure cluster
- Configured 2 queues with 2 separate label partitions 
- Verified that a job will successfully run on the correctly labelled node by 
specifying a non-default (but queue-accessible) label. 
- Verified that a distributed shell job would keep non-AM containers running 
across an App Master attempt restart. 
- Verified that preemption happens as expected (sort of). I say "sort of" 
because about twice as many containers were preempted as I thought should have 
been, but once the other underserved app began to run, it stopped preempting. 
Also, it didn't preempt between 2 queues with the same partition label. 
Partition preemption may not be supported in 2.7, so this is probably also okay.


Thanks!
Eric Payne





From: Vinod Kumar Vavilapalli <vino...@apache.org>
To: "common-...@hadoop.apache.org" <common-...@hadoop.apache.org>; 
hdfs-...@hadoop.apache.org; yarn-...@hadoop.apache.org; 
"mapreduce-dev@hadoop.apache.org" <mapreduce-dev@hadoop.apache.org> 
Cc: Vinod Kumar Vavilapalli <vino...@apache.org>
Sent: Wednesday, August 17, 2016 9:05 PM
Subject: [VOTE] Release Apache Hadoop 2.7.3 RC2


Hi all,

I've created a new release candidate RC2 for Apache Hadoop 2.7.3.

As discussed before, this is the next maintenance release to follow up 2.7.2.

The RC is available for validation at: 
http://home.apache.org/~vinodkv/hadoop-2.7.3-RC2/ 
<http://home.apache.org/~vinodkv/hadoop-2.7.3-RC2/>

The RC tag in git is: release-2.7.3-RC2

The maven artifacts are available via repository.apache.org 
<http://repository.apache.org/> at 
https://repository.apache.org/content/repositories/orgapachehadoop-1046 
<https://repository.apache.org/content/repositories/orgapachehadoop-1046>

The release-notes are inside the tar-balls at location 
hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html. I hosted 
this at http://home.apache.org/~vinodkv/hadoop-2.7.3-RC2/releasenotes.html 
<http://home.apache.org/~vinodkv/hadoop-2.7.3-RC2/releasenotes.html> for your 
quick perusal.

As you may have noted,
- few issues with RC0 forced a RC1 [1]
- few more issues with RC1 forced a RC2 [2]
- a very long fix-cycle for the License & Notice issues (HADOOP-12893) caused 
2.7.3 (along with every other Hadoop release) to slip by quite a bit. This 
release's related discussion thread is linked below: [3].

Please try the release and vote; the vote will run for the usual 5 days.

Thanks,
Vinod

[1] [VOTE] Release Apache Hadoop 2.7.3 RC0: 
https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/index.html#26106 
<https://www.mail-archive.com/hdfs-dev@hadoop.apache.org/index.html#26106>
[2] [VOTE] Release Apache Hadoop 2.7.3 RC1: 
https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/msg26336.html 
<https://www.mail-archive.com/hdfs-dev@hadoop.apache.org/msg26336.html>
[3] 2.7.3 release plan: 
https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/msg24439.html 
<http://markmail.org/thread/6yv2fyrs4jlepmmr>

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 2.7.3 RC0

[Eric Payne]

+1 (non-binding)
Thanks, Vinod, for all of your hard work and congratulations in completing this 
release.
After downloading and building the source, I installed Hadoop 2.7.3 RC0 on a 
3-node, multi-tenant, insecure cluster. I ran manual tests to ensure the 
following:
- Ensure that user limit percent is honored for multiple users in the same queue
- Ensure that cross-queue preemption occurs when a preemptable queue is over 
its guaranteed capacity
- Ensure that jobs submitted to labelled queues only run on the labelled nodes.
- Ensure that RM UI shows correct queue resource metrics when jobs are running 
in labelled queues (YARN-4751).
- Ensure that a yarn distributed shell application can be launched and complete 
successfully.

Eric Payne
  From: Vinod Kumar Vavilapalli <vino...@apache.org>
 To: "common-...@hadoop.apache.org" <common-...@hadoop.apache.org>; 
hdfs-...@hadoop.apache.org; yarn-...@hadoop.apache.org; 
"mapreduce-dev@hadoop.apache.org" <mapreduce-dev@hadoop.apache.org> 
Cc: Vinod Kumar Vavilapalli <vino...@apache.org>
 Sent: Friday, July 22, 2016 9:15 PM
 Subject: [VOTE] Release Apache Hadoop 2.7.3 RC0
   
Hi all,

I've created a release candidate RC0 for Apache Hadoop 2.7.3.

As discussed before, this is the next maintenance release to follow up 2.7.2.

The RC is available for validation at: 
http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/ 
<http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/>

The RC tag in git is: release-2.7.3-RC0

The maven artifacts are available via repository.apache.org 
<http://repository.apache.org/> at 
https://repository.apache.org/content/repositories/orgapachehadoop-1040/ 
<https://repository.apache.org/content/repositories/orgapachehadoop-1040/>

The release-notes are inside the tar-balls at location 
hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html. I hosted 
this at http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/releasenotes.html 
<http://people.apache.org/~vinodkv/hadoop-2.7.2-RC1/releasenotes.html> for your 
quick perusal.

As you may have noted, a very long fix-cycle for the License & Notice issues 
(HADOOP-12893) caused 2.7.3 (along with every other Hadoop release) to slip by 
quite a bit. This release's related discussion thread is linked below: [1].

Please try the release and vote; the vote will run for the usual 5 days.

Thanks,
Vinod

[1]: 2.7.3 release plan: 
https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/msg24439.html 
<http://markmail.org/thread/6yv2fyrs4jlepmmr>

   

Re: [VOTE] Release Apache Hadoop 2.7.2 RC2

[Eric Payne]

Vinod, I have opened https://issues.apache.org/jira/browse/YARN-4751 to cover 
this issue.It happens in 2.7, not 2.8.Thanks,-Eric


  From: Vinod Kumar Vavilapalli <vino...@apache.org>
 To: hdfs-...@hadoop.apache.org; Eric Payne <eric.payne1...@yahoo.com> 
Cc: Hadoop Common <common-...@hadoop.apache.org>; "yarn-...@hadoop.apache.org" 
<yarn-...@hadoop.apache.org>; "mapreduce-dev@hadoop.apache.org" 
<mapreduce-dev@hadoop.apache.org>
 Sent: Monday, January 25, 2016 11:16 AM
 Subject: Re: [VOTE] Release Apache Hadoop 2.7.2 RC2
   
Eric, is there a JIRA for this?
+Vinod

On Jan 19, 2016, at 2:27 PM, Eric Payne <eric.payne1...@yahoo.com.INVALID> 
wrote:
One minor issue is that when the job was running on the labelled queue, it ran 
fine but the cluster scheduler web UI showed a blank status bar for that queue 
in the "Application Queues" section.



  

Re: [VOTE] Release Apache Hadoop 2.6.4 RC0

[Eric Payne]

Hi Junping Du. Thank you for your work preparing this release.
I did the following things to test release Hadoop 2.6.4 rc0:- Downloaded 
hadoop-2.6.4-RC0-src.tar.gz- built from source both package, install, and 
eclipse:eclipse- Set up a 3-node, unsecured cluster with 3 queues, one of which 
has preemption enabled- Ran a successful test to ensure that preemption would 
happen to containers on the preemptable queue if they were needed for an 
application on another queue.- Ran successful streaming and yarn shell tests
Junping, I did have a concern about labelled nodes and queues. Is full label 
support backported to 2.6.4? I see that the syntax for the rmadmin command 
lists label commands like -addToClusterNodeLabels and -replaceLabelsOnNode. I 
was able to add a label (using -addToClusterNodeLabels) and I was able to 
define a queue whose accessible node label was listed with my specified label. 
However, when I tried to set the node label to a specific node using 
-replaceLabelsOnNode, the label does not show up on the specified node in 
cluster nodes UI (http://RM:8088/cluster/nodes). I also confirmed that 
submitting a job to the labelled queue gets accepted but never runs, which is 
the behavior I would expect if no node had the specified label. I will also add 
that this procedure works fine in 2.7.
Thanks,-Eric Payne

  From: Junping Du <j...@hortonworks.com>
 To: "hdfs-...@hadoop.apache.org" <hdfs-...@hadoop.apache.org>; 
"yarn-...@hadoop.apache.org" <yarn-...@hadoop.apache.org>; 
"mapreduce-dev@hadoop.apache.org" <mapreduce-dev@hadoop.apache.org>; 
"common-...@hadoop.apache.org" <common-...@hadoop.apache.org> 
 Sent: Wednesday, February 3, 2016 1:01 AM
 Subject: [VOTE] Release Apache Hadoop 2.6.4 RC0
   
Hi community folks,
  I've created a release candidate RC0 for Apache Hadoop 2.6.4 (the next 
maintenance release to follow up 2.6.3.) according to email thread of release 
plan 2.6.4 [1]. Below is details of this release candidate:

The RC is available for validation at:
*http://people.apache.org/~junping_du/hadoop-2.6.4-RC0/
<http://people.apache.org/~junping_du/hadoop-2.6.4-RC0/>*

The RC tag in git is: release-2.6.4-RC0

The maven artifacts are staged via repository.apache.org at:
*https://repository.apache.org/content/repositories/orgapachehadoop-1028/?
<https://repository.apache.org/content/repositories/orgapachehadoop-1028/>*

You can find my public key at:
http://svn.apache.org/repos/asf/hadoop/common/dist/KEYS

Please try the release and vote. The vote will run for the usual 5 days.

Thanks!


Cheers,

Junping


[1]: 2.6.4 release plan: http://markmail.org/message/fk3ud3c665lscvx5?


  

Re: [VOTE] Release Apache Hadoop 2.6.4 RC0

[Eric Payne]

Naganarasimha Garla, thanks for the reply.

Yes, I used the node ID. I did not include a port. Here are the steps I used, 
which work for me on 2.7:

- yarn rmadmin -addToClusterNodeLabels abc
- yarn rmadmin -replaceLabelsOnNode hostname.company.com=abc
- configure queue properties as appropriate
- yarn rmadmin -refreshQueues

As I say, this works for me when I try it on 2.7 and later. It's probably 
something with my environment. I will continue to look into it.

Thanks for your help
-Eric



From: Naganarasimha Garla <naganarasimha...@gmail.com>
To: mapreduce-dev@hadoop.apache.org; Eric Payne <eric.payne1...@yahoo.com> 
Cc: "common-...@hadoop.apache.org" <common-...@hadoop.apache.org>; 
"hdfs-...@hadoop.apache.org" <hdfs-...@hadoop.apache.org>; 
"yarn-...@hadoop.apache.org" <yarn-...@hadoop.apache.org>
Sent: Monday, February 8, 2016 1:01 PM
Subject: Re: [VOTE] Release Apache Hadoop 2.6.4 RC0



+1 (non binding)

* Downloaded hadoop-2.6.4-RC0-src.tar.gz- built from source both package, 
install, and verified the MD5 checksum

* Did a Pseudo cluster and tested basic hdfs operations
* Ran sleep job and Pi job
* Added node label and ran job under the label by configuring 
default-node-label-expression and it ran fine

Eric Payne,
Hope you tried adding/replacing the labels using NodeId/Node Address and not 
the HTTP address!
I executed the following command to configure the label and node
  "./yarn rmadmin -replaceLabelsOnNode  "localhost:43795,test1"  "
After this was able to submit the job for a label

Regards,
+ Naga


On Mon, Feb 8, 2016 at 11:06 PM, Eric Payne <eric.payne1...@yahoo.com.invalid> 
wrote:

Hi Junping Du. Thank you for your work preparing this release.
>I did the following things to test release Hadoop 2.6.4 rc0:- Downloaded 
>hadoop-2.6.4-RC0-src.tar.gz- built from source both package, install, and 
>eclipse:eclipse- Set up a 3-node, unsecured cluster with 3 queues, one of 
>which has preemption enabled- Ran a successful test to ensure that preemption 
>would happen to containers on the preemptable queue if they were needed for an 
>application on another queue.- Ran successful streaming and yarn shell tests
>Junping, I did have a concern about labelled nodes and queues. Is full label 
>support backported to 2.6.4? I see that the syntax for the rmadmin command 
>lists label commands like -addToClusterNodeLabels and -replaceLabelsOnNode. I 
>was able to add a label (using -addToClusterNodeLabels) and I was able to 
>define a queue whose accessible node label was listed with my specified label. 
>However, when I tried to set the node label to a specific node using 
>-replaceLabelsOnNode, the label does not show up on the specified node in 
>cluster nodes UI (http://RM:8088/cluster/nodes). I also confirmed that 
>submitting a job to the labelled queue gets accepted but never runs, which is 
>the behavior I would expect if no node had the specified label. I will also 
>add that this procedure works fine in 2.7.
>Thanks,-Eric Payne
>
>  From: Junping Du <j...@hortonworks.com>
> To: "hdfs-...@hadoop.apache.org" <hdfs-...@hadoop.apache.org>; 
> "yarn-...@hadoop.apache.org" <yarn-...@hadoop.apache.org>; 
> "mapreduce-dev@hadoop.apache.org" <mapreduce-dev@hadoop.apache.org>; 
> "common-...@hadoop.apache.org" <common-...@hadoop.apache.org>
> Sent: Wednesday, February 3, 2016 1:01 AM
> Subject: [VOTE] Release Apache Hadoop 2.6.4 RC0
>
>
>Hi community folks,
>  I've created a release candidate RC0 for Apache Hadoop 2.6.4 (the next 
> maintenance release to follow up 2.6.3.) according to email thread of release 
> plan 2.6.4 [1]. Below is details of this release candidate:
>
>The RC is available for validation at:
>*http://people.apache.org/~junping_du/hadoop-2.6.4-RC0/
><http://people.apache.org/~junping_du/hadoop-2.6.4-RC0/>*
>
>The RC tag in git is: release-2.6.4-RC0
>
>The maven artifacts are staged via repository.apache.org at:
>*https://repository.apache.org/content/repositories/orgapachehadoop-1028/?
><https://repository.apache.org/content/repositories/orgapachehadoop-1028/>*
>
>You can find my public key at:
>http://svn.apache.org/repos/asf/hadoop/common/dist/KEYS
>
>Please try the release and vote. The vote will run for the usual 5 days.
>
>Thanks!
>
>
>Cheers,
>
>Junping
>
>
>[1]: 2.6.4 release plan: http://markmail.org/message/fk3ud3c665lscvx5?
>
>
>  

Re: [VOTE] Release Apache Hadoop 2.7.2 RC2

[Eric Payne]

+1 (non-binding)

- Downloaded and built the source

- Installed on 3-node, non-secured cluster
- Ran 2 jobs, one in preemptable queue and one not. Containers were preempted 
until the app in the non-preemptable queue was balanced with the app running in 
the preemptable queue, and then preemption stopped, as expected.

- Labelled 1 queue (of 3) and ran jobs on labelled queue. All containers 
(including AM) ran on labelled nodes, as expected. One minor issue is that when 
the job was running on the labelled queue, it ran fine but the cluster 
scheduler web UI showed a blank status bar for that queue in the "Application 
Queues" section.
- Tested to make sure application data is preserved in the generic application 
history server when the AM crashes during startup (e.g., exceeds max splits).

-Eric Payne




From: Vinod Kumar Vavilapalli <vino...@apache.org>
To: Hadoop Common <common-...@hadoop.apache.org>; hdfs-...@hadoop.apache.org; 
yarn-...@hadoop.apache.org; mapreduce-dev@hadoop.apache.org 
Sent: Thursday, January 14, 2016 10:57 PM
Subject: [VOTE] Release Apache Hadoop 2.7.2 RC2


Hi all,

I've created an updated release candidate RC2 for Apache Hadoop 2.7.2.

As discussed before, this is the next maintenance release to follow up 2.7.1.

The RC is available for validation at: 
http://people.apache.org/~vinodkv/hadoop-2.7.2-RC2/

The RC tag in git is: release-2.7.2-RC2

The maven artifacts are available via repository.apache.org 
<http://repository.apache.org/> at 
https://repository.apache.org/content/repositories/orgapachehadoop-1027 
<https://repository.apache.org/content/repositories/orgapachehadoop-1027>

The release-notes are inside the tar-balls at location 
hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html. I hosted 
this at http://people.apache.org/~vinodkv/hadoop-2.7.2-RC2/releasenotes.html 
<http://people.apache.org/~vinodkv/hadoop-2.7.2-RC1/releasenotes.html> for your 
quick perusal.

As you may have noted,
- I terminated the RC1 related voting thread after finding out that we didn’t 
have a bunch of patches that are already in the released 2.6.3 version. After a 
brief discussion, we decided to keep the parallel 2.6.x and 2.7.x releases 
incremental, see [4] for this discussion.
- The RC0 related voting thread got halted due to some critical issues. It took 
a while again for getting all those blockers out of the way. See the previous 
voting thread [3] for details.
- Before RC0, an unusually long 2.6.3 release caused 2.7.2 to slip by quite a 
bit. This release's related discussion threads are linked below: [1] and [2].

Please try the release and vote; the vote will run for the usual 5 days.

Thanks,
Vinod

[1]: 2.7.2 release plan: http://markmail.org/message/oozq3gvd4nhzsaes 
<http://markmail.org/message/oozq3gvd4nhzsaes>
[2]: Planning Apache Hadoop 2.7.2 http://markmail.org/message/iktqss2qdeykgpqk 
<http://markmail.org/message/iktqss2qdeykgpqk>
[3]: [VOTE] Release Apache Hadoop 2.7.2 RC0: 
http://markmail.org/message/5txhvr2qdiqglrwc 
<http://markmail.org/message/5txhvr2qdiqglrwc>
[4] Retracted [VOTE] Release Apache Hadoop 2.7.2 RC1: 
http://markmail.org/thread/n7ljbsnquihn3wlw

[jira] [Resolved] (MAPREDUCE-2011) Reduce number of getFileStatus call made from every task(TaskDistributedCache) setup

[Eric Payne (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-2011?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eric Payne resolved MAPREDUCE-2011.
---
Resolution: Won't Fix

[~knoguchi], here are [~jlowe]'s comments from an offline discussion:
I think the distributed cache already behaves the way you desire, at least in 
YARN. When a resource request arrives at the nodemanager, it tries to lookup 
the local resource info based on that request. If it finds it (i.e.: a hit in 
the cache) then it just increments the refcount of the resource – I don't see 
any attempt to stat HDFS to verify it's still there in HDFS. The only time I 
see the timestamp of the request compared with HDFS is when it tries to 
download the resource from HDFS.

> Reduce number of getFileStatus call made from every 
> task(TaskDistributedCache) setup
> 
>
> Key: MAPREDUCE-2011
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2011
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>  Components: distributed-cache
>Reporter: Koji Noguchi
>
> On our cluster, we had jobs with 20 dist cache and very short-lived tasks 
> resulting in 500 map tasks launched per second resulting in  10,000 
> getFileStatus calls to the namenode.  Namenode can handle this but asking to 
> see if we can reduce this somehow.  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [VOTE] Release Apache Hadoop 2.6.2

[Eric Payne]

Thank you, Sangjin Lee, for producing and managing this build! 
+1
- Downloaded source, compiled native code, and installed on 2-node cluster.
- Ran sleep jobs in 2 queues, 1 with preemption enabled, 1 without. Preemption 
occurred, but preemption was done unnecessarily because containers were 
aggressively given back to preempted queue (see YARN-3769).
- Verified resource metrics on Application UI were correct by comparing against 
time spent in each container as calculated by the timeline server.
- Wordcount test

-Eric Payne
 
  From: Sangjin Lee <sj...@apache.org>
 To: "common-...@hadoop.apache.org" <common-...@hadoop.apache.org>; 
"yarn-...@hadoop.apache.org" <yarn-...@hadoop.apache.org>; 
"hdfs-...@hadoop.apache.org" <hdfs-...@hadoop.apache.org>; 
"mapreduce-dev@hadoop.apache.org" <mapreduce-dev@hadoop.apache.org> 
Cc: Vinod Kumar Vavilapalli <vino...@apache.org> 
 Sent: Thursday, October 22, 2015 4:14 PM
 Subject: [VOTE] Release Apache Hadoop 2.6.2
   
Hi all,

I have created a release candidate (RC0) for Hadoop 2.6.2.

The RC is available at: http://people.apache.org/~sjlee/hadoop-2.6.2-RC0/

The RC tag in git is: release-2.6.2-RC0

The list of JIRAs committed for 2.6.2:
https://issues.apache.org/jira/browse/YARN-4101?jql=project%20in%20(HADOOP%2C%20HDFS%2C%20YARN%2C%20MAPREDUCE)%20AND%20fixVersion%20%3D%202.6.2

The maven artifacts are staged at
https://repository.apache.org/content/repositories/orgapachehadoop-1022/

Please try out the release candidate and vote. The vote will run for 5 days.

Thanks,
Sangjin


   
 

Re: [VOTE] Release Apache Hadoop 2.6.1 RC1

[Eric Payne]

Hi Vinod and everyone else who helped on this release!

Thank you very much for going through the work and effort to put this release 
together.
While doing my testing for Hadoop 2.6.1 RC1, I encountered the following 
issue:[MAPREDUCE-6334] Fetcher#copyMapOutput is leaking usedMemory upon 
IOException during InMemoryMapOutput shuffle handler - ASF JIRA

|   |
|   |   |   |   |   |
| [MAPREDUCE-6334] Fetcher#copyMapOutput is leaking usedMemory upon IOException 
during ...We are seeing this happen when an NM's disk goes bad during the 
creation of map output(s) the reducer's fetcher can read the shuffle header and 
reserve the memory  |
|  |
| View on issues.apache.org | Preview by Yahoo |
|  |
|   |


This is actually something we ran across frequently when running 2.6. You may 
want to consider pulling that in to 2.6.
Other than that, it looks fine. I did the following manual testing on a 
one-node cluster:
- successfully downloaded and compiled the code- successfully ran streaming 
jobs- successfully ran distributed shell jobs-- as part of the distributed 
shell testing, I started the jobs with the 
'-keep_containers_across_application_attempts' property set, killed the AM 
container, and verified that it would in fact keep the containers running 
across AM restarts.- sucessfully ran wordcount jobs.
Thank you,
-Eric Payne
 
  From: Vinod Kumar Vavilapalli <vino...@apache.org>
 To: common-...@hadoop.apache.org; yarn-...@hadoop.apache.org; 
hdfs-...@hadoop.apache.org; mapreduce-dev@hadoop.apache.org 
Cc: vino...@apache.org; Sangjin Lee <sjl...@gmail.com>; Akira AJISAKA 
<ajisa...@oss.nttdata.co.jp> 
 Sent: Wednesday, September 16, 2015 9:10 PM
 Subject: [VOTE] Release Apache Hadoop 2.6.1 RC1
   
Hi all,

After a nearly month long [1] toil, with loads of help from Sangjin Lee and
Akira Ajisaka, and 153 (RC0)+7(RC1) commits later, I've created a release
candidate RC1 for hadoop-2.6.1.

RC1 is RC0 [0] (for which I opened and closed a vote last week) + UI fixes
for the issue Sangjin raised (YARN-3171 and the dependencies YARN-3779,
YARN-3248), additional fix to avoid incompatibility (YARN-3740), other UI
bugs (YARN-1884, YARN-3544) and the MiniYARNCluster issue (right patch for
YARN-2890) that Jeff Zhang raised.

The RC is available at: http://people.apache.org/~vinodkv/hadoop-2.6.1-RC1/

The RC tag in git is: release-2.6.1-RC1

The maven artifacts are available via repository.apache.org at
https://repository.apache.org/content/repositories/orgapachehadoop-1021

Some notes from our release process
 -  - Sangjin and I moved out a bunch of items pending from 2.6.1 [2] -
non-committed but desired patches. 2.6.1 is already big as is and is late
by any standard, we can definitely include them in the next release.
 - The 2.6.1 wiki page [3] captures some (but not all) of the context of
the patches that we pushed in.
 - Given the number of fixes pushed [4] in, we had to make a bunch of
changes to our original plan - we added a few improvements that helped us
backport patches easier (or in many cases made backports possible), and we
dropped a few that didn't make sense (HDFS-7831, HDFS-7926, HDFS-7676,
HDFS-7611, HDFS-7843, HDFS-8850).
 - I ran all the unit tests which (surprisingly?) passed. (Except for one,
which pointed out a missing fix HDFS-7552).

As discussed before [5]
 - This release is the first point release after 2.6.0
 - I’d like to use this as a starting release for 2.6.2 in a few weeks and
then follow up with more of these.

Please try the release and vote; the vote will run for the usual 5 days.

Thanks,
Vinod

[0] Hadoop 2.6.1 RC0 vote: http://markmail.org/thread/ubut2rn3lodc55iy
[1] Hadoop 2.6.1 Release process thread:
http://markmail.org/thread/wkbgkxkhntx5tlux
[2] 2.6.1 Pending tickets:
https://issues.apache.org/jira/issues/?filter=12331711
[3] 2.6.1 Wiki page: https://wiki.apache.org/hadoop/Release-2.6.1
-Working-Notes
[4] List of 2.6.1 patches pushed:
https://issues.apache.org/jira/issues/?jql=fixVersion%20%3D%202.6.1
%20and%20labels%20%3D%20%222.6.1-candidate%22
[5] Planning Hadoop 2.6.1 release:
http://markmail.org/thread/sbykjn5xgnksh6wg

PS:
 - Note that branch-2.6 which will be the base for 2.6.2 doesn't have these
fixes yet. Once 2.6.1 goes through, I plan to rebase branch-2.6 based off
2.6.1.
 - The additional patches in RC1 that got into 2.6.1 all the way from 2.8
are NOT in 2.7.2 yet, this will be done as a followup.

   

Re: [VOTE] Release Apache Hadoop 2.7.1 RC0

[Eric Payne]

+1 (non-binding)
+ Downloaded and built source+ Installed on one-node cluster+ Ran simple manual 
tests+ spot-checked unit tests
Thanks, Vinod, for managing this release!-Eric Payne
 
  From: Vinod Kumar Vavilapalli vino...@apache.org
 To: common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
yarn-...@hadoop.apache.org; mapreduce-dev@hadoop.apache.org 
Cc: vino...@apache.org 
 Sent: Monday, June 29, 2015 3:45 AM
 Subject: [VOTE] Release Apache Hadoop 2.7.1 RC0
   
Hi all,

I've created a release candidate RC0 for Apache Hadoop 2.7.1.

As discussed before, this is the next stable release to follow up 2.6.0,
and the first stable one in the 2.7.x line.

The RC is available for validation at:
*http://people.apache.org/~vinodkv/hadoop-2.7.1-RC0/
http://people.apache.org/~vinodkv/hadoop-2.7.1-RC0/*

The RC tag in git is: release-2.7.1-RC0

The maven artifacts are available via repository.apache.org at
*https://repository.apache.org/content/repositories/orgapachehadoop-1019/
https://repository.apache.org/content/repositories/orgapachehadoop-1019/*

Please try the release and vote; the vote will run for the usual 5 days.

Thanks,
Vinod

PS: It took 2 months instead of the planned [1] 2 weeks in getting this
release out: post-mortem in a separate thread.

[1]: A 2.7.1 release to follow up 2.7.0
http://markmail.org/thread/zwzze6cqqgwq4rmw


   

Re: Bug Bash Post-mortem

[Eric Payne]

Thank you Allen and Barbara for organizing the bug bash and for the 
post-mortem. I would definitely like to have another one in the fall.
 
  From: Allen Wittenauer a...@altiscale.com
 To: common-...@hadoop.apache.org common-...@hadoop.apache.org 
Cc: mapreduce-dev@hadoop.apache.org mapreduce-dev@hadoop.apache.org; 
hdfs-...@hadoop.apache.org hdfs-...@hadoop.apache.org; 
yarn-...@hadoop.apache.org yarn-...@hadoop.apache.org 
 Sent: Friday, May 15, 2015 8:05 PM
 Subject: Bug Bash Post-mortem
   
    
Hey gang.

    I promised a post-mortem, but I got distracted with making everyone's lives 
miserable with new test-patch code.  Luckily my coworker Barbara (who some of 
you met!) wrote some wrap up info about it on our corporate blog, including 
some interesting stats….

    
https://www.altiscale.com/hadoop-blog/the-power-of-community-apache-hadoop-global-bug-bash/



   

Re: [VOTE] Release Apache Hadoop 2.7.0 RC0

[Eric Payne]

+1 (non-binding)
Thanks,Vinod, for the time and effort you put into creating this release.

I downloaded and built the source for 2.7.0, RC0. I started a local cluster and 
ran wordcount, sleep, and streaming jobs.

In addition, I did the following:
    o I ran a distributed shell job which tested preserving containers across 
AM restarts. I did this by setting the 
-keep_containers_across_application_attempts flag and killing the first AM once 
the containers started. I checked the results by querying the timeline server 
and comparing the start and end times of the non-AM containers against the 
start and end times of the later AM containers. This verified that the non-AM 
containers started during the first AM's run and finished after the first AM 
had been killed and after the second AM had started.

    o I enabled the preemption feature and verified that containers were 
preempted from preemptable queues and not from non-preemptable ones.
    o I ran unit tests for hadoop-yarn-server-resourcemanage. All passed with 
the exception of TestFifoScheduler#testResourceOverCommit. 
    o I ran unit tests for hadoop-yarn-server-nodemanager. All passed with the 
exception of TestResourceLocalizationService#testFailedDirsResourceRelease.

    o I ran unit tests for hadoop-hdfs. All passed with the exception of 
TestReplaceDatanodeOnFailure#testReplaceDatanodeOnFailure,TestBalancer#testUnknownDatanode,
 TestBPOfferService#testBasicFunctionality, and 
TestWebHdfsFileSystemContract#testGetFileBlockLocations.
  Thank you,-Eric Payne
 



  From: Vinod Kumar Vavilapalli vino...@apache.org
 To: common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
yarn-...@hadoop.apache.org; mapreduce-dev@hadoop.apache.org 
Cc: vino...@apache.org 
 Sent: Friday, April 10, 2015 6:44 PM
 Subject: [VOTE] Release Apache Hadoop 2.7.0 RC0
   
Hi all,

I've created a release candidate RC0 for Apache Hadoop 2.7.0.

 The RC is available at: http://people.apache.org/~vinodkv/hadoop-2.7.0-RC0/

The RC tag in git is: release-2.7.0-RC0

 The maven artifacts are available via repository.apache.org at
https://repository.apache.org/content/repositories/orgapachehadoop-1017/

As discussed before
 - This release will only work with JDK 1.7 and above
 - I’d like to use this as a starting release for 2.7.x [1], depending on
how it goes, get it stabilized and potentially use a 2.7.1 in a few
weeks as the stable release.

 Please try the release and vote; the vote will run for the usual 5 days.

 Thanks,
 Vinod

 [1]: A 2.7.1 release to follow up 2.7.0
http://markmail.org/thread/zwzze6cqqgwq4rmw

   

[jira] [Resolved] (MAPREDUCE-3286) Unit tests for MAPREDUCE-3186 - User jobs are getting hanged if the Resource manager process goes down and comes up while job is getting executed.

[Eric Payne (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-3286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eric Payne resolved MAPREDUCE-3286.
---
  Resolution: Invalid
Target Version/s:   (was: )
Release Note:   (was: New Yarn configuration property:

Name: yarn.app.mapreduce.am.scheduler.connection.retries
Description: Number of times AM should retry to contact RM if connection is 
lost.)

RM has been refactored and restructured a few times in the past 3.5 years.

Closing as invalid.

 Unit tests for MAPREDUCE-3186 - User jobs are getting hanged if the Resource 
 manager process goes down and comes up while job is getting executed.
 --

 Key: MAPREDUCE-3286
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3286
 Project: Hadoop Map/Reduce
  Issue Type: Test
  Components: mrv2
Affects Versions: 0.23.0
 Environment: linux
Reporter: Eric Payne
Assignee: Eric Payne
  Labels: test

 If the resource manager is restarted while the job execution is in progress, 
 the job is getting hanged.
 UI shows the job as running.
 In the RM log, it is throwing an error ERROR 
 org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService: 
 AppAttemptId doesnt exist in cache appattempt_1318579738195_0004_01
 In the console MRAppMaster and Runjar processes are not getting killed



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (MAPREDUCE-6174) Combine common stream code into parent class for InMemoryMapOutput and OnDiskMapOutput.

[Eric Payne (JIRA)]

Eric Payne created MAPREDUCE-6174:
-

 Summary: Combine common stream code into parent class for 
InMemoryMapOutput and OnDiskMapOutput.
 Key: MAPREDUCE-6174
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6174
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
  Components: mrv2
Affects Versions: 3.0.0, 2.6.0
Reporter: Eric Payne
Assignee: Eric Payne


Once MAPREDUCE-6166 is complete, both InMemoryMapOutput and OnDiskMapOutput 
will be doing similar things with regards to IFile streams.

In order to make it explicit that InMemoryMapOutput and OnDiskMapOutput are 
different from 3rd-party implementations, this JIRA will make them subclass a 
common class (see 
https://issues.apache.org/jira/browse/MAPREDUCE-6166?focusedCommentId=14223368page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14223368)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [VOTE] Release Apache Hadoop 2.6.0

[Eric Payne]

+1 . Thanks Arun, for producing this release. I downloaded and built the 
source. I started local cluster and ran wordcount, sleep, and streaming jobs.

 - I ran a distributed shell job which tested preserving containers across AM 
restart by setting the -keep_containers_across_application_attempts flag and 
killing the first AM once the containers start. I checked the results by 
looking in the timeline server and comparing the start times of the non-AM 
containers against the start times of the later AM container. - I enabled the 
preemption feature and verified containers were preempted and queues were 
adjusted to guaranteed levels.
- I ran unit tests for hadoop-yarn-server-resourcemanage. All passed with the 
exception of TestContainerResourceUsage. - I ran unit tests for hadoop-hdfs. 
All passed with the exception of TestBPOfferService#testBasicFunctionality 
(HDFS-3930) Thank you,-Eric Payne

 


  From: Arun C Murthy a...@hortonworks.com
 To: common-...@hadoop.apache.org common-...@hadoop.apache.org; 
hdfs-...@hadoop.apache.org hdfs-...@hadoop.apache.org; 
yarn-...@hadoop.apache.org yarn-...@hadoop.apache.org; 
mapreduce-dev@hadoop.apache.org mapreduce-dev@hadoop.apache.org 
 Sent: Thursday, November 13, 2014 5:08 PM
 Subject: [VOTE] Release Apache Hadoop 2.6.0 
   
Folks,

I've created another release candidate (rc1) for hadoop-2.6.0 based on the 
feedback.

The RC is available at: http://people.apache.org/~acmurthy/hadoop-2.6.0-rc1
The RC tag in git is: release-2.6.0-rc1

The maven artifacts are available via repository.apache.org at 
https://repository.apache.org/content/repositories/orgapachehadoop-1013.

Please try the release and vote; the vote will run for the usual 5 days.

thanks,
Arun


-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.


   

Re: [VOTE] Release Apache Hadoop 2.6.0

[Eric Payne]

+1
I downloaded and built source. Started local cluster and ran wordcount, sleep, 
and simple streaming job.

Also, I ran a distributed shell job which tested preserving containers across 
AM restart by setting the -keep_containers_across_application_attempts flag and 
killing the first AM once the containers start.
Enabled the preemption feature and verified containers were preempted and 
queues were levelized.
Ran unit tests for hadoop-yarn-server-resourcemanagerRan unit tests for 
hadoop-hdfs
 Thank you,-Eric Payne

 
  From: Arun C Murthy a...@hortonworks.com
 To: common-...@hadoop.apache.org common-...@hadoop.apache.org; 
hdfs-...@hadoop.apache.org hdfs-...@hadoop.apache.org; 
yarn-...@hadoop.apache.org yarn-...@hadoop.apache.org; 
mapreduce-dev@hadoop.apache.org mapreduce-dev@hadoop.apache.org 
 Sent: Monday, November 10, 2014 8:52 PM
 Subject: [VOTE] Release Apache Hadoop 2.6.0
   
Folks,

I've created a release candidate (rc0) for hadoop-2.6.0 that I would like to 
see released.

The RC is available at: http://people.apache.org/~acmurthy/hadoop-2.6.0-rc0
The RC tag in git is: release-2.6.0-rc0

The maven artifacts are available via repository.apache.org at 
https://repository.apache.org/content/repositories/orgapachehadoop-1012.

Please try the release and vote; the vote will run for the usual 5 days.

thanks,
Arun


-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.


   

[jira] [Created] (MAPREDUCE-3975) Default value not set for Configuration parameter mapreduce.job.local.dir

[Eric Payne (Created) (JIRA)]

Default value not set for Configuration parameter mapreduce.job.local.dir
-

 Key: MAPREDUCE-3975
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3975
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mrv2
Affects Versions: 0.23.1, 0.23.2
Reporter: Eric Payne
Assignee: Eric Payne
Priority: Blocker


mapreduce.job.local.dir (formerly job.local.dir in 0.20) is not set by default. 
This is a regression from 0.20.205.

In 0.20.205, JobLocalizer.createWorkDir() constructs the 
$mapred.local.dir/taskTracker/$user/jobcache/$jobid/work path based on $user 
and $jobid, and then sets TaskTracker.JOB_LOCAL_DIR in the job's JobConf.

So far, I haven't found where this is done in 0.23. It could be that this is 
what should be done by LocalJobRunner.setupChildMapredLocalDirs(), but I am 
still investigating.



--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (MAPREDUCE-3856) Instances of RunningJob class givs incorrect job tracking urls when mutiple jobs are submitted from same client jvm.

[Eric Payne (Created) (JIRA)]

Instances of RunningJob class givs incorrect job tracking urls when mutiple 
jobs are submitted from same client jvm.


 Key: MAPREDUCE-3856
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3856
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mrv2
Affects Versions: 0.23.1, 0.23.2
Reporter: Eric Payne
Assignee: Eric Payne
Priority: Critical


When multiple jobs are submitted from the same client JVM, each call to 
RunningJob.getTrackingURL() always returns the tracking URL from the first job.

This happens even if the jobs are submitted and the client waits for the job to 
complete before submitting the subsequent job. Each job runs fine and is 
definitely a new, unique job, but the call to getTrackingURL() still returns 
the URL for the first job.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (MAPREDUCE-3456) $HADOOP_PREFIX/bin/yarn should set defaults for $HADOOP_*_HOME

[Eric Payne (Created) (JIRA)]

$HADOOP_PREFIX/bin/yarn should set defaults for $HADOOP_*_HOME
--

 Key: MAPREDUCE-3456
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3456
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mrv2
Affects Versions: 0.23.0
Reporter: Eric Payne


If the $HADOOP_PREFIX/hadoop-dist/target/hadoop-0.23.0-SNAPSHOT.tar.gz tarball 
is used to distribute hadoop, all of the HADOOP components (HDFS, MAPRED, 
COMMON) are all under one directory. In this use case, HADOOP_PREFIX should be 
set and should point to the root directory for all components, and it should 
not be necessary to set HADOOP_HDFS_HOME, HADOOP_COMMON_HOME, and 
HADOOP_MAPRED_HOME. However, the $HADOOP_PREFIX/bin/yarn script requires these 
3 to be set explicitly in the calling environment or it won't run.

$HADOOP_PREFIX/bin/yarn should check if $HADOOP_PREFIX is set and, if it is, 
use that value for the 3 other HADOOP_*_HOME variables.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (MAPREDUCE-3286) Unit tests for MAPREDUCE-3186 - User jobs are getting hanged if the Resource manager process goes down and comes up while job is getting executed.

[Eric Payne (Created) (JIRA)]

Unit tests for MAPREDUCE-3186 - User jobs are getting hanged if the Resource 
manager process goes down and comes up while job is getting executed.
--

 Key: MAPREDUCE-3286
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3286
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mrv2
Affects Versions: 0.23.0
 Environment: linux
Reporter: Eric Payne
Assignee: Eric Payne
Priority: Blocker


If the resource manager is restarted while the job execution is in progress, 
the job is getting hanged.
UI shows the job as running.
In the RM log, it is throwing an error ERROR 
org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService: 
AppAttemptId doesnt exist in cache appattempt_1318579738195_0004_01
In the console MRAppMaster and Runjar processes are not getting killed

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira