date:20170123

hadoop git commit: HDFS-11352. Potential deadlock in NN when failing over. Contributed by Eric Krogen.

2017-01-23 Thread aajisaka

Repository: hadoop
Updated Branches:
  refs/heads/branch-2.6 67065e228 -> 8aff157e2


HDFS-11352. Potential deadlock in NN when failing over. Contributed by Eric 
Krogen.


Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/8aff157e
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/8aff157e
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/8aff157e

Branch: refs/heads/branch-2.6
Commit: 8aff157e2db814faca830e53c9eaadae726ef81a
Parents: 67065e2
Author: Akira Ajisaka 
Authored: Tue Jan 24 15:02:20 2017 +0900
Committer: Akira Ajisaka 
Committed: Tue Jan 24 15:02:20 2017 +0900

--
 hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt   | 3 +++
 .../java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java | 2 +-
 2 files changed, 4 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/hadoop/blob/8aff157e/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
--
diff --git a/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt 
b/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
index 9b06f2f..4b17365 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
+++ b/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
@@ -18,6 +18,9 @@ Release 2.6.6 - UNRELEASED
 HDFS-11290. TestFSNameSystemMBean should wait until JMX cache is cleared.
 (Eric Krogen via aajisaka)
 
+HDFS-11352. Potential deadlock in NN when failing over.
+(Eric Krogen via aajisaka)
+
 Release 2.6.5 - 2016-10-08
 
   INCOMPATIBLE CHANGES

http://git-wip-us.apache.org/repos/asf/hadoop/blob/8aff157e/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
--
diff --git 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
index ba92508..c88080a 100644
--- 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
+++ 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
@@ -5368,7 +5368,7 @@ public class FSNamesystem implements Namesystem, 
FSClusterStats,
   @Metric({"TransactionsSinceLastCheckpoint",
   "Number of transactions since last checkpoint"})
   public long getTransactionsSinceLastCheckpoint() {
-return getEditLog().getLastWrittenTxId() -
+return getEditLog().getLastWrittenTxIdWithoutLock() -
 getFSImage().getStorage().getMostRecentCheckpointTxId();
   }
   


-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org

hadoop git commit: HDFS-11352. Add missing entry in CHANGES.txt.

2017-01-23 Thread aajisaka

Repository: hadoop
Updated Branches:
  refs/heads/branch-2.7 3acc96edb -> ab6248424


HDFS-11352. Add missing entry in CHANGES.txt.


Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/ab624842
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/ab624842
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/ab624842

Branch: refs/heads/branch-2.7
Commit: ab624842402847dfd2c2147c197730da9fdfcdc2
Parents: 3acc96e
Author: Akira Ajisaka 
Authored: Tue Jan 24 14:57:21 2017 +0900
Committer: Akira Ajisaka 
Committed: Tue Jan 24 14:57:21 2017 +0900

--
 hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt | 3 +++
 1 file changed, 3 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/hadoop/blob/ab624842/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
--
diff --git a/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt 
b/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
index c4580d9..9455f7b 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
+++ b/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
@@ -216,6 +216,9 @@ Release 2.7.4 - UNRELEASED
 HDFS-11290. TestFSNameSystemMBean should wait until JMX cache is cleared.
 (Eric Krogen via aajisaka)
 
+HDFS-11352. Potential deadlock in NN when failing over.
+(Eric Krogen via aajisaka)
+
 Release 2.7.3 - 2016-08-25
 
   INCOMPATIBLE CHANGES


-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org

hadoop git commit: HDFS-11352. Potential deadlock in NN when failing over. Contributed by Erik Krogen.

2017-01-23 Thread aajisaka

Repository: hadoop
Updated Branches:
  refs/heads/branch-2.7 b0b396685 -> 3acc96edb


HDFS-11352. Potential deadlock in NN when failing over. Contributed by Erik 
Krogen.


Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/3acc96ed
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/3acc96ed
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/3acc96ed

Branch: refs/heads/branch-2.7
Commit: 3acc96edbf3f389a780face80d778c57087199d3
Parents: b0b3966
Author: Akira Ajisaka 
Authored: Tue Jan 24 14:55:27 2017 +0900
Committer: Akira Ajisaka 
Committed: Tue Jan 24 14:55:27 2017 +0900

--
 .../java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java  | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/hadoop/blob/3acc96ed/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
--
diff --git 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
index c7f8508..96bcfef 100644
--- 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
+++ 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
@@ -4768,7 +4768,7 @@ public class FSNamesystem implements Namesystem, 
FSNamesystemMBean,
   @Metric({"TransactionsSinceLastCheckpoint",
   "Number of transactions since last checkpoint"})
   public long getTransactionsSinceLastCheckpoint() {
-return getEditLog().getLastWrittenTxId() -
+return getEditLog().getLastWrittenTxIdWithoutLock() -
 getFSImage().getStorage().getMostRecentCheckpointTxId();
   }
   


-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org

hadoop git commit: HDFS-11290. TestFSNameSystemMBean should wait until JMX cache is cleared. Contributed by Erik Krogen.

2017-01-23 Thread aajisaka

Repository: hadoop
Updated Branches:
  refs/heads/branch-2.6 a4e062ce9 -> 67065e228


HDFS-11290. TestFSNameSystemMBean should wait until JMX cache is cleared. 
Contributed by Erik Krogen.

(cherry picked from commit b1a9ec856b572894e769f052aea2340fc3f23c78)
(cherry picked from commit db47bd15dd80a4a5727607832c16635cb0ec7726)


Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/67065e22
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/67065e22
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/67065e22

Branch: refs/heads/branch-2.6
Commit: 67065e228c1c5416dd225466ed6750bcf7520916
Parents: a4e062c
Author: Akira Ajisaka 
Authored: Wed Jan 18 17:53:21 2017 +0900
Committer: Akira Ajisaka 
Committed: Tue Jan 24 14:48:00 2017 +0900

--
 hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt   |  3 +++
 .../hdfs/server/namenode/TestFSNamesystemMBean.java   | 10 ++
 2 files changed, 13 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/hadoop/blob/67065e22/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
--
diff --git a/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt 
b/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
index b884f7f..9b06f2f 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
+++ b/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
@@ -15,6 +15,9 @@ Release 2.6.6 - UNRELEASED
 HDFS-11180. Intermittent deadlock in NameNode when failover happens.
 (aajisaka)
 
+HDFS-11290. TestFSNameSystemMBean should wait until JMX cache is cleared.
+(Eric Krogen via aajisaka)
+
 Release 2.6.5 - 2016-10-08
 
   INCOMPATIBLE CHANGES

http://git-wip-us.apache.org/repos/asf/hadoop/blob/67065e22/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSNamesystemMBean.java
--
diff --git 
a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSNamesystemMBean.java
 
b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSNamesystemMBean.java
index 853bb53..0341ca3 100644
--- 
a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSNamesystemMBean.java
+++ 
b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSNamesystemMBean.java
@@ -32,6 +32,8 @@ import javax.management.ObjectName;
 
 import org.apache.hadoop.conf.Configuration;
 import org.apache.hadoop.hdfs.MiniDFSCluster;
+import org.apache.hadoop.metrics2.impl.ConfigBuilder;
+import org.apache.hadoop.metrics2.impl.TestMetricsConfig;
 import org.junit.Test;
 import org.mortbay.util.ajax.JSON;
 
@@ -132,12 +134,16 @@ public class TestFSNamesystemMBean {
 MiniDFSCluster cluster = null;
 FSNamesystem fsn = null;
 
+int jmxCachePeriod = 1;
+new ConfigBuilder().add("namenode.period", jmxCachePeriod)
+.save(TestMetricsConfig.getTestFilename("hadoop-metrics2-namenode"));
 try {
   cluster = new MiniDFSCluster.Builder(conf).build();
   cluster.waitActive();
 
   fsn = cluster.getNameNode().namesystem;
   fsn.writeLock();
+  Thread.sleep(jmxCachePeriod * 1000);
 
   MBeanClient client = new MBeanClient();
   client.start();
@@ -160,11 +166,15 @@ public class TestFSNamesystemMBean {
   @Test
   public void testWithFSEditLogLock() throws Exception {
 Configuration conf = new Configuration();
+int jmxCachePeriod = 1;
+new ConfigBuilder().add("namenode.period", jmxCachePeriod)
+.save(TestMetricsConfig.getTestFilename("hadoop-metrics2-namenode"));
 MiniDFSCluster cluster = null;
 try {
   cluster = new MiniDFSCluster.Builder(conf).build();
   cluster.waitActive();
   synchronized (cluster.getNameNode().getFSImage().getEditLog()) {
+Thread.sleep(jmxCachePeriod * 1000);
 MBeanClient client = new MBeanClient();
 client.start();
 client.join(2);


-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org

hadoop git commit: YARN-6082. Invalid REST api response for getApps since queueUsagePercentage is coming as INF. Contributed by Sunil G.

2017-01-23 Thread rohithsharmaks

Repository: hadoop
Updated Branches:
  refs/heads/branch-2 94b326f2c -> 26c4cfb65


YARN-6082. Invalid REST api response for getApps since queueUsagePercentage is 
coming as INF. Contributed by Sunil G.

(cherry picked from commit 0101267d9d801eab4cb3b4df289c402ecb591685)


Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/26c4cfb6
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/26c4cfb6
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/26c4cfb6

Branch: refs/heads/branch-2
Commit: 26c4cfb653c587ac1f9dd24f4ea3c279a46c745b
Parents: 94b326f
Author: Rohith Sharma K S 
Authored: Tue Jan 24 10:59:55 2017 +0530
Committer: Rohith Sharma K S 
Committed: Tue Jan 24 11:01:52 2017 +0530

--
 .../scheduler/SchedulerApplicationAttempt.java  |  9 ++---
 .../scheduler/TestSchedulerApplicationAttempt.java  | 12 
 2 files changed, 18 insertions(+), 3 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/hadoop/blob/26c4cfb6/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerApplicationAttempt.java
--
diff --git 
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerApplicationAttempt.java
 
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerApplicationAttempt.java
index aab6811..5647ebd 100644
--- 
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerApplicationAttempt.java
+++ 
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerApplicationAttempt.java
@@ -901,9 +901,12 @@ public class SchedulerApplicationAttempt implements 
SchedulableEntity {
   float queueUsagePerc = 0.0f;
   float clusterUsagePerc = 0.0f;
   if (!calc.isInvalidDivisor(cluster)) {
-queueUsagePerc = calc.divide(cluster, usedResourceClone, Resources
-.multiply(cluster, queue.getQueueInfo(false, false).getCapacity()))
-* 100;
+float queueCapacityPerc = queue.getQueueInfo(false, false)
+.getCapacity();
+if (queueCapacityPerc != 0) {
+  queueUsagePerc = calc.divide(cluster, usedResourceClone,
+  Resources.multiply(cluster, queueCapacityPerc)) * 100;
+}
 clusterUsagePerc = calc.divide(cluster, usedResourceClone, cluster)
 * 100;
   }

http://git-wip-us.apache.org/repos/asf/hadoop/blob/26c4cfb6/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/TestSchedulerApplicationAttempt.java
--
diff --git 
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/TestSchedulerApplicationAttempt.java
 
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/TestSchedulerApplicationAttempt.java
index ff5dc02..0a3075e 100644
--- 
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/TestSchedulerApplicationAttempt.java
+++ 
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/TestSchedulerApplicationAttempt.java
@@ -228,6 +228,18 @@ public class TestSchedulerApplicationAttempt {
 0.01f);
 assertEquals(60.0f,
 app.getResourceUsageReport().getClusterUsagePercentage(), 0.01f);
+
+queue = createQueue("test3", null, 0.0f);
+app = new SchedulerApplicationAttempt(appAttId, user, queue,
+queue.getActiveUsersManager(), rmContext);
+
+// Resource request
+app.attemptResourceUsage.incUsed(requestedResource);
+
+assertEquals(0.0f, app.getResourceUsageReport().getQueueUsagePercentage(),
+0.01f);
+assertEquals(15.0f,
+app.getResourceUsageReport().getClusterUsagePercentage(), 0.01f);
   }
 
   @Test


-
To unsubscribe, e-mail:

hadoop git commit: YARN-6082. Invalid REST api response for getApps since queueUsagePercentage is coming as INF. Contributed by Sunil G.

2017-01-23 Thread rohithsharmaks

Repository: hadoop
Updated Branches:
  refs/heads/trunk dc6ec9704 -> 0101267d9


YARN-6082. Invalid REST api response for getApps since queueUsagePercentage is 
coming as INF. Contributed by Sunil G.


Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/0101267d
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/0101267d
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/0101267d

Branch: refs/heads/trunk
Commit: 0101267d9d801eab4cb3b4df289c402ecb591685
Parents: dc6ec97
Author: Rohith Sharma K S 
Authored: Tue Jan 24 10:59:55 2017 +0530
Committer: Rohith Sharma K S 
Committed: Tue Jan 24 10:59:55 2017 +0530

--
 .../scheduler/SchedulerApplicationAttempt.java  |  9 ++---
 .../scheduler/TestSchedulerApplicationAttempt.java  | 12 
 2 files changed, 18 insertions(+), 3 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/hadoop/blob/0101267d/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerApplicationAttempt.java
--
diff --git 
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerApplicationAttempt.java
 
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerApplicationAttempt.java
index 7558eac..3e41cb4 100644
--- 
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerApplicationAttempt.java
+++ 
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerApplicationAttempt.java
@@ -1037,9 +1037,12 @@ public class SchedulerApplicationAttempt implements 
SchedulableEntity {
   float queueUsagePerc = 0.0f;
   float clusterUsagePerc = 0.0f;
   if (!calc.isInvalidDivisor(cluster)) {
-queueUsagePerc = calc.divide(cluster, usedResourceClone, Resources
-.multiply(cluster, queue.getQueueInfo(false, false).getCapacity()))
-* 100;
+float queueCapacityPerc = queue.getQueueInfo(false, false)
+.getCapacity();
+if (queueCapacityPerc != 0) {
+  queueUsagePerc = calc.divide(cluster, usedResourceClone,
+  Resources.multiply(cluster, queueCapacityPerc)) * 100;
+}
 clusterUsagePerc = calc.divide(cluster, usedResourceClone, cluster)
 * 100;
   }

http://git-wip-us.apache.org/repos/asf/hadoop/blob/0101267d/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/TestSchedulerApplicationAttempt.java
--
diff --git 
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/TestSchedulerApplicationAttempt.java
 
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/TestSchedulerApplicationAttempt.java
index 9a6c8d4..c5e5183 100644
--- 
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/TestSchedulerApplicationAttempt.java
+++ 
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/TestSchedulerApplicationAttempt.java
@@ -227,6 +227,18 @@ public class TestSchedulerApplicationAttempt {
 0.01f);
 assertEquals(60.0f,
 app.getResourceUsageReport().getClusterUsagePercentage(), 0.01f);
+
+queue = createQueue("test3", null, 0.0f);
+app = new SchedulerApplicationAttempt(appAttId, user, queue,
+queue.getActiveUsersManager(), rmContext);
+
+// Resource request
+app.attemptResourceUsage.incUsed(requestedResource);
+
+assertEquals(0.0f, app.getResourceUsageReport().getQueueUsagePercentage(),
+0.01f);
+assertEquals(15.0f,
+app.getResourceUsageReport().getClusterUsagePercentage(), 0.01f);
   }
 
   @Test


-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands,

hadoop git commit: HDFS-11290. TestFSNameSystemMBean should wait until JMX cache is cleared. Contributed by Erik Krogen.

2017-01-23 Thread aajisaka

Repository: hadoop
Updated Branches:
  refs/heads/branch-2.7 b13492a0a -> b0b396685


HDFS-11290. TestFSNameSystemMBean should wait until JMX cache is cleared. 
Contributed by Erik Krogen.

(cherry picked from commit b1a9ec856b572894e769f052aea2340fc3f23c78)
(cherry picked from commit db47bd15dd80a4a5727607832c16635cb0ec7726)


Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/b0b39668
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/b0b39668
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/b0b39668

Branch: refs/heads/branch-2.7
Commit: b0b39668519dfb7066da6a5f0c6163fd106a6946
Parents: b13492a
Author: Akira Ajisaka 
Authored: Wed Jan 18 17:53:21 2017 +0900
Committer: Akira Ajisaka 
Committed: Tue Jan 24 14:19:21 2017 +0900

--
 hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt   |  3 +++
 .../hdfs/server/namenode/TestFSNamesystemMBean.java   | 10 ++
 2 files changed, 13 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/hadoop/blob/b0b39668/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
--
diff --git a/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt 
b/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
index ad49975..c4580d9 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
+++ b/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
@@ -213,6 +213,9 @@ Release 2.7.4 - UNRELEASED
 HDFS-11263. ClassCastException when we use Bzipcodec for Fsimage
 compression. (Brahma Reddy Battula)
 
+HDFS-11290. TestFSNameSystemMBean should wait until JMX cache is cleared.
+(Eric Krogen via aajisaka)
+
 Release 2.7.3 - 2016-08-25
 
   INCOMPATIBLE CHANGES

http://git-wip-us.apache.org/repos/asf/hadoop/blob/b0b39668/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSNamesystemMBean.java
--
diff --git 
a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSNamesystemMBean.java
 
b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSNamesystemMBean.java
index 764c627..ef016ec 100644
--- 
a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSNamesystemMBean.java
+++ 
b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSNamesystemMBean.java
@@ -32,6 +32,8 @@ import javax.management.ObjectName;
 
 import org.apache.hadoop.conf.Configuration;
 import org.apache.hadoop.hdfs.MiniDFSCluster;
+import org.apache.hadoop.metrics2.impl.ConfigBuilder;
+import org.apache.hadoop.metrics2.impl.TestMetricsConfig;
 import org.junit.Test;
 import org.mortbay.util.ajax.JSON;
 
@@ -137,12 +139,16 @@ public class TestFSNamesystemMBean {
 MiniDFSCluster cluster = null;
 FSNamesystem fsn = null;
 
+int jmxCachePeriod = 1;
+new ConfigBuilder().add("namenode.period", jmxCachePeriod)
+.save(TestMetricsConfig.getTestFilename("hadoop-metrics2-namenode"));
 try {
   cluster = new MiniDFSCluster.Builder(conf).build();
   cluster.waitActive();
 
   fsn = cluster.getNameNode().namesystem;
   fsn.writeLock();
+  Thread.sleep(jmxCachePeriod * 1000);
 
   MBeanClient client = new MBeanClient();
   client.start();
@@ -165,11 +171,15 @@ public class TestFSNamesystemMBean {
   @Test
   public void testWithFSEditLogLock() throws Exception {
 Configuration conf = new Configuration();
+int jmxCachePeriod = 1;
+new ConfigBuilder().add("namenode.period", jmxCachePeriod)
+.save(TestMetricsConfig.getTestFilename("hadoop-metrics2-namenode"));
 MiniDFSCluster cluster = null;
 try {
   cluster = new MiniDFSCluster.Builder(conf).build();
   cluster.waitActive();
   synchronized (cluster.getNameNode().getFSImage().getEditLog()) {
+Thread.sleep(jmxCachePeriod * 1000);
 MBeanClient client = new MBeanClient();
 client.start();
 client.join(2);


-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org

hadoop git commit: HDFS-11290. TestFSNameSystemMBean should wait until JMX cache is cleared. Contributed by Erik Krogen.

2017-01-23 Thread aajisaka

Repository: hadoop
Updated Branches:
  refs/heads/branch-2.8 e72f485d8 -> 71cbc4f6b


HDFS-11290. TestFSNameSystemMBean should wait until JMX cache is cleared. 
Contributed by Erik Krogen.

(cherry picked from commit b1a9ec856b572894e769f052aea2340fc3f23c78)
(cherry picked from commit db47bd15dd80a4a5727607832c16635cb0ec7726)


Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/71cbc4f6
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/71cbc4f6
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/71cbc4f6

Branch: refs/heads/branch-2.8
Commit: 71cbc4f6b51fc8b48ee1a026b1ba1617bb5a6986
Parents: e72f485
Author: Akira Ajisaka 
Authored: Wed Jan 18 17:53:21 2017 +0900
Committer: Akira Ajisaka 
Committed: Tue Jan 24 14:10:54 2017 +0900

--
 .../hdfs/server/namenode/TestFSNamesystemMBean.java   | 10 ++
 1 file changed, 10 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/hadoop/blob/71cbc4f6/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSNamesystemMBean.java
--
diff --git 
a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSNamesystemMBean.java
 
b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSNamesystemMBean.java
index e8db18f..b66bdbc 100644
--- 
a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSNamesystemMBean.java
+++ 
b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSNamesystemMBean.java
@@ -34,6 +34,8 @@ import org.apache.hadoop.fs.FileSystem;
 import org.apache.hadoop.fs.Path;
 import org.apache.hadoop.conf.Configuration;
 import org.apache.hadoop.hdfs.MiniDFSCluster;
+import org.apache.hadoop.metrics2.impl.ConfigBuilder;
+import org.apache.hadoop.metrics2.impl.TestMetricsConfig;
 import org.junit.Test;
 import org.mortbay.util.ajax.JSON;
 
@@ -139,12 +141,16 @@ public class TestFSNamesystemMBean {
 MiniDFSCluster cluster = null;
 FSNamesystem fsn = null;
 
+int jmxCachePeriod = 1;
+new ConfigBuilder().add("namenode.period", jmxCachePeriod)
+.save(TestMetricsConfig.getTestFilename("hadoop-metrics2-namenode"));
 try {
   cluster = new MiniDFSCluster.Builder(conf).build();
   cluster.waitActive();
 
   fsn = cluster.getNameNode().namesystem;
   fsn.writeLock();
+  Thread.sleep(jmxCachePeriod * 1000);
 
   MBeanClient client = new MBeanClient();
   client.start();
@@ -167,11 +173,15 @@ public class TestFSNamesystemMBean {
   @Test
   public void testWithFSEditLogLock() throws Exception {
 Configuration conf = new Configuration();
+int jmxCachePeriod = 1;
+new ConfigBuilder().add("namenode.period", jmxCachePeriod)
+.save(TestMetricsConfig.getTestFilename("hadoop-metrics2-namenode"));
 MiniDFSCluster cluster = null;
 try {
   cluster = new MiniDFSCluster.Builder(conf).build();
   cluster.waitActive();
   synchronized (cluster.getNameNode().getFSImage().getEditLog()) {
+Thread.sleep(jmxCachePeriod * 1000);
 MBeanClient client = new MBeanClient();
 client.start();
 client.join(2);


-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org

hadoop git commit: YARN-6117. SharedCacheManager does not start up. Contributed by Chris Trezzo.

2017-01-23 Thread sjlee

Repository: hadoop
Updated Branches:
  refs/heads/trunk a2c50127d -> dc6ec9704


YARN-6117. SharedCacheManager does not start up. Contributed by Chris Trezzo.


Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/dc6ec970
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/dc6ec970
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/dc6ec970

Branch: refs/heads/trunk
Commit: dc6ec9704829f180ce0e182c436fe1a435744c88
Parents: a2c5012
Author: Sangjin Lee 
Authored: Mon Jan 23 21:07:25 2017 -0800
Committer: Sangjin Lee 
Committed: Mon Jan 23 21:07:25 2017 -0800

--
 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/pom.xml  | 1 +
 .../hadoop-yarn-common/src/main/resources/webapps/sharedcache/.keep | 0
 2 files changed, 1 insertion(+)
--


http://git-wip-us.apache.org/repos/asf/hadoop/blob/dc6ec970/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/pom.xml
--
diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/pom.xml 
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/pom.xml
index ca410f6..5707444 100644
--- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/pom.xml
+++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/pom.xml
@@ -220,6 +220,7 @@
 src/main/resources/webapps/yarn/.keep
 
src/main/resources/webapps/applicationhistory/.keep
 src/main/resources/webapps/timeline/.keep
+src/main/resources/webapps/sharedcache/.keep
 src/main/resources/webapps/cluster/.keep
 src/main/resources/webapps/test/.keep
 src/main/resources/webapps/proxy/.keep

http://git-wip-us.apache.org/repos/asf/hadoop/blob/dc6ec970/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/webapps/sharedcache/.keep
--
diff --git 
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/webapps/sharedcache/.keep
 
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/webapps/sharedcache/.keep
new file mode 100644
index 000..e69de29


-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org

hadoop git commit: YARN-6012. Remove node label (removeFromClusterNodeLabels) document is missing (Contributed by Ying Zhang via Daniel Templeton)

2017-01-23 Thread templedf

Repository: hadoop
Updated Branches:
  refs/heads/trunk ce832059d -> a2c50127d


YARN-6012. Remove node label (removeFromClusterNodeLabels) document is missing 
(Contributed by Ying Zhang via Daniel Templeton)


Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/a2c50127
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/a2c50127
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/a2c50127

Branch: refs/heads/trunk
Commit: a2c50127d12b0aefcf1b050e0ab0d2df2abb30b3
Parents: ce83205
Author: Daniel Templeton 
Authored: Mon Jan 23 17:40:44 2017 -0800
Committer: Daniel Templeton 
Committed: Mon Jan 23 17:42:54 2017 -0800

--
 .../hadoop-yarn-site/src/site/markdown/NodeLabel.md  | 8 
 1 file changed, 8 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/hadoop/blob/a2c50127/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/NodeLabel.md
--
diff --git 
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/NodeLabel.md
 
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/NodeLabel.md
index a87658d..73c9f41 100644
--- 
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/NodeLabel.md
+++ 
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/NodeLabel.md
@@ -20,6 +20,7 @@ YARN Node Labels
 * [Configuration](#Configuration)
 * [Setting up ResourceManager to enable Node 
Labels](#Setting_up_ResourceManager_to_enable_Node_Labels)
 * [Add/modify node labels list to 
YARN](#Add/modify_node_labels_list_to_YARN)
+* [Remove node labels from YARN](#Remove_node_labels_from_YARN)
 * [Add/modify node-to-labels mapping to 
YARN](#Add/modify_node-to-labels_mapping_to_YARN)
 * [Configuration of Schedulers for node 
labels](#Configuration_of_Schedulers_for_node_labels)
 * [Specifying node label for 
application](#Specifying_node_label_for_application)
@@ -86,6 +87,13 @@ Notes:
 * If user donât specify â(exclusive=â¦)â, exclusive will be 
```true``` by default.
 * Run ```yarn cluster --list-node-labels``` to check added node labels are 
visible in the cluster.
 
+###Remove node labels from YARN
+
+* Remove cluster node labels:
+* To remove one or more node labels, execute the following command: 
```yarn rmadmin -removeFromClusterNodeLabels "[,,...]"```. The 
command argument should be a comma-separated list of node labels to remove.
+* It is not allowed to remove a label which has been associated with 
queues, i.e., one or more queues have access to this label.
+* To verify if specified node labels have been successfully removed, run 
```yarn cluster --list-node-labels```.
+
 ###Add/modify node-to-labels mapping to YARN
 
 * Configuring nodes to labels mapping in **Centralized** NodeLabel setup


-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org

hadoop git commit: HDFS-11028: libhdfs++: FileSystem needs to be able to cancel pending connections. Contributed by James Clampffer

2017-01-23 Thread jhc

Repository: hadoop
Updated Branches:
  refs/heads/HDFS-8707 7368178e2 -> ff2b18c45


HDFS-11028: libhdfs++: FileSystem needs to be able to cancel pending 
connections.  Contributed by James Clampffer


Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/ff2b18c4
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/ff2b18c4
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/ff2b18c4

Branch: refs/heads/HDFS-8707
Commit: ff2b18c4587a6c29f5a351435e87d949e1f1865f
Parents: 7368178
Author: James 
Authored: Mon Jan 23 18:19:13 2017 -0500
Committer: James 
Committed: Mon Jan 23 18:19:13 2017 -0500

--
 .../native/libhdfspp/examples/c/CMakeLists.txt  |   1 +
 .../examples/c/connect_cancel/CMakeLists.txt|  35 
 .../examples/c/connect_cancel/connect_cancel.c  | 107 +
 .../libhdfspp/examples/cpp/CMakeLists.txt   |   1 +
 .../examples/cpp/connect_cancel/CMakeLists.txt  |  29 
 .../cpp/connect_cancel/connect_cancel.cc| 158 +++
 .../native/libhdfspp/include/hdfspp/hdfs_ext.h  |  31 
 .../native/libhdfspp/include/hdfspp/hdfspp.h|   6 +
 .../native/libhdfspp/lib/bindings/c/hdfs.cc | 108 +
 .../libhdfspp/lib/common/continuation/asio.h|  86 --
 .../lib/common/continuation/continuation.h  |  12 --
 .../libhdfspp/lib/common/hdfs_configuration.cc  |  25 ++-
 .../libhdfspp/lib/common/namenode_info.cc   | 123 ++-
 .../src/main/native/libhdfspp/lib/common/util.h |  68 
 .../main/native/libhdfspp/lib/fs/filesystem.cc  |  42 -
 .../main/native/libhdfspp/lib/fs/filesystem.h   |   8 +
 .../libhdfspp/lib/fs/namenode_operations.cc |   4 +
 .../libhdfspp/lib/fs/namenode_operations.h  |   2 +
 .../main/native/libhdfspp/lib/rpc/rpc_engine.cc |  21 ++-
 .../main/native/libhdfspp/lib/rpc/rpc_engine.h  |   5 +
 20 files changed, 729 insertions(+), 143 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/hadoop/blob/ff2b18c4/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/examples/c/CMakeLists.txt
--
diff --git 
a/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/examples/c/CMakeLists.txt
 
b/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/examples/c/CMakeLists.txt
index 76880cd..a73d2bc 100644
--- 
a/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/examples/c/CMakeLists.txt
+++ 
b/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/examples/c/CMakeLists.txt
@@ -17,3 +17,4 @@
 #
 
 add_subdirectory(cat)
+add_subdirectory(connect_cancel)

http://git-wip-us.apache.org/repos/asf/hadoop/blob/ff2b18c4/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/examples/c/connect_cancel/CMakeLists.txt
--
diff --git 
a/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/examples/c/connect_cancel/CMakeLists.txt
 
b/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/examples/c/connect_cancel/CMakeLists.txt
new file mode 100644
index 000..56aeeea
--- /dev/null
+++ 
b/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/examples/c/connect_cancel/CMakeLists.txt
@@ -0,0 +1,35 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# Default LIBHDFSPP_DIR to the default install location.  You can override
+#it by add -DLIBHDFSPP_DIR=... to your cmake invocation
+set(LIBHDFSPP_DIR CACHE STRING ${CMAKE_INSTALL_PREFIX})
+
+include_directories( ${LIBHDFSPP_DIR}/include )
+link_directories( ${LIBHDFSPP_DIR}/lib )
+
+add_executable(connect_cancel_c connect_cancel.c)
+target_link_libraries(connect_cancel_c hdfspp uriparser2)
+
+# Several examples in different languages need to produce executables with
+# same names. To allow executables with same names we keep their CMake
+# names different, but specify their executable names

[1/3] hadoop git commit: YARN-5864. Capacity Scheduler - Queue Priorities. (wangda)

2017-01-23 Thread wangda

Repository: hadoop
Updated Branches:
  refs/heads/trunk 69fa81679 -> ce832059d


http://git-wip-us.apache.org/repos/asf/hadoop/blob/ce832059/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestContainerAllocation.java
--
diff --git 
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestContainerAllocation.java
 
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestContainerAllocation.java
index bb400ef..dd6b25b 100644
--- 
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestContainerAllocation.java
+++ 
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestContainerAllocation.java
@@ -776,4 +776,115 @@ public class TestContainerAllocation {
 Resources.createResource(20 * GB), "", true).getMemorySize());
 rm1.close();
   }
+
+
+  @Test(timeout = 6)
+  public void testQueuePriorityOrdering() throws Exception {
+CapacitySchedulerConfiguration newConf =
+(CapacitySchedulerConfiguration) TestUtils
+.getConfigurationWithMultipleQueues(conf);
+
+// Set ordering policy
+newConf.setQueueOrderingPolicy(CapacitySchedulerConfiguration.ROOT,
+
CapacitySchedulerConfiguration.QUEUE_PRIORITY_UTILIZATION_ORDERING_POLICY);
+
+// Set maximum capacity of A to 20
+newConf.setMaximumCapacity(CapacitySchedulerConfiguration.ROOT + ".a", 20);
+newConf.setQueuePriority(CapacitySchedulerConfiguration.ROOT + ".c", 1);
+newConf.setQueuePriority(CapacitySchedulerConfiguration.ROOT + ".b", 2);
+newConf.setQueuePriority(CapacitySchedulerConfiguration.ROOT + ".a", 3);
+
+MockRM rm1 = new MockRM(newConf);
+
+rm1.getRMContext().setNodeLabelManager(mgr);
+rm1.start();
+MockNM nm1 = rm1.registerNode("h1:1234", 100 * GB);
+
+// launch an app to queue A, AM container should be launched in nm1
+RMApp app1 = rm1.submitApp(2 * GB, "app", "user", null, "a");
+MockAM am1 = MockRM.launchAndRegisterAM(app1, rm1, nm1);
+
+// launch an app to queue B, AM container should be launched in nm1
+RMApp app2 = rm1.submitApp(2 * GB, "app", "user", null, "b");
+MockAM am2 = MockRM.launchAndRegisterAM(app2, rm1, nm1);
+
+// launch an app to queue C, AM container should be launched in nm1
+RMApp app3 = rm1.submitApp(2 * GB, "app", "user", null, "c");
+MockAM am3 = MockRM.launchAndRegisterAM(app3, rm1, nm1);
+
+// Each application asks 10 * 5GB containers
+am1.allocate("*", 5 * GB, 10, null);
+am2.allocate("*", 5 * GB, 10, null);
+am3.allocate("*", 5 * GB, 10, null);
+
+CapacityScheduler cs = (CapacityScheduler) rm1.getResourceScheduler();
+RMNode rmNode1 = rm1.getRMContext().getRMNodes().get(nm1.getNodeId());
+
+FiCaSchedulerApp schedulerApp1 =
+cs.getApplicationAttempt(am1.getApplicationAttemptId());
+FiCaSchedulerApp schedulerApp2 =
+cs.getApplicationAttempt(am2.getApplicationAttemptId());
+FiCaSchedulerApp schedulerApp3 =
+cs.getApplicationAttempt(am3.getApplicationAttemptId());
+
+// container will be allocated to am1
+// App1 will get 2 container allocated (plus AM container)
+cs.handle(new NodeUpdateSchedulerEvent(rmNode1));
+Assert.assertEquals(2, schedulerApp1.getLiveContainers().size());
+Assert.assertEquals(1, schedulerApp2.getLiveContainers().size());
+Assert.assertEquals(1, schedulerApp3.getLiveContainers().size());
+
+// container will be allocated to am1 again,
+// App1 will get 3 container allocated (plus AM container)
+cs.handle(new NodeUpdateSchedulerEvent(rmNode1));
+Assert.assertEquals(3, schedulerApp1.getLiveContainers().size());
+Assert.assertEquals(1, schedulerApp2.getLiveContainers().size());
+Assert.assertEquals(1, schedulerApp3.getLiveContainers().size());
+
+// (Now usages of queues: a=12G (satisfied), b=2G, c=2G)
+
+// container will be allocated to am2 (since app1 reaches its guaranteed
+// capacity)
+// App2 will get 2 container allocated (plus AM container)
+cs.handle(new NodeUpdateSchedulerEvent(rmNode1));
+Assert.assertEquals(3, schedulerApp1.getLiveContainers().size());
+Assert.assertEquals(2, schedulerApp2.getLiveContainers().size());
+Assert.assertEquals(1, schedulerApp3.getLiveContainers().size());
+
+// Do this 3 times
+// container will be allocated to am2 (since app1 reaches its guaranteed
+// capacity)
+

[2/3] hadoop git commit: YARN-5864. Capacity Scheduler - Queue Priorities. (wangda)

2017-01-23 Thread wangda

http://git-wip-us.apache.org/repos/asf/hadoop/blob/ce832059/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/policy/PriorityUtilizationQueueOrderingPolicy.java
--
diff --git 
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/policy/PriorityUtilizationQueueOrderingPolicy.java
 
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/policy/PriorityUtilizationQueueOrderingPolicy.java
new file mode 100644
index 000..fe60611
--- /dev/null
+++ 
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/policy/PriorityUtilizationQueueOrderingPolicy.java
@@ -0,0 +1,186 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package 
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.policy;
+
+import org.apache.commons.lang3.StringUtils;
+import 
org.apache.hadoop.yarn.server.resourcemanager.nodelabels.RMNodeLabelsManager;
+import 
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CSQueue;
+import 
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerConfiguration;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.Iterator;
+import java.util.List;
+import java.util.function.Supplier;
+
+/**
+ * For two queues with the same priority:
+ * - The queue with less relative used-capacity goes first - todayâs 
behavior.
+ * - The default priority for all queues is 0 and equal. So, we get todayâs
+ *   behaviour at every level - the queue with the lowest used-capacity
+ *   percentage gets the resources
+ *
+ * For two queues with different priorities:
+ * - Both the queues are under their guaranteed capacities: The queue with
+ *   the higher priority gets resources
+ * - Both the queues are over or meeting their guaranteed capacities:
+ *   The queue with the higher priority gets resources
+ * - One of the queues is over or meeting their guaranteed capacities and the
+ *   other is under: The queue that is under its capacity guarantee gets the
+ *   resources.
+ */
+public class PriorityUtilizationQueueOrderingPolicy implements 
QueueOrderingPolicy {
+  private List queues;
+  private boolean respectPriority;
+
+  // This makes multiple threads can sort queues at the same time
+  // For different partitions.
+  private static ThreadLocal partitionToLookAt =
+  ThreadLocal.withInitial(new Supplier() {
+@Override
+public String get() {
+  return RMNodeLabelsManager.NO_LABEL;
+}
+  });
+
+  /**
+   * Compare two queues with possibly different priority and assigned capacity,
+   * Will be used by preemption policy as well.
+   *
+   * @param relativeAssigned1 relativeAssigned1
+   * @param relativeAssigned2 relativeAssigned2
+   * @param priority1 p1
+   * @param priority2 p2
+   * @return compared result
+   */
+  public static int compare(double relativeAssigned1, double relativeAssigned2,
+  int priority1, int priority2) {
+if (priority1 == priority2) {
+  // The queue with less relative used-capacity goes first
+  return Double.compare(relativeAssigned1, relativeAssigned2);
+} else {
+  // When priority is different:
+  if ((relativeAssigned1 < 1.0f && relativeAssigned2 < 1.0f) || (
+  relativeAssigned1 >= 1.0f && relativeAssigned2 >= 1.0f)) {
+// When both the queues are under their guaranteed capacities,
+// Or both the queues are over or meeting their guaranteed capacities
+// queue with higher used-capacity goes first
+return Integer.compare(priority2, priority1);
+  } else {
+// Otherwise, when one of the queues is over or meeting their
+// guaranteed capacities and the other is under: The queue

[3/3] hadoop git commit: YARN-5864. Capacity Scheduler - Queue Priorities. (wangda)

2017-01-23 Thread wangda

YARN-5864. Capacity Scheduler - Queue Priorities. (wangda)


Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/ce832059
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/ce832059
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/ce832059

Branch: refs/heads/trunk
Commit: ce832059db077fa95922198b066a737ed4f609fe
Parents: 69fa816
Author: Wangda Tan 
Authored: Mon Jan 23 10:52:14 2017 -0800
Committer: Wangda Tan 
Committed: Mon Jan 23 10:52:14 2017 -0800

--
 .../dev-support/findbugs-exclude.xml|   4 -
 .../AbstractPreemptableResourceCalculator.java  |  14 +-
 .../capacity/PreemptionCandidatesSelector.java  |   3 +-
 .../ProportionalCapacityPreemptionPolicy.java   |  37 ++
 ...QueuePriorityContainerCandidateSelector.java | 510 +
 .../monitor/capacity/TempQueuePerPartition.java |  32 +-
 .../monitor/capacity/TempSchedulerNode.java | 120 
 .../rmcontainer/RMContainerImpl.java|   7 +-
 .../scheduler/SchedulerNode.java|   2 +-
 .../scheduler/capacity/AbstractCSQueue.java |  10 +
 .../scheduler/capacity/CSQueue.java |   7 +
 .../scheduler/capacity/CapacityScheduler.java   |  75 ++-
 .../CapacitySchedulerConfiguration.java | 236 +++-
 .../capacity/CapacitySchedulerContext.java  |   4 -
 .../capacity/CapacitySchedulerQueueManager.java |   3 -
 .../scheduler/capacity/LeafQueue.java   |   4 +-
 .../scheduler/capacity/ParentQueue.java |  70 ++-
 .../capacity/PartitionedQueueComparator.java|  72 ---
 .../PriorityUtilizationQueueOrderingPolicy.java | 186 ++
 .../capacity/policy/QueueOrderingPolicy.java|  52 ++
 .../scheduler/common/fica/FiCaSchedulerApp.java |  96 +++-
 ...alCapacityPreemptionPolicyMockFramework.java |  64 +++
 .../TestPreemptionForQueueWithPriorities.java   | 361 
 ...estProportionalCapacityPreemptionPolicy.java |  31 +-
 ...pacityPreemptionPolicyForNodePartitions.java |  11 +-
 ...alCapacityPreemptionPolicyMockFramework.java |  13 +-
 .../CapacitySchedulerPreemptionTestBase.java|  22 +-
 .../capacity/TestApplicationLimits.java |   9 -
 .../TestApplicationLimitsByPartition.java   |   3 -
 ...TestCapacitySchedulerSurgicalPreemption.java | 572 ++-
 .../scheduler/capacity/TestChildQueueOrder.java |   3 -
 .../capacity/TestContainerAllocation.java   | 111 
 .../scheduler/capacity/TestLeafQueue.java   |  13 +-
 .../scheduler/capacity/TestParentQueue.java |   3 -
 .../scheduler/capacity/TestReservations.java|   2 -
 ...tPriorityUtilizationQueueOrderingPolicy.java | 222 +++
 .../policy/TestFairOrderingPolicy.java  |  11 +-
 37 files changed, 2764 insertions(+), 231 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/hadoop/blob/ce832059/hadoop-yarn-project/hadoop-yarn/dev-support/findbugs-exclude.xml
--
diff --git a/hadoop-yarn-project/hadoop-yarn/dev-support/findbugs-exclude.xml 
b/hadoop-yarn-project/hadoop-yarn/dev-support/findbugs-exclude.xml
index ab36a4e..c090749 100644
--- a/hadoop-yarn-project/hadoop-yarn/dev-support/findbugs-exclude.xml
+++ b/hadoop-yarn-project/hadoop-yarn/dev-support/findbugs-exclude.xml
@@ -183,10 +183,6 @@
 
 
   
-  
-
-
-  
   
   
 

http://git-wip-us.apache.org/repos/asf/hadoop/blob/ce832059/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/monitor/capacity/AbstractPreemptableResourceCalculator.java
--
diff --git 
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/monitor/capacity/AbstractPreemptableResourceCalculator.java
 
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/monitor/capacity/AbstractPreemptableResourceCalculator.java
index 8255a30..a80f317 100644
--- 
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/monitor/capacity/AbstractPreemptableResourceCalculator.java
+++ 
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/monitor/capacity/AbstractPreemptableResourceCalculator.java
@@ -19,6 +19,7 @@
 package org.apache.hadoop.yarn.server.resourcemanager.monitor.capacity;
 
 import org.apache.hadoop.yarn.api.records.Resource;
+import

hadoop git commit: YARN-5910. Support for multi-cluster delegation tokens. Contributed by Jian He (cherry picked from commit 69fa81679f59378fd19a2c65db8019393d7c05a2)

2017-01-23 Thread jlowe

Repository: hadoop
Updated Branches:
  refs/heads/branch-2 f8e49bb34 -> 94b326f2c


YARN-5910. Support for multi-cluster delegation tokens. Contributed by Jian He
(cherry picked from commit 69fa81679f59378fd19a2c65db8019393d7c05a2)

Conflicts:


hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManager.java


Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/94b326f2
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/94b326f2
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/94b326f2

Branch: refs/heads/branch-2
Commit: 94b326f2c812cb30eba606518d332c3bb16097ea
Parents: f8e49bb
Author: Jason Lowe 
Authored: Mon Jan 23 17:23:32 2017 +
Committer: Jason Lowe 
Committed: Mon Jan 23 17:23:32 2017 +

--
 .../apache/hadoop/mapreduce/MRJobConfig.java|   2 +
 .../src/main/resources/mapred-default.xml   |  18 +++
 .../org/apache/hadoop/mapred/YARNRunner.java|  36 ++
 .../apache/hadoop/mapred/TestYARNRunner.java|  39 ++
 .../api/records/ContainerLaunchContext.java |  16 +++
 .../hadoop/yarn/conf/YarnConfiguration.java |   7 +-
 .../src/main/proto/yarn_protos.proto|   2 +
 .../impl/pb/ContainerLaunchContextPBImpl.java   |  26 
 .../src/main/resources/yarn-default.xml |  10 ++
 .../hadoop/yarn/server/utils/BuilderUtils.java  |  32 +
 .../server/resourcemanager/ClientRMService.java |  19 ++-
 .../server/resourcemanager/RMAppManager.java|  30 +
 .../server/resourcemanager/rmapp/RMAppImpl.java |  23 ++--
 .../security/DelegationTokenRenewer.java|  74 +++
 .../yarn/server/resourcemanager/MockRM.java |  36 --
 .../server/resourcemanager/TestAppManager.java  |  41 ++
 .../server/resourcemanager/TestRMRestart.java   |  37 +++---
 .../TestRMAppLogAggregationStatus.java  |   8 +-
 .../rmapp/TestRMAppTransitions.java |   2 +-
 .../scheduler/fair/FairSchedulerTestBase.java   |   9 +-
 .../security/TestDelegationTokenRenewer.java| 127 +--
 21 files changed, 448 insertions(+), 146 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/hadoop/blob/94b326f2/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java
--
diff --git 
a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java
 
b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java
index 015c1f2..51353cd 100644
--- 
a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java
+++ 
b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java
@@ -996,4 +996,6 @@ public interface MRJobConfig {
* A comma-separated list of properties whose value will be redacted.
*/
   String MR_JOB_REDACTED_PROPERTIES = "mapreduce.job.redacted-properties";
+
+  String MR_JOB_SEND_TOKEN_CONF = "mapreduce.job.send-token-conf";
 }

http://git-wip-us.apache.org/repos/asf/hadoop/blob/94b326f2/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
--
diff --git 
a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
 
b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
index 523dd28..5aa2e71 100644
--- 
a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
+++ 
b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
@@ -1955,4 +1955,22 @@
   mapreduce.job.redacted-properties
   
 
+
+
+  
+This configuration is a regex expression. The list of configurations that
+match the regex expression will be sent to RM. RM will use these
+configurations for renewing tokens.
+This configuration is added for below scenario: User needs to run distcp
+jobs across two clusters, but the RM does not have necessary hdfs
+configurations to connect to the remote hdfs cluster. Hence, user relies on
+this config to send the configurations to RM and RM uses these
+configurations to renew tokens.
+For example the following regex expression

hadoop git commit: YARN-5910. Support for multi-cluster delegation tokens. Contributed by Jian He

2017-01-23 Thread jlowe

Repository: hadoop
Updated Branches:
  refs/heads/trunk 3fa0d540d -> 69fa81679


YARN-5910. Support for multi-cluster delegation tokens. Contributed by Jian He


Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/69fa8167
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/69fa8167
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/69fa8167

Branch: refs/heads/trunk
Commit: 69fa81679f59378fd19a2c65db8019393d7c05a2
Parents: 3fa0d54
Author: Jason Lowe 
Authored: Mon Jan 23 17:12:51 2017 +
Committer: Jason Lowe 
Committed: Mon Jan 23 17:12:51 2017 +

--
 .../apache/hadoop/mapreduce/MRJobConfig.java|   2 +
 .../src/main/resources/mapred-default.xml   |  18 +++
 .../org/apache/hadoop/mapred/YARNRunner.java|  36 ++
 .../apache/hadoop/mapred/TestYARNRunner.java|  39 ++
 .../api/records/ContainerLaunchContext.java |  16 +++
 .../hadoop/yarn/conf/YarnConfiguration.java |   7 +-
 .../src/main/proto/yarn_protos.proto|   2 +
 .../impl/pb/ContainerLaunchContextPBImpl.java   |  26 
 .../src/main/resources/yarn-default.xml |  10 ++
 .../hadoop/yarn/server/utils/BuilderUtils.java  |  32 +
 .../server/resourcemanager/ClientRMService.java |  19 ++-
 .../server/resourcemanager/RMAppManager.java|  30 +
 .../server/resourcemanager/rmapp/RMAppImpl.java |  23 ++--
 .../security/DelegationTokenRenewer.java|  74 +++
 .../yarn/server/resourcemanager/MockRM.java |  36 --
 .../server/resourcemanager/TestAppManager.java  |  41 ++
 .../server/resourcemanager/TestRMRestart.java   |  37 +++---
 .../TestRMAppLogAggregationStatus.java  |   8 +-
 .../rmapp/TestRMAppTransitions.java |   2 +-
 .../scheduler/fair/FairSchedulerTestBase.java   |   9 +-
 .../security/TestDelegationTokenRenewer.java| 127 +--
 21 files changed, 448 insertions(+), 146 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/hadoop/blob/69fa8167/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java
--
diff --git 
a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java
 
b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java
index 7add7df..ab48255 100644
--- 
a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java
+++ 
b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java
@@ -1013,4 +1013,6 @@ public interface MRJobConfig {
* A comma-separated list of properties whose value will be redacted.
*/
   String MR_JOB_REDACTED_PROPERTIES = "mapreduce.job.redacted-properties";
+
+  String MR_JOB_SEND_TOKEN_CONF = "mapreduce.job.send-token-conf";
 }

http://git-wip-us.apache.org/repos/asf/hadoop/blob/69fa8167/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
--
diff --git 
a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
 
b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
index 1b4f373..a163fbd 100644
--- 
a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
+++ 
b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
@@ -1989,4 +1989,22 @@
   mapreduce.job.redacted-properties
   
 
+
+
+  
+This configuration is a regex expression. The list of configurations that
+match the regex expression will be sent to RM. RM will use these
+configurations for renewing tokens.
+This configuration is added for below scenario: User needs to run distcp
+jobs across two clusters, but the RM does not have necessary hdfs
+configurations to connect to the remote hdfs cluster. Hence, user relies on
+this config to send the configurations to RM and RM uses these
+configurations to renew tokens.
+For example the following regex expression indicates the minimum required
+configs for RM to connect to a remote hdfs cluster:
+
dfs.nameservices|^dfs.namenode.rpc-address.*$|^dfs.ha.namenodes.*$|^dfs.client.failover.proxy.provider.*$|dfs.namenode.kerberos.principal
+  
+

hadoop git commit: HDFS-11352. Potential deadlock in NN when failing over. Contributed by Eric Krogen.

hadoop git commit: HDFS-11352. Add missing entry in CHANGES.txt.

hadoop git commit: HDFS-11352. Potential deadlock in NN when failing over. Contributed by Erik Krogen.

hadoop git commit: HDFS-11290. TestFSNameSystemMBean should wait until JMX cache is cleared. Contributed by Erik Krogen.

hadoop git commit: YARN-6082. Invalid REST api response for getApps since queueUsagePercentage is coming as INF. Contributed by Sunil G.

hadoop git commit: YARN-6082. Invalid REST api response for getApps since queueUsagePercentage is coming as INF. Contributed by Sunil G.

hadoop git commit: HDFS-11290. TestFSNameSystemMBean should wait until JMX cache is cleared. Contributed by Erik Krogen.

hadoop git commit: HDFS-11290. TestFSNameSystemMBean should wait until JMX cache is cleared. Contributed by Erik Krogen.

hadoop git commit: YARN-6117. SharedCacheManager does not start up. Contributed by Chris Trezzo.

hadoop git commit: YARN-6012. Remove node label (removeFromClusterNodeLabels) document is missing (Contributed by Ying Zhang via Daniel Templeton)

hadoop git commit: HDFS-11028: libhdfs++: FileSystem needs to be able to cancel pending connections. Contributed by James Clampffer

[1/3] hadoop git commit: YARN-5864. Capacity Scheduler - Queue Priorities. (wangda)

[2/3] hadoop git commit: YARN-5864. Capacity Scheduler - Queue Priorities. (wangda)

[3/3] hadoop git commit: YARN-5864. Capacity Scheduler - Queue Priorities. (wangda)

hadoop git commit: YARN-5910. Support for multi-cluster delegation tokens. Contributed by Jian He (cherry picked from commit 69fa81679f59378fd19a2c65db8019393d7c05a2)

hadoop git commit: YARN-5910. Support for multi-cluster delegation tokens. Contributed by Jian He

16 matches

Site Navigation

Mail list logo

Footer information