from:"rchiang"

svn commit: r1834138 - in /hadoop/common/site/main: author/src/documentation/content/xdocs/ publish/ publish/skin/images/

2018-06-22 Thread rchiang

Author: rchiang
Date: Fri Jun 22 14:11:23 2018
New Revision: 1834138

URL: http://svn.apache.org/viewvc?rev=1834138=rev
Log:
Add rchiang to PMC

Modified:
hadoop/common/site/main/author/src/documentation/content/xdocs/who.xml
hadoop/common/site/main/publish/bylaws.pdf
hadoop/common/site/main/publish/committer_criteria.pdf
hadoop/common/site/main/publish/index.pdf
hadoop/common/site/main/publish/issue_tracking.pdf
hadoop/common/site/main/publish/linkmap.pdf
hadoop/common/site/main/publish/mailing_lists.pdf
hadoop/common/site/main/publish/privacy_policy.pdf
hadoop/common/site/main/publish/releases.pdf
hadoop/common/site/main/publish/skin/images/rc-b-l-15-1body-2menu-3menu.png
hadoop/common/site/main/publish/skin/images/rc-b-r-15-1body-2menu-3menu.png

hadoop/common/site/main/publish/skin/images/rc-b-r-5-1header-2tab-selected-3tab-selected.png

hadoop/common/site/main/publish/skin/images/rc-t-l-5-1header-2searchbox-3searchbox.png

hadoop/common/site/main/publish/skin/images/rc-t-l-5-1header-2tab-selected-3tab-selected.png

hadoop/common/site/main/publish/skin/images/rc-t-l-5-1header-2tab-unselected-3tab-unselected.png
hadoop/common/site/main/publish/skin/images/rc-t-r-15-1body-2menu-3menu.png

hadoop/common/site/main/publish/skin/images/rc-t-r-5-1header-2searchbox-3searchbox.png

hadoop/common/site/main/publish/skin/images/rc-t-r-5-1header-2tab-selected-3tab-selected.png

hadoop/common/site/main/publish/skin/images/rc-t-r-5-1header-2tab-unselected-3tab-unselected.png
hadoop/common/site/main/publish/version_control.pdf
hadoop/common/site/main/publish/versioning.pdf
hadoop/common/site/main/publish/who.html
hadoop/common/site/main/publish/who.pdf

Modified: hadoop/common/site/main/author/src/documentation/content/xdocs/who.xml
URL: 
http://svn.apache.org/viewvc/hadoop/common/site/main/author/src/documentation/content/xdocs/who.xml?rev=1834138=1834137=1834138=diff
==
--- hadoop/common/site/main/author/src/documentation/content/xdocs/who.xml 
(original)
+++ hadoop/common/site/main/author/src/documentation/content/xdocs/who.xml Fri 
Jun 22 14:11:23 2018
@@ -399,6 +399,14 @@
  -8

 
+   
+ rchiang
+ Ray Chiang
+ Cloudera
+ 
+ -8
+   
+
 
   rkanter
   Robert Kanter

Modified: hadoop/common/site/main/publish/bylaws.pdf
URL: 
http://svn.apache.org/viewvc/hadoop/common/site/main/publish/bylaws.pdf?rev=1834138=1834137=1834138=diff
==
Binary files - no diff available.

Modified: hadoop/common/site/main/publish/committer_criteria.pdf
URL: 
http://svn.apache.org/viewvc/hadoop/common/site/main/publish/committer_criteria.pdf?rev=1834138=1834137=1834138=diff
==
Binary files - no diff available.

Modified: hadoop/common/site/main/publish/index.pdf
URL: 
http://svn.apache.org/viewvc/hadoop/common/site/main/publish/index.pdf?rev=1834138=1834137=1834138=diff
==
Binary files - no diff available.

Modified: hadoop/common/site/main/publish/issue_tracking.pdf
URL: 
http://svn.apache.org/viewvc/hadoop/common/site/main/publish/issue_tracking.pdf?rev=1834138=1834137=1834138=diff
==
Binary files - no diff available.

Modified: hadoop/common/site/main/publish/linkmap.pdf
URL: 
http://svn.apache.org/viewvc/hadoop/common/site/main/publish/linkmap.pdf?rev=1834138=1834137=1834138=diff
==
Binary files - no diff available.

Modified: hadoop/common/site/main/publish/mailing_lists.pdf
URL: 
http://svn.apache.org/viewvc/hadoop/common/site/main/publish/mailing_lists.pdf?rev=1834138=1834137=1834138=diff
==
Binary files - no diff available.

Modified: hadoop/common/site/main/publish/privacy_policy.pdf
URL: 
http://svn.apache.org/viewvc/hadoop/common/site/main/publish/privacy_policy.pdf?rev=1834138=1834137=1834138=diff
==
Binary files - no diff available.

Modified: hadoop/common/site/main/publish/releases.pdf
URL: 
http://svn.apache.org/viewvc/hadoop/common/site/main/publish/releases.pdf?rev=1834138=1834137=1834138=diff
==
Binary files - no diff available.

Modified: 
hadoop/common/site/main/publish/skin/images/rc-b-l-15-1body-2menu-3menu.png
URL: 
http://svn.apache.org/viewvc/hadoop/common/site/main/publish/skin/images/rc-b-l-15-1body-2menu-3menu.png?rev=1834138=1834137=1834138=diff

hadoop git commit: HADOOP-14652. Update metrics-core version to 3.2.4. (rchiang)

2018-03-01 Thread rchiang

Repository: hadoop
Updated Branches:
  refs/heads/trunk d4cc50f62 -> 923e1773e


HADOOP-14652. Update metrics-core version to 3.2.4. (rchiang)


Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/923e1773
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/923e1773
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/923e1773

Branch: refs/heads/trunk
Commit: 923e1773e181ce125855639b882dc5a1a58e80ee
Parents: d4cc50f
Author: Ray Chiang <rchi...@apache.org>
Authored: Thu Mar 1 15:34:54 2018 -0800
Committer: Ray Chiang <rchi...@apache.org>
Committed: Thu Mar 1 15:36:10 2018 -0800

--
 NOTICE.txt| 14 ++
 hadoop-common-project/hadoop-kms/pom.xml  |  2 +-
 hadoop-project/pom.xml|  4 ++--
 hadoop-tools/hadoop-sls/pom.xml   |  2 +-
 .../hadoop-yarn-server-nodemanager/pom.xml|  2 +-
 5 files changed, 19 insertions(+), 5 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/hadoop/blob/923e1773/NOTICE.txt
--
diff --git a/NOTICE.txt b/NOTICE.txt
index bc7a26f..95a670d 100644
--- a/NOTICE.txt
+++ b/NOTICE.txt
@@ -591,3 +591,17 @@ The binary distribution of this product bundles binaries of
 swagger-annotations (https://github.com/swagger-api/swagger-core),
 which has the following notices:
  * Copyright 2016 SmartBear Software
+
+The binary distribution of this product bundles binaries of
+metrics-core 3.2.4
+which has the following notices:
+ * Copyright 2010-2013 Coda Hale and Yammer, Inc.
+
+   This product includes software developed by Coda Hale and Yammer, Inc.
+
+   This product includes code derived from the JSR-166 project 
(ThreadLocalRandom, Striped64,
+   LongAdder), which was released with the following comments:
+
+   Written by Doug Lea with assistance from members of JCP JSR-166
+   Expert Group and released to the public domain, as explained at
+   http://creativecommons.org/publicdomain/zero/1.0/

http://git-wip-us.apache.org/repos/asf/hadoop/blob/923e1773/hadoop-common-project/hadoop-kms/pom.xml
--
diff --git a/hadoop-common-project/hadoop-kms/pom.xml 
b/hadoop-common-project/hadoop-kms/pom.xml
index c1c5285..03a68c5 100644
--- a/hadoop-common-project/hadoop-kms/pom.xml
+++ b/hadoop-common-project/hadoop-kms/pom.xml
@@ -159,7 +159,7 @@
   compile
 
 
-  com.codahale.metrics
+  io.dropwizard.metrics
   metrics-core
   compile
 

http://git-wip-us.apache.org/repos/asf/hadoop/blob/923e1773/hadoop-project/pom.xml
--
diff --git a/hadoop-project/pom.xml b/hadoop-project/pom.xml
index 7cc68bb..b72e497 100644
--- a/hadoop-project/pom.xml
+++ b/hadoop-project/pom.xml
@@ -1107,9 +1107,9 @@
 ${hsqldb.version}
   
   
-com.codahale.metrics
+io.dropwizard.metrics
 metrics-core
-3.0.1
+3.2.4
   
   
 org.apache.hadoop

http://git-wip-us.apache.org/repos/asf/hadoop/blob/923e1773/hadoop-tools/hadoop-sls/pom.xml
--
diff --git a/hadoop-tools/hadoop-sls/pom.xml b/hadoop-tools/hadoop-sls/pom.xml
index ef5ac54..10f6294 100644
--- a/hadoop-tools/hadoop-sls/pom.xml
+++ b/hadoop-tools/hadoop-sls/pom.xml
@@ -49,7 +49,7 @@
   hadoop-rumen
 
 
-  com.codahale.metrics
+  io.dropwizard.metrics
   metrics-core
   compile
 

http://git-wip-us.apache.org/repos/asf/hadoop/blob/923e1773/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/pom.xml
--
diff --git 
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/pom.xml
 
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/pom.xml
index 072395a..5146820 100644
--- 
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/pom.xml
+++ 
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/pom.xml
@@ -122,7 +122,7 @@
 
 
 
-  com.codahale.metrics
+  io.dropwizard.metrics
   metrics-core

hadoop git commit: YARN-4227. Ignore expired containers from removed nodes in FairScheduler. (Wilfred Spiegelenburg via rchiang)

2018-01-08 Thread rchiang

Repository: hadoop
Updated Branches:
  refs/heads/branch-2 eadd3cecf -> 18c398285


YARN-4227. Ignore expired containers from removed nodes in FairScheduler. 
(Wilfred Spiegelenburg via rchiang)

(cherry picked from commit bc2d67d6c10619716ef7acce263f3269a86c3150)


Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/18c39828
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/18c39828
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/18c39828

Branch: refs/heads/branch-2
Commit: 18c39828514247994b3eba6a3a9afa0ad23d3b82
Parents: eadd3ce
Author: Ray Chiang <rchi...@apache.org>
Authored: Mon Jan 8 15:32:25 2018 -0800
Committer: Ray Chiang <rchi...@apache.org>
Committed: Mon Jan 8 16:23:29 2018 -0800

--
 .../scheduler/fair/FairScheduler.java   | 29 ++
 .../scheduler/fair/TestFairScheduler.java   | 59 
 2 files changed, 78 insertions(+), 10 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/hadoop/blob/18c39828/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairScheduler.java
--
diff --git 
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairScheduler.java
 
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairScheduler.java
index a14b2ec..b7fcbe8 100644
--- 
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairScheduler.java
+++ 
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairScheduler.java
@@ -693,27 +693,36 @@ public class FairScheduler extends
   ApplicationId appId =
   container.getId().getApplicationAttemptId().getApplicationId();
   if (application == null) {
-LOG.info(
-"Container " + container + " of" + " finished application " + appId
-+ " completed with event " + event);
+LOG.info("Container " + container + " of finished application " +
+appId + " completed with event " + event);
 return;
   }
 
   // Get the node on which the container was allocated
-  FSSchedulerNode node = getFSSchedulerNode(container.getNodeId());
-
+  NodeId nodeID = container.getNodeId();
+  FSSchedulerNode node = getFSSchedulerNode(nodeID);
+  // node could be null if the thread was waiting for the lock and the node
+  // was removed in another thread
   if (rmContainer.getState() == RMContainerState.RESERVED) {
-application.unreserve(rmContainer.getReservedSchedulerKey(), node);
-  } else{
+if (node != null) {
+  application.unreserve(rmContainer.getReservedSchedulerKey(), node);
+} else if (LOG.isDebugEnabled()) {
+  LOG.debug("Skipping unreserve on removed node: " + nodeID);
+}
+  } else {
 application.containerCompleted(rmContainer, containerStatus, event);
-node.releaseContainer(rmContainer.getContainerId(), false);
+if (node != null) {
+  node.releaseContainer(rmContainer.getContainerId(), false);
+} else if (LOG.isDebugEnabled()) {
+  LOG.debug("Skipping container release on removed node: " + nodeID);
+}
 updateRootQueueMetrics();
   }
 
   if (LOG.isDebugEnabled()) {
 LOG.debug("Application attempt " + 
application.getApplicationAttemptId()
-+ " released container " + container.getId() + " on node: " + node
-+ " with event: " + event);
++ " released container " + container.getId() + " on node: " +
+(node == null ? nodeID : node) + " with event: " + event);
   }
 } finally {
   writeLock.unlock();

http://git-wip-us.apache.org/repos/asf/hadoop/blob/18c39828/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairScheduler.java
--
diff --git 
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/

hadoop git commit: YARN-4227. Ignore expired containers from removed nodes in FairScheduler. (Wilfred Spiegelenburg via rchiang)

2018-01-08 Thread rchiang

Repository: hadoop
Updated Branches:
  refs/heads/trunk 73ff09b79 -> 59ab5da0a


YARN-4227. Ignore expired containers from removed nodes in FairScheduler. 
(Wilfred Spiegelenburg via rchiang)


Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/59ab5da0
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/59ab5da0
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/59ab5da0

Branch: refs/heads/trunk
Commit: 59ab5da0a0337c49a58bc9b2db9d1a89f4d5b9dd
Parents: 73ff09b
Author: Ray Chiang <rchi...@apache.org>
Authored: Mon Jan 8 15:32:25 2018 -0800
Committer: Ray Chiang <rchi...@apache.org>
Committed: Mon Jan 8 16:19:02 2018 -0800

--
 .../scheduler/fair/FairScheduler.java   | 29 ++
 .../scheduler/fair/TestFairScheduler.java   | 59 
 2 files changed, 78 insertions(+), 10 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/hadoop/blob/59ab5da0/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairScheduler.java
--
diff --git 
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairScheduler.java
 
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairScheduler.java
index ebc7222..86d9fd7 100644
--- 
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairScheduler.java
+++ 
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairScheduler.java
@@ -669,27 +669,36 @@ public class FairScheduler extends
   ApplicationId appId =
   container.getId().getApplicationAttemptId().getApplicationId();
   if (application == null) {
-LOG.info(
-"Container " + container + " of" + " finished application " + appId
-+ " completed with event " + event);
+LOG.info("Container " + container + " of finished application " +
+appId + " completed with event " + event);
 return;
   }
 
   // Get the node on which the container was allocated
-  FSSchedulerNode node = getFSSchedulerNode(container.getNodeId());
-
+  NodeId nodeID = container.getNodeId();
+  FSSchedulerNode node = getFSSchedulerNode(nodeID);
+  // node could be null if the thread was waiting for the lock and the node
+  // was removed in another thread
   if (rmContainer.getState() == RMContainerState.RESERVED) {
-application.unreserve(rmContainer.getReservedSchedulerKey(), node);
-  } else{
+if (node != null) {
+  application.unreserve(rmContainer.getReservedSchedulerKey(), node);
+} else if (LOG.isDebugEnabled()) {
+  LOG.debug("Skipping unreserve on removed node: " + nodeID);
+}
+  } else {
 application.containerCompleted(rmContainer, containerStatus, event);
-node.releaseContainer(rmContainer.getContainerId(), false);
+if (node != null) {
+  node.releaseContainer(rmContainer.getContainerId(), false);
+} else if (LOG.isDebugEnabled()) {
+  LOG.debug("Skipping container release on removed node: " + nodeID);
+}
 updateRootQueueMetrics();
   }
 
   if (LOG.isDebugEnabled()) {
 LOG.debug("Application attempt " + 
application.getApplicationAttemptId()
-+ " released container " + container.getId() + " on node: " + node
-+ " with event: " + event);
++ " released container " + container.getId() + " on node: " +
+(node == null ? nodeID : node) + " with event: " + event);
   }
 } finally {
   writeLock.unlock();

http://git-wip-us.apache.org/repos/asf/hadoop/blob/59ab5da0/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairScheduler.java
--
diff --git 
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairScheduler.java
 
b/hadoop-yarn-pr

hadoop git commit: YARN-7219. Make AllocateRequestProto compatible with branch-2/branch-2.8. (rchiang)

2017-10-03 Thread rchiang

Repository: hadoop
Updated Branches:
  refs/heads/branch-3.0 f982ffe33 -> 01b047b41


YARN-7219.  Make AllocateRequestProto compatible with branch-2/branch-2.8. 
(rchiang)

(cherry picked from commit 67e12169b0ae8f3a9edb6511d3ea7ffd57b82e75)


Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/01b047b4
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/01b047b4
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/01b047b4

Branch: refs/heads/branch-3.0
Commit: 01b047b41bb43200f9d95ec8db11196b8ff45be6
Parents: f982ffe
Author: Ray Chiang <rchi...@apache.org>
Authored: Tue Oct 3 14:57:56 2017 -0700
Committer: Ray Chiang <rchi...@apache.org>
Committed: Tue Oct 3 15:16:31 2017 -0700

--
 .../hadoop-yarn-api/src/main/proto/yarn_service_protos.proto   | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/hadoop/blob/01b047b4/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/yarn_service_protos.proto
--
diff --git 
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/yarn_service_protos.proto
 
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/yarn_service_protos.proto
index 7a7f035..2498142 100644
--- 
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/yarn_service_protos.proto
+++ 
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/yarn_service_protos.proto
@@ -87,7 +87,7 @@ message AllocateRequestProto {
   optional ResourceBlacklistRequestProto blacklist_request = 3;
   optional int32 response_id = 4;
   optional float progress = 5;
-  repeated UpdateContainerRequestProto update_requests = 6;
+  repeated UpdateContainerRequestProto update_requests = 7;
 }
 
 message NMTokenProto {


-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org

hadoop git commit: YARN-7219. Make AllocateRequestProto compatible with branch-2/branch-2.8. (rchiang)

2017-10-03 Thread rchiang

Repository: hadoop
Updated Branches:
  refs/heads/trunk c9b525bd9 -> 67e12169b


YARN-7219.  Make AllocateRequestProto compatible with branch-2/branch-2.8. 
(rchiang)


Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/67e12169
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/67e12169
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/67e12169

Branch: refs/heads/trunk
Commit: 67e12169b0ae8f3a9edb6511d3ea7ffd57b82e75
Parents: c9b525b
Author: Ray Chiang <rchi...@apache.org>
Authored: Tue Oct 3 14:57:56 2017 -0700
Committer: Ray Chiang <rchi...@apache.org>
Committed: Tue Oct 3 15:06:07 2017 -0700

--
 .../hadoop-yarn-api/src/main/proto/yarn_service_protos.proto   | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/hadoop/blob/67e12169/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/yarn_service_protos.proto
--
diff --git 
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/yarn_service_protos.proto
 
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/yarn_service_protos.proto
index 3da4ee7..54a3c69 100644
--- 
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/yarn_service_protos.proto
+++ 
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/yarn_service_protos.proto
@@ -88,7 +88,7 @@ message AllocateRequestProto {
   optional ResourceBlacklistRequestProto blacklist_request = 3;
   optional int32 response_id = 4;
   optional float progress = 5;
-  repeated UpdateContainerRequestProto update_requests = 6;
+  repeated UpdateContainerRequestProto update_requests = 7;
 }
 
 message NMTokenProto {


-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org

hadoop git commit: HADOOP-14903. Add json-smart explicitly to pom.xml. (rchiang)

2017-09-25 Thread rchiang

Repository: hadoop
Updated Branches:
  refs/heads/branch-3.0 bfad0ca3b -> da6ed36ed


HADOOP-14903. Add json-smart explicitly to pom.xml. (rchiang)

(cherry picked from commit cde804b91f2fed514935d186d89ea7cbbf2701f3)


Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/da6ed36e
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/da6ed36e
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/da6ed36e

Branch: refs/heads/branch-3.0
Commit: da6ed36edb759bbfcb41d886c7f444e8dddf9f2c
Parents: bfad0ca
Author: Ray Chiang <rchi...@apache.org>
Authored: Mon Sep 25 15:30:51 2017 -0700
Committer: Ray Chiang <rchi...@apache.org>
Committed: Mon Sep 25 15:32:15 2017 -0700

--
 hadoop-project/pom.xml | 9 +
 1 file changed, 9 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/hadoop/blob/da6ed36e/hadoop-project/pom.xml
--
diff --git a/hadoop-project/pom.xml b/hadoop-project/pom.xml
index 1c74c02..a106c17 100755
--- a/hadoop-project/pom.xml
+++ b/hadoop-project/pom.xml
@@ -1207,6 +1207,15 @@
 
   
   
+
+net.minidev
+json-smart
+2.3
+  
+  
 org.skyscreamer
 jsonassert
 1.3.0


-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org

hadoop git commit: HADOOP-14903. Add json-smart explicitly to pom.xml. (rchiang)

2017-09-25 Thread rchiang

Repository: hadoop
Updated Branches:
  refs/heads/trunk 0889e5a8b -> cde804b91


HADOOP-14903. Add json-smart explicitly to pom.xml. (rchiang)


Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/cde804b9
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/cde804b9
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/cde804b9

Branch: refs/heads/trunk
Commit: cde804b91f2fed514935d186d89ea7cbbf2701f3
Parents: 0889e5a
Author: Ray Chiang <rchi...@apache.org>
Authored: Mon Sep 25 15:30:51 2017 -0700
Committer: Ray Chiang <rchi...@apache.org>
Committed: Mon Sep 25 15:30:51 2017 -0700

--
 hadoop-project/pom.xml | 9 +
 1 file changed, 9 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/hadoop/blob/cde804b9/hadoop-project/pom.xml
--
diff --git a/hadoop-project/pom.xml b/hadoop-project/pom.xml
index 7f657ca..7b342c4 100755
--- a/hadoop-project/pom.xml
+++ b/hadoop-project/pom.xml
@@ -1207,6 +1207,15 @@
 
   
   
+
+net.minidev
+json-smart
+2.3
+  
+  
 org.skyscreamer
 jsonassert
 1.3.0


-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org

hadoop git commit: HADOOP-14856. Fix AWS, Jetty, HBase, Ehcache entries for NOTICE.txt. (rchiang)

2017-09-12 Thread rchiang

Repository: hadoop
Updated Branches:
  refs/heads/branch-3.0 5f6e387ba -> 99e7b2644


HADOOP-14856. Fix AWS, Jetty, HBase, Ehcache entries for NOTICE.txt. (rchiang)

(cherry picked from commit 2ffe93ab609dfb54c3d1a53273ac2bc5ad15a5dd)


Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/99e7b264
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/99e7b264
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/99e7b264

Branch: refs/heads/branch-3.0
Commit: 99e7b264431ba59987693b3e39e392afbbe86302
Parents: 5f6e387
Author: Ray Chiang <rchi...@apache.org>
Authored: Tue Sep 12 13:37:38 2017 -0700
Committer: Ray Chiang <rchi...@apache.org>
Committed: Tue Sep 12 13:38:12 2017 -0700

--
 NOTICE.txt | 56 ++--
 1 file changed, 26 insertions(+), 30 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/hadoop/blob/99e7b264/NOTICE.txt
--
diff --git a/NOTICE.txt b/NOTICE.txt
index b5c9c5d..0718909 100644
--- a/NOTICE.txt
+++ b/NOTICE.txt
@@ -8,19 +8,10 @@ following notices:
 * Copyright 2011 FuseSource Corp. http://fusesource.com
 
 The binary distribution of this product bundles binaries of
-org.fusesource.hawtjni:hawtjni-runtime (https://github.com/fusesource/hawtjni),
-which has the following notices:
-* This product includes software developed by FuseSource Corp.
-  http://fusesource.com
-* This product includes software developed at
-  Progress Software Corporation and/or its  subsidiaries or affiliates.
-* This product includes software developed by IBM Corporation and others.
-
-The binary distribution of this product bundles binaries of
-AWS SDK for Java - Core 1.11.45,
-AWS Java SDK for AWS KMS 1.11.45,
-AWS Java SDK for Amazon S3 1.11.45,
-AWS Java SDK for AWS STS 1.11.45,
+AWS SDK for Java - Bundle 1.11.134,
+AWS Java SDK for AWS KMS 1.11.134,
+AWS Java SDK for Amazon S3 1.11.134,
+AWS Java SDK for AWS STS 1.11.134,
 JMES Path Query library 1.0,
 which has the following notices:
  * This software includes third party software subject to the following
@@ -303,15 +294,15 @@ which has the following notices:
   notice.
 
 The binary distribution of this product bundles binaries of
-Jetty :: Http Utility 9.3.11.,
-Jetty :: IO Utility 9.3.11.,
-Jetty :: Security 9.3.11.,
-Jetty :: Server Core 9.3.11.,
-Jetty :: Servlet Handling 9.3.11.,
-Jetty :: Utilities 9.3.11.,
+Jetty :: Http Utility 9.3.19.,
+Jetty :: IO Utility 9.3.19.,
+Jetty :: Security 9.3.19.,
+Jetty :: Server Core 9.3.19.,
+Jetty :: Servlet Handling 9.3.19.,
+Jetty :: Utilities 9.3.19.,
 Jetty :: Utilities :: Ajax,
-Jetty :: Webapp Application Support 9.3.11.,
-Jetty :: XML utilities 9.3.11.,
+Jetty :: Webapp Application Support 9.3.19.,
+Jetty :: XML utilities 9.3.19.,
 which has the following notices:
  * ==
 Jetty Web Container
@@ -481,15 +472,15 @@ which has the following notices:
Copyright (C) 1999-2012, QOS.ch. All rights reserved.
 
 The binary distribution of this product bundles binaries of
-Apache HBase - Annotations 1.2.4,
-Apache HBase - Client 1.2.4,
-Apache HBase - Common 1.2.4,
-Apache HBase - Hadoop Compatibility 1.2.4,
-Apache HBase - Hadoop Two Compatibility 1.2.4,
-Apache HBase - Prefix Tree 1.2.4,
-Apache HBase - Procedure 1.2.4,
-Apache HBase - Protocol 1.2.4,
-Apache HBase - Server 1.2.4,
+Apache HBase - Annotations 1.2.6,
+Apache HBase - Client 1.2.6,
+Apache HBase - Common 1.2.6,
+Apache HBase - Hadoop Compatibility 1.2.6,
+Apache HBase - Hadoop Two Compatibility 1.2.6,
+Apache HBase - Prefix Tree 1.2.6,
+Apache HBase - Procedure 1.2.6,
+Apache HBase - Protocol 1.2.6,
+Apache HBase - Server 1.2.6,
 which has the following notices:
  * Apache HBase
Copyright 2007-2015 The Apache Software Foundation
@@ -585,3 +576,8 @@ which has the following notices:

=
This product includes software developed by
Joda.org (http://www.joda.org/).
+
+The binary distribution of this product bundles binaries of
+Ehcache 3.3.1,
+which has the following notices:
+ * Ehcache V3 Copyright 2014-2016 Terracotta, Inc.


-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org

hadoop git commit: HADOOP-14856. Fix AWS, Jetty, HBase, Ehcache entries for NOTICE.txt. (rchiang)

2017-09-12 Thread rchiang

Repository: hadoop
Updated Branches:
  refs/heads/trunk f1d751bd0 -> 2ffe93ab6


HADOOP-14856. Fix AWS, Jetty, HBase, Ehcache entries for NOTICE.txt. (rchiang)


Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/2ffe93ab
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/2ffe93ab
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/2ffe93ab

Branch: refs/heads/trunk
Commit: 2ffe93ab609dfb54c3d1a53273ac2bc5ad15a5dd
Parents: f1d751b
Author: Ray Chiang <rchi...@apache.org>
Authored: Tue Sep 12 13:37:38 2017 -0700
Committer: Ray Chiang <rchi...@apache.org>
Committed: Tue Sep 12 13:37:38 2017 -0700

--
 NOTICE.txt | 56 ++--
 1 file changed, 26 insertions(+), 30 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/hadoop/blob/2ffe93ab/NOTICE.txt
--
diff --git a/NOTICE.txt b/NOTICE.txt
index b5c9c5d..0718909 100644
--- a/NOTICE.txt
+++ b/NOTICE.txt
@@ -8,19 +8,10 @@ following notices:
 * Copyright 2011 FuseSource Corp. http://fusesource.com
 
 The binary distribution of this product bundles binaries of
-org.fusesource.hawtjni:hawtjni-runtime (https://github.com/fusesource/hawtjni),
-which has the following notices:
-* This product includes software developed by FuseSource Corp.
-  http://fusesource.com
-* This product includes software developed at
-  Progress Software Corporation and/or its  subsidiaries or affiliates.
-* This product includes software developed by IBM Corporation and others.
-
-The binary distribution of this product bundles binaries of
-AWS SDK for Java - Core 1.11.45,
-AWS Java SDK for AWS KMS 1.11.45,
-AWS Java SDK for Amazon S3 1.11.45,
-AWS Java SDK for AWS STS 1.11.45,
+AWS SDK for Java - Bundle 1.11.134,
+AWS Java SDK for AWS KMS 1.11.134,
+AWS Java SDK for Amazon S3 1.11.134,
+AWS Java SDK for AWS STS 1.11.134,
 JMES Path Query library 1.0,
 which has the following notices:
  * This software includes third party software subject to the following
@@ -303,15 +294,15 @@ which has the following notices:
   notice.
 
 The binary distribution of this product bundles binaries of
-Jetty :: Http Utility 9.3.11.,
-Jetty :: IO Utility 9.3.11.,
-Jetty :: Security 9.3.11.,
-Jetty :: Server Core 9.3.11.,
-Jetty :: Servlet Handling 9.3.11.,
-Jetty :: Utilities 9.3.11.,
+Jetty :: Http Utility 9.3.19.,
+Jetty :: IO Utility 9.3.19.,
+Jetty :: Security 9.3.19.,
+Jetty :: Server Core 9.3.19.,
+Jetty :: Servlet Handling 9.3.19.,
+Jetty :: Utilities 9.3.19.,
 Jetty :: Utilities :: Ajax,
-Jetty :: Webapp Application Support 9.3.11.,
-Jetty :: XML utilities 9.3.11.,
+Jetty :: Webapp Application Support 9.3.19.,
+Jetty :: XML utilities 9.3.19.,
 which has the following notices:
  * ==
 Jetty Web Container
@@ -481,15 +472,15 @@ which has the following notices:
Copyright (C) 1999-2012, QOS.ch. All rights reserved.
 
 The binary distribution of this product bundles binaries of
-Apache HBase - Annotations 1.2.4,
-Apache HBase - Client 1.2.4,
-Apache HBase - Common 1.2.4,
-Apache HBase - Hadoop Compatibility 1.2.4,
-Apache HBase - Hadoop Two Compatibility 1.2.4,
-Apache HBase - Prefix Tree 1.2.4,
-Apache HBase - Procedure 1.2.4,
-Apache HBase - Protocol 1.2.4,
-Apache HBase - Server 1.2.4,
+Apache HBase - Annotations 1.2.6,
+Apache HBase - Client 1.2.6,
+Apache HBase - Common 1.2.6,
+Apache HBase - Hadoop Compatibility 1.2.6,
+Apache HBase - Hadoop Two Compatibility 1.2.6,
+Apache HBase - Prefix Tree 1.2.6,
+Apache HBase - Procedure 1.2.6,
+Apache HBase - Protocol 1.2.6,
+Apache HBase - Server 1.2.6,
 which has the following notices:
  * Apache HBase
Copyright 2007-2015 The Apache Software Foundation
@@ -585,3 +576,8 @@ which has the following notices:

=
This product includes software developed by
Joda.org (http://www.joda.org/).
+
+The binary distribution of this product bundles binaries of
+Ehcache 3.3.1,
+which has the following notices:
+ * Ehcache V3 Copyright 2014-2016 Terracotta, Inc.


-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org

hadoop git commit: HADOOP-14797. Update re2j version to 1.1. (rchiang)

2017-09-12 Thread rchiang

Repository: hadoop
Updated Branches:
  refs/heads/branch-3.0 4e458e93f -> 5f6e387ba


HADOOP-14797. Update re2j version to 1.1. (rchiang)

(cherry picked from commit 288c3b12d46fccdb9638d823b1b6b0cbc076ec16)


Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/5f6e387b
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/5f6e387b
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/5f6e387b

Branch: refs/heads/branch-3.0
Commit: 5f6e387ba9306db9e42f9932afc56862954fef74
Parents: 4e458e9
Author: Ray Chiang <rchi...@apache.org>
Authored: Tue Sep 12 11:34:53 2017 -0700
Committer: Ray Chiang <rchi...@apache.org>
Committed: Tue Sep 12 11:35:15 2017 -0700

--
 LICENSE.txt| 2 +-
 hadoop-project/pom.xml | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/hadoop/blob/5f6e387b/LICENSE.txt
--
diff --git a/LICENSE.txt b/LICENSE.txt
index 3800adc..3f50521 100644
--- a/LICENSE.txt
+++ b/LICENSE.txt
@@ -359,7 +359,7 @@ OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF 
SUCH DAMAGE.
 
 The binary distribution of this product bundles these dependencies under the
 following license:
-re2j 1.0
+re2j 1.1
 -
 (GO license)
 This is a work derived from Russ Cox's RE2 in Go, whose license

http://git-wip-us.apache.org/repos/asf/hadoop/blob/5f6e387b/hadoop-project/pom.xml
--
diff --git a/hadoop-project/pom.xml b/hadoop-project/pom.xml
index c34b1c9..d6c5d66 100755
--- a/hadoop-project/pom.xml
+++ b/hadoop-project/pom.xml
@@ -76,7 +76,7 @@
 1.7.25
 
 
-1.0
+1.1
 
 
 


-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org

hadoop git commit: HADOOP-14797. Update re2j version to 1.1. (rchiang)

2017-09-12 Thread rchiang

Repository: hadoop
Updated Branches:
  refs/heads/trunk 80ee89b28 -> f1d751bd0


HADOOP-14797. Update re2j version to 1.1. (rchiang)


Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/f1d751bd
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/f1d751bd
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/f1d751bd

Branch: refs/heads/trunk
Commit: f1d751bd05988fb77a85e4ad5b07026a4c5a86af
Parents: 80ee89b
Author: Ray Chiang <rchi...@apache.org>
Authored: Tue Sep 12 11:34:53 2017 -0700
Committer: Ray Chiang <rchi...@apache.org>
Committed: Tue Sep 12 11:35:21 2017 -0700

--
 LICENSE.txt| 2 +-
 hadoop-project/pom.xml | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/hadoop/blob/f1d751bd/LICENSE.txt
--
diff --git a/LICENSE.txt b/LICENSE.txt
index 3800adc..3f50521 100644
--- a/LICENSE.txt
+++ b/LICENSE.txt
@@ -359,7 +359,7 @@ OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF 
SUCH DAMAGE.
 
 The binary distribution of this product bundles these dependencies under the
 following license:
-re2j 1.0
+re2j 1.1
 -
 (GO license)
 This is a work derived from Russ Cox's RE2 in Go, whose license

http://git-wip-us.apache.org/repos/asf/hadoop/blob/f1d751bd/hadoop-project/pom.xml
--
diff --git a/hadoop-project/pom.xml b/hadoop-project/pom.xml
index fc31452..4fa6b6e 100755
--- a/hadoop-project/pom.xml
+++ b/hadoop-project/pom.xml
@@ -76,7 +76,7 @@
 1.7.25
 
 
-1.0
+1.1
 
 
 


-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org

hadoop git commit: HADOOP-14653. Update joda-time version to 2.9.9. (rchiang)

2017-09-12 Thread rchiang

Repository: hadoop
Updated Branches:
  refs/heads/branch-3.0 85db3b2ae -> 619d61080


HADOOP-14653. Update joda-time version to 2.9.9. (rchiang)

(cherry picked from commit a6432ba5a177ec3d3a95fa79e313a9bbc531a1e7)


Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/619d6108
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/619d6108
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/619d6108

Branch: refs/heads/branch-3.0
Commit: 619d61080c4c436a13f4f1828a47e2cc8680be1f
Parents: 85db3b2
Author: Ray Chiang <rchi...@apache.org>
Authored: Tue Sep 12 11:12:44 2017 -0700
Committer: Ray Chiang <rchi...@apache.org>
Committed: Tue Sep 12 11:13:21 2017 -0700

--
 NOTICE.txt | 9 +
 hadoop-project/pom.xml | 2 +-
 2 files changed, 10 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/hadoop/blob/619d6108/NOTICE.txt
--
diff --git a/NOTICE.txt b/NOTICE.txt
index c41972b..b5c9c5d 100644
--- a/NOTICE.txt
+++ b/NOTICE.txt
@@ -576,3 +576,12 @@ The binary distribution of this product bundles binaries of
 software.amazon.ion:ion-java 1.0.1,
 which has the following notices:
  * Amazon Ion Java Copyright 2007-2016 Amazon.com, Inc. or its affiliates. All 
Rights Reserved.
+
+The binary distribution of this product bundles binaries of
+joda-time:joda-time:2.9.9
+which has the following notices:
+ * 
=
+   = NOTICE file corresponding to section 4d of the Apache License Version 2.0 
=
+   
=
+   This product includes software developed by
+   Joda.org (http://www.joda.org/).

http://git-wip-us.apache.org/repos/asf/hadoop/blob/619d6108/hadoop-project/pom.xml
--
diff --git a/hadoop-project/pom.xml b/hadoop-project/pom.xml
index a0420a4..c34b1c9 100755
--- a/hadoop-project/pom.xml
+++ b/hadoop-project/pom.xml
@@ -90,7 +90,7 @@
 
 11.0.2
 4.0
-2.9.4
+2.9.9
 
 
 2.0.0-M21


-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org

hadoop git commit: HADOOP-14653. Update joda-time version to 2.9.9. (rchiang)

2017-09-12 Thread rchiang

Repository: hadoop
Updated Branches:
  refs/heads/trunk 39818259c -> a6432ba5a


HADOOP-14653. Update joda-time version to 2.9.9. (rchiang)


Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/a6432ba5
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/a6432ba5
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/a6432ba5

Branch: refs/heads/trunk
Commit: a6432ba5a177ec3d3a95fa79e313a9bbc531a1e7
Parents: 3981825
Author: Ray Chiang <rchi...@apache.org>
Authored: Tue Sep 12 11:12:44 2017 -0700
Committer: Ray Chiang <rchi...@apache.org>
Committed: Tue Sep 12 11:12:44 2017 -0700

--
 NOTICE.txt | 9 +
 hadoop-project/pom.xml | 2 +-
 2 files changed, 10 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/hadoop/blob/a6432ba5/NOTICE.txt
--
diff --git a/NOTICE.txt b/NOTICE.txt
index c41972b..b5c9c5d 100644
--- a/NOTICE.txt
+++ b/NOTICE.txt
@@ -576,3 +576,12 @@ The binary distribution of this product bundles binaries of
 software.amazon.ion:ion-java 1.0.1,
 which has the following notices:
  * Amazon Ion Java Copyright 2007-2016 Amazon.com, Inc. or its affiliates. All 
Rights Reserved.
+
+The binary distribution of this product bundles binaries of
+joda-time:joda-time:2.9.9
+which has the following notices:
+ * 
=
+   = NOTICE file corresponding to section 4d of the Apache License Version 2.0 
=
+   
=
+   This product includes software developed by
+   Joda.org (http://www.joda.org/).

http://git-wip-us.apache.org/repos/asf/hadoop/blob/a6432ba5/hadoop-project/pom.xml
--
diff --git a/hadoop-project/pom.xml b/hadoop-project/pom.xml
index bb4bf44..fc31452 100755
--- a/hadoop-project/pom.xml
+++ b/hadoop-project/pom.xml
@@ -90,7 +90,7 @@
 
 11.0.2
 4.0
-2.9.4
+2.9.9
 
 
 2.0.0-M21


-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org

hadoop git commit: HADOOP-14648. Bump commons-configuration2 to 2.1.1. (rchiang)

2017-09-12 Thread rchiang

Repository: hadoop
Updated Branches:
  refs/heads/branch-3.0 fb5d12524 -> 85db3b2ae


HADOOP-14648. Bump commons-configuration2 to 2.1.1. (rchiang)

(cherry picked from commit 39818259c3b28a52b2cd7ee65b2422212d584664)


Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/85db3b2a
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/85db3b2a
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/85db3b2a

Branch: refs/heads/branch-3.0
Commit: 85db3b2aeb00bea61bf62a65864000d0a2cd4ef0
Parents: fb5d125
Author: Ray Chiang <rchi...@apache.org>
Authored: Tue Sep 12 10:53:48 2017 -0700
Committer: Ray Chiang <rchi...@apache.org>
Committed: Tue Sep 12 10:54:10 2017 -0700

--
 hadoop-project/pom.xml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/hadoop/blob/85db3b2a/hadoop-project/pom.xml
--
diff --git a/hadoop-project/pom.xml b/hadoop-project/pom.xml
index 82c3906..a0420a4 100755
--- a/hadoop-project/pom.xml
+++ b/hadoop-project/pom.xml
@@ -869,7 +869,7 @@
   
 org.apache.commons
 commons-configuration2
-2.1
+2.1.1
 
   
 org.apache.commons


-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org

hadoop git commit: HADOOP-14648. Bump commons-configuration2 to 2.1.1. (rchiang)

2017-09-12 Thread rchiang

Repository: hadoop
Updated Branches:
  refs/heads/trunk af45cd1be -> 39818259c


HADOOP-14648. Bump commons-configuration2 to 2.1.1. (rchiang)


Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/39818259
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/39818259
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/39818259

Branch: refs/heads/trunk
Commit: 39818259c3b28a52b2cd7ee65b2422212d584664
Parents: af45cd1
Author: Ray Chiang <rchi...@apache.org>
Authored: Tue Sep 12 10:53:48 2017 -0700
Committer: Ray Chiang <rchi...@apache.org>
Committed: Tue Sep 12 10:53:48 2017 -0700

--
 hadoop-project/pom.xml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/hadoop/blob/39818259/hadoop-project/pom.xml
--
diff --git a/hadoop-project/pom.xml b/hadoop-project/pom.xml
index c586a9a..bb4bf44 100755
--- a/hadoop-project/pom.xml
+++ b/hadoop-project/pom.xml
@@ -869,7 +869,7 @@
   
 org.apache.commons
 commons-configuration2
-2.1
+2.1.1
 
   
 org.apache.commons


-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org

hadoop git commit: HADOOP-14796. Update json-simple version to 1.1.1. (rchiang)

2017-09-12 Thread rchiang

Repository: hadoop
Updated Branches:
  refs/heads/branch-3.0 656c6d1ac -> fb5d12524


HADOOP-14796. Update json-simple version to 1.1.1. (rchiang)

(cherry picked from commit af45cd1be63c3eee968f0c820e7d70230a7bf246)


Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/fb5d1252
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/fb5d1252
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/fb5d1252

Branch: refs/heads/branch-3.0
Commit: fb5d12524ca79a1a592e162dcbfe7c670695f41d
Parents: 656c6d1
Author: Ray Chiang <rchi...@apache.org>
Authored: Tue Sep 12 10:36:04 2017 -0700
Committer: Ray Chiang <rchi...@apache.org>
Committed: Tue Sep 12 10:36:28 2017 -0700

--
 hadoop-project/pom.xml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/hadoop/blob/fb5d1252/hadoop-project/pom.xml
--
diff --git a/hadoop-project/pom.xml b/hadoop-project/pom.xml
index 5b994d5..82c3906 100755
--- a/hadoop-project/pom.xml
+++ b/hadoop-project/pom.xml
@@ -1026,7 +1026,7 @@
   
 com.googlecode.json-simple
 json-simple
-1.1
+1.1.1
   
 
   


-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org

hadoop git commit: HADOOP-14796. Update json-simple version to 1.1.1. (rchiang)

2017-09-12 Thread rchiang

Repository: hadoop
Updated Branches:
  refs/heads/trunk 556812c17 -> af45cd1be


HADOOP-14796. Update json-simple version to 1.1.1. (rchiang)


Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/af45cd1b
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/af45cd1b
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/af45cd1b

Branch: refs/heads/trunk
Commit: af45cd1be63c3eee968f0c820e7d70230a7bf246
Parents: 556812c
Author: Ray Chiang <rchi...@apache.org>
Authored: Tue Sep 12 10:36:04 2017 -0700
Committer: Ray Chiang <rchi...@apache.org>
Committed: Tue Sep 12 10:36:04 2017 -0700

--
 hadoop-project/pom.xml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/hadoop/blob/af45cd1b/hadoop-project/pom.xml
--
diff --git a/hadoop-project/pom.xml b/hadoop-project/pom.xml
index ee08754..c586a9a 100755
--- a/hadoop-project/pom.xml
+++ b/hadoop-project/pom.xml
@@ -1026,7 +1026,7 @@
   
 com.googlecode.json-simple
 json-simple
-1.1
+1.1.1
   
 
   


-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org

hadoop git commit: HADOOP-14799. Update nimbus-jose-jwt to 4.41.1. (rchiang)

2017-09-12 Thread rchiang

Repository: hadoop
Updated Branches:
  refs/heads/branch-3.0 f811306c8 -> 656c6d1ac


HADOOP-14799. Update nimbus-jose-jwt to 4.41.1. (rchiang)

(cherry picked from commit 556812c179aa094c21acf610439a8d69fe6420ab)


Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/656c6d1a
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/656c6d1a
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/656c6d1a

Branch: refs/heads/branch-3.0
Commit: 656c6d1acc4b2c06bb573a257c0d28bde2c355b4
Parents: f811306
Author: Ray Chiang <rchi...@apache.org>
Authored: Tue Sep 12 10:19:34 2017 -0700
Committer: Ray Chiang <rchi...@apache.org>
Committed: Tue Sep 12 10:19:58 2017 -0700

--
 .../JWTRedirectAuthenticationHandler.java   |   4 +-
 .../TestJWTRedirectAuthenticationHandler.java   | 476 ++
 .../TestJWTRedirectAuthentictionHandler.java| 481 ---
 .../mapreduce/task/reduce/TestFetcher.java  |   4 -
 hadoop-project/pom.xml  |   2 +-
 5 files changed, 480 insertions(+), 487 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/hadoop/blob/656c6d1a/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/server/JWTRedirectAuthenticationHandler.java
--
diff --git 
a/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/server/JWTRedirectAuthenticationHandler.java
 
b/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/server/JWTRedirectAuthenticationHandler.java
index 61f5b9e..884398c 100644
--- 
a/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/server/JWTRedirectAuthenticationHandler.java
+++ 
b/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/server/JWTRedirectAuthenticationHandler.java
@@ -28,6 +28,7 @@ import java.text.ParseException;
 
 import java.security.interfaces.RSAPublicKey;
 
+import com.google.common.annotations.VisibleForTesting;
 import 
org.apache.hadoop.security.authentication.client.AuthenticationException;
 import org.apache.hadoop.security.authentication.util.CertificateUtil;
 import org.slf4j.Logger;
@@ -216,7 +217,8 @@ public class JWTRedirectAuthenticationHandler extends
* @param request for getting the original request URL
* @return url to use as login url for redirect
*/
-  protected String constructLoginURL(HttpServletRequest request) {
+  @VisibleForTesting
+  String constructLoginURL(HttpServletRequest request) {
 String delimiter = "?";
 if (authenticationProviderUrl.contains("?")) {
   delimiter = "&";

http://git-wip-us.apache.org/repos/asf/hadoop/blob/656c6d1a/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/server/TestJWTRedirectAuthenticationHandler.java
--
diff --git 
a/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/server/TestJWTRedirectAuthenticationHandler.java
 
b/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/server/TestJWTRedirectAuthenticationHandler.java
new file mode 100644
index 000..5a2db9b
--- /dev/null
+++ 
b/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/server/TestJWTRedirectAuthenticationHandler.java
@@ -0,0 +1,476 @@
+/**
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License. See accompanying LICENSE file.
+ */
+package org.apache.hadoop.security.authentication.server;
+
+import static org.junit.Assert.assertTrue;
+import static org.junit.Assert.fail;
+
+import java.io.File;
+import java.security.KeyPair;
+import java.security.KeyPairGenerator;
+import java.security.NoSuchAlgorithmException;
+import java.security.interfaces.RSAPrivateKey;
+import java.security.interfaces.RSAPublicKey;
+import java.util.List;
+import java.util.ArrayList;
+import java.util.Properties;
+import java.util.Vector;
+import java.util.Date;
+
+import javax.servlet.ServletException;
+import javax.servlet.http.Cookie;
+import javax.servlet.ht

hadoop git commit: HADOOP-14799. Update nimbus-jose-jwt to 4.41.1. (rchiang)

2017-09-12 Thread rchiang

Repository: hadoop
Updated Branches:
  refs/heads/trunk ad7469180 -> 556812c17


HADOOP-14799. Update nimbus-jose-jwt to 4.41.1. (rchiang)


Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/556812c1
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/556812c1
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/556812c1

Branch: refs/heads/trunk
Commit: 556812c179aa094c21acf610439a8d69fe6420ab
Parents: ad74691
Author: Ray Chiang <rchi...@apache.org>
Authored: Tue Sep 12 10:19:34 2017 -0700
Committer: Ray Chiang <rchi...@apache.org>
Committed: Tue Sep 12 10:19:34 2017 -0700

--
 .../JWTRedirectAuthenticationHandler.java   |   4 +-
 .../TestJWTRedirectAuthenticationHandler.java   | 476 ++
 .../TestJWTRedirectAuthentictionHandler.java| 481 ---
 .../mapreduce/task/reduce/TestFetcher.java  |   4 -
 hadoop-project/pom.xml  |   2 +-
 5 files changed, 480 insertions(+), 487 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/hadoop/blob/556812c1/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/server/JWTRedirectAuthenticationHandler.java
--
diff --git 
a/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/server/JWTRedirectAuthenticationHandler.java
 
b/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/server/JWTRedirectAuthenticationHandler.java
index 61f5b9e..884398c 100644
--- 
a/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/server/JWTRedirectAuthenticationHandler.java
+++ 
b/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/server/JWTRedirectAuthenticationHandler.java
@@ -28,6 +28,7 @@ import java.text.ParseException;
 
 import java.security.interfaces.RSAPublicKey;
 
+import com.google.common.annotations.VisibleForTesting;
 import 
org.apache.hadoop.security.authentication.client.AuthenticationException;
 import org.apache.hadoop.security.authentication.util.CertificateUtil;
 import org.slf4j.Logger;
@@ -216,7 +217,8 @@ public class JWTRedirectAuthenticationHandler extends
* @param request for getting the original request URL
* @return url to use as login url for redirect
*/
-  protected String constructLoginURL(HttpServletRequest request) {
+  @VisibleForTesting
+  String constructLoginURL(HttpServletRequest request) {
 String delimiter = "?";
 if (authenticationProviderUrl.contains("?")) {
   delimiter = "&";

http://git-wip-us.apache.org/repos/asf/hadoop/blob/556812c1/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/server/TestJWTRedirectAuthenticationHandler.java
--
diff --git 
a/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/server/TestJWTRedirectAuthenticationHandler.java
 
b/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/server/TestJWTRedirectAuthenticationHandler.java
new file mode 100644
index 000..5a2db9b
--- /dev/null
+++ 
b/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/server/TestJWTRedirectAuthenticationHandler.java
@@ -0,0 +1,476 @@
+/**
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License. See accompanying LICENSE file.
+ */
+package org.apache.hadoop.security.authentication.server;
+
+import static org.junit.Assert.assertTrue;
+import static org.junit.Assert.fail;
+
+import java.io.File;
+import java.security.KeyPair;
+import java.security.KeyPairGenerator;
+import java.security.NoSuchAlgorithmException;
+import java.security.interfaces.RSAPrivateKey;
+import java.security.interfaces.RSAPublicKey;
+import java.util.List;
+import java.util.ArrayList;
+import java.util.Properties;
+import java.util.Vector;
+import java.util.Date;
+
+import javax.servlet.ServletException;
+import javax.servlet.http.Cookie;
+import javax.servlet.http.HttpServletRequest;
+import javax.servlet.http.HttpServletResponse;

hadoop git commit: HADOOP-14798. Update sshd-core and related mina-core library versions. (rchiang)

2017-09-12 Thread rchiang

Repository: hadoop
Updated Branches:
  refs/heads/branch-3.0 d2e381882 -> f811306c8


HADOOP-14798. Update sshd-core and related mina-core library versions. (rchiang)

(cherry picked from commit 3876fe4e500efcb88fff93a86acf9626d961c8c7)


Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/f811306c
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/f811306c
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/f811306c

Branch: refs/heads/branch-3.0
Commit: f811306c861183e17e94554a34042516b1213eae
Parents: d2e3818
Author: Ray Chiang <rchi...@apache.org>
Authored: Mon Sep 11 16:52:47 2017 -0700
Committer: Ray Chiang <rchi...@apache.org>
Committed: Tue Sep 12 10:00:25 2017 -0700

--
 .../apache/hadoop/fs/sftp/TestSFTPFileSystem.java   | 16 
 hadoop-project/pom.xml  |  4 ++--
 2 files changed, 10 insertions(+), 10 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/hadoop/blob/f811306c/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/sftp/TestSFTPFileSystem.java
--
diff --git 
a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/sftp/TestSFTPFileSystem.java
 
b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/sftp/TestSFTPFileSystem.java
index e8eedd9..af48cb6 100644
--- 
a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/sftp/TestSFTPFileSystem.java
+++ 
b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/sftp/TestSFTPFileSystem.java
@@ -34,16 +34,16 @@ import org.apache.hadoop.fs.LocalFileSystem;
 import org.apache.hadoop.fs.Path;
 import org.apache.hadoop.test.GenericTestUtils;
 
-import org.apache.sshd.SshServer;
+import org.apache.sshd.server.SshServer;
 import org.apache.sshd.common.NamedFactory;
 import org.apache.sshd.server.Command;
-import org.apache.sshd.server.PasswordAuthenticator;
-import org.apache.sshd.server.UserAuth;
-import org.apache.sshd.server.auth.UserAuthPassword;
+import org.apache.sshd.server.auth.password.PasswordAuthenticator;
+import org.apache.sshd.server.auth.UserAuth;
+import org.apache.sshd.server.auth.password.UserAuthPasswordFactory;
 import org.apache.sshd.server.keyprovider.SimpleGeneratorHostKeyProvider;
 import org.apache.sshd.server.session.ServerSession;
-import org.apache.sshd.server.sftp.SftpSubsystem;
 
+import org.apache.sshd.server.subsystem.sftp.SftpSubsystemFactory;
 import org.junit.AfterClass;
 import org.junit.BeforeClass;
 import org.junit.Rule;
@@ -76,7 +76,7 @@ public class TestSFTPFileSystem {
 
 List<NamedFactory> userAuthFactories =
 new ArrayList<NamedFactory>();
-userAuthFactories.add(new UserAuthPassword.Factory());
+userAuthFactories.add(new UserAuthPasswordFactory());
 
 sshd.setUserAuthFactories(userAuthFactories);
 
@@ -92,7 +92,7 @@ public class TestSFTPFileSystem {
 });
 
 sshd.setSubsystemFactories(
-Arrays.<NamedFactory>asList(new SftpSubsystem.Factory()));
+Arrays.<NamedFactory>asList(new SftpSubsystemFactory()));
 
 sshd.start();
 port = sshd.getPort();
@@ -140,7 +140,7 @@ public class TestSFTPFileSystem {
 if (sshd != null) {
   try {
 sshd.stop(true);
-  } catch (InterruptedException e) {
+  } catch (IOException e) {
 // ignore
   }
 }

http://git-wip-us.apache.org/repos/asf/hadoop/blob/f811306c/hadoop-project/pom.xml
--
diff --git a/hadoop-project/pom.xml b/hadoop-project/pom.xml
index 5a3bd50..3e9d114 100755
--- a/hadoop-project/pom.xml
+++ b/hadoop-project/pom.xml
@@ -824,12 +824,12 @@
   
 org.apache.mina
 mina-core
-2.0.0-M5
+2.0.16
   
   
 org.apache.sshd
 sshd-core
-0.14.0
+1.6.0
   
   
 org.apache.ftpserver


-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org

hadoop git commit: HADOOP-14798. Update sshd-core and related mina-core library versions. (rchiang)

2017-09-12 Thread rchiang

Repository: hadoop
Updated Branches:
  refs/heads/trunk da0b6a354 -> ad7469180


HADOOP-14798. Update sshd-core and related mina-core library versions. (rchiang)


Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/ad746918
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/ad746918
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/ad746918

Branch: refs/heads/trunk
Commit: ad74691807c51713bd9071e731d4481ee212567e
Parents: da0b6a3
Author: Ray Chiang <rchi...@apache.org>
Authored: Mon Sep 11 16:52:47 2017 -0700
Committer: Ray Chiang <rchi...@apache.org>
Committed: Tue Sep 12 10:04:22 2017 -0700

--
 .../apache/hadoop/fs/sftp/TestSFTPFileSystem.java   | 16 
 hadoop-project/pom.xml  |  4 ++--
 2 files changed, 10 insertions(+), 10 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/hadoop/blob/ad746918/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/sftp/TestSFTPFileSystem.java
--
diff --git 
a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/sftp/TestSFTPFileSystem.java
 
b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/sftp/TestSFTPFileSystem.java
index e8eedd9..af48cb6 100644
--- 
a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/sftp/TestSFTPFileSystem.java
+++ 
b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/sftp/TestSFTPFileSystem.java
@@ -34,16 +34,16 @@ import org.apache.hadoop.fs.LocalFileSystem;
 import org.apache.hadoop.fs.Path;
 import org.apache.hadoop.test.GenericTestUtils;
 
-import org.apache.sshd.SshServer;
+import org.apache.sshd.server.SshServer;
 import org.apache.sshd.common.NamedFactory;
 import org.apache.sshd.server.Command;
-import org.apache.sshd.server.PasswordAuthenticator;
-import org.apache.sshd.server.UserAuth;
-import org.apache.sshd.server.auth.UserAuthPassword;
+import org.apache.sshd.server.auth.password.PasswordAuthenticator;
+import org.apache.sshd.server.auth.UserAuth;
+import org.apache.sshd.server.auth.password.UserAuthPasswordFactory;
 import org.apache.sshd.server.keyprovider.SimpleGeneratorHostKeyProvider;
 import org.apache.sshd.server.session.ServerSession;
-import org.apache.sshd.server.sftp.SftpSubsystem;
 
+import org.apache.sshd.server.subsystem.sftp.SftpSubsystemFactory;
 import org.junit.AfterClass;
 import org.junit.BeforeClass;
 import org.junit.Rule;
@@ -76,7 +76,7 @@ public class TestSFTPFileSystem {
 
 List<NamedFactory> userAuthFactories =
 new ArrayList<NamedFactory>();
-userAuthFactories.add(new UserAuthPassword.Factory());
+userAuthFactories.add(new UserAuthPasswordFactory());
 
 sshd.setUserAuthFactories(userAuthFactories);
 
@@ -92,7 +92,7 @@ public class TestSFTPFileSystem {
 });
 
 sshd.setSubsystemFactories(
-Arrays.<NamedFactory>asList(new SftpSubsystem.Factory()));
+Arrays.<NamedFactory>asList(new SftpSubsystemFactory()));
 
 sshd.start();
 port = sshd.getPort();
@@ -140,7 +140,7 @@ public class TestSFTPFileSystem {
 if (sshd != null) {
   try {
 sshd.stop(true);
-  } catch (InterruptedException e) {
+  } catch (IOException e) {
 // ignore
   }
 }

http://git-wip-us.apache.org/repos/asf/hadoop/blob/ad746918/hadoop-project/pom.xml
--
diff --git a/hadoop-project/pom.xml b/hadoop-project/pom.xml
index a414fcd..24c779c 100755
--- a/hadoop-project/pom.xml
+++ b/hadoop-project/pom.xml
@@ -824,12 +824,12 @@
   
 org.apache.mina
 mina-core
-2.0.0-M5
+2.0.16
   
   
 org.apache.sshd
 sshd-core
-0.14.0
+1.6.0
   
   
 org.apache.ftpserver


-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org

hadoop git commit: HADOOP-14655. Update httpcore version to 4.4.6. (rchiang)

2017-09-11 Thread rchiang

Repository: hadoop
Updated Branches:
  refs/heads/branch-3.0 14bec7043 -> 0b69a22f8


HADOOP-14655. Update httpcore version to 4.4.6. (rchiang)

(cherry picked from commit 088a18b44d87c6994b571696edbf87c95528dcaa)


Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/0b69a22f
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/0b69a22f
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/0b69a22f

Branch: refs/heads/branch-3.0
Commit: 0b69a22f8688ae98e3c506b0bd1d26f97ccb391e
Parents: 14bec70
Author: Ray Chiang <rchi...@apache.org>
Authored: Mon Sep 11 13:54:40 2017 -0700
Committer: Ray Chiang <rchi...@apache.org>
Committed: Mon Sep 11 13:55:49 2017 -0700

--
 hadoop-project/pom.xml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/hadoop/blob/0b69a22f/hadoop-project/pom.xml
--
diff --git a/hadoop-project/pom.xml b/hadoop-project/pom.xml
index 5b39f60..5a3bd50 100755
--- a/hadoop-project/pom.xml
+++ b/hadoop-project/pom.xml
@@ -576,7 +576,7 @@
   
 org.apache.httpcomponents
 httpcore
-4.4.4
+4.4.6
   
   
 commons-codec


-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org

hadoop git commit: HADOOP-14655. Update httpcore version to 4.4.6. (rchiang)

2017-09-11 Thread rchiang

Repository: hadoop
Updated Branches:
  refs/heads/trunk 1118ca9b0 -> 088a18b44


HADOOP-14655. Update httpcore version to 4.4.6. (rchiang)


Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/088a18b4
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/088a18b4
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/088a18b4

Branch: refs/heads/trunk
Commit: 088a18b44d87c6994b571696edbf87c95528dcaa
Parents: 1118ca9
Author: Ray Chiang <rchi...@apache.org>
Authored: Mon Sep 11 13:54:40 2017 -0700
Committer: Ray Chiang <rchi...@apache.org>
Committed: Mon Sep 11 13:54:40 2017 -0700

--
 hadoop-project/pom.xml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/hadoop/blob/088a18b4/hadoop-project/pom.xml
--
diff --git a/hadoop-project/pom.xml b/hadoop-project/pom.xml
index 6bfb213..a414fcd 100755
--- a/hadoop-project/pom.xml
+++ b/hadoop-project/pom.xml
@@ -576,7 +576,7 @@
   
 org.apache.httpcomponents
 httpcore
-4.4.4
+4.4.6
   
   
 commons-codec


-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org

hadoop git commit: HADOOP-14654. Update httpclient version to 4.5.3. (rchiang)

2017-09-11 Thread rchiang

Repository: hadoop
Updated Branches:
  refs/heads/trunk d2d8f665e -> 1118ca9b0


HADOOP-14654. Update httpclient version to 4.5.3. (rchiang)


Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/1118ca9b
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/1118ca9b
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/1118ca9b

Branch: refs/heads/trunk
Commit: 1118ca9b07f8a92821de7d9c9b56272a7eb66183
Parents: d2d8f66
Author: Ray Chiang <rchi...@apache.org>
Authored: Mon Sep 11 13:33:42 2017 -0700
Committer: Ray Chiang <rchi...@apache.org>
Committed: Mon Sep 11 13:33:42 2017 -0700

--
 hadoop-project/pom.xml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/hadoop/blob/1118ca9b/hadoop-project/pom.xml
--
diff --git a/hadoop-project/pom.xml b/hadoop-project/pom.xml
index f74c0e8..6bfb213 100755
--- a/hadoop-project/pom.xml
+++ b/hadoop-project/pom.xml
@@ -571,7 +571,7 @@
   
 org.apache.httpcomponents
 httpclient
-4.5.2
+4.5.3
   
   
 org.apache.httpcomponents


-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org

hadoop git commit: HADOOP-14654. Update httpclient version to 4.5.3. (rchiang)

2017-09-11 Thread rchiang

Repository: hadoop
Updated Branches:
  refs/heads/branch-3.0 cb95c442d -> 14bec7043


HADOOP-14654. Update httpclient version to 4.5.3. (rchiang)


Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/14bec704
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/14bec704
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/14bec704

Branch: refs/heads/branch-3.0
Commit: 14bec7043a622a4226b0b267818efae32accd1bf
Parents: cb95c44
Author: Ray Chiang <rchi...@apache.org>
Authored: Mon Sep 11 13:43:48 2017 -0700
Committer: Ray Chiang <rchi...@apache.org>
Committed: Mon Sep 11 13:43:48 2017 -0700

--
 hadoop-project/pom.xml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/hadoop/blob/14bec704/hadoop-project/pom.xml
--
diff --git a/hadoop-project/pom.xml b/hadoop-project/pom.xml
index 9a52d76..5b39f60 100755
--- a/hadoop-project/pom.xml
+++ b/hadoop-project/pom.xml
@@ -571,7 +571,7 @@
   
 org.apache.httpcomponents
 httpclient
-4.5.2
+4.5.3
   
   
 org.apache.httpcomponents


-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org

hadoop git commit: HADOOP-14649. Update aliyun-sdk-oss version to 2.8.1. (Genmao Yu via rchiang)

2017-08-23 Thread rchiang

Repository: hadoop
Updated Branches:
  refs/heads/trunk 7e6463d2f -> 26d8c8fa5


HADOOP-14649. Update aliyun-sdk-oss version to 2.8.1. (Genmao Yu via rchiang)


Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/26d8c8fa
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/26d8c8fa
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/26d8c8fa

Branch: refs/heads/trunk
Commit: 26d8c8fa586a634ae91993940b28b3a1452d4be6
Parents: 7e6463d
Author: Ray Chiang <rchi...@apache.org>
Authored: Wed Aug 23 15:22:24 2017 -0700
Committer: Ray Chiang <rchi...@apache.org>
Committed: Wed Aug 23 15:22:24 2017 -0700

--
 hadoop-project/pom.xml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/hadoop/blob/26d8c8fa/hadoop-project/pom.xml
--
diff --git a/hadoop-project/pom.xml b/hadoop-project/pom.xml
index 8c1d374..938ef05 100755
--- a/hadoop-project/pom.xml
+++ b/hadoop-project/pom.xml
@@ -1140,7 +1140,7 @@
   
 com.aliyun.oss
 aliyun-sdk-oss
-2.4.1
+2.8.1
 
   
 org.apache.httpcomponents


-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org

hadoop git commit: YARN-6798. Fix NM startup failure with old state store due to version mismatch. (Botong Huang via rchiang)

2017-07-18 Thread rchiang

Repository: hadoop
Updated Branches:
  refs/heads/trunk 0b7afc060 -> f5f14a2ad


YARN-6798. Fix NM startup failure with old state store due to version mismatch. 
(Botong Huang via rchiang)


Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/f5f14a2a
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/f5f14a2a
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/f5f14a2a

Branch: refs/heads/trunk
Commit: f5f14a2ad67f91064a73685b44369c6314f0e1cd
Parents: 0b7afc0
Author: Ray Chiang <rchi...@apache.org>
Authored: Tue Jul 18 12:35:08 2017 -0700
Committer: Ray Chiang <rchi...@apache.org>
Committed: Tue Jul 18 12:35:08 2017 -0700

--
 .../nodemanager/recovery/NMLeveldbStateStoreService.java   | 6 +-
 1 file changed, 5 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/hadoop/blob/f5f14a2a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/recovery/NMLeveldbStateStoreService.java
--
diff --git 
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/recovery/NMLeveldbStateStoreService.java
 
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/recovery/NMLeveldbStateStoreService.java
index a0502df..c556b39 100644
--- 
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/recovery/NMLeveldbStateStoreService.java
+++ 
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/recovery/NMLeveldbStateStoreService.java
@@ -85,7 +85,11 @@ public class NMLeveldbStateStoreService extends 
NMStateStoreService {
   private static final String DB_NAME = "yarn-nm-state";
   private static final String DB_SCHEMA_VERSION_KEY = "nm-schema-version";
 
-  private static final Version CURRENT_VERSION_INFO = Version.newInstance(3, 
0);
+  /**
+   * Changes from 1.0 to 1.1: Save AMRMProxy state in NMSS.
+   * Changes from 1.2 to 1.2: Save queued container information.
+   */
+  private static final Version CURRENT_VERSION_INFO = Version.newInstance(1, 
2);
 
   private static final String DELETION_TASK_KEY_PREFIX =
   "DeletionService/deltask_";


-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org

hadoop git commit: YARN-6331. Fix flakiness in TestFairScheduler#testDumpState. (Yufei Gu via rchiang)

2017-03-14 Thread rchiang

Repository: hadoop
Updated Branches:
  refs/heads/trunk fa67a96d7 -> 4c66a8d19


YARN-6331. Fix flakiness in TestFairScheduler#testDumpState. (Yufei Gu via 
rchiang)


Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/4c66a8d1
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/4c66a8d1
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/4c66a8d1

Branch: refs/heads/trunk
Commit: 4c66a8d19b7d503095ad27aeed39d62238b9cb47
Parents: fa67a96
Author: Ray Chiang <rchi...@apache.org>
Authored: Tue Mar 14 14:37:18 2017 -0700
Committer: Ray Chiang <rchi...@apache.org>
Committed: Tue Mar 14 15:09:47 2017 -0700

--
 .../server/resourcemanager/scheduler/fair/TestFairScheduler.java  | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/hadoop/blob/4c66a8d1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairScheduler.java
--
diff --git 
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairScheduler.java
 
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairScheduler.java
index 537d3d0..baf7434 100644
--- 
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairScheduler.java
+++ 
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairScheduler.java
@@ -5201,6 +5201,7 @@ public class TestFairScheduler extends 
FairSchedulerTestBase {
 FSParentQueue parent =
 scheduler.getQueueManager().getParentQueue("parent", false);
 parent.setMaxShare(resource);
+parent.updateDemand();
 
 String parentQueueString = "{Name: root.parent,"
 + " Weight: ,"
@@ -5210,7 +5211,7 @@ public class TestFairScheduler extends 
FairSchedulerTestBase {
 + " MaxShare: <memory:4096, vCores:4>,"
 + " MinShare: <memory:0, vCores:0>,"
 + " ResourceUsage: <memory:4096, vCores:4>,"
-+ " Demand: <memory:0, vCores:0>,"
++ " Demand: <memory:4096, vCores:4>,"
 + " MaxAMShare: 0.5,"
 + " Runnable: 0}";
 


-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org

hadoop git commit: YARN-6042. Dump scheduler and queue state information into FairScheduler DEBUG log. (Yufei Gu via rchiang)

2017-03-14 Thread rchiang

Repository: hadoop
Updated Branches:
  refs/heads/branch-2 0e7879052 -> fa59f4e49


YARN-6042. Dump scheduler and queue state information into FairScheduler DEBUG 
log. (Yufei Gu via rchiang)


Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/fa59f4e4
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/fa59f4e4
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/fa59f4e4

Branch: refs/heads/branch-2
Commit: fa59f4e4907d2c37841d59656d79b3162e774310
Parents: 0e78790
Author: Ray Chiang <rchi...@apache.org>
Authored: Tue Mar 14 14:45:13 2017 -0700
Committer: Ray Chiang <rchi...@apache.org>
Committed: Tue Mar 14 14:45:13 2017 -0700

--
 .../src/main/conf/log4j.properties  |  9 +++
 .../scheduler/fair/FSAppAttempt.java| 51 +++---
 .../scheduler/fair/FSLeafQueue.java | 21 ++
 .../scheduler/fair/FSParentQueue.java   | 21 ++
 .../resourcemanager/scheduler/fair/FSQueue.java | 41 ++-
 .../scheduler/fair/FairScheduler.java   | 28 +---
 .../scheduler/fair/TestFairScheduler.java   | 72 
 7 files changed, 209 insertions(+), 34 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/hadoop/blob/fa59f4e4/hadoop-common-project/hadoop-common/src/main/conf/log4j.properties
--
diff --git a/hadoop-common-project/hadoop-common/src/main/conf/log4j.properties 
b/hadoop-common-project/hadoop-common/src/main/conf/log4j.properties
index 95afc61..7c02b20 100644
--- a/hadoop-common-project/hadoop-common/src/main/conf/log4j.properties
+++ b/hadoop-common-project/hadoop-common/src/main/conf/log4j.properties
@@ -321,3 +321,12 @@ 
log4j.appender.EWMA=org.apache.hadoop.yarn.util.Log4jWarningErrorMetricsAppender
 log4j.appender.EWMA.cleanupInterval=${yarn.ewma.cleanupInterval}
 log4j.appender.EWMA.messageAgeLimitSeconds=${yarn.ewma.messageAgeLimitSeconds}
 log4j.appender.EWMA.maxUniqueMessages=${yarn.ewma.maxUniqueMessages}
+
+# Fair scheduler requests log on state dump
+log4j.logger.org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.statedump=DEBUG,FSLOGGER
+log4j.appender.FSLOGGER=org.apache.log4j.RollingFileAppender
+log4j.appender.FSLOGGER.File=${hadoop.log.dir}/fairscheduler-statedump.log
+log4j.appender.FSLOGGER.layout=org.apache.log4j.PatternLayout
+log4j.appender.FSLOGGER.layout.ConversionPattern=%d{ISO8601} %p %c: %m%n
+log4j.appender.FSLOGGER.MaxFileSize=${hadoop.log.maxfilesize}
+log4j.appender.FSLOGGER.MaxBackupIndex=${hadoop.log.maxbackupindex}
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/hadoop/blob/fa59f4e4/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSAppAttempt.java
--
diff --git 
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSAppAttempt.java
 
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSAppAttempt.java
index 60902a2..2e08343 100644
--- 
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSAppAttempt.java
+++ 
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSAppAttempt.java
@@ -834,24 +834,26 @@ public class FSAppAttempt extends 
SchedulerApplicationAttempt
   return capability;
 }
 
+if (LOG.isDebugEnabled()) {
+  LOG.debug("Resource request: " + capability + " exceeds the available"
+  + " resources of the node.");
+}
+
 // The desired container won't fit here, so reserve
 if (isReservable(capability) &&
 reserve(request, node, reservedContainer, type, schedulerKey)) {
-  if (isWaitingForAMContainer()) {
-updateAMDiagnosticMsg(capability,
-" exceed the available resources of the node and the request is"
-+ " reserved");
+  updateAMDiagnosticMsg(capability, " exceeds the available resources of "
+  + "the node and the request is reserved)");
+  if (LOG.isDebugEnabled()) {
+LOG.debug(getName() + "'s resource request is reserved.");
   }
   return FairScheduler.CONTAINER_RESERVED;
 } else {
-  if (isWaitingForAMContainer())

hadoop git commit: YARN-6042. Dump scheduler and queue state information into FairScheduler DEBUG log. (Yufei Gu via rchiang)

2017-03-10 Thread rchiang

Repository: hadoop
Updated Branches:
  refs/heads/trunk 229c7c9f8 -> 4db9cc70d


YARN-6042. Dump scheduler and queue state information into FairScheduler DEBUG 
log. (Yufei Gu via rchiang)


Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/4db9cc70
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/4db9cc70
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/4db9cc70

Branch: refs/heads/trunk
Commit: 4db9cc70d0178703fb28f451eb84d97f2bf63af8
Parents: 229c7c9
Author: Ray Chiang <rchi...@apache.org>
Authored: Fri Mar 10 16:13:31 2017 -0800
Committer: Ray Chiang <rchi...@apache.org>
Committed: Fri Mar 10 16:13:31 2017 -0800

--
 .../src/main/conf/log4j.properties  |  9 +++
 .../scheduler/fair/FSAppAttempt.java| 49 +++---
 .../scheduler/fair/FSLeafQueue.java | 21 ++
 .../scheduler/fair/FSParentQueue.java   | 21 ++
 .../resourcemanager/scheduler/fair/FSQueue.java | 41 ++-
 .../scheduler/fair/FairScheduler.java   | 28 +---
 .../scheduler/fair/TestFairScheduler.java   | 71 
 7 files changed, 206 insertions(+), 34 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/hadoop/blob/4db9cc70/hadoop-common-project/hadoop-common/src/main/conf/log4j.properties
--
diff --git a/hadoop-common-project/hadoop-common/src/main/conf/log4j.properties 
b/hadoop-common-project/hadoop-common/src/main/conf/log4j.properties
index b8c84e7..6026763 100644
--- a/hadoop-common-project/hadoop-common/src/main/conf/log4j.properties
+++ b/hadoop-common-project/hadoop-common/src/main/conf/log4j.properties
@@ -321,3 +321,12 @@ 
log4j.appender.EWMA=org.apache.hadoop.yarn.util.Log4jWarningErrorMetricsAppender
 log4j.appender.EWMA.cleanupInterval=${yarn.ewma.cleanupInterval}
 log4j.appender.EWMA.messageAgeLimitSeconds=${yarn.ewma.messageAgeLimitSeconds}
 log4j.appender.EWMA.maxUniqueMessages=${yarn.ewma.maxUniqueMessages}
+
+# Fair scheduler requests log on state dump
+log4j.logger.org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.statedump=DEBUG,FSLOGGER
+log4j.appender.FSLOGGER=org.apache.log4j.RollingFileAppender
+log4j.appender.FSLOGGER.File=${hadoop.log.dir}/fairscheduler-statedump.log
+log4j.appender.FSLOGGER.layout=org.apache.log4j.PatternLayout
+log4j.appender.FSLOGGER.layout.ConversionPattern=%d{ISO8601} %p %c: %m%n
+log4j.appender.FSLOGGER.MaxFileSize=${hadoop.log.maxfilesize}
+log4j.appender.FSLOGGER.MaxBackupIndex=${hadoop.log.maxbackupindex}
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/hadoop/blob/4db9cc70/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSAppAttempt.java
--
diff --git 
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSAppAttempt.java
 
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSAppAttempt.java
index 3a9c94e..ccfcffb 100644
--- 
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSAppAttempt.java
+++ 
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSAppAttempt.java
@@ -835,25 +835,27 @@ public class FSAppAttempt extends 
SchedulerApplicationAttempt
   return capability;
 }
 
+if (LOG.isDebugEnabled()) {
+  LOG.debug("Resource request: " + capability + " exceeds the available"
+  + " resources of the node.");
+}
+
 // The desired container won't fit here, so reserve
 if (isReservable(capability) &&
 reserve(pendingAsk.getPerAllocationResource(), node, reservedContainer,
 type, schedulerKey)) {
-  if (isWaitingForAMContainer()) {
-updateAMDiagnosticMsg(capability,
-" exceed the available resources of the node and the request is"
-+ " reserved");
+  updateAMDiagnosticMsg(capability, " exceeds the available resources of "
+  + "the node and the request is reserved)");
+  if (LOG.isDebugEnabled()) {
+LOG.debug(getName() + "'s resource request is reserved.");
   }
   return FairScheduler.CONTAINER_RESERVED;
 } else {
-

hadoop git commit: YARN-5665. Enhance documentation for yarn.resourcemanager.scheduler.class property. (Yufei Gu via rchiang)

2017-03-06 Thread rchiang

Repository: hadoop
Updated Branches:
  refs/heads/branch-2 cf5a1390a -> d94348014


YARN-5665. Enhance documentation for yarn.resourcemanager.scheduler.class 
property. (Yufei Gu via rchiang)

(cherry picked from commit d9dc444dc73fbe23f9e553d63baf83f12c636fa7)


Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/d9434801
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/d9434801
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/d9434801

Branch: refs/heads/branch-2
Commit: d943480147cb25e99f0a858ac4001e05909fa5ac
Parents: cf5a139
Author: Ray Chiang <rchi...@apache.org>
Authored: Mon Mar 6 14:02:49 2017 -0800
Committer: Ray Chiang <rchi...@apache.org>
Committed: Mon Mar 6 14:05:09 2017 -0800

--
 .../hadoop-common/src/site/markdown/ClusterSetup.md| 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/hadoop/blob/d9434801/hadoop-common-project/hadoop-common/src/site/markdown/ClusterSetup.md
--
diff --git 
a/hadoop-common-project/hadoop-common/src/site/markdown/ClusterSetup.md 
b/hadoop-common-project/hadoop-common/src/site/markdown/ClusterSetup.md
index a1c2307..d33c225 100644
--- a/hadoop-common-project/hadoop-common/src/site/markdown/ClusterSetup.md
+++ b/hadoop-common-project/hadoop-common/src/site/markdown/ClusterSetup.md
@@ -146,7 +146,7 @@ This section deals with important parameters to be 
specified in the given config
 | `yarn.resourcemanager.admin.address` | `ResourceManager` host:port for 
administrative commands. | *host:port*Â If set, overrides the hostname set in 
`yarn.resourcemanager.hostname`. |
 | `yarn.resourcemanager.webapp.address` | `ResourceManager` web-ui host:port. 
| *host:port*Â If set, overrides the hostname set in 
`yarn.resourcemanager.hostname`. |
 | `yarn.resourcemanager.hostname` | `ResourceManager` host. | *host*Â Single 
hostname that can be set in place of setting all `yarn.resourcemanager*address` 
resources. Results in default ports for ResourceManager components. |
-| `yarn.resourcemanager.scheduler.class` | `ResourceManager` Scheduler class. 
| `CapacityScheduler` (recommended), `FairScheduler` (also recommended), or 
`FifoScheduler` |
+| `yarn.resourcemanager.scheduler.class` | `ResourceManager` Scheduler class. 
| `CapacityScheduler` (recommended), `FairScheduler` (also recommended), or 
`FifoScheduler`. Use a fully qualified class name, e.g., 
`org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler`. |
 | `yarn.scheduler.minimum-allocation-mb` | Minimum limit of memory to allocate 
to each container request at the `Resource Manager`. | In MBs |
 | `yarn.scheduler.maximum-allocation-mb` | Maximum limit of memory to allocate 
to each container request at the `Resource Manager`. | In MBs |
 | `yarn.resourcemanager.nodes.include-path` / 
`yarn.resourcemanager.nodes.exclude-path` | List of permitted/excluded 
NodeManagers. | If necessary, use these files to control the list of allowable 
NodeManagers. |


-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org

hadoop git commit: YARN-5665. Enhance documentation for yarn.resourcemanager.scheduler.class property. (Yufei Gu via rchiang)

2017-03-06 Thread rchiang

Repository: hadoop
Updated Branches:
  refs/heads/trunk ec839b94c -> d9dc444dc


YARN-5665. Enhance documentation for yarn.resourcemanager.scheduler.class 
property. (Yufei Gu via rchiang)


Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/d9dc444d
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/d9dc444d
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/d9dc444d

Branch: refs/heads/trunk
Commit: d9dc444dc73fbe23f9e553d63baf83f12c636fa7
Parents: ec839b9
Author: Ray Chiang <rchi...@apache.org>
Authored: Mon Mar 6 14:02:49 2017 -0800
Committer: Ray Chiang <rchi...@apache.org>
Committed: Mon Mar 6 14:02:49 2017 -0800

--
 .../hadoop-common/src/site/markdown/ClusterSetup.md| 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/hadoop/blob/d9dc444d/hadoop-common-project/hadoop-common/src/site/markdown/ClusterSetup.md
--
diff --git 
a/hadoop-common-project/hadoop-common/src/site/markdown/ClusterSetup.md 
b/hadoop-common-project/hadoop-common/src/site/markdown/ClusterSetup.md
index 1d9e9da..7be6a19 100644
--- a/hadoop-common-project/hadoop-common/src/site/markdown/ClusterSetup.md
+++ b/hadoop-common-project/hadoop-common/src/site/markdown/ClusterSetup.md
@@ -139,7 +139,7 @@ This section deals with important parameters to be 
specified in the given config
 | `yarn.resourcemanager.admin.address` | `ResourceManager` host:port for 
administrative commands. | *host:port*Â If set, overrides the hostname set in 
`yarn.resourcemanager.hostname`. |
 | `yarn.resourcemanager.webapp.address` | `ResourceManager` web-ui host:port. 
| *host:port*Â If set, overrides the hostname set in 
`yarn.resourcemanager.hostname`. |
 | `yarn.resourcemanager.hostname` | `ResourceManager` host. | *host*Â Single 
hostname that can be set in place of setting all `yarn.resourcemanager*address` 
resources. Results in default ports for ResourceManager components. |
-| `yarn.resourcemanager.scheduler.class` | `ResourceManager` Scheduler class. 
| `CapacityScheduler` (recommended), `FairScheduler` (also recommended), or 
`FifoScheduler` |
+| `yarn.resourcemanager.scheduler.class` | `ResourceManager` Scheduler class. 
| `CapacityScheduler` (recommended), `FairScheduler` (also recommended), or 
`FifoScheduler`. Use a fully qualified class name, e.g., 
`org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler`. |
 | `yarn.scheduler.minimum-allocation-mb` | Minimum limit of memory to allocate 
to each container request at the `Resource Manager`. | In MBs |
 | `yarn.scheduler.maximum-allocation-mb` | Maximum limit of memory to allocate 
to each container request at the `Resource Manager`. | In MBs |
 | `yarn.resourcemanager.nodes.include-path` / 
`yarn.resourcemanager.nodes.exclude-path` | List of permitted/excluded 
NodeManagers. | If necessary, use these files to control the list of allowable 
NodeManagers. |


-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org

hadoop git commit: YARN-6218. Fix TestAMRMClient when using FairScheduler. (Miklos Szegedi via rchiang)

2017-03-03 Thread rchiang

Repository: hadoop
Updated Branches:
  refs/heads/branch-2 5eca427da -> 303ee13e3


YARN-6218. Fix TestAMRMClient when using FairScheduler. (Miklos Szegedi via 
rchiang)

(cherry picked from commit 30b91ff9540e35195af834d1bd5983114a556c6d)


Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/303ee13e
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/303ee13e
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/303ee13e

Branch: refs/heads/branch-2
Commit: 303ee13e3c6836e8f59535b6e2d6771bdcf8c299
Parents: 5eca427
Author: Ray Chiang <rchi...@apache.org>
Authored: Fri Mar 3 12:55:45 2017 -0800
Committer: Ray Chiang <rchi...@apache.org>
Committed: Fri Mar 3 12:57:17 2017 -0800

--
 .../yarn/client/api/impl/TestAMRMClient.java| 139 ---
 .../scheduler/fair/FairScheduler.java   |   3 +-
 2 files changed, 87 insertions(+), 55 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/hadoop/blob/303ee13e/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestAMRMClient.java
--
diff --git 
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestAMRMClient.java
 
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestAMRMClient.java
index 239189a..06ae828 100644
--- 
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestAMRMClient.java
+++ 
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestAMRMClient.java
@@ -86,47 +86,71 @@ import org.apache.hadoop.yarn.exceptions.YarnException;
 import org.apache.hadoop.yarn.ipc.YarnRPC;
 import org.apache.hadoop.yarn.security.AMRMTokenIdentifier;
 import org.apache.hadoop.yarn.server.MiniYARNCluster;
+import org.apache.hadoop.yarn.server.resourcemanager.RMContext;
 import 
org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttempt;
 import 
org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptState;
+import org.apache.hadoop.yarn.server.resourcemanager.rmnode.RMNode;
+import 
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler;
+import 
org.apache.hadoop.yarn.server.resourcemanager.scheduler.event.NodeUpdateSchedulerEvent;
+import 
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler;
 import 
org.apache.hadoop.yarn.server.resourcemanager.security.AMRMTokenSecretManager;
 import org.apache.hadoop.yarn.server.utils.BuilderUtils;
 import org.apache.hadoop.yarn.util.Records;
 import org.junit.After;
-import org.junit.AfterClass;
 import org.junit.Assert;
+import org.junit.Assume;
 import org.junit.Before;
-import org.junit.BeforeClass;
 import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
 import org.mockito.invocation.InvocationOnMock;
 import org.mockito.stubbing.Answer;
 import org.mortbay.log.Log;
 
 import com.google.common.base.Supplier;
 
+/**
+ * Test application master client class to resource manager.
+ */
+@RunWith(value = Parameterized.class)
 public class TestAMRMClient {
-  static Configuration conf = null;
-  static MiniYARNCluster yarnCluster = null;
-  static YarnClient yarnClient = null;
-  static List nodeReports = null;
-  static ApplicationAttemptId attemptId = null;
-  static int nodeCount = 3;
+  private String schedulerName = null;
+  private Configuration conf = null;
+  private MiniYARNCluster yarnCluster = null;
+  private YarnClient yarnClient = null;
+  private List nodeReports = null;
+  private ApplicationAttemptId attemptId = null;
+  private int nodeCount = 3;
   
   static final int rolling_interval_sec = 13;
   static final long am_expire_ms = 4000;
 
-  static Resource capability;
-  static Priority priority;
-  static Priority priority2;
-  static String node;
-  static String rack;
-  static String[] nodes;
-  static String[] racks;
+  private Resource capability;
+  private Priority priority;
+  private Priority priority2;
+  private String node;
+  private String rack;
+  private String[] nodes;
+  private String[] racks;
   private final static int DEFAULT_ITERATION = 3;
 
-  @BeforeClass
-  public static void setup() throws Exception {
+  public TestAMRMClient(String schedulerName) {
+this.schedulerName = schedulerName;
+  }
+
+  @Parameterized.Parameters
+  public static Collection<Object[]> data() {
+List<Object[]> list = new ArrayList<Object[]>(2);
+list.add(new Object[] {CapacityScheduler.class.getName()});
+list.add(new Object[] {FairScheduler.class.getName()});
+return list;

hadoop git commit: YARN-6218. Fix TestAMRMClient when using FairScheduler. (Miklos Szegedi via rchiang)

2017-03-03 Thread rchiang

Repository: hadoop
Updated Branches:
  refs/heads/trunk 3085a6043 -> 2148b8399


YARN-6218. Fix TestAMRMClient when using FairScheduler. (Miklos Szegedi via 
rchiang)


Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/2148b839
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/2148b839
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/2148b839

Branch: refs/heads/trunk
Commit: 2148b83993fd8ce73bcbc7677c57ee5028a59cd4
Parents: 3085a60
Author: Ray Chiang <rchi...@apache.org>
Authored: Fri Mar 3 12:55:45 2017 -0800
Committer: Ray Chiang <rchi...@apache.org>
Committed: Fri Mar 3 13:04:51 2017 -0800

--
 .../yarn/client/api/impl/TestAMRMClient.java| 139 ---
 .../scheduler/fair/FairScheduler.java   |   3 +-
 2 files changed, 87 insertions(+), 55 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/hadoop/blob/2148b839/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestAMRMClient.java
--
diff --git 
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestAMRMClient.java
 
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestAMRMClient.java
index 4f73bac..43c0271 100644
--- 
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestAMRMClient.java
+++ 
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestAMRMClient.java
@@ -64,47 +64,71 @@ import org.apache.hadoop.yarn.exceptions.YarnException;
 import org.apache.hadoop.yarn.ipc.YarnRPC;
 import org.apache.hadoop.yarn.security.AMRMTokenIdentifier;
 import org.apache.hadoop.yarn.server.MiniYARNCluster;
+import org.apache.hadoop.yarn.server.resourcemanager.RMContext;
 import 
org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttempt;
 import 
org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptState;
+import org.apache.hadoop.yarn.server.resourcemanager.rmnode.RMNode;
+import 
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler;
+import 
org.apache.hadoop.yarn.server.resourcemanager.scheduler.event.NodeUpdateSchedulerEvent;
+import 
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler;
 import 
org.apache.hadoop.yarn.server.resourcemanager.security.AMRMTokenSecretManager;
 import org.apache.hadoop.yarn.server.utils.BuilderUtils;
 import org.apache.hadoop.yarn.util.Records;
 import org.junit.After;
-import org.junit.AfterClass;
 import org.junit.Assert;
+import org.junit.Assume;
 import org.junit.Before;
-import org.junit.BeforeClass;
 import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
 import org.mockito.invocation.InvocationOnMock;
 import org.mockito.stubbing.Answer;
 import org.eclipse.jetty.util.log.Log;
 
 import com.google.common.base.Supplier;
 
+/**
+ * Test application master client class to resource manager.
+ */
+@RunWith(value = Parameterized.class)
 public class TestAMRMClient {
-  static Configuration conf = null;
-  static MiniYARNCluster yarnCluster = null;
-  static YarnClient yarnClient = null;
-  static List nodeReports = null;
-  static ApplicationAttemptId attemptId = null;
-  static int nodeCount = 3;
+  private String schedulerName = null;
+  private Configuration conf = null;
+  private MiniYARNCluster yarnCluster = null;
+  private YarnClient yarnClient = null;
+  private List nodeReports = null;
+  private ApplicationAttemptId attemptId = null;
+  private int nodeCount = 3;
   
   static final int rolling_interval_sec = 13;
   static final long am_expire_ms = 4000;
 
-  static Resource capability;
-  static Priority priority;
-  static Priority priority2;
-  static String node;
-  static String rack;
-  static String[] nodes;
-  static String[] racks;
+  private Resource capability;
+  private Priority priority;
+  private Priority priority2;
+  private String node;
+  private String rack;
+  private String[] nodes;
+  private String[] racks;
   private final static int DEFAULT_ITERATION = 3;
 
-  @BeforeClass
-  public static void setup() throws Exception {
+  public TestAMRMClient(String schedulerName) {
+this.schedulerName = schedulerName;
+  }
+
+  @Parameterized.Parameters
+  public static Collection<Object[]> data() {
+List<Object[]> list = new ArrayList<Object[]>(2);
+list.add(new Object[] {CapacityScheduler.class.getName()});
+list.add(new Object[] {FairScheduler.class.getName()});
+return list;
+  }
+
+  @Before
+  public void setup() throws Exception {

hadoop git commit: YARN-6106. Document FairScheduler 'allowPreemptionFrom' queue property. (Yufei Gu via rchiang)

2017-01-30 Thread rchiang

Repository: hadoop
Updated Branches:
  refs/heads/branch-2 c9824c5ff -> 9f0b81335


YARN-6106.  Document FairScheduler 'allowPreemptionFrom' queue property. (Yufei 
Gu via rchiang)

(cherry picked from commit 3f970ac363dc2faf4f54180ee5b64d0c60807097)


Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/9f0b8133
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/9f0b8133
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/9f0b8133

Branch: refs/heads/branch-2
Commit: 9f0b813350c61dfb807c893e1b22de1f01ba71bb
Parents: c9824c5
Author: Ray Chiang <rchi...@apache.org>
Authored: Fri Jan 20 14:04:10 2017 -0800
Committer: Ray Chiang <rchi...@apache.org>
Committed: Mon Jan 30 13:42:11 2017 -0800

--
 .../hadoop-yarn-site/src/site/markdown/FairScheduler.md| 2 ++
 1 file changed, 2 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/hadoop/blob/9f0b8133/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/FairScheduler.md
--
diff --git 
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/FairScheduler.md
 
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/FairScheduler.md
index 91e7d8d..bcf56cb 100644
--- 
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/FairScheduler.md
+++ 
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/FairScheduler.md
@@ -117,6 +117,8 @@ The allocation file must be in XML format. The format 
contains five types of ele
 
 * fairSharePreemptionThreshold: the fair share preemption threshold for 
the queue. If the queue waits fairSharePreemptionTimeout without receiving 
fairSharePreemptionThreshold\*fairShare resources, it is allowed to preempt 
containers to take resources from other queues. If not set, the queue will 
inherit the value from its parent queue.
 
+* allowPreemptionFrom: determines whether the scheduler is allowed to 
preempt resources from the queue. The default is true. If a queue has this 
property set to false, this property will apply recursively to all child queues.
+
 * **User elements**: which represent settings governing the behavior of 
individual users. They can contain a single property: maxRunningApps, a limit 
on the number of running apps for a particular user.
 
 * **A userMaxAppsDefault element**: which sets the default running app limit 
for any users whose limit is not otherwise specified.


-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org

hadoop git commit: YARN-6106. Document FairScheduler 'allowPreemptionFrom' queue property. (Yufei Gu via rchiang)

2017-01-20 Thread rchiang

Repository: hadoop
Updated Branches:
  refs/heads/trunk 355b907c0 -> 9bab85cad


YARN-6106.  Document FairScheduler 'allowPreemptionFrom' queue property. (Yufei 
Gu via rchiang)


Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/9bab85ca
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/9bab85ca
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/9bab85ca

Branch: refs/heads/trunk
Commit: 9bab85cadfdfa0d9071303452f72c0a5d008480a
Parents: 355b907
Author: Ray Chiang <rchi...@apache.org>
Authored: Fri Jan 20 14:04:10 2017 -0800
Committer: Ray Chiang <rchi...@apache.org>
Committed: Fri Jan 20 14:08:25 2017 -0800

--
 .../hadoop-yarn-site/src/site/markdown/FairScheduler.md| 2 ++
 1 file changed, 2 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/hadoop/blob/9bab85ca/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/FairScheduler.md
--
diff --git 
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/FairScheduler.md
 
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/FairScheduler.md
index ae4c3ab..5d7b3dc 100644
--- 
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/FairScheduler.md
+++ 
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/FairScheduler.md
@@ -119,6 +119,8 @@ The allocation file must be in XML format. The format 
contains five types of ele
 
 * fairSharePreemptionThreshold: the fair share preemption threshold for 
the queue. If the queue waits fairSharePreemptionTimeout without receiving 
fairSharePreemptionThreshold\*fairShare resources, it is allowed to preempt 
containers to take resources from other queues. If not set, the queue will 
inherit the value from its parent queue.
 
+* allowPreemptionFrom: determines whether the scheduler is allowed to 
preempt resources from the queue. The default is true. If a queue has this 
property set to false, this property will apply recursively to all child queues.
+
 * **User elements**: which represent settings governing the behavior of 
individual users. They can contain a single property: maxRunningApps, a limit 
on the number of running apps for a particular user.
 
 * **A userMaxAppsDefault element**: which sets the default running app limit 
for any users whose limit is not otherwise specified.


-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org

hadoop git commit: YARN-5549. AMLauncher#createAMContainerLaunchContext() should not log the command to be launched indiscriminately. (Daniel Templeton via rchiang)

2016-09-02 Thread rchiang

Repository: hadoop
Updated Branches:
  refs/heads/branch-2.8 9b7e079d4 -> 02272a6a7


YARN-5549. AMLauncher#createAMContainerLaunchContext() should not log the 
command to be launched indiscriminately. (Daniel Templeton via rchiang)

(cherry picked from commit 4ee1729cb04e72f3015666d750ad1e54257345d2)


Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/02272a6a
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/02272a6a
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/02272a6a

Branch: refs/heads/branch-2.8
Commit: 02272a6a77ce958cab54242933cdc67ff22d2ca7
Parents: 9b7e079
Author: Ray Chiang <rchi...@apache.org>
Authored: Fri Sep 2 14:57:05 2016 -0700
Committer: Ray Chiang <rchi...@apache.org>
Committed: Fri Sep 2 15:00:24 2016 -0700

--
 .../hadoop/yarn/conf/YarnConfiguration.java | 12 +
 .../src/main/resources/yarn-default.xml | 13 +
 .../resourcemanager/amlauncher/AMLauncher.java  | 28 +++-
 3 files changed, 47 insertions(+), 6 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/hadoop/blob/02272a6a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
--
diff --git 
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
 
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
index 5ce171c..96c4f6f 100644
--- 
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
+++ 
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
@@ -403,6 +403,18 @@ public class YarnConfiguration extends Configuration {
   public static final int 
DEFAULT_RM_SYSTEM_METRICS_PUBLISHER_DISPATCHER_POOL_SIZE =
   10;
 
+  /**
+   * The {@code AMLauncher.createAMContainerLaunchContext()} method will log 
the
+   * command being executed to the RM log if this property is true. Commands
+   * may contain sensitive information, such as application or service
+   * passwords, making logging the commands a security risk. In cases where
+   * the cluster may be running applications with such commands, this property
+   * should be set to false. Commands are only logged at the debug level.
+   */
+  public static final String RM_AMLAUNCHER_LOG_COMMAND =
+  RM_PREFIX + "amlauncher.log.command";
+  public static final boolean DEFAULT_RM_AMLAUNCHER_LOG_COMMAND = false;
+
   //RM delegation token related keys
   public static final String RM_DELEGATION_KEY_UPDATE_INTERVAL_KEY =
 RM_PREFIX + "delegation.key.update-interval";

http://git-wip-us.apache.org/repos/asf/hadoop/blob/02272a6a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
--
diff --git 
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
 
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
index 788b0fd..512db81 100644
--- 
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
+++ 
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
@@ -299,6 +299,19 @@
   
 
   
+
+  The resource manager will log all commands being executed to the RM log
+  if this property is true. Commands may contain sensitive information,
+  such as application or service passwords, making logging the commands a
+  security risk. In cases where the cluster may be running applications 
with
+  such commands this property should be set to false. Commands are only
+  logged at the debug level.
+
+yarn.resourcemanager.amlauncher.log.command
+false
+  
+
+  
 The class to use as the resource scheduler.
 yarn.resourcemanager.scheduler.class
 
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler

http://git-wip-us.apache.org/repos/asf/hadoop/blob/02272a6a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/amlauncher/AMLauncher.java
--
diff --git 
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/amlauncher/AMLauncher.java
 
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourc

hadoop git commit: YARN-5549. AMLauncher#createAMContainerLaunchContext() should not log the command to be launched indiscriminately. (Daniel Templeton via rchiang)

2016-09-02 Thread rchiang

Repository: hadoop
Updated Branches:
  refs/heads/branch-2 40eab7756 -> 4ee1729cb


YARN-5549. AMLauncher#createAMContainerLaunchContext() should not log the 
command to be launched indiscriminately. (Daniel Templeton via rchiang)


Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/4ee1729c
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/4ee1729c
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/4ee1729c

Branch: refs/heads/branch-2
Commit: 4ee1729cb04e72f3015666d750ad1e54257345d2
Parents: 40eab77
Author: Ray Chiang <rchi...@apache.org>
Authored: Fri Sep 2 14:57:05 2016 -0700
Committer: Ray Chiang <rchi...@apache.org>
Committed: Fri Sep 2 14:57:05 2016 -0700

--
 .../hadoop/yarn/conf/YarnConfiguration.java | 12 +
 .../src/main/resources/yarn-default.xml | 13 +
 .../resourcemanager/amlauncher/AMLauncher.java  | 28 +++-
 3 files changed, 47 insertions(+), 6 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/hadoop/blob/4ee1729c/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
--
diff --git 
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
 
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
index 15ef0c1..0936ebf 100644
--- 
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
+++ 
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
@@ -508,6 +508,18 @@ public class YarnConfiguration extends Configuration {
   public static final int 
DEFAULT_RM_SYSTEM_METRICS_PUBLISHER_DISPATCHER_POOL_SIZE =
   10;
 
+  /**
+   * The {@code AMLauncher.createAMContainerLaunchContext()} method will log 
the
+   * command being executed to the RM log if this property is true. Commands
+   * may contain sensitive information, such as application or service
+   * passwords, making logging the commands a security risk. In cases where
+   * the cluster may be running applications with such commands, this property
+   * should be set to false. Commands are only logged at the debug level.
+   */
+  public static final String RM_AMLAUNCHER_LOG_COMMAND =
+  RM_PREFIX + "amlauncher.log.command";
+  public static final boolean DEFAULT_RM_AMLAUNCHER_LOG_COMMAND = false;
+
   //RM delegation token related keys
   public static final String RM_DELEGATION_KEY_UPDATE_INTERVAL_KEY =
 RM_PREFIX + "delegation.key.update-interval";

http://git-wip-us.apache.org/repos/asf/hadoop/blob/4ee1729c/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
--
diff --git 
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
 
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
index 3d2d593..3862cc8 100644
--- 
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
+++ 
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
@@ -299,6 +299,19 @@
   
 
   
+
+  The resource manager will log all commands being executed to the RM log
+  if this property is true. Commands may contain sensitive information,
+  such as application or service passwords, making logging the commands a
+  security risk. In cases where the cluster may be running applications 
with
+  such commands this property should be set to false. Commands are only
+  logged at the debug level.
+
+yarn.resourcemanager.amlauncher.log.command
+false
+  
+
+  
 The class to use as the resource scheduler.
 yarn.resourcemanager.scheduler.class
 
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler

http://git-wip-us.apache.org/repos/asf/hadoop/blob/4ee1729c/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/amlauncher/AMLauncher.java
--
diff --git 
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/amlauncher/AMLauncher.java
 
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/amlauncher/AMLauncher.java
index 4c840e7..e7105f9 100644
---

hadoop git commit: YARN-5549. AMLauncher#createAMContainerLaunchContext() should not log the command to be launched indiscriminately. (Daniel Templeton via rchiang)

2016-09-02 Thread rchiang

Repository: hadoop
Updated Branches:
  refs/heads/trunk 5a8c5064d -> 378f624a3


YARN-5549. AMLauncher#createAMContainerLaunchContext() should not log the 
command to be launched indiscriminately. (Daniel Templeton via rchiang)


Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/378f624a
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/378f624a
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/378f624a

Branch: refs/heads/trunk
Commit: 378f624a392550770d1db33cb4cee3ef7d5facd4
Parents: 5a8c506
Author: Ray Chiang <rchi...@apache.org>
Authored: Fri Sep 2 11:07:39 2016 -0700
Committer: Ray Chiang <rchi...@apache.org>
Committed: Fri Sep 2 11:14:35 2016 -0700

--
 .../hadoop/yarn/conf/YarnConfiguration.java | 12 +
 .../src/main/resources/yarn-default.xml | 13 ++
 .../resourcemanager/amlauncher/AMLauncher.java  | 26 
 3 files changed, 46 insertions(+), 5 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/hadoop/blob/378f624a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
--
diff --git 
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
 
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
index 46e3323..86e8a95 100644
--- 
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
+++ 
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
@@ -534,6 +534,18 @@ public class YarnConfiguration extends Configuration {
   public static final int
   DEFAULT_RM_SYSTEM_METRICS_PUBLISHER_DISPATCHER_POOL_SIZE = 10;
 
+  /**
+   * The {@code AMLauncher.createAMContainerLaunchContext()} method will log 
the
+   * command being executed to the RM log if this property is true. Commands
+   * may contain sensitive information, such as application or service
+   * passwords, making logging the commands a security risk. In cases where
+   * the cluster may be running applications with such commands, this property
+   * should be set to false. Commands are only logged at the debug level.
+   */
+  public static final String RM_AMLAUNCHER_LOG_COMMAND =
+  RM_PREFIX + "amlauncher.log.command";
+  public static final boolean DEFAULT_RM_AMLAUNCHER_LOG_COMMAND = false;
+
   //RM delegation token related keys
   public static final String RM_DELEGATION_KEY_UPDATE_INTERVAL_KEY =
 RM_PREFIX + "delegation.key.update-interval";

http://git-wip-us.apache.org/repos/asf/hadoop/blob/378f624a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
--
diff --git 
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
 
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
index e956507..423b78b 100644
--- 
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
+++ 
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
@@ -299,6 +299,19 @@
   
 
   
+
+  The resource manager will log all commands being executed to the RM log
+  if this property is true. Commands may contain sensitive information,
+  such as application or service passwords, making logging the commands a
+  security risk. In cases where the cluster may be running applications 
with
+  such commands this property should be set to false. Commands are only
+  logged at the debug level.
+
+yarn.resourcemanager.amlauncher.log.command
+false
+  
+
+  
 The class to use as the resource scheduler.
 yarn.resourcemanager.scheduler.class
 
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler

http://git-wip-us.apache.org/repos/asf/hadoop/blob/378f624a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/amlauncher/AMLauncher.java
--
diff --git 
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/amlauncher/AMLauncher.java
 
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/amlauncher/AMLauncher.java
index 4aace2c..181463a 100644
--- 
a/hadoop-y

hadoop git commit: Revert "YARN-5567. Fix script exit code checking in NodeHealthScriptRunner#reportHealthStatus. (Yufei Gu via rchiang)"

2016-09-01 Thread rchiang

Repository: hadoop
Updated Branches:
  refs/heads/branch-2.8 af26fba62 -> 2dd809ead


Revert "YARN-5567. Fix script exit code checking in 
NodeHealthScriptRunner#reportHealthStatus. (Yufei Gu via rchiang)"

This reverts commit 9ad3a7a43b35dd4bb9d10bd322fbb956d3b01ff2.


Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/2dd809ea
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/2dd809ea
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/2dd809ea

Branch: refs/heads/branch-2.8
Commit: 2dd809eadabb966a18c2b9f183eaef2ea66354a6
Parents: af26fba
Author: Ray Chiang <rchi...@apache.org>
Authored: Thu Sep 1 15:10:37 2016 -0700
Committer: Ray Chiang <rchi...@apache.org>
Committed: Thu Sep 1 15:10:37 2016 -0700

--
 .../java/org/apache/hadoop/util/NodeHealthScriptRunner.java   | 3 +--
 .../org/apache/hadoop/util/TestNodeHealthScriptRunner.java| 7 ---
 2 files changed, 1 insertion(+), 9 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/hadoop/blob/2dd809ea/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/NodeHealthScriptRunner.java
--
diff --git 
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/NodeHealthScriptRunner.java
 
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/NodeHealthScriptRunner.java
index c3bef37..fc392c4 100644
--- 
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/NodeHealthScriptRunner.java
+++ 
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/NodeHealthScriptRunner.java
@@ -106,7 +106,6 @@ public class NodeHealthScriptRunner extends AbstractService 
{
 shexec.execute();
   } catch (ExitCodeException e) {
 // ignore the exit code of the script
-exceptionStackTrace = StringUtils.stringifyException(e);
 status = HealthCheckerExitStatus.FAILED_WITH_EXIT_CODE;
 // On Windows, we will not hit the Stream closed IOException
 // thrown by stdout buffered reader for timeout event.
@@ -163,7 +162,7 @@ public class NodeHealthScriptRunner extends AbstractService 
{
 setHealthStatus(false, exceptionStackTrace);
 break;
   case FAILED_WITH_EXIT_CODE:
-setHealthStatus(false, exceptionStackTrace);
+setHealthStatus(true, "", now);
 break;
   case FAILED:
 setHealthStatus(false, shexec.getOutput());

http://git-wip-us.apache.org/repos/asf/hadoop/blob/2dd809ea/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestNodeHealthScriptRunner.java
--
diff --git 
a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestNodeHealthScriptRunner.java
 
b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestNodeHealthScriptRunner.java
index db61f5a..8fc64d1 100644
--- 
a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestNodeHealthScriptRunner.java
+++ 
b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestNodeHealthScriptRunner.java
@@ -91,7 +91,6 @@ public class TestNodeHealthScriptRunner {
   public void testNodeHealthScript() throws Exception {
 String errorScript = "echo ERROR\n echo \"Tracker not healthy\"";
 String normalScript = "echo \"I am all fine\"";
-String failWithExitCodeScript = "echo \"Not healthy\"; exit -1";
 String timeOutScript =
   Shell.WINDOWS ? "@echo off\nping -n 4 127.0.0.1 >nul\necho \"I am fine\""
   : "sleep 4\necho \"I am fine\"";
@@ -125,12 +124,6 @@ public class TestNodeHealthScriptRunner {
 nodeHealthScriptRunner.isHealthy());
 Assert.assertEquals("", nodeHealthScriptRunner.getHealthReport());
 
-// Script which fails with exit code.
-writeNodeHealthScriptFile(failWithExitCodeScript, true);
-timerTask.run();
-Assert.assertFalse("Node health status reported healthy",
-nodeHealthScriptRunner.isHealthy());
-
 // Timeout script.
 writeNodeHealthScriptFile(timeOutScript, true);
 timerTask.run();


-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org

hadoop git commit: Revert "YARN-5567. Fix script exit code checking in NodeHealthScriptRunner#reportHealthStatus. (Yufei Gu via rchiang)"

2016-09-01 Thread rchiang

Repository: hadoop
Updated Branches:
  refs/heads/branch-2 736dd1646 -> c14204808


Revert "YARN-5567. Fix script exit code checking in 
NodeHealthScriptRunner#reportHealthStatus. (Yufei Gu via rchiang)"

This reverts commit 3b19bcb8f0f70606a8080bc46fa64cd34f6280d7.


Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/c1420480
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/c1420480
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/c1420480

Branch: refs/heads/branch-2
Commit: c142048084f0bf97ccb9c383003fd4b6ee95b222
Parents: 736dd16
Author: Ray Chiang <rchi...@apache.org>
Authored: Thu Sep 1 15:10:53 2016 -0700
Committer: Ray Chiang <rchi...@apache.org>
Committed: Thu Sep 1 15:10:53 2016 -0700

--
 .../java/org/apache/hadoop/util/NodeHealthScriptRunner.java   | 3 +--
 .../org/apache/hadoop/util/TestNodeHealthScriptRunner.java| 7 ---
 2 files changed, 1 insertion(+), 9 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/hadoop/blob/c1420480/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/NodeHealthScriptRunner.java
--
diff --git 
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/NodeHealthScriptRunner.java
 
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/NodeHealthScriptRunner.java
index c3bef37..fc392c4 100644
--- 
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/NodeHealthScriptRunner.java
+++ 
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/NodeHealthScriptRunner.java
@@ -106,7 +106,6 @@ public class NodeHealthScriptRunner extends AbstractService 
{
 shexec.execute();
   } catch (ExitCodeException e) {
 // ignore the exit code of the script
-exceptionStackTrace = StringUtils.stringifyException(e);
 status = HealthCheckerExitStatus.FAILED_WITH_EXIT_CODE;
 // On Windows, we will not hit the Stream closed IOException
 // thrown by stdout buffered reader for timeout event.
@@ -163,7 +162,7 @@ public class NodeHealthScriptRunner extends AbstractService 
{
 setHealthStatus(false, exceptionStackTrace);
 break;
   case FAILED_WITH_EXIT_CODE:
-setHealthStatus(false, exceptionStackTrace);
+setHealthStatus(true, "", now);
 break;
   case FAILED:
 setHealthStatus(false, shexec.getOutput());

http://git-wip-us.apache.org/repos/asf/hadoop/blob/c1420480/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestNodeHealthScriptRunner.java
--
diff --git 
a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestNodeHealthScriptRunner.java
 
b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestNodeHealthScriptRunner.java
index db61f5a..8fc64d1 100644
--- 
a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestNodeHealthScriptRunner.java
+++ 
b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestNodeHealthScriptRunner.java
@@ -91,7 +91,6 @@ public class TestNodeHealthScriptRunner {
   public void testNodeHealthScript() throws Exception {
 String errorScript = "echo ERROR\n echo \"Tracker not healthy\"";
 String normalScript = "echo \"I am all fine\"";
-String failWithExitCodeScript = "echo \"Not healthy\"; exit -1";
 String timeOutScript =
   Shell.WINDOWS ? "@echo off\nping -n 4 127.0.0.1 >nul\necho \"I am fine\""
   : "sleep 4\necho \"I am fine\"";
@@ -125,12 +124,6 @@ public class TestNodeHealthScriptRunner {
 nodeHealthScriptRunner.isHealthy());
 Assert.assertEquals("", nodeHealthScriptRunner.getHealthReport());
 
-// Script which fails with exit code.
-writeNodeHealthScriptFile(failWithExitCodeScript, true);
-timerTask.run();
-Assert.assertFalse("Node health status reported healthy",
-nodeHealthScriptRunner.isHealthy());
-
 // Timeout script.
 writeNodeHealthScriptFile(timeOutScript, true);
 timerTask.run();


-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org

hadoop git commit: YARN-5567. Fix script exit code checking in NodeHealthScriptRunner#reportHealthStatus. (Yufei Gu via rchiang)

2016-08-29 Thread rchiang

Repository: hadoop
Updated Branches:
  refs/heads/branch-2 3ad4f0823 -> 3b19bcb8f


YARN-5567. Fix script exit code checking in 
NodeHealthScriptRunner#reportHealthStatus. (Yufei Gu via rchiang)

(cherry picked from commit 05ede003868871addc30162e9707c3dc14ed6b7a)


Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/3b19bcb8
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/3b19bcb8
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/3b19bcb8

Branch: refs/heads/branch-2
Commit: 3b19bcb8f0f70606a8080bc46fa64cd34f6280d7
Parents: 3ad4f08
Author: Ray Chiang <rchi...@apache.org>
Authored: Mon Aug 29 15:55:33 2016 -0700
Committer: Ray Chiang <rchi...@apache.org>
Committed: Mon Aug 29 15:56:30 2016 -0700

--
 .../java/org/apache/hadoop/util/NodeHealthScriptRunner.java   | 3 ++-
 .../org/apache/hadoop/util/TestNodeHealthScriptRunner.java| 7 +++
 2 files changed, 9 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/hadoop/blob/3b19bcb8/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/NodeHealthScriptRunner.java
--
diff --git 
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/NodeHealthScriptRunner.java
 
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/NodeHealthScriptRunner.java
index fc392c4..c3bef37 100644
--- 
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/NodeHealthScriptRunner.java
+++ 
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/NodeHealthScriptRunner.java
@@ -106,6 +106,7 @@ public class NodeHealthScriptRunner extends AbstractService 
{
 shexec.execute();
   } catch (ExitCodeException e) {
 // ignore the exit code of the script
+exceptionStackTrace = StringUtils.stringifyException(e);
 status = HealthCheckerExitStatus.FAILED_WITH_EXIT_CODE;
 // On Windows, we will not hit the Stream closed IOException
 // thrown by stdout buffered reader for timeout event.
@@ -162,7 +163,7 @@ public class NodeHealthScriptRunner extends AbstractService 
{
 setHealthStatus(false, exceptionStackTrace);
 break;
   case FAILED_WITH_EXIT_CODE:
-setHealthStatus(true, "", now);
+setHealthStatus(false, exceptionStackTrace);
 break;
   case FAILED:
 setHealthStatus(false, shexec.getOutput());

http://git-wip-us.apache.org/repos/asf/hadoop/blob/3b19bcb8/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestNodeHealthScriptRunner.java
--
diff --git 
a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestNodeHealthScriptRunner.java
 
b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestNodeHealthScriptRunner.java
index 8fc64d1..db61f5a 100644
--- 
a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestNodeHealthScriptRunner.java
+++ 
b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestNodeHealthScriptRunner.java
@@ -91,6 +91,7 @@ public class TestNodeHealthScriptRunner {
   public void testNodeHealthScript() throws Exception {
 String errorScript = "echo ERROR\n echo \"Tracker not healthy\"";
 String normalScript = "echo \"I am all fine\"";
+String failWithExitCodeScript = "echo \"Not healthy\"; exit -1";
 String timeOutScript =
   Shell.WINDOWS ? "@echo off\nping -n 4 127.0.0.1 >nul\necho \"I am fine\""
   : "sleep 4\necho \"I am fine\"";
@@ -124,6 +125,12 @@ public class TestNodeHealthScriptRunner {
 nodeHealthScriptRunner.isHealthy());
 Assert.assertEquals("", nodeHealthScriptRunner.getHealthReport());
 
+// Script which fails with exit code.
+writeNodeHealthScriptFile(failWithExitCodeScript, true);
+timerTask.run();
+Assert.assertFalse("Node health status reported healthy",
+nodeHealthScriptRunner.isHealthy());
+
 // Timeout script.
 writeNodeHealthScriptFile(timeOutScript, true);
 timerTask.run();


-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org

hadoop git commit: YARN-5567. Fix script exit code checking in NodeHealthScriptRunner#reportHealthStatus. (Yufei Gu via rchiang)

2016-08-29 Thread rchiang

Repository: hadoop
Updated Branches:
  refs/heads/trunk 8b57be108 -> 05ede0038


YARN-5567. Fix script exit code checking in 
NodeHealthScriptRunner#reportHealthStatus. (Yufei Gu via rchiang)


Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/05ede003
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/05ede003
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/05ede003

Branch: refs/heads/trunk
Commit: 05ede003868871addc30162e9707c3dc14ed6b7a
Parents: 8b57be1
Author: Ray Chiang <rchi...@apache.org>
Authored: Mon Aug 29 15:55:33 2016 -0700
Committer: Ray Chiang <rchi...@apache.org>
Committed: Mon Aug 29 15:55:33 2016 -0700

--
 .../java/org/apache/hadoop/util/NodeHealthScriptRunner.java   | 3 ++-
 .../org/apache/hadoop/util/TestNodeHealthScriptRunner.java| 7 +++
 2 files changed, 9 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/hadoop/blob/05ede003/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/NodeHealthScriptRunner.java
--
diff --git 
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/NodeHealthScriptRunner.java
 
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/NodeHealthScriptRunner.java
index fc392c4..c3bef37 100644
--- 
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/NodeHealthScriptRunner.java
+++ 
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/NodeHealthScriptRunner.java
@@ -106,6 +106,7 @@ public class NodeHealthScriptRunner extends AbstractService 
{
 shexec.execute();
   } catch (ExitCodeException e) {
 // ignore the exit code of the script
+exceptionStackTrace = StringUtils.stringifyException(e);
 status = HealthCheckerExitStatus.FAILED_WITH_EXIT_CODE;
 // On Windows, we will not hit the Stream closed IOException
 // thrown by stdout buffered reader for timeout event.
@@ -162,7 +163,7 @@ public class NodeHealthScriptRunner extends AbstractService 
{
 setHealthStatus(false, exceptionStackTrace);
 break;
   case FAILED_WITH_EXIT_CODE:
-setHealthStatus(true, "", now);
+setHealthStatus(false, exceptionStackTrace);
 break;
   case FAILED:
 setHealthStatus(false, shexec.getOutput());

http://git-wip-us.apache.org/repos/asf/hadoop/blob/05ede003/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestNodeHealthScriptRunner.java
--
diff --git 
a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestNodeHealthScriptRunner.java
 
b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestNodeHealthScriptRunner.java
index 8fc64d1..db61f5a 100644
--- 
a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestNodeHealthScriptRunner.java
+++ 
b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestNodeHealthScriptRunner.java
@@ -91,6 +91,7 @@ public class TestNodeHealthScriptRunner {
   public void testNodeHealthScript() throws Exception {
 String errorScript = "echo ERROR\n echo \"Tracker not healthy\"";
 String normalScript = "echo \"I am all fine\"";
+String failWithExitCodeScript = "echo \"Not healthy\"; exit -1";
 String timeOutScript =
   Shell.WINDOWS ? "@echo off\nping -n 4 127.0.0.1 >nul\necho \"I am fine\""
   : "sleep 4\necho \"I am fine\"";
@@ -124,6 +125,12 @@ public class TestNodeHealthScriptRunner {
 nodeHealthScriptRunner.isHealthy());
 Assert.assertEquals("", nodeHealthScriptRunner.getHealthReport());
 
+// Script which fails with exit code.
+writeNodeHealthScriptFile(failWithExitCodeScript, true);
+timerTask.run();
+Assert.assertFalse("Node health status reported healthy",
+nodeHealthScriptRunner.isHealthy());
+
 // Timeout script.
 writeNodeHealthScriptFile(timeOutScript, true);
 timerTask.run();


-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org

hadoop git commit: YARN-5567. Fix script exit code checking in NodeHealthScriptRunner#reportHealthStatus. (Yufei Gu via rchiang)

2016-08-29 Thread rchiang

Repository: hadoop
Updated Branches:
  refs/heads/branch-2.8 765139e4d -> 9ad3a7a43


YARN-5567. Fix script exit code checking in 
NodeHealthScriptRunner#reportHealthStatus. (Yufei Gu via rchiang)

(cherry picked from commit 05ede003868871addc30162e9707c3dc14ed6b7a)
(cherry picked from commit 3b19bcb8f0f70606a8080bc46fa64cd34f6280d7)


Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/9ad3a7a4
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/9ad3a7a4
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/9ad3a7a4

Branch: refs/heads/branch-2.8
Commit: 9ad3a7a43b35dd4bb9d10bd322fbb956d3b01ff2
Parents: 765139e
Author: Ray Chiang <rchi...@apache.org>
Authored: Mon Aug 29 15:55:33 2016 -0700
Committer: Ray Chiang <rchi...@apache.org>
Committed: Mon Aug 29 16:15:39 2016 -0700

--
 .../java/org/apache/hadoop/util/NodeHealthScriptRunner.java   | 3 ++-
 .../org/apache/hadoop/util/TestNodeHealthScriptRunner.java| 7 +++
 2 files changed, 9 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/hadoop/blob/9ad3a7a4/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/NodeHealthScriptRunner.java
--
diff --git 
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/NodeHealthScriptRunner.java
 
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/NodeHealthScriptRunner.java
index fc392c4..c3bef37 100644
--- 
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/NodeHealthScriptRunner.java
+++ 
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/NodeHealthScriptRunner.java
@@ -106,6 +106,7 @@ public class NodeHealthScriptRunner extends AbstractService 
{
 shexec.execute();
   } catch (ExitCodeException e) {
 // ignore the exit code of the script
+exceptionStackTrace = StringUtils.stringifyException(e);
 status = HealthCheckerExitStatus.FAILED_WITH_EXIT_CODE;
 // On Windows, we will not hit the Stream closed IOException
 // thrown by stdout buffered reader for timeout event.
@@ -162,7 +163,7 @@ public class NodeHealthScriptRunner extends AbstractService 
{
 setHealthStatus(false, exceptionStackTrace);
 break;
   case FAILED_WITH_EXIT_CODE:
-setHealthStatus(true, "", now);
+setHealthStatus(false, exceptionStackTrace);
 break;
   case FAILED:
 setHealthStatus(false, shexec.getOutput());

http://git-wip-us.apache.org/repos/asf/hadoop/blob/9ad3a7a4/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestNodeHealthScriptRunner.java
--
diff --git 
a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestNodeHealthScriptRunner.java
 
b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestNodeHealthScriptRunner.java
index 8fc64d1..db61f5a 100644
--- 
a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestNodeHealthScriptRunner.java
+++ 
b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestNodeHealthScriptRunner.java
@@ -91,6 +91,7 @@ public class TestNodeHealthScriptRunner {
   public void testNodeHealthScript() throws Exception {
 String errorScript = "echo ERROR\n echo \"Tracker not healthy\"";
 String normalScript = "echo \"I am all fine\"";
+String failWithExitCodeScript = "echo \"Not healthy\"; exit -1";
 String timeOutScript =
   Shell.WINDOWS ? "@echo off\nping -n 4 127.0.0.1 >nul\necho \"I am fine\""
   : "sleep 4\necho \"I am fine\"";
@@ -124,6 +125,12 @@ public class TestNodeHealthScriptRunner {
 nodeHealthScriptRunner.isHealthy());
 Assert.assertEquals("", nodeHealthScriptRunner.getHealthReport());
 
+// Script which fails with exit code.
+writeNodeHealthScriptFile(failWithExitCodeScript, true);
+timerTask.run();
+Assert.assertFalse("Node health status reported healthy",
+nodeHealthScriptRunner.isHealthy());
+
 // Timeout script.
 writeNodeHealthScriptFile(timeOutScript, true);
 timerTask.run();


-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org

hadoop git commit: MAPREDUCE-6751. Add debug log message when splitting is not possible due to unsplittable compression. (Peter Vary via rchiang)

2016-08-16 Thread rchiang

Repository: hadoop
Updated Branches:
  refs/heads/branch-2 96e9f8bec -> d1c4ba2d7


MAPREDUCE-6751. Add debug log message when splitting is not possible due to 
unsplittable compression. (Peter Vary via rchiang)

(cherry picked from commit 6c154abd33279475315b5f7f78dc47f1b0aa7028)


Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/d1c4ba2d
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/d1c4ba2d
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/d1c4ba2d

Branch: refs/heads/branch-2
Commit: d1c4ba2d7cd22c79ad10bb4729bd6385963cc3d6
Parents: 96e9f8b
Author: Ray Chiang <rchi...@apache.org>
Authored: Tue Aug 16 12:13:22 2016 -0700
Committer: Ray Chiang <rchi...@apache.org>
Committed: Tue Aug 16 12:14:30 2016 -0700

--
 .../main/java/org/apache/hadoop/mapred/FileInputFormat.java   | 7 +++
 .../hadoop/mapreduce/lib/input/CombineFileInputFormat.java| 4 
 .../apache/hadoop/mapreduce/lib/input/FileInputFormat.java| 7 +++
 3 files changed, 18 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/hadoop/blob/d1c4ba2d/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/FileInputFormat.java
--
diff --git 
a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/FileInputFormat.java
 
b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/FileInputFormat.java
index 2c58ebe..5803d60 100644
--- 
a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/FileInputFormat.java
+++ 
b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/FileInputFormat.java
@@ -369,6 +369,13 @@ public abstract class FileInputFormat<K, V> implements 
InputFormat<K, V> {
 splitHosts[0], splitHosts[1]));
   }
 } else {
+  if (LOG.isDebugEnabled()) {
+// Log only if the file is big enough to be splitted
+if (length > Math.min(file.getBlockSize(), minSize)) {
+  LOG.debug("File is not splittable so no parallelization "
+  + "is possible: " + file.getPath());
+}
+  }
   String[][] splitHosts = 
getSplitHostsAndCachedHosts(blkLocations,0,length,clusterMap);
   splits.add(makeSplit(path, 0, length, splitHosts[0], splitHosts[1]));
 }

http://git-wip-us.apache.org/repos/asf/hadoop/blob/d1c4ba2d/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/input/CombineFileInputFormat.java
--
diff --git 
a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/input/CombineFileInputFormat.java
 
b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/input/CombineFileInputFormat.java
index b2b7656..8f9699e 100644
--- 
a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/input/CombineFileInputFormat.java
+++ 
b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/input/CombineFileInputFormat.java
@@ -600,6 +600,10 @@ public abstract class CombineFileInputFormat<K, V>
 if (!isSplitable) {
   // if the file is not splitable, just create the one block with
   // full file length
+  if (LOG.isDebugEnabled()) {
+LOG.debug("File is not splittable so no parallelization "
++ "is possible: " + stat.getPath());
+  }
   blocks = new OneBlockInfo[1];
   fileSize = stat.getLen();
   blocks[0] = new OneBlockInfo(stat.getPath(), 0, fileSize,

http://git-wip-us.apache.org/repos/asf/hadoop/blob/d1c4ba2d/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/input/FileInputFormat.java
--
diff --git 
a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/input/FileInputFormat.java
 
b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/input/FileInputFormat.java
index 0c5

hadoop git commit: MAPREDUCE-6751. Add debug log message when splitting is not possible due to unsplittable compression. (Peter Vary via rchiang)

2016-08-16 Thread rchiang

Repository: hadoop
Updated Branches:
  refs/heads/trunk b047bc727 -> 6c154abd3


MAPREDUCE-6751. Add debug log message when splitting is not possible due to 
unsplittable compression. (Peter Vary via rchiang)


Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/6c154abd
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/6c154abd
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/6c154abd

Branch: refs/heads/trunk
Commit: 6c154abd33279475315b5f7f78dc47f1b0aa7028
Parents: b047bc7
Author: Ray Chiang <rchi...@apache.org>
Authored: Tue Aug 16 12:13:22 2016 -0700
Committer: Ray Chiang <rchi...@apache.org>
Committed: Tue Aug 16 12:13:22 2016 -0700

--
 .../main/java/org/apache/hadoop/mapred/FileInputFormat.java   | 7 +++
 .../hadoop/mapreduce/lib/input/CombineFileInputFormat.java| 4 
 .../apache/hadoop/mapreduce/lib/input/FileInputFormat.java| 7 +++
 3 files changed, 18 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/hadoop/blob/6c154abd/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/FileInputFormat.java
--
diff --git 
a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/FileInputFormat.java
 
b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/FileInputFormat.java
index 2c58ebe..5803d60 100644
--- 
a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/FileInputFormat.java
+++ 
b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/FileInputFormat.java
@@ -369,6 +369,13 @@ public abstract class FileInputFormat<K, V> implements 
InputFormat<K, V> {
 splitHosts[0], splitHosts[1]));
   }
 } else {
+  if (LOG.isDebugEnabled()) {
+// Log only if the file is big enough to be splitted
+if (length > Math.min(file.getBlockSize(), minSize)) {
+  LOG.debug("File is not splittable so no parallelization "
+  + "is possible: " + file.getPath());
+}
+  }
   String[][] splitHosts = 
getSplitHostsAndCachedHosts(blkLocations,0,length,clusterMap);
   splits.add(makeSplit(path, 0, length, splitHosts[0], splitHosts[1]));
 }

http://git-wip-us.apache.org/repos/asf/hadoop/blob/6c154abd/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/input/CombineFileInputFormat.java
--
diff --git 
a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/input/CombineFileInputFormat.java
 
b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/input/CombineFileInputFormat.java
index b2b7656..8f9699e 100644
--- 
a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/input/CombineFileInputFormat.java
+++ 
b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/input/CombineFileInputFormat.java
@@ -600,6 +600,10 @@ public abstract class CombineFileInputFormat<K, V>
 if (!isSplitable) {
   // if the file is not splitable, just create the one block with
   // full file length
+  if (LOG.isDebugEnabled()) {
+LOG.debug("File is not splittable so no parallelization "
++ "is possible: " + stat.getPath());
+  }
   blocks = new OneBlockInfo[1];
   fileSize = stat.getLen();
   blocks[0] = new OneBlockInfo(stat.getPath(), 0, fileSize,

http://git-wip-us.apache.org/repos/asf/hadoop/blob/6c154abd/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/input/FileInputFormat.java
--
diff --git 
a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/input/FileInputFormat.java
 
b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/input/FileInputFormat.java
index 0c5ede9..7ec882f 100644
--- 
a/hadoop-mapreduce-project/hadoop-mapreduce-client/hado

hadoop git commit: YARN-5137. Make DiskChecker pluggable in NodeManager. (Yufei Gu via rchiang)

2016-08-10 Thread rchiang

Repository: hadoop
Updated Branches:
  refs/heads/branch-2 01fc975ed -> 372f6f845


YARN-5137. Make DiskChecker pluggable in NodeManager. (Yufei Gu via rchiang)

(cherry picked from commit dbe9e70cc084220ea1f68da850cdb092281b5e96)


Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/372f6f84
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/372f6f84
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/372f6f84

Branch: refs/heads/branch-2
Commit: 372f6f8456b1a4d06fcc5dd751041e7dd52d3fbc
Parents: 01fc975
Author: Ray Chiang <rchi...@apache.org>
Authored: Wed Aug 10 16:03:12 2016 -0700
Committer: Ray Chiang <rchi...@apache.org>
Committed: Wed Aug 10 19:26:49 2016 -0700

--
 .../hadoop/yarn/conf/YarnConfiguration.java |  4 
 .../src/main/resources/yarn-default.xml |  8 +++
 .../server/nodemanager/DirectoryCollection.java | 22 +---
 .../localizer/ContainerLocalizer.java   | 13 
 .../localizer/ResourceLocalizationService.java  | 15 -
 5 files changed, 54 insertions(+), 8 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/hadoop/blob/372f6f84/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
--
diff --git 
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
 
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
index c341022..84fb0d4 100644
--- 
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
+++ 
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
@@ -897,6 +897,10 @@ public class YarnConfiguration extends Configuration {
   NM_PREFIX + "resourcemanager.minimum.version";
   public static final String DEFAULT_NM_RESOURCEMANAGER_MINIMUM_VERSION = 
"NONE";
 
+  /** Disk Validator. */
+  public static final String DISK_VALIDATOR = NM_PREFIX + "disk-validator";
+  public static final String DEFAULT_DISK_VALIDATOR = "basic";
+
   /**
* Maximum size of contain's diagnostics to keep for relaunching container
* case.

http://git-wip-us.apache.org/repos/asf/hadoop/blob/372f6f84/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
--
diff --git 
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
 
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
index 2a8bd2d..723dd1a 100644
--- 
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
+++ 
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
@@ -2892,6 +2892,14 @@
 
   
 
+  The name of disk validator.
+
+yarn.nodemanager.disk-validator
+basic
+  
+
+  
+  
   Enable the CSRF filter for the timeline service web app
 
 yarn.timeline-service.webapp.rest-csrf.enabled

http://git-wip-us.apache.org/repos/asf/hadoop/blob/372f6f84/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DirectoryCollection.java
--
diff --git 
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DirectoryCollection.java
 
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DirectoryCollection.java
index 417b207..72c32e8 100644
--- 
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DirectoryCollection.java
+++ 
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DirectoryCollection.java
@@ -42,7 +42,11 @@ import org.apache.hadoop.fs.FileAlreadyExistsException;
 import org.apache.hadoop.fs.FileContext;
 import org.apache.hadoop.fs.Path;
 import org.apache.hadoop.fs.permission.FsPermission;
-import org.apache.hadoop.util.DiskChecker;
+import org.apache.hadoop.util.DiskValidator;
+import org.apache.hadoop.util.DiskValidatorFactory;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.yarn.conf.YarnConfiguration;
+import org.apache.hadoop.yarn.exceptions.YarnRuntimeException;

hadoop git commit: YARN-5137. Make DiskChecker pluggable in NodeManager. (Yufei Gu via rchiang)

2016-08-10 Thread rchiang

Repository: hadoop
Updated Branches:
  refs/heads/trunk e83be44af -> aea3e6574


YARN-5137. Make DiskChecker pluggable in NodeManager. (Yufei Gu via rchiang)


Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/aea3e657
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/aea3e657
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/aea3e657

Branch: refs/heads/trunk
Commit: aea3e65749333b97d8c45280077f03d4ebddb5a8
Parents: e83be44
Author: Ray Chiang <rchi...@apache.org>
Authored: Wed Aug 10 16:03:12 2016 -0700
Committer: Ray Chiang <rchi...@apache.org>
Committed: Wed Aug 10 19:23:29 2016 -0700

--
 .../hadoop/yarn/conf/YarnConfiguration.java |  4 
 .../src/main/resources/yarn-default.xml |  8 +++
 .../server/nodemanager/DirectoryCollection.java | 22 +---
 .../localizer/ContainerLocalizer.java   | 13 
 .../localizer/ResourceLocalizationService.java  | 15 -
 5 files changed, 54 insertions(+), 8 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/hadoop/blob/aea3e657/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
--
diff --git 
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
 
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
index d1f410b..8899ccd 100644
--- 
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
+++ 
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
@@ -935,6 +935,10 @@ public class YarnConfiguration extends Configuration {
   NM_PREFIX + "resourcemanager.minimum.version";
   public static final String DEFAULT_NM_RESOURCEMANAGER_MINIMUM_VERSION = 
"NONE";
 
+  /** Disk Validator. */
+  public static final String DISK_VALIDATOR = NM_PREFIX + "disk-validator";
+  public static final String DEFAULT_DISK_VALIDATOR = "basic";
+
   /**
* Maximum size of contain's diagnostics to keep for relaunching container
* case.

http://git-wip-us.apache.org/repos/asf/hadoop/blob/aea3e657/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
--
diff --git 
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
 
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
index 7305283..e77e990 100644
--- 
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
+++ 
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
@@ -2975,6 +2975,14 @@
 
   
 
+  The name of disk validator.
+
+yarn.nodemanager.disk-validator
+basic
+  
+
+  
+  
   Enable the CSRF filter for the timeline service web app
 
 yarn.timeline-service.webapp.rest-csrf.enabled

http://git-wip-us.apache.org/repos/asf/hadoop/blob/aea3e657/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DirectoryCollection.java
--
diff --git 
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DirectoryCollection.java
 
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DirectoryCollection.java
index 417b207..72c32e8 100644
--- 
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DirectoryCollection.java
+++ 
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DirectoryCollection.java
@@ -42,7 +42,11 @@ import org.apache.hadoop.fs.FileAlreadyExistsException;
 import org.apache.hadoop.fs.FileContext;
 import org.apache.hadoop.fs.Path;
 import org.apache.hadoop.fs.permission.FsPermission;
-import org.apache.hadoop.util.DiskChecker;
+import org.apache.hadoop.util.DiskValidator;
+import org.apache.hadoop.util.DiskValidatorFactory;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.yarn.conf.YarnConfiguration;
+import org.apache.hadoop.yarn.exceptions.YarnRuntimeException;
 
 import com.google.common.annotations.VisibleForTesting;
 
@@ -52,6 +56,8 @@

hadoop git commit: YARN-5272. Handle queue names consistently in FairScheduler. (Wilfred Spiegelenburg via rchiang)

2016-07-15 Thread rchiang

Repository: hadoop
Updated Branches:
  refs/heads/branch-2 9d31c0b53 -> 66a501be5


YARN-5272. Handle queue names consistently in FairScheduler. (Wilfred 
Spiegelenburg via rchiang)

(cherry picked from commit 2df1b1e72b18422606e38a9fbe311988637cb9a5)


Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/66a501be
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/66a501be
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/66a501be

Branch: refs/heads/branch-2
Commit: 66a501be5e3243df89332ea2bacc4683085d5ca2
Parents: 9d31c0b
Author: Ray Chiang <rchi...@apache.org>
Authored: Fri Jul 15 14:35:50 2016 -0700
Committer: Ray Chiang <rchi...@apache.org>
Committed: Fri Jul 15 14:39:28 2016 -0700

--
 .../fair/AllocationFileLoaderService.java   |  4 ++-
 .../scheduler/fair/QueueManager.java|  6 -
 .../fair/TestAllocationFileLoaderService.java   | 28 
 .../scheduler/fair/TestFairScheduler.java   | 11 
 .../scheduler/fair/TestQueueManager.java|  3 +++
 5 files changed, 50 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/hadoop/blob/66a501be/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/AllocationFileLoaderService.java
--
diff --git 
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/AllocationFileLoaderService.java
 
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/AllocationFileLoaderService.java
index 661caa7..fab536d 100644
--- 
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/AllocationFileLoaderService.java
+++ 
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/AllocationFileLoaderService.java
@@ -53,6 +53,7 @@ import org.w3c.dom.NodeList;
 import org.w3c.dom.Text;
 import org.xml.sax.SAXException;
 
+import com.google.common.base.CharMatcher;
 import com.google.common.annotations.VisibleForTesting;
 
 @Public
@@ -445,7 +446,8 @@ public class AllocationFileLoaderService extends 
AbstractService {
   Set reservableQueues,
   Set nonPreemptableQueues)
   throws AllocationConfigurationException {
-String queueName = element.getAttribute("name").trim();
+String queueName = CharMatcher.WHITESPACE.trimFrom(
+element.getAttribute("name"));
 
 if (queueName.contains(".")) {
   throw new AllocationConfigurationException("Bad fair scheduler config "

http://git-wip-us.apache.org/repos/asf/hadoop/blob/66a501be/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/QueueManager.java
--
diff --git 
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/QueueManager.java
 
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/QueueManager.java
index aeadcf6..8d308dc 100644
--- 
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/QueueManager.java
+++ 
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/QueueManager.java
@@ -37,6 +37,7 @@ import org.apache.hadoop.conf.Configuration;
 import org.apache.hadoop.yarn.conf.YarnConfiguration;
 import org.xml.sax.SAXException;
 
+import com.google.common.base.CharMatcher;
 import com.google.common.annotations.VisibleForTesting;
 /**
  * Maintains a list of queues as well as scheduling parameters for each queue,
@@ -457,6 +458,9 @@ public class QueueManager {
*/
   @VisibleForTesting
   boolean isQueueNameValid(String node) {
-return !node.isEmpty() && node.equals(node.trim());
+// use the same white space trim as in QueueMetrics() otherwise things fail
+// guava uses a different definition for whitespace than java.
+return !node.isEmpty

hadoop git commit: YARN-5272. Handle queue names consistently in FairScheduler. (Wilfred Spiegelenburg via rchiang)

2016-07-15 Thread rchiang

Repository: hadoop
Updated Branches:
  refs/heads/trunk e905a42a2 -> f5f1c81e7


YARN-5272. Handle queue names consistently in FairScheduler. (Wilfred 
Spiegelenburg via rchiang)


Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/f5f1c81e
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/f5f1c81e
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/f5f1c81e

Branch: refs/heads/trunk
Commit: f5f1c81e7dcae0272e71ef4e6bedfc00b8c677d6
Parents: e905a42
Author: Ray Chiang <rchi...@apache.org>
Authored: Fri Jul 15 14:35:50 2016 -0700
Committer: Ray Chiang <rchi...@apache.org>
Committed: Fri Jul 15 14:38:50 2016 -0700

--
 .../fair/AllocationFileLoaderService.java   |  4 ++-
 .../scheduler/fair/QueueManager.java|  6 -
 .../fair/TestAllocationFileLoaderService.java   | 28 
 .../scheduler/fair/TestFairScheduler.java   | 11 
 .../scheduler/fair/TestQueueManager.java|  3 +++
 5 files changed, 50 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/hadoop/blob/f5f1c81e/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/AllocationFileLoaderService.java
--
diff --git 
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/AllocationFileLoaderService.java
 
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/AllocationFileLoaderService.java
index 661caa7..fab536d 100644
--- 
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/AllocationFileLoaderService.java
+++ 
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/AllocationFileLoaderService.java
@@ -53,6 +53,7 @@ import org.w3c.dom.NodeList;
 import org.w3c.dom.Text;
 import org.xml.sax.SAXException;
 
+import com.google.common.base.CharMatcher;
 import com.google.common.annotations.VisibleForTesting;
 
 @Public
@@ -445,7 +446,8 @@ public class AllocationFileLoaderService extends 
AbstractService {
   Set reservableQueues,
   Set nonPreemptableQueues)
   throws AllocationConfigurationException {
-String queueName = element.getAttribute("name").trim();
+String queueName = CharMatcher.WHITESPACE.trimFrom(
+element.getAttribute("name"));
 
 if (queueName.contains(".")) {
   throw new AllocationConfigurationException("Bad fair scheduler config "

http://git-wip-us.apache.org/repos/asf/hadoop/blob/f5f1c81e/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/QueueManager.java
--
diff --git 
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/QueueManager.java
 
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/QueueManager.java
index aeadcf6..8d308dc 100644
--- 
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/QueueManager.java
+++ 
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/QueueManager.java
@@ -37,6 +37,7 @@ import org.apache.hadoop.conf.Configuration;
 import org.apache.hadoop.yarn.conf.YarnConfiguration;
 import org.xml.sax.SAXException;
 
+import com.google.common.base.CharMatcher;
 import com.google.common.annotations.VisibleForTesting;
 /**
  * Maintains a list of queues as well as scheduling parameters for each queue,
@@ -457,6 +458,9 @@ public class QueueManager {
*/
   @VisibleForTesting
   boolean isQueueNameValid(String node) {
-return !node.isEmpty() && node.equals(node.trim());
+// use the same white space trim as in QueueMetrics() otherwise things fail
+// guava uses a different definition for whitespace than java.
+return !node.isEmpty() &&
+node.equals(CharMatcher.WHITESPACE.trimFrom(node));
   }
 }

svn commit: r1751944 - in /hadoop/common/site/main: author/src/documentation/content/xdocs/ publish/ publish/skin/images/

2016-07-08 Thread rchiang

Author: rchiang
Date: Fri Jul  8 17:37:03 2016
New Revision: 1751944

URL: http://svn.apache.org/viewvc?rev=1751944=rev
Log:
Adding myself (rchiang) to committers list.

Modified:
hadoop/common/site/main/author/src/documentation/content/xdocs/who.xml
hadoop/common/site/main/publish/bylaws.pdf
hadoop/common/site/main/publish/index.pdf
hadoop/common/site/main/publish/issue_tracking.pdf
hadoop/common/site/main/publish/linkmap.pdf
hadoop/common/site/main/publish/mailing_lists.pdf
hadoop/common/site/main/publish/privacy_policy.pdf
hadoop/common/site/main/publish/releases.pdf
hadoop/common/site/main/publish/skin/images/rc-b-l-15-1body-2menu-3menu.png
hadoop/common/site/main/publish/skin/images/rc-b-r-15-1body-2menu-3menu.png

hadoop/common/site/main/publish/skin/images/rc-b-r-5-1header-2tab-selected-3tab-selected.png

hadoop/common/site/main/publish/skin/images/rc-t-l-5-1header-2searchbox-3searchbox.png

hadoop/common/site/main/publish/skin/images/rc-t-l-5-1header-2tab-selected-3tab-selected.png

hadoop/common/site/main/publish/skin/images/rc-t-l-5-1header-2tab-unselected-3tab-unselected.png
hadoop/common/site/main/publish/skin/images/rc-t-r-15-1body-2menu-3menu.png

hadoop/common/site/main/publish/skin/images/rc-t-r-5-1header-2searchbox-3searchbox.png

hadoop/common/site/main/publish/skin/images/rc-t-r-5-1header-2tab-selected-3tab-selected.png

hadoop/common/site/main/publish/skin/images/rc-t-r-5-1header-2tab-unselected-3tab-unselected.png
hadoop/common/site/main/publish/version_control.pdf
hadoop/common/site/main/publish/who.html
hadoop/common/site/main/publish/who.pdf

Modified: hadoop/common/site/main/author/src/documentation/content/xdocs/who.xml
URL: 
http://svn.apache.org/viewvc/hadoop/common/site/main/author/src/documentation/content/xdocs/who.xml?rev=1751944=1751943=1751944=diff
==
--- hadoop/common/site/main/author/src/documentation/content/xdocs/who.xml 
(original)
+++ hadoop/common/site/main/author/src/documentation/content/xdocs/who.xml Fri 
Jul  8 17:37:03 2016
@@ -1163,6 +1163,14 @@

 

+ rchiang
+ Ray Chiang
+ Cloudera
+ 
+ -8
+   
+
+   
  rkanter
  http://people.apache.org/~rkanter;>Robert Kanter
  Cloudera

Modified: hadoop/common/site/main/publish/bylaws.pdf
URL: 
http://svn.apache.org/viewvc/hadoop/common/site/main/publish/bylaws.pdf?rev=1751944=1751943=1751944=diff
==
Binary files - no diff available.

Modified: hadoop/common/site/main/publish/index.pdf
URL: 
http://svn.apache.org/viewvc/hadoop/common/site/main/publish/index.pdf?rev=1751944=1751943=1751944=diff
==
Binary files - no diff available.

Modified: hadoop/common/site/main/publish/issue_tracking.pdf
URL: 
http://svn.apache.org/viewvc/hadoop/common/site/main/publish/issue_tracking.pdf?rev=1751944=1751943=1751944=diff
==
Binary files - no diff available.

Modified: hadoop/common/site/main/publish/linkmap.pdf
URL: 
http://svn.apache.org/viewvc/hadoop/common/site/main/publish/linkmap.pdf?rev=1751944=1751943=1751944=diff
==
Binary files - no diff available.

Modified: hadoop/common/site/main/publish/mailing_lists.pdf
URL: 
http://svn.apache.org/viewvc/hadoop/common/site/main/publish/mailing_lists.pdf?rev=1751944=1751943=1751944=diff
==
Binary files - no diff available.

Modified: hadoop/common/site/main/publish/privacy_policy.pdf
URL: 
http://svn.apache.org/viewvc/hadoop/common/site/main/publish/privacy_policy.pdf?rev=1751944=1751943=1751944=diff
==
Binary files - no diff available.

Modified: hadoop/common/site/main/publish/releases.pdf
URL: 
http://svn.apache.org/viewvc/hadoop/common/site/main/publish/releases.pdf?rev=1751944=1751943=1751944=diff
==
Binary files - no diff available.

Modified: 
hadoop/common/site/main/publish/skin/images/rc-b-l-15-1body-2menu-3menu.png
URL: 
http://svn.apache.org/viewvc/hadoop/common/site/main/publish/skin/images/rc-b-l-15-1body-2menu-3menu.png?rev=1751944=1751943=1751944=diff
==
Binary files - no diff available.

Modified: 
hadoop/common/site/main/publish/skin/images/rc-b-r-15-1body-2menu-3menu.png
URL: 
http://svn.apache.org/viewvc/hadoop/common/site/main/publish/skin/images/rc-b-r-15-1body-2menu-3menu.png?rev=1751944=1751943=1751944=d

hadoop git commit: YARN-4568. Fix message when NodeManager runs into errors initializing the recovery directory. (rchiang)

2016-07-01 Thread rchiang

Repository: hadoop
Updated Branches:
  refs/heads/branch-2 44e6ae8cc -> fb9327eda


YARN-4568. Fix message when NodeManager runs into errors initializing the 
recovery directory. (rchiang)

(cherry picked from commit 0a5def155eff4564b5dc7685e7460952f51bbd24)


Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/fb9327ed
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/fb9327ed
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/fb9327ed

Branch: refs/heads/branch-2
Commit: fb9327edae0c169ff74ac4f9b740776b855e921d
Parents: 44e6ae8
Author: Ray Chiang <rchi...@cloudera.com>
Authored: Fri Jul 1 15:49:30 2016 -0700
Committer: Ray Chiang <rchi...@cloudera.com>
Committed: Fri Jul 1 15:55:56 2016 -0700

--
 .../apache/hadoop/yarn/server/nodemanager/NodeManager.java  | 9 -
 1 file changed, 8 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/hadoop/blob/fb9327ed/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeManager.java
--
diff --git 
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeManager.java
 
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeManager.java
index 6ca7ffe..f7d226e 100644
--- 
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeManager.java
+++ 
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeManager.java
@@ -287,7 +287,14 @@ public class NodeManager extends CompositeService
 .RM_WORK_PRESERVING_RECOVERY_ENABLED,
 YarnConfiguration.DEFAULT_RM_WORK_PRESERVING_RECOVERY_ENABLED);
 
-initAndStartRecoveryStore(conf);
+try {
+  initAndStartRecoveryStore(conf);
+} catch (IOException e) {
+  String recoveryDirName = conf.get(YarnConfiguration.NM_RECOVERY_DIR);
+  throw new
+  YarnRuntimeException("Unable to initialize recovery directory at "
+  + recoveryDirName, e);
+}
 
 NMContainerTokenSecretManager containerTokenSecretManager =
 new NMContainerTokenSecretManager(conf, nmStore);


-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org

hadoop git commit: YARN-4568. Fix message when NodeManager runs into errors initializing the recovery directory. (rchiang)

2016-07-01 Thread rchiang

Repository: hadoop
Updated Branches:
  refs/heads/trunk 36cd0bce8 -> 0a5def155


YARN-4568. Fix message when NodeManager runs into errors initializing the 
recovery directory. (rchiang)


Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/0a5def15
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/0a5def15
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/0a5def15

Branch: refs/heads/trunk
Commit: 0a5def155eff4564b5dc7685e7460952f51bbd24
Parents: 36cd0bc
Author: Ray Chiang <rchi...@cloudera.com>
Authored: Fri Jul 1 15:49:30 2016 -0700
Committer: Ray Chiang <rchi...@cloudera.com>
Committed: Fri Jul 1 15:53:46 2016 -0700

--
 .../apache/hadoop/yarn/server/nodemanager/NodeManager.java  | 9 -
 1 file changed, 8 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/hadoop/blob/0a5def15/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeManager.java
--
diff --git 
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeManager.java
 
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeManager.java
index 6ca7ffe..f7d226e 100644
--- 
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeManager.java
+++ 
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeManager.java
@@ -287,7 +287,14 @@ public class NodeManager extends CompositeService
 .RM_WORK_PRESERVING_RECOVERY_ENABLED,
 YarnConfiguration.DEFAULT_RM_WORK_PRESERVING_RECOVERY_ENABLED);
 
-initAndStartRecoveryStore(conf);
+try {
+  initAndStartRecoveryStore(conf);
+} catch (IOException e) {
+  String recoveryDirName = conf.get(YarnConfiguration.NM_RECOVERY_DIR);
+  throw new
+  YarnRuntimeException("Unable to initialize recovery directory at "
+  + recoveryDirName, e);
+}
 
 NMContainerTokenSecretManager containerTokenSecretManager =
 new NMContainerTokenSecretManager(conf, nmStore);


-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org

54 matches

Mail list logo