hadoop git commit: HADOOP-15040 Upgrade AWS SDK to 1.11.271: NPE bug spams logs w/ Yarn Log Aggregation

2018-02-13 Thread fabbri 
Repository: hadoop
Updated Branches:
  refs/heads/branch-3.0 16b2cad8e -> e6f6c0bfc


HADOOP-15040 Upgrade AWS SDK to 1.11.271: NPE bug spams logs w/ Yarn Log 
Aggregation


Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/e6f6c0bf
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/e6f6c0bf
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/e6f6c0bf

Branch: refs/heads/branch-3.0
Commit: e6f6c0bfc2db629485eca60f822aaccd2b6d8609
Parents: 16b2cad
Author: Aaron Fabbri 
Authored: Tue Feb 13 18:38:22 2018 -0800
Committer: Aaron Fabbri 
Committed: Tue Feb 13 20:20:46 2018 -0800

--
 hadoop-project/pom.xml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/hadoop/blob/e6f6c0bf/hadoop-project/pom.xml
--
diff --git a/hadoop-project/pom.xml b/hadoop-project/pom.xml
index b957ab9..616cf73 100644
--- a/hadoop-project/pom.xml
+++ b/hadoop-project/pom.xml
@@ -135,7 +135,7 @@
 1.0-beta-1
 1.0-alpha-8
 900
-1.11.199
+1.11.271
 2.3.4
 1.5

hadoop git commit: HADOOP-15040 Upgrade AWS SDK to 1.11.271: NPE bug spams logs w/ Yarn Log Aggregation

2018-02-13 Thread fabbri 
Repository: hadoop
Updated Branches:
  refs/heads/trunk 332269de0 -> 60971b819


HADOOP-15040 Upgrade AWS SDK to 1.11.271: NPE bug spams logs w/ Yarn Log 
Aggregation


Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/60971b81
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/60971b81
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/60971b81

Branch: refs/heads/trunk
Commit: 60971b8195c954c109e83cdbd1c94c700da4a271
Parents: 332269d
Author: Aaron Fabbri 
Authored: Tue Feb 13 18:38:22 2018 -0800
Committer: Aaron Fabbri 
Committed: Tue Feb 13 18:51:22 2018 -0800

--
 hadoop-project/pom.xml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/hadoop/blob/60971b81/hadoop-project/pom.xml
--
diff --git a/hadoop-project/pom.xml b/hadoop-project/pom.xml
index dd8465a..c27596c 100644
--- a/hadoop-project/pom.xml
+++ b/hadoop-project/pom.xml
@@ -137,7 +137,7 @@
 1.0-beta-1
 1.0-alpha-8
 900
-1.11.199
+1.11.271
 2.3.4
 1.5

[3/3] hadoop git commit: HADOOP-14799. Update nimbus-jose-jwt to 4.41.1. (rchiang)

2018-02-13 Thread brahma 
HADOOP-14799. Update nimbus-jose-jwt to 4.41.1. (rchiang)

(cherry picked from commit 556812c179aa094c21acf610439a8d69fe6420ab)


Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/557d6f28
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/557d6f28
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/557d6f28

Branch: refs/heads/branch-2.9
Commit: 557d6f28145e46c8ff21b291362808e88010bbde
Parents: b8b67e4
Author: Ray Chiang 
Authored: Tue Sep 12 10:19:34 2017 -0700
Committer: Brahma Reddy Battula 
Committed: Wed Feb 14 00:24:14 2018 +0530

--
 .../JWTRedirectAuthenticationHandler.java   |   4 +-
 .../TestJWTRedirectAuthenticationHandler.java   | 476 ++
 .../TestJWTRedirectAuthentictionHandler.java| 481 ---
 .../mapreduce/task/reduce/TestFetcher.java  |   4 -
 hadoop-project/pom.xml  |   2 +-
 5 files changed, 480 insertions(+), 487 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/hadoop/blob/557d6f28/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/server/JWTRedirectAuthenticationHandler.java
--
diff --git 
a/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/server/JWTRedirectAuthenticationHandler.java
 
b/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/server/JWTRedirectAuthenticationHandler.java
index 61f5b9e..884398c 100644
--- 
a/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/server/JWTRedirectAuthenticationHandler.java
+++ 
b/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/server/JWTRedirectAuthenticationHandler.java
@@ -28,6 +28,7 @@ import java.text.ParseException;
 
 import java.security.interfaces.RSAPublicKey;
 
+import com.google.common.annotations.VisibleForTesting;
 import 
org.apache.hadoop.security.authentication.client.AuthenticationException;
 import org.apache.hadoop.security.authentication.util.CertificateUtil;
 import org.slf4j.Logger;
@@ -216,7 +217,8 @@ public class JWTRedirectAuthenticationHandler extends
* @param request for getting the original request URL
* @return url to use as login url for redirect
*/
-  protected String constructLoginURL(HttpServletRequest request) {
+  @VisibleForTesting
+  String constructLoginURL(HttpServletRequest request) {
 String delimiter = "?";
 if (authenticationProviderUrl.contains("?")) {
   delimiter = "&";

http://git-wip-us.apache.org/repos/asf/hadoop/blob/557d6f28/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/server/TestJWTRedirectAuthenticationHandler.java
--
diff --git 
a/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/server/TestJWTRedirectAuthenticationHandler.java
 
b/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/server/TestJWTRedirectAuthenticationHandler.java
new file mode 100644
index 000..5a2db9b
--- /dev/null
+++ 
b/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/server/TestJWTRedirectAuthenticationHandler.java
@@ -0,0 +1,476 @@
+/**
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License. See accompanying LICENSE file.
+ */
+package org.apache.hadoop.security.authentication.server;
+
+import static org.junit.Assert.assertTrue;
+import static org.junit.Assert.fail;
+
+import java.io.File;
+import java.security.KeyPair;
+import java.security.KeyPairGenerator;
+import java.security.NoSuchAlgorithmException;
+import java.security.interfaces.RSAPrivateKey;
+import java.security.interfaces.RSAPublicKey;
+import java.util.List;
+import java.util.ArrayList;
+import java.util.Properties;
+import java.util.Vector;
+import java.util.Date;
+
+import javax.servlet.ServletException;
+import javax.servlet.http.Cookie;
+import javax.servlet.http.HttpServletRequest;
+import javax.servlet.http.HttpServletResponse;
+
+import org.apache.hadoop.minikdc.KerberosSecurityTestcase;
+import

[2/3] hadoop git commit: HADOOP-14799. Update nimbus-jose-jwt to 4.41.1. (rchiang)

2018-02-13 Thread brahma 
HADOOP-14799. Update nimbus-jose-jwt to 4.41.1. (rchiang)

(cherry picked from commit 556812c179aa094c21acf610439a8d69fe6420ab)


Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/bfd8fb30
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/bfd8fb30
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/bfd8fb30

Branch: refs/heads/branch-2.8
Commit: bfd8fb309d61ef249f8a51e7a0fbd5bb2845e074
Parents: 486fa88
Author: Ray Chiang 
Authored: Tue Sep 12 10:19:34 2017 -0700
Committer: Brahma Reddy Battula 
Committed: Wed Feb 14 00:08:02 2018 +0530

--
 .../JWTRedirectAuthenticationHandler.java   |   4 +-
 .../TestJWTRedirectAuthenticationHandler.java   | 476 ++
 .../TestJWTRedirectAuthentictionHandler.java| 481 ---
 .../mapreduce/task/reduce/TestFetcher.java  |   4 -
 hadoop-project/pom.xml  |   2 +-
 5 files changed, 480 insertions(+), 487 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/hadoop/blob/bfd8fb30/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/server/JWTRedirectAuthenticationHandler.java
--
diff --git 
a/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/server/JWTRedirectAuthenticationHandler.java
 
b/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/server/JWTRedirectAuthenticationHandler.java
index 61f5b9e..884398c 100644
--- 
a/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/server/JWTRedirectAuthenticationHandler.java
+++ 
b/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/server/JWTRedirectAuthenticationHandler.java
@@ -28,6 +28,7 @@ import java.text.ParseException;
 
 import java.security.interfaces.RSAPublicKey;
 
+import com.google.common.annotations.VisibleForTesting;
 import 
org.apache.hadoop.security.authentication.client.AuthenticationException;
 import org.apache.hadoop.security.authentication.util.CertificateUtil;
 import org.slf4j.Logger;
@@ -216,7 +217,8 @@ public class JWTRedirectAuthenticationHandler extends
* @param request for getting the original request URL
* @return url to use as login url for redirect
*/
-  protected String constructLoginURL(HttpServletRequest request) {
+  @VisibleForTesting
+  String constructLoginURL(HttpServletRequest request) {
 String delimiter = "?";
 if (authenticationProviderUrl.contains("?")) {
   delimiter = "&";

http://git-wip-us.apache.org/repos/asf/hadoop/blob/bfd8fb30/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/server/TestJWTRedirectAuthenticationHandler.java
--
diff --git 
a/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/server/TestJWTRedirectAuthenticationHandler.java
 
b/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/server/TestJWTRedirectAuthenticationHandler.java
new file mode 100644
index 000..5a2db9b
--- /dev/null
+++ 
b/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/server/TestJWTRedirectAuthenticationHandler.java
@@ -0,0 +1,476 @@
+/**
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License. See accompanying LICENSE file.
+ */
+package org.apache.hadoop.security.authentication.server;
+
+import static org.junit.Assert.assertTrue;
+import static org.junit.Assert.fail;
+
+import java.io.File;
+import java.security.KeyPair;
+import java.security.KeyPairGenerator;
+import java.security.NoSuchAlgorithmException;
+import java.security.interfaces.RSAPrivateKey;
+import java.security.interfaces.RSAPublicKey;
+import java.util.List;
+import java.util.ArrayList;
+import java.util.Properties;
+import java.util.Vector;
+import java.util.Date;
+
+import javax.servlet.ServletException;
+import javax.servlet.http.Cookie;
+import javax.servlet.http.HttpServletRequest;
+import javax.servlet.http.HttpServletResponse;
+
+import org.apache.hadoop.minikdc.KerberosSecurityTestcase;
+import

[1/3] hadoop git commit: HADOOP-14799. Update nimbus-jose-jwt to 4.41.1. (rchiang)

2018-02-13 Thread brahma 
Repository: hadoop
Updated Branches:
  refs/heads/branch-2 08feac4c8 -> 577a26719
  refs/heads/branch-2.8 486fa889d -> bfd8fb309
  refs/heads/branch-2.9 b8b67e4b8 -> 557d6f281


HADOOP-14799. Update nimbus-jose-jwt to 4.41.1. (rchiang)

(cherry picked from commit 556812c179aa094c21acf610439a8d69fe6420ab)


Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/577a2671
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/577a2671
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/577a2671

Branch: refs/heads/branch-2
Commit: 577a267198b1d745eee85d1ad9af64d4506bbd0a
Parents: 08feac4
Author: Ray Chiang 
Authored: Tue Sep 12 10:19:34 2017 -0700
Committer: Brahma Reddy Battula 
Committed: Wed Feb 14 00:07:09 2018 +0530

--
 .../JWTRedirectAuthenticationHandler.java   |   4 +-
 .../TestJWTRedirectAuthenticationHandler.java   | 476 ++
 .../TestJWTRedirectAuthentictionHandler.java| 481 ---
 .../mapreduce/task/reduce/TestFetcher.java  |   4 -
 hadoop-project/pom.xml  |   2 +-
 5 files changed, 480 insertions(+), 487 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/hadoop/blob/577a2671/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/server/JWTRedirectAuthenticationHandler.java
--
diff --git 
a/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/server/JWTRedirectAuthenticationHandler.java
 
b/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/server/JWTRedirectAuthenticationHandler.java
index 61f5b9e..884398c 100644
--- 
a/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/server/JWTRedirectAuthenticationHandler.java
+++ 
b/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/server/JWTRedirectAuthenticationHandler.java
@@ -28,6 +28,7 @@ import java.text.ParseException;
 
 import java.security.interfaces.RSAPublicKey;
 
+import com.google.common.annotations.VisibleForTesting;
 import 
org.apache.hadoop.security.authentication.client.AuthenticationException;
 import org.apache.hadoop.security.authentication.util.CertificateUtil;
 import org.slf4j.Logger;
@@ -216,7 +217,8 @@ public class JWTRedirectAuthenticationHandler extends
* @param request for getting the original request URL
* @return url to use as login url for redirect
*/
-  protected String constructLoginURL(HttpServletRequest request) {
+  @VisibleForTesting
+  String constructLoginURL(HttpServletRequest request) {
 String delimiter = "?";
 if (authenticationProviderUrl.contains("?")) {
   delimiter = "&";

http://git-wip-us.apache.org/repos/asf/hadoop/blob/577a2671/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/server/TestJWTRedirectAuthenticationHandler.java
--
diff --git 
a/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/server/TestJWTRedirectAuthenticationHandler.java
 
b/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/server/TestJWTRedirectAuthenticationHandler.java
new file mode 100644
index 000..5a2db9b
--- /dev/null
+++ 
b/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/server/TestJWTRedirectAuthenticationHandler.java
@@ -0,0 +1,476 @@
+/**
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License. See accompanying LICENSE file.
+ */
+package org.apache.hadoop.security.authentication.server;
+
+import static org.junit.Assert.assertTrue;
+import static org.junit.Assert.fail;
+
+import java.io.File;
+import java.security.KeyPair;
+import java.security.KeyPairGenerator;
+import java.security.NoSuchAlgorithmException;
+import java.security.interfaces.RSAPrivateKey;
+import java.security.interfaces.RSAPublicKey;
+import java.util.List;
+import java.util.ArrayList;
+import java.util.Properties;
+import java.util.Vector;
+import java.util.Date;
+
+import javax.servlet.ServletException;
+import javax.servlet.http.Cookie;

[1/3] hadoop git commit: HADOOP-12897. KerberosAuthenticator.authenticate to include URL on IO failures. Contributed by Ajay Kumar.

2018-02-13 Thread arp 
Repository: hadoop
Updated Branches:
  refs/heads/branch-3.0 bc2d4ba50 -> 16b2cad8e
  refs/heads/branch-3.1 1fcc3eac8 -> e88265ea5
  refs/heads/trunk c5e6e3de1 -> 332269de0


HADOOP-12897. KerberosAuthenticator.authenticate to include URL on IO failures. 
Contributed by Ajay Kumar.


Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/332269de
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/332269de
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/332269de

Branch: refs/heads/trunk
Commit: 332269de065d0f40eb54ee5e53b765217c24081e
Parents: c5e6e3d
Author: Arpit Agarwal 
Authored: Tue Feb 13 10:14:16 2018 -0800
Committer: Arpit Agarwal 
Committed: Tue Feb 13 10:14:16 2018 -0800

--
 .../client/KerberosAuthenticator.java   | 80 +---
 .../client/TestKerberosAuthenticator.java   | 29 +++
 2 files changed, 82 insertions(+), 27 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/hadoop/blob/332269de/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/client/KerberosAuthenticator.java
--
diff --git 
a/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/client/KerberosAuthenticator.java
 
b/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/client/KerberosAuthenticator.java
index 942d13c..64d4330 100644
--- 
a/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/client/KerberosAuthenticator.java
+++ 
b/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/client/KerberosAuthenticator.java
@@ -13,6 +13,8 @@
  */
 package org.apache.hadoop.security.authentication.client;
 
+import com.google.common.annotations.VisibleForTesting;
+import java.lang.reflect.Constructor;
 import org.apache.commons.codec.binary.Base64;
 import org.apache.hadoop.security.authentication.server.HttpConstants;
 import org.apache.hadoop.security.authentication.util.AuthToken;
@@ -177,41 +179,65 @@ public class KerberosAuthenticator implements 
Authenticator {
*/
   @Override
   public void authenticate(URL url, AuthenticatedURL.Token token)
-throws IOException, AuthenticationException {
+  throws IOException, AuthenticationException {
 if (!token.isSet()) {
   this.url = url;
   base64 = new Base64(0);
-  HttpURLConnection conn = token.openConnection(url, connConfigurator);
-  conn.setRequestMethod(AUTH_HTTP_METHOD);
-  conn.connect();
-  
-  boolean needFallback = false;
-  if (conn.getResponseCode() == HttpURLConnection.HTTP_OK) {
-LOG.debug("JDK performed authentication on our behalf.");
-// If the JDK already did the SPNEGO back-and-forth for
-// us, just pull out the token.
-AuthenticatedURL.extractToken(conn, token);
-if (isTokenKerberos(token)) {
-  return;
+  try {
+HttpURLConnection conn = token.openConnection(url, connConfigurator);
+conn.setRequestMethod(AUTH_HTTP_METHOD);
+conn.connect();
+
+boolean needFallback = false;
+if (conn.getResponseCode() == HttpURLConnection.HTTP_OK) {
+  LOG.debug("JDK performed authentication on our behalf.");
+  // If the JDK already did the SPNEGO back-and-forth for
+  // us, just pull out the token.
+  AuthenticatedURL.extractToken(conn, token);
+  if (isTokenKerberos(token)) {
+return;
+  }
+  needFallback = true;
 }
-needFallback = true;
-  }
-  if (!needFallback && isNegotiate(conn)) {
-LOG.debug("Performing our own SPNEGO sequence.");
-doSpnegoSequence(token);
-  } else {
-LOG.debug("Using fallback authenticator sequence.");
-Authenticator auth = getFallBackAuthenticator();
-// Make sure that the fall back authenticator have the same
-// ConnectionConfigurator, since the method might be overridden.
-// Otherwise the fall back authenticator might not have the information
-// to make the connection (e.g., SSL certificates)
-auth.setConnectionConfigurator(connConfigurator);
-auth.authenticate(url, token);
+if (!needFallback && isNegotiate(conn)) {
+  LOG.debug("Performing our own SPNEGO sequence.");
+  doSpnegoSequence(token);
+} else {
+  LOG.debug("Using fallback authenticator sequence.");
+  Authenticator auth = getFallBackAuthenticator();
+  // Make sure that the fall back authenticator have the same
+  // ConnectionConfigurator, since the method might be overridden.
+  //

[3/3] hadoop git commit: HADOOP-12897. KerberosAuthenticator.authenticate to include URL on IO failures. Contributed by Ajay Kumar.

2018-02-13 Thread arp 
HADOOP-12897. KerberosAuthenticator.authenticate to include URL on IO failures. 
Contributed by Ajay Kumar.


Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/e88265ea
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/e88265ea
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/e88265ea

Branch: refs/heads/branch-3.1
Commit: e88265ea56c7ca3abfbe9c78819c9df9f789af39
Parents: 1fcc3ea
Author: Arpit Agarwal 
Authored: Tue Feb 13 10:14:16 2018 -0800
Committer: Arpit Agarwal 
Committed: Tue Feb 13 10:14:38 2018 -0800

--
 .../client/KerberosAuthenticator.java   | 80 +---
 .../client/TestKerberosAuthenticator.java   | 29 +++
 2 files changed, 82 insertions(+), 27 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/hadoop/blob/e88265ea/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/client/KerberosAuthenticator.java
--
diff --git 
a/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/client/KerberosAuthenticator.java
 
b/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/client/KerberosAuthenticator.java
index 942d13c..64d4330 100644
--- 
a/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/client/KerberosAuthenticator.java
+++ 
b/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/client/KerberosAuthenticator.java
@@ -13,6 +13,8 @@
  */
 package org.apache.hadoop.security.authentication.client;
 
+import com.google.common.annotations.VisibleForTesting;
+import java.lang.reflect.Constructor;
 import org.apache.commons.codec.binary.Base64;
 import org.apache.hadoop.security.authentication.server.HttpConstants;
 import org.apache.hadoop.security.authentication.util.AuthToken;
@@ -177,41 +179,65 @@ public class KerberosAuthenticator implements 
Authenticator {
*/
   @Override
   public void authenticate(URL url, AuthenticatedURL.Token token)
-throws IOException, AuthenticationException {
+  throws IOException, AuthenticationException {
 if (!token.isSet()) {
   this.url = url;
   base64 = new Base64(0);
-  HttpURLConnection conn = token.openConnection(url, connConfigurator);
-  conn.setRequestMethod(AUTH_HTTP_METHOD);
-  conn.connect();
-  
-  boolean needFallback = false;
-  if (conn.getResponseCode() == HttpURLConnection.HTTP_OK) {
-LOG.debug("JDK performed authentication on our behalf.");
-// If the JDK already did the SPNEGO back-and-forth for
-// us, just pull out the token.
-AuthenticatedURL.extractToken(conn, token);
-if (isTokenKerberos(token)) {
-  return;
+  try {
+HttpURLConnection conn = token.openConnection(url, connConfigurator);
+conn.setRequestMethod(AUTH_HTTP_METHOD);
+conn.connect();
+
+boolean needFallback = false;
+if (conn.getResponseCode() == HttpURLConnection.HTTP_OK) {
+  LOG.debug("JDK performed authentication on our behalf.");
+  // If the JDK already did the SPNEGO back-and-forth for
+  // us, just pull out the token.
+  AuthenticatedURL.extractToken(conn, token);
+  if (isTokenKerberos(token)) {
+return;
+  }
+  needFallback = true;
 }
-needFallback = true;
-  }
-  if (!needFallback && isNegotiate(conn)) {
-LOG.debug("Performing our own SPNEGO sequence.");
-doSpnegoSequence(token);
-  } else {
-LOG.debug("Using fallback authenticator sequence.");
-Authenticator auth = getFallBackAuthenticator();
-// Make sure that the fall back authenticator have the same
-// ConnectionConfigurator, since the method might be overridden.
-// Otherwise the fall back authenticator might not have the information
-// to make the connection (e.g., SSL certificates)
-auth.setConnectionConfigurator(connConfigurator);
-auth.authenticate(url, token);
+if (!needFallback && isNegotiate(conn)) {
+  LOG.debug("Performing our own SPNEGO sequence.");
+  doSpnegoSequence(token);
+} else {
+  LOG.debug("Using fallback authenticator sequence.");
+  Authenticator auth = getFallBackAuthenticator();
+  // Make sure that the fall back authenticator have the same
+  // ConnectionConfigurator, since the method might be overridden.
+  // Otherwise the fall back authenticator might not have the
+  // information to make the connection (e.g., SSL certificates)
+

[2/3] hadoop git commit: HADOOP-12897. KerberosAuthenticator.authenticate to include URL on IO failures. Contributed by Ajay Kumar.

2018-02-13 Thread arp 
HADOOP-12897. KerberosAuthenticator.authenticate to include URL on IO failures. 
Contributed by Ajay Kumar.


Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/16b2cad8
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/16b2cad8
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/16b2cad8

Branch: refs/heads/branch-3.0
Commit: 16b2cad8e8f5215325c51eb582f58640a386b06b
Parents: bc2d4ba
Author: Arpit Agarwal 
Authored: Tue Feb 13 10:14:16 2018 -0800
Committer: Arpit Agarwal 
Committed: Tue Feb 13 10:14:29 2018 -0800

--
 .../client/KerberosAuthenticator.java   | 80 +---
 .../client/TestKerberosAuthenticator.java   | 29 +++
 2 files changed, 82 insertions(+), 27 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/hadoop/blob/16b2cad8/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/client/KerberosAuthenticator.java
--
diff --git 
a/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/client/KerberosAuthenticator.java
 
b/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/client/KerberosAuthenticator.java
index 942d13c..64d4330 100644
--- 
a/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/client/KerberosAuthenticator.java
+++ 
b/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/client/KerberosAuthenticator.java
@@ -13,6 +13,8 @@
  */
 package org.apache.hadoop.security.authentication.client;
 
+import com.google.common.annotations.VisibleForTesting;
+import java.lang.reflect.Constructor;
 import org.apache.commons.codec.binary.Base64;
 import org.apache.hadoop.security.authentication.server.HttpConstants;
 import org.apache.hadoop.security.authentication.util.AuthToken;
@@ -177,41 +179,65 @@ public class KerberosAuthenticator implements 
Authenticator {
*/
   @Override
   public void authenticate(URL url, AuthenticatedURL.Token token)
-throws IOException, AuthenticationException {
+  throws IOException, AuthenticationException {
 if (!token.isSet()) {
   this.url = url;
   base64 = new Base64(0);
-  HttpURLConnection conn = token.openConnection(url, connConfigurator);
-  conn.setRequestMethod(AUTH_HTTP_METHOD);
-  conn.connect();
-  
-  boolean needFallback = false;
-  if (conn.getResponseCode() == HttpURLConnection.HTTP_OK) {
-LOG.debug("JDK performed authentication on our behalf.");
-// If the JDK already did the SPNEGO back-and-forth for
-// us, just pull out the token.
-AuthenticatedURL.extractToken(conn, token);
-if (isTokenKerberos(token)) {
-  return;
+  try {
+HttpURLConnection conn = token.openConnection(url, connConfigurator);
+conn.setRequestMethod(AUTH_HTTP_METHOD);
+conn.connect();
+
+boolean needFallback = false;
+if (conn.getResponseCode() == HttpURLConnection.HTTP_OK) {
+  LOG.debug("JDK performed authentication on our behalf.");
+  // If the JDK already did the SPNEGO back-and-forth for
+  // us, just pull out the token.
+  AuthenticatedURL.extractToken(conn, token);
+  if (isTokenKerberos(token)) {
+return;
+  }
+  needFallback = true;
 }
-needFallback = true;
-  }
-  if (!needFallback && isNegotiate(conn)) {
-LOG.debug("Performing our own SPNEGO sequence.");
-doSpnegoSequence(token);
-  } else {
-LOG.debug("Using fallback authenticator sequence.");
-Authenticator auth = getFallBackAuthenticator();
-// Make sure that the fall back authenticator have the same
-// ConnectionConfigurator, since the method might be overridden.
-// Otherwise the fall back authenticator might not have the information
-// to make the connection (e.g., SSL certificates)
-auth.setConnectionConfigurator(connConfigurator);
-auth.authenticate(url, token);
+if (!needFallback && isNegotiate(conn)) {
+  LOG.debug("Performing our own SPNEGO sequence.");
+  doSpnegoSequence(token);
+} else {
+  LOG.debug("Using fallback authenticator sequence.");
+  Authenticator auth = getFallBackAuthenticator();
+  // Make sure that the fall back authenticator have the same
+  // ConnectionConfigurator, since the method might be overridden.
+  // Otherwise the fall back authenticator might not have the
+  // information to make the connection (e.g., SSL certificates)
+

hadoop git commit: YARN-7813: Capacity Scheduler Intra-queue Preemption should be configurable for each queue

2018-02-13 Thread epayne 
Repository: hadoop
Updated Branches:
  refs/heads/branch-3.1 08053c4ea -> 1fcc3eac8


YARN-7813: Capacity Scheduler Intra-queue Preemption should be configurable for 
each queue

(cherry picked from commit c5e6e3de1c31eda052f89eddd7bba288625936b9)


Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/1fcc3eac
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/1fcc3eac
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/1fcc3eac

Branch: refs/heads/branch-3.1
Commit: 1fcc3eac885af23d79b170b5b80f271d7877641a
Parents: 08053c4
Author: Eric Payne 
Authored: Tue Feb 13 10:11:02 2018 -0600
Committer: Eric Payne 
Committed: Tue Feb 13 10:20:35 2018 -0600

--
 .../hadoop/yarn/api/records/QueueInfo.java  | 35 +++
 .../src/main/proto/yarn_protos.proto|  1 +
 .../apache/hadoop/yarn/client/cli/QueueCLI.java |  6 ++
 .../hadoop/yarn/client/ProtocolHATestBase.java  |  3 +-
 .../hadoop/yarn/client/cli/TestYarnCLI.java | 96 ++--
 .../api/records/impl/pb/QueueInfoPBImpl.java| 13 +++
 .../hadoop/yarn/api/TestPBImplRecords.java  |  2 +-
 .../capacity/IntraQueueCandidatesSelector.java  |  4 +-
 .../scheduler/capacity/AbstractCSQueue.java | 72 +--
 .../scheduler/capacity/CSQueue.java | 16 +++-
 .../CapacitySchedulerConfiguration.java | 15 +++
 .../webapp/CapacitySchedulerPage.java   |  5 +-
 .../dao/CapacitySchedulerLeafQueueInfo.java |  6 ++
 .../TestConfigurationMutationACLPolicies.java   |  2 +-
 .../TestSchedulerApplicationAttempt.java|  2 +-
 .../scheduler/capacity/TestLeafQueue.java   |  2 +-
 .../src/site/markdown/CapacityScheduler.md  |  3 +-
 17 files changed, 257 insertions(+), 26 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/hadoop/blob/1fcc3eac/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/QueueInfo.java
--
diff --git 
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/QueueInfo.java
 
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/QueueInfo.java
index 897b442..57ea9bf 100644
--- 
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/QueueInfo.java
+++ 
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/QueueInfo.java
@@ -94,6 +94,26 @@ public abstract class QueueInfo {
 return queueInfo;
   }
 
+  @Private
+  @Unstable
+  public static QueueInfo newInstance(String queueName, float capacity,
+  float maximumCapacity, float currentCapacity,
+  List childQueues, List applications,
+  QueueState queueState, Set accessibleNodeLabels,
+  String defaultNodeLabelExpression, QueueStatistics queueStatistics,
+  boolean preemptionDisabled,
+  Map queueConfigurations,
+  boolean intraQueuePreemptionDisabled) {
+QueueInfo queueInfo = QueueInfo.newInstance(queueName, capacity,
+maximumCapacity, currentCapacity,
+childQueues, applications,
+queueState, accessibleNodeLabels,
+defaultNodeLabelExpression, queueStatistics,
+preemptionDisabled, queueConfigurations);
+queueInfo.setIntraQueuePreemptionDisabled(intraQueuePreemptionDisabled);
+return queueInfo;
+  }
+
   /**
* Get the name of the queue.
* @return name of the queue
@@ -261,4 +281,19 @@ public abstract class QueueInfo {
   @Unstable
   public abstract void setQueueConfigurations(
   Map queueConfigurations);
+
+
+  /**
+   * Get the intra-queue preemption status of the queue.
+   * @return if property is not in proto, return null;
+   *otherwise, return intra-queue preemption status of the queue
+   */
+  @Public
+  @Stable
+  public abstract Boolean getIntraQueuePreemptionDisabled();
+
+  @Private
+  @Unstable
+  public abstract void setIntraQueuePreemptionDisabled(
+  boolean intraQueuePreemptionDisabled);
 }

http://git-wip-us.apache.org/repos/asf/hadoop/blob/1fcc3eac/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/yarn_protos.proto
--
diff --git 
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/yarn_protos.proto
 
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/yarn_protos.proto
index 25c8569..b978761 100644
--- 
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/yarn_protos.proto
+++ 
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/yarn_protos.proto
@@ -569,6 +569,7 @@ message

hadoop git commit: YARN-7813: Capacity Scheduler Intra-queue Preemption should be configurable for each queue

2018-02-13 Thread epayne 
Repository: hadoop
Updated Branches:
  refs/heads/trunk 0c5d7d71a -> c5e6e3de1


YARN-7813: Capacity Scheduler Intra-queue Preemption should be configurable for 
each queue


Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/c5e6e3de
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/c5e6e3de
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/c5e6e3de

Branch: refs/heads/trunk
Commit: c5e6e3de1c31eda052f89eddd7bba288625936b9
Parents: 0c5d7d7
Author: Eric Payne 
Authored: Tue Feb 13 10:11:02 2018 -0600
Committer: Eric Payne 
Committed: Tue Feb 13 10:11:02 2018 -0600

--
 .../hadoop/yarn/api/records/QueueInfo.java  | 35 +++
 .../src/main/proto/yarn_protos.proto|  1 +
 .../apache/hadoop/yarn/client/cli/QueueCLI.java |  6 ++
 .../hadoop/yarn/client/ProtocolHATestBase.java  |  3 +-
 .../hadoop/yarn/client/cli/TestYarnCLI.java | 96 ++--
 .../api/records/impl/pb/QueueInfoPBImpl.java| 13 +++
 .../hadoop/yarn/api/TestPBImplRecords.java  |  2 +-
 .../capacity/IntraQueueCandidatesSelector.java  |  4 +-
 .../scheduler/capacity/AbstractCSQueue.java | 72 +--
 .../scheduler/capacity/CSQueue.java | 16 +++-
 .../CapacitySchedulerConfiguration.java | 15 +++
 .../webapp/CapacitySchedulerPage.java   |  5 +-
 .../dao/CapacitySchedulerLeafQueueInfo.java |  6 ++
 .../TestConfigurationMutationACLPolicies.java   |  2 +-
 .../TestSchedulerApplicationAttempt.java|  2 +-
 .../scheduler/capacity/TestLeafQueue.java   |  2 +-
 .../src/site/markdown/CapacityScheduler.md  |  3 +-
 17 files changed, 257 insertions(+), 26 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/hadoop/blob/c5e6e3de/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/QueueInfo.java
--
diff --git 
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/QueueInfo.java
 
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/QueueInfo.java
index 897b442..57ea9bf 100644
--- 
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/QueueInfo.java
+++ 
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/QueueInfo.java
@@ -94,6 +94,26 @@ public abstract class QueueInfo {
 return queueInfo;
   }
 
+  @Private
+  @Unstable
+  public static QueueInfo newInstance(String queueName, float capacity,
+  float maximumCapacity, float currentCapacity,
+  List childQueues, List applications,
+  QueueState queueState, Set accessibleNodeLabels,
+  String defaultNodeLabelExpression, QueueStatistics queueStatistics,
+  boolean preemptionDisabled,
+  Map queueConfigurations,
+  boolean intraQueuePreemptionDisabled) {
+QueueInfo queueInfo = QueueInfo.newInstance(queueName, capacity,
+maximumCapacity, currentCapacity,
+childQueues, applications,
+queueState, accessibleNodeLabels,
+defaultNodeLabelExpression, queueStatistics,
+preemptionDisabled, queueConfigurations);
+queueInfo.setIntraQueuePreemptionDisabled(intraQueuePreemptionDisabled);
+return queueInfo;
+  }
+
   /**
* Get the name of the queue.
* @return name of the queue
@@ -261,4 +281,19 @@ public abstract class QueueInfo {
   @Unstable
   public abstract void setQueueConfigurations(
   Map queueConfigurations);
+
+
+  /**
+   * Get the intra-queue preemption status of the queue.
+   * @return if property is not in proto, return null;
+   *otherwise, return intra-queue preemption status of the queue
+   */
+  @Public
+  @Stable
+  public abstract Boolean getIntraQueuePreemptionDisabled();
+
+  @Private
+  @Unstable
+  public abstract void setIntraQueuePreemptionDisabled(
+  boolean intraQueuePreemptionDisabled);
 }

http://git-wip-us.apache.org/repos/asf/hadoop/blob/c5e6e3de/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/yarn_protos.proto
--
diff --git 
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/yarn_protos.proto
 
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/yarn_protos.proto
index 25c8569..b978761 100644
--- 
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/yarn_protos.proto
+++ 
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/yarn_protos.proto
@@ -569,6 +569,7 @@ message QueueInfoProto {
   optional QueueStatisticsProto queueStatistics = 10;
   optional bool

[Hadoop Wiki] Update of "Books" by SteveLoughran

2018-02-13 Thread Apache Wiki 
Dear Wiki user,

You have subscribed to a wiki page or wiki category on "Hadoop Wiki" for change 
notification.

The "Books" page has been changed by SteveLoughran:
https://wiki.apache.org/hadoop/Books?action=diff=44=45

Comment:
revert. That's Hbase, not hadoop. And I'm thinking we should cut all videos out 
from here

  
  
  == Hadoop Videos ==
- === Learn by Example : HBase - The Hadoop Database (Video) ===
  
- '''Name:'''  
[[https://www.packtpub.com/application-development/learn-example-hbase-hadoop-database-video|Learn
 by Example : HBase - The Hadoop Database (Video)]]
- 
- '''Author:''' Loonycorn
- 
- '''Publisher:''' Packt
- 
- '''Date of Publishing:''' December  2017
- 
- 25 solved examples to get you up to speed with HBase
  
  === The Ultimate Hands-on Hadoop (Video) ===
  

-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org