hadoop git commit: MAPREDUCE-7165. mapred-site.xml is misformatted in single node setup document. Contributed by Zhaohui Xin.
Repository: hadoop Updated Branches: refs/heads/branch-3.1 0b83c95ff -> 298c76c48 MAPREDUCE-7165. mapred-site.xml is misformatted in single node setup document. Contributed by Zhaohui Xin. (cherry picked from commit c9bfca217f4b15a3a367db51147d0dc2075ca274) Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/298c76c4 Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/298c76c4 Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/298c76c4 Branch: refs/heads/branch-3.1 Commit: 298c76c48530e759ebdda46474ff44ff28ff9a81 Parents: 0b83c95 Author: Akira Ajisaka Authored: Fri Nov 30 13:13:23 2018 +0900 Committer: Akira Ajisaka Committed: Fri Nov 30 13:30:52 2018 +0900 -- .../hadoop-common/src/site/markdown/SingleCluster.md.vm | 3 --- 1 file changed, 3 deletions(-) -- http://git-wip-us.apache.org/repos/asf/hadoop/blob/298c76c4/hadoop-common-project/hadoop-common/src/site/markdown/SingleCluster.md.vm -- diff --git a/hadoop-common-project/hadoop-common/src/site/markdown/SingleCluster.md.vm b/hadoop-common-project/hadoop-common/src/site/markdown/SingleCluster.md.vm index e7a0a08..18fb52d 100644 --- a/hadoop-common-project/hadoop-common/src/site/markdown/SingleCluster.md.vm +++ b/hadoop-common-project/hadoop-common/src/site/markdown/SingleCluster.md.vm @@ -190,9 +190,6 @@ The following instructions assume that 1. ~ 4. steps of [the above instructions] mapreduce.framework.name yarn - - - mapreduce.application.classpath $HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/* - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
hadoop git commit: MAPREDUCE-7165. mapred-site.xml is misformatted in single node setup document. Contributed by Zhaohui Xin.
Repository: hadoop Updated Branches: refs/heads/branch-3.0 689555004 -> c9c4511a0 MAPREDUCE-7165. mapred-site.xml is misformatted in single node setup document. Contributed by Zhaohui Xin. (cherry picked from commit c9bfca217f4b15a3a367db51147d0dc2075ca274) Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/c9c4511a Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/c9c4511a Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/c9c4511a Branch: refs/heads/branch-3.0 Commit: c9c4511a0e8da82423cdae4ceba7ee809845314c Parents: 6895550 Author: Akira Ajisaka Authored: Fri Nov 30 13:13:23 2018 +0900 Committer: Akira Ajisaka Committed: Fri Nov 30 13:31:13 2018 +0900 -- .../hadoop-common/src/site/markdown/SingleCluster.md.vm | 3 --- 1 file changed, 3 deletions(-) -- http://git-wip-us.apache.org/repos/asf/hadoop/blob/c9c4511a/hadoop-common-project/hadoop-common/src/site/markdown/SingleCluster.md.vm -- diff --git a/hadoop-common-project/hadoop-common/src/site/markdown/SingleCluster.md.vm b/hadoop-common-project/hadoop-common/src/site/markdown/SingleCluster.md.vm index e7a0a08..18fb52d 100644 --- a/hadoop-common-project/hadoop-common/src/site/markdown/SingleCluster.md.vm +++ b/hadoop-common-project/hadoop-common/src/site/markdown/SingleCluster.md.vm @@ -190,9 +190,6 @@ The following instructions assume that 1. ~ 4. steps of [the above instructions] mapreduce.framework.name yarn - - - mapreduce.application.classpath $HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/* - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
hadoop git commit: HDFS-13870. WebHDFS: Document ALLOWSNAPSHOT and DISALLOWSNAPSHOT API doc. Contributed by Siyao Meng.
Repository: hadoop Updated Branches: refs/heads/trunk bad12031f -> 0e36e935d HDFS-13870. WebHDFS: Document ALLOWSNAPSHOT and DISALLOWSNAPSHOT API doc. Contributed by Siyao Meng. Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/0e36e935 Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/0e36e935 Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/0e36e935 Branch: refs/heads/trunk Commit: 0e36e935d909862401890d0a5410204504f48b31 Parents: bad1203 Author: Yiqun Lin Authored: Fri Nov 30 11:31:34 2018 +0800 Committer: Yiqun Lin Committed: Fri Nov 30 11:31:34 2018 +0800 -- .../hadoop-hdfs/src/site/markdown/WebHDFS.md| 24 1 file changed, 24 insertions(+) -- http://git-wip-us.apache.org/repos/asf/hadoop/blob/0e36e935/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/WebHDFS.md -- diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/WebHDFS.md b/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/WebHDFS.md index 383eda0..8661659 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/WebHDFS.md +++ b/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/WebHDFS.md @@ -64,6 +64,8 @@ The HTTP REST API supports the complete [FileSystem](../../api/org/apache/hadoop * [`SETTIMES`](#Set_Access_or_Modification_Time) (see [FileSystem](../../api/org/apache/hadoop/fs/FileSystem.html).setTimes) * [`RENEWDELEGATIONTOKEN`](#Renew_Delegation_Token) (see [DelegationTokenAuthenticator](../../api/org/apache/hadoop/security/token/delegation/web/DelegationTokenAuthenticator.html).renewDelegationToken) * [`CANCELDELEGATIONTOKEN`](#Cancel_Delegation_Token) (see [DelegationTokenAuthenticator](../../api/org/apache/hadoop/security/token/delegation/web/DelegationTokenAuthenticator.html).cancelDelegationToken) +* [`ALLOWSNAPSHOT`](#Allow_Snapshot) +* [`DISALLOWSNAPSHOT`](#Disallow_Snapshot) * [`CREATESNAPSHOT`](#Create_Snapshot) (see [FileSystem](../../api/org/apache/hadoop/fs/FileSystem.html).createSnapshot) * [`RENAMESNAPSHOT`](#Rename_Snapshot) (see [FileSystem](../../api/org/apache/hadoop/fs/FileSystem.html).renameSnapshot) * [`SETXATTR`](#Set_XAttr) (see [FileSystem](../../api/org/apache/hadoop/fs/FileSystem.html).setXAttr) @@ -1302,6 +1304,28 @@ See also: [HDFSErasureCoding](./HDFSErasureCoding.html#Administrative_commands). Snapshot Operations --- +### Allow Snapshot + +* Submit a HTTP PUT request. + +curl -i -X PUT "http://:/webhdfs/v1/?op=ALLOWSNAPSHOT" + +The client receives a response with zero content length on success: + +HTTP/1.1 200 OK +Content-Length: 0 + +### Disallow Snapshot + +* Submit a HTTP PUT request. + +curl -i -X PUT "http://:/webhdfs/v1/?op=DISALLOWSNAPSHOT" + +The client receives a response with zero content length on success: + +HTTP/1.1 200 OK +Content-Length: 0 + ### Create Snapshot * Submit a HTTP PUT request. - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
hadoop git commit: HADOOP-15959. Revert "HADOOP-12751. While using kerberos Hadoop incorrectly assumes names with '@' to be non-simple"
Repository: hadoop
Updated Branches:
refs/heads/branch-3.2.0 fa48363e2 -> 542b224d1
HADOOP-15959. Revert "HADOOP-12751. While using kerberos Hadoop incorrectly
assumes names with '@' to be non-simple"
This reverts commit 829a2e4d271f05afb209ddc834cd4a0e85492eda.
(cherry picked from commit d0edd37269bb40290b409d583bcf3b70897c13e0)
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/542b224d
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/542b224d
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/542b224d
Branch: refs/heads/branch-3.2.0
Commit: 542b224d1060b77c6f486870ca678d4f6f371cc7
Parents: fa48363
Author: Steve Loughran
Authored: Thu Nov 29 17:52:11 2018 +
Committer: Sunil G
Committed: Fri Nov 30 05:36:03 2018 +0530
--
.../authentication/util/KerberosName.java | 9 ++--
.../TestKerberosAuthenticationHandler.java | 7 ++-
.../authentication/util/TestKerberosName.java | 17 ++--
.../java/org/apache/hadoop/security/KDiag.java | 46 +---
.../src/site/markdown/SecureMode.md | 6 ---
.../org/apache/hadoop/security/TestKDiag.java | 16 ---
.../security/TestUserGroupInformation.java | 27
7 files changed, 33 insertions(+), 95 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/hadoop/blob/542b224d/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/util/KerberosName.java
--
diff --git
a/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/util/KerberosName.java
b/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/util/KerberosName.java
index 4e7ee3c..287bb13 100644
---
a/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/util/KerberosName.java
+++
b/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/util/KerberosName.java
@@ -324,8 +324,8 @@ public class KerberosName {
}
}
if (result != null && nonSimplePattern.matcher(result).find()) {
-LOG.info("Non-simple name {} after auth_to_local rule {}",
-result, this);
+throw new NoMatchingRule("Non-simple name " + result +
+ " after auth_to_local rule " + this);
}
if (toLowerCase && result != null) {
result = result.toLowerCase(Locale.ENGLISH);
@@ -378,7 +378,7 @@ public class KerberosName {
/**
* Get the translation of the principal name into an operating system
* user name.
- * @return the user name
+ * @return the short name
* @throws IOException throws if something is wrong with the rules
*/
public String getShortName() throws IOException {
@@ -398,8 +398,7 @@ public class KerberosName {
return result;
}
}
-LOG.info("No auth_to_local rules applied to {}", this);
-return toString();
+throw new NoMatchingRule("No rules applied to " + toString());
}
/**
http://git-wip-us.apache.org/repos/asf/hadoop/blob/542b224d/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/server/TestKerberosAuthenticationHandler.java
--
diff --git
a/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/server/TestKerberosAuthenticationHandler.java
b/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/server/TestKerberosAuthenticationHandler.java
index e672391..8b4bc15 100644
---
a/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/server/TestKerberosAuthenticationHandler.java
+++
b/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/server/TestKerberosAuthenticationHandler.java
@@ -108,7 +108,12 @@ public class TestKerberosAuthenticationHandler
kn = new KerberosName("bar@BAR");
Assert.assertEquals("bar", kn.getShortName());
kn = new KerberosName("bar@FOO");
-Assert.assertEquals("bar@FOO", kn.getShortName());
+try {
+ kn.getShortName();
+ Assert.fail();
+}
+catch (Exception ex) {
+}
}
@Test(timeout=6)
http://git-wip-us.apache.org/repos/asf/hadoop/blob/542b224d/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/util/TestKerberosName.java
--
diff --git
a/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/util/TestKerberosName.java
hadoop git commit: YARN-9010. Fix the incorrect trailing slash deletion in constructor method of CGroupsHandlerImpl. (Zhankun Tang via wangda)
Repository: hadoop
Updated Branches:
refs/heads/trunk 0081b02e3 -> bad12031f
YARN-9010. Fix the incorrect trailing slash deletion in constructor method of
CGroupsHandlerImpl. (Zhankun Tang via wangda)
Change-Id: Iaecc66d57781cc10f19ead4647e47fc9556676da
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/bad12031
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/bad12031
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/bad12031
Branch: refs/heads/trunk
Commit: bad12031f603347a701249a1e3ef5d879a5f1c8f
Parents: 0081b02
Author: Wangda Tan
Authored: Thu Nov 29 14:56:07 2018 -0800
Committer: Wangda Tan
Committed: Thu Nov 29 14:56:07 2018 -0800
--
.../linux/resources/CGroupsHandlerImpl.java | 3 +-
.../linux/resources/TestCGroupsHandlerImpl.java | 38
2 files changed, 40 insertions(+), 1 deletion(-)
--
http://git-wip-us.apache.org/repos/asf/hadoop/blob/bad12031/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/CGroupsHandlerImpl.java
--
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/CGroupsHandlerImpl.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/CGroupsHandlerImpl.java
index 050d0a8..1b2c780 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/CGroupsHandlerImpl.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/CGroupsHandlerImpl.java
@@ -87,9 +87,10 @@ class CGroupsHandlerImpl implements CGroupsHandler {
CGroupsHandlerImpl(Configuration conf, PrivilegedOperationExecutor
privilegedOperationExecutor, String mtab)
throws ResourceHandlerException {
+// Remove leading and trialing slash(es)
this.cGroupPrefix = conf.get(YarnConfiguration.
NM_LINUX_CONTAINER_CGROUPS_HIERARCHY, "/hadoop-yarn")
-.replaceAll("^/", "").replaceAll("$/", "");
+.replaceAll("^/+", "").replaceAll("/+$", "");
this.enableCGroupMount = conf.getBoolean(YarnConfiguration.
NM_LINUX_CONTAINER_CGROUPS_MOUNT, false);
this.cGroupMountPath = conf.get(YarnConfiguration.
http://git-wip-us.apache.org/repos/asf/hadoop/blob/bad12031/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/TestCGroupsHandlerImpl.java
--
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/TestCGroupsHandlerImpl.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/TestCGroupsHandlerImpl.java
index ea6fb52..70badaf 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/TestCGroupsHandlerImpl.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/TestCGroupsHandlerImpl.java
@@ -598,4 +598,42 @@ public class TestCGroupsHandlerImpl {
FileUtils.deleteQuietly(cpu);
}
}
+
+ // Remove leading and trailing slashes
+ @Test
+ public void testCgroupsHierarchySetting() throws ResourceHandlerException {
+YarnConfiguration conf = new YarnConfiguration();
+conf.set(YarnConfiguration.NM_LINUX_CONTAINER_CGROUPS_MOUNT_PATH, tmpPath);
+conf.set(YarnConfiguration.NM_LINUX_CONTAINER_CGROUPS_HIERARCHY,
+"/hadoop-yarn");
+CGroupsHandlerImpl cGroupsHandler = new CGroupsHandlerImpl(conf, null);
+String expectedRelativePath = "hadoop-yarn/c1";
+Assert.assertEquals(expectedRelativePath,
+cGroupsHandler.getRelativePathForCGroup("c1"));
+
+conf.set(YarnConfiguration.NM_LINUX_CONTAINER_CGROUPS_HIERARCHY,
+"hadoop-yarn");
+cGroupsHandler = new CGroupsHandlerImpl(conf, null);
+
hadoop git commit: HDDS-884. Fix merge issue that causes NPE OzoneManager#httpServer. Contributed by Xiaoyu Yao.
Repository: hadoop
Updated Branches:
refs/heads/HDDS-4 187bbbe68 -> e0a65bb08
HDDS-884. Fix merge issue that causes NPE OzoneManager#httpServer. Contributed
by Xiaoyu Yao.
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/e0a65bb0
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/e0a65bb0
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/e0a65bb0
Branch: refs/heads/HDDS-4
Commit: e0a65bb08d6af67fdecddbd526e22384dcc02cfb
Parents: 187bbbe
Author: Xiaoyu Yao
Authored: Thu Nov 29 13:39:30 2018 -0800
Committer: Xiaoyu Yao
Committed: Thu Nov 29 13:39:30 2018 -0800
--
.../src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java | 1 +
1 file changed, 1 insertion(+)
--
http://git-wip-us.apache.org/repos/asf/hadoop/blob/e0a65bb0/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java
--
diff --git
a/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java
b/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java
index 1e49779..cb99b9e 100644
---
a/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java
+++
b/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java
@@ -702,6 +702,7 @@ public final class OzoneManager extends
ServiceRuntimeInfoImpl
keyManager.start(configuration);
omRpcServer.start();
try {
+ httpServer = new OzoneManagerHttpServer(configuration, this);
httpServer.start();
} catch (Exception ex) {
// Allow OM to start as Http Server failure is not fatal.
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
hadoop git commit: HDFS-14112. Avoid recursive call to external authorizer for getContentSummary.
Repository: hadoop
Updated Branches:
refs/heads/branch-3.2 e2fa9e8cd -> 9d508f719
HDFS-14112. Avoid recursive call to external authorizer for getContentSummary.
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/9d508f71
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/9d508f71
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/9d508f71
Branch: refs/heads/branch-3.2
Commit: 9d508f719be9c1c40fbc133d19256613f4b8fd75
Parents: e2fa9e8
Author: Tsz Wo Nicholas Sze
Authored: Thu Nov 29 13:55:21 2018 -0800
Committer: Tsz Wo Nicholas Sze
Committed: Thu Nov 29 13:56:43 2018 -0800
--
.../main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java | 4
.../hdfs/server/namenode/FSDirStatAndListingOp.java | 5 +
.../apache/hadoop/hdfs/server/namenode/FSDirectory.java | 7 +++
.../hadoop-hdfs/src/main/resources/hdfs-default.xml | 10 ++
4 files changed, 26 insertions(+)
--
http://git-wip-us.apache.org/repos/asf/hadoop/blob/9d508f71/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
--
diff --git
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
index d8024dc..c9f10e1 100644
---
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
+++
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
@@ -280,6 +280,10 @@ public class DFSConfigKeys extends CommonConfigurationKeys
{
HdfsClientConfigKeys.DFS_WEBHDFS_USER_PATTERN_DEFAULT;
public static final String DFS_PERMISSIONS_ENABLED_KEY =
HdfsClientConfigKeys.DeprecatedKeys.DFS_PERMISSIONS_ENABLED_KEY;
+ public static final String DFS_PERMISSIONS_CONTENT_SUMMARY_SUBACCESS_KEY
+ = "dfs.permissions.ContentSummary.subAccess";
+ public static final boolean DFS_PERMISSIONS_CONTENT_SUMMARY_SUBACCESS_DEFAULT
+ = false;
public static final boolean DFS_PERMISSIONS_ENABLED_DEFAULT = true;
public static final String DFS_PERMISSIONS_SUPERUSERGROUP_KEY =
HdfsClientConfigKeys.DeprecatedKeys.DFS_PERMISSIONS_SUPERUSERGROUP_KEY;
http://git-wip-us.apache.org/repos/asf/hadoop/blob/9d508f71/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirStatAndListingOp.java
--
diff --git
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirStatAndListingOp.java
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirStatAndListingOp.java
index 01de236..052e522 100644
---
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirStatAndListingOp.java
+++
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirStatAndListingOp.java
@@ -128,6 +128,11 @@ class FSDirStatAndListingOp {
static ContentSummary getContentSummary(
FSDirectory fsd, FSPermissionChecker pc, String src) throws IOException {
final INodesInPath iip = fsd.resolvePath(pc, src, DirOp.READ_LINK);
+if (fsd.isPermissionEnabled() &&
fsd.isPermissionContentSummarySubAccess()) {
+ fsd.checkPermission(pc, iip, false, null, null, null,
+ FsAction.READ_EXECUTE);
+ pc = null;
+}
// getContentSummaryInt() call will check access (if enabled) when
// traversing all sub directories.
return getContentSummaryInt(fsd, pc, iip);
http://git-wip-us.apache.org/repos/asf/hadoop/blob/9d508f71/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java
--
diff --git
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java
index 2a976d2..c49e00f 100644
---
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java
+++
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java
@@ -175,6 +175,7 @@ public class FSDirectory implements Closeable {
private final ReentrantReadWriteLock dirLock;
private final boolean isPermissionEnabled;
+ private final boolean isPermissionContentSummarySubAccess;
/**
* Support for ACLs is controlled by a configuration flag. If the
* configuration flag is false, then the NameNode will reject all
@@ -274,6 +275,9 @@ public class FSDirectory implements
hadoop git commit: HDFS-13547. Add ingress port based sasl resolver. Contributed by Chen Liang.
Repository: hadoop
Updated Branches:
refs/heads/branch-3 28b780277 -> bf90a27b5
HDFS-13547. Add ingress port based sasl resolver. Contributed by Chen Liang.
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/bf90a27b
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/bf90a27b
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/bf90a27b
Branch: refs/heads/branch-3
Commit: bf90a27b51b1f1ac102fa861eb28025d21aad19b
Parents: 28b7802
Author: Chen Liang
Authored: Thu Nov 29 13:31:58 2018 -0800
Committer: Chen Liang
Committed: Thu Nov 29 13:31:58 2018 -0800
--
.../security/IngressPortBasedResolver.java | 100 +++
.../hadoop/security/SaslPropertiesResolver.java | 47 -
.../hadoop/security/WhitelistBasedResolver.java | 20 +---
.../security/TestIngressPortBasedResolver.java | 59 +++
4 files changed, 207 insertions(+), 19 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/hadoop/blob/bf90a27b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/IngressPortBasedResolver.java
--
diff --git
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/IngressPortBasedResolver.java
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/IngressPortBasedResolver.java
new file mode 100644
index 000..a30e4a8
--- /dev/null
+++
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/IngressPortBasedResolver.java
@@ -0,0 +1,100 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.security;
+
+import com.google.common.annotations.VisibleForTesting;
+import java.net.InetAddress;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.Map;
+import org.apache.hadoop.conf.Configuration;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * An implementation of SaslPropertiesResolver. Used on server side,
+ * returns SASL properties based on the port the client is connecting
+ * to. This should be used along with server side enabling multiple ports
+ * TODO: when NN multiple listener is enabled, automatically use this
+ * resolver without having to set in config.
+ *
+ * For configuration, for example if server runs on two ports 9000 and 9001,
+ * and we want to specify 9000 to use auth-conf and 9001 to use auth.
+ *
+ * We need to set the following configuration properties:
+ * ingress.port.sasl.configured.ports=9000,9001
+ * ingress.port.sasl.prop.9000=privacy
+ * ingress.port.sasl.prop.9001=authentication
+ *
+ * One note is that, if there is misconfiguration that a port, say, 9002 is
+ * given in ingress.port.sasl.configured.ports, but it's sasl prop is not
+ * set, a default of QOP of privacy (auth-conf) will be used. In addition,
+ * if a port is not given even in ingress.port.sasl.configured.ports, but
+ * is being checked in getServerProperties(), the default SASL prop will
+ * be returned. Both of these two cases are considered misconfiguration.
+ */
+public class IngressPortBasedResolver extends SaslPropertiesResolver {
+
+ public static final Logger LOG =
+ LoggerFactory.getLogger(IngressPortBasedResolver.class.getName());
+
+ static final String INGRESS_PORT_SASL_PROP_PREFIX = "ingress.port.sasl.prop";
+
+ static final String INGRESS_PORT_SASL_CONFIGURED_PORTS =
+ "ingress.port.sasl.configured.ports";
+
+ // no need to concurrent map, because after setConf() it never change,
+ // only for read.
+ private HashMap> portPropMapping;
+
+ @Override
+ public void setConf(Configuration conf) {
+super.setConf(conf);
+portPropMapping = new HashMap<>();
+Collection portStrings =
+conf.getTrimmedStringCollection(INGRESS_PORT_SASL_CONFIGURED_PORTS);
+for (String portString : portStrings) {
+ int port = Integer.parseInt(portString);
+ String configKey = INGRESS_PORT_SASL_PROP_PREFIX + "." + portString;
+ Map props =
hadoop git commit: HDFS-13547. Add ingress port based sasl resolver. Contributed by Chen Liang.
Repository: hadoop
Updated Branches:
refs/heads/branch-3.1.1 2b9a8c1d3 -> 7caf768a8
HDFS-13547. Add ingress port based sasl resolver. Contributed by Chen Liang.
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/7caf768a
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/7caf768a
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/7caf768a
Branch: refs/heads/branch-3.1.1
Commit: 7caf768a8c9a639b6139b2cae8656c89e3d8c58d
Parents: 2b9a8c1d
Author: Chen Liang
Authored: Thu Nov 29 13:17:01 2018 -0800
Committer: Chen Liang
Committed: Thu Nov 29 13:17:01 2018 -0800
--
.../security/IngressPortBasedResolver.java | 100 +++
.../hadoop/security/SaslPropertiesResolver.java | 47 -
.../hadoop/security/WhitelistBasedResolver.java | 20 +---
.../security/TestIngressPortBasedResolver.java | 59 +++
4 files changed, 207 insertions(+), 19 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/hadoop/blob/7caf768a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/IngressPortBasedResolver.java
--
diff --git
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/IngressPortBasedResolver.java
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/IngressPortBasedResolver.java
new file mode 100644
index 000..a30e4a8
--- /dev/null
+++
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/IngressPortBasedResolver.java
@@ -0,0 +1,100 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.security;
+
+import com.google.common.annotations.VisibleForTesting;
+import java.net.InetAddress;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.Map;
+import org.apache.hadoop.conf.Configuration;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * An implementation of SaslPropertiesResolver. Used on server side,
+ * returns SASL properties based on the port the client is connecting
+ * to. This should be used along with server side enabling multiple ports
+ * TODO: when NN multiple listener is enabled, automatically use this
+ * resolver without having to set in config.
+ *
+ * For configuration, for example if server runs on two ports 9000 and 9001,
+ * and we want to specify 9000 to use auth-conf and 9001 to use auth.
+ *
+ * We need to set the following configuration properties:
+ * ingress.port.sasl.configured.ports=9000,9001
+ * ingress.port.sasl.prop.9000=privacy
+ * ingress.port.sasl.prop.9001=authentication
+ *
+ * One note is that, if there is misconfiguration that a port, say, 9002 is
+ * given in ingress.port.sasl.configured.ports, but it's sasl prop is not
+ * set, a default of QOP of privacy (auth-conf) will be used. In addition,
+ * if a port is not given even in ingress.port.sasl.configured.ports, but
+ * is being checked in getServerProperties(), the default SASL prop will
+ * be returned. Both of these two cases are considered misconfiguration.
+ */
+public class IngressPortBasedResolver extends SaslPropertiesResolver {
+
+ public static final Logger LOG =
+ LoggerFactory.getLogger(IngressPortBasedResolver.class.getName());
+
+ static final String INGRESS_PORT_SASL_PROP_PREFIX = "ingress.port.sasl.prop";
+
+ static final String INGRESS_PORT_SASL_CONFIGURED_PORTS =
+ "ingress.port.sasl.configured.ports";
+
+ // no need to concurrent map, because after setConf() it never change,
+ // only for read.
+ private HashMap> portPropMapping;
+
+ @Override
+ public void setConf(Configuration conf) {
+super.setConf(conf);
+portPropMapping = new HashMap<>();
+Collection portStrings =
+conf.getTrimmedStringCollection(INGRESS_PORT_SASL_CONFIGURED_PORTS);
+for (String portString : portStrings) {
+ int port = Integer.parseInt(portString);
+ String configKey = INGRESS_PORT_SASL_PROP_PREFIX + "." + portString;
+ Map props =
[16/50] [abbrv] hadoop git commit: HDDS-6. Enable SCM kerberos auth. Contributed by Ajay Kumar.
HDDS-6. Enable SCM kerberos auth. Contributed by Ajay Kumar. Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/fcd705a5 Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/fcd705a5 Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/fcd705a5 Branch: refs/heads/HDDS-4 Commit: fcd705a59420dcb5d6a05e2d49536a77ae4f1d5b Parents: 2119be4 Author: Xiaoyu Yao Authored: Wed May 9 15:56:03 2018 -0700 Committer: Xiaoyu Yao Committed: Thu Nov 29 11:56:49 2018 -0800 -- hadoop-hdds/common/src/main/resources/ozone-default.xml | 4 1 file changed, 4 insertions(+) -- http://git-wip-us.apache.org/repos/asf/hadoop/blob/fcd705a5/hadoop-hdds/common/src/main/resources/ozone-default.xml -- diff --git a/hadoop-hdds/common/src/main/resources/ozone-default.xml b/hadoop-hdds/common/src/main/resources/ozone-default.xml index d3e352b..4d79e8c 100644 --- a/hadoop-hdds/common/src/main/resources/ozone-default.xml +++ b/hadoop-hdds/common/src/main/resources/ozone-default.xml @@ -198,6 +198,7 @@ +<<< HEAD dfs.ratis.client.request.timeout.duration 3s OZONE, RATIS, MANAGEMENT @@ -255,6 +256,9 @@ hdds.container.report.interval +=== +ozone.container.report.interval +>>> HDDS-6. Enable SCM kerberos auth. Contributed by Ajay Kumar. 6ms OZONE, CONTAINER, MANAGEMENT Time interval of the datanode to send container report. Each - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
[13/50] [abbrv] hadoop git commit: HADOOP-15959. Revert "HADOOP-12751. While using kerberos Hadoop incorrectly assumes names with '@' to be non-simple"
HADOOP-15959. Revert "HADOOP-12751. While using kerberos Hadoop incorrectly
assumes names with '@' to be non-simple"
This reverts commit 829a2e4d271f05afb209ddc834cd4a0e85492eda.
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/d0edd372
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/d0edd372
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/d0edd372
Branch: refs/heads/HDDS-4
Commit: d0edd37269bb40290b409d583bcf3b70897c13e0
Parents: 5e102f9
Author: Steve Loughran
Authored: Thu Nov 29 17:52:11 2018 +
Committer: Steve Loughran
Committed: Thu Nov 29 17:52:11 2018 +
--
.../authentication/util/KerberosName.java | 9 ++--
.../TestKerberosAuthenticationHandler.java | 7 ++-
.../authentication/util/TestKerberosName.java | 17 ++--
.../java/org/apache/hadoop/security/KDiag.java | 46 +---
.../src/site/markdown/SecureMode.md | 6 ---
.../org/apache/hadoop/security/TestKDiag.java | 16 ---
.../security/TestUserGroupInformation.java | 27
7 files changed, 33 insertions(+), 95 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/hadoop/blob/d0edd372/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/util/KerberosName.java
--
diff --git
a/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/util/KerberosName.java
b/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/util/KerberosName.java
index 4e7ee3c..287bb13 100644
---
a/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/util/KerberosName.java
+++
b/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/util/KerberosName.java
@@ -324,8 +324,8 @@ public class KerberosName {
}
}
if (result != null && nonSimplePattern.matcher(result).find()) {
-LOG.info("Non-simple name {} after auth_to_local rule {}",
-result, this);
+throw new NoMatchingRule("Non-simple name " + result +
+ " after auth_to_local rule " + this);
}
if (toLowerCase && result != null) {
result = result.toLowerCase(Locale.ENGLISH);
@@ -378,7 +378,7 @@ public class KerberosName {
/**
* Get the translation of the principal name into an operating system
* user name.
- * @return the user name
+ * @return the short name
* @throws IOException throws if something is wrong with the rules
*/
public String getShortName() throws IOException {
@@ -398,8 +398,7 @@ public class KerberosName {
return result;
}
}
-LOG.info("No auth_to_local rules applied to {}", this);
-return toString();
+throw new NoMatchingRule("No rules applied to " + toString());
}
/**
http://git-wip-us.apache.org/repos/asf/hadoop/blob/d0edd372/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/server/TestKerberosAuthenticationHandler.java
--
diff --git
a/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/server/TestKerberosAuthenticationHandler.java
b/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/server/TestKerberosAuthenticationHandler.java
index e672391..8b4bc15 100644
---
a/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/server/TestKerberosAuthenticationHandler.java
+++
b/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/server/TestKerberosAuthenticationHandler.java
@@ -108,7 +108,12 @@ public class TestKerberosAuthenticationHandler
kn = new KerberosName("bar@BAR");
Assert.assertEquals("bar", kn.getShortName());
kn = new KerberosName("bar@FOO");
-Assert.assertEquals("bar@FOO", kn.getShortName());
+try {
+ kn.getShortName();
+ Assert.fail();
+}
+catch (Exception ex) {
+}
}
@Test(timeout=6)
http://git-wip-us.apache.org/repos/asf/hadoop/blob/d0edd372/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/util/TestKerberosName.java
--
diff --git
a/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/util/TestKerberosName.java
b/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/util/TestKerberosName.java
index c584fce..2db0df4 100644
---
[36/50] [abbrv] hadoop git commit: HDDS-8. Add OzoneManager Delegation Token support. Contributed by Ajay Kumar.
HDDS-8. Add OzoneManager Delegation Token support. Contributed by Ajay Kumar.
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/59767be1
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/59767be1
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/59767be1
Branch: refs/heads/HDDS-4
Commit: 59767be13a37557b6d51cd5356c58ebdfad63896
Parents: 9c0baf0
Author: Ajay Kumar
Authored: Thu Nov 15 12:18:19 2018 -0800
Committer: Xiaoyu Yao
Committed: Thu Nov 29 11:57:47 2018 -0800
--
.../apache/hadoop/ozone/OzoneConfigKeys.java| 4 +
.../org/apache/hadoop/ozone/OzoneConsts.java| 1 +
.../common/src/main/resources/ozone-default.xml | 33 +
.../ozone/client/protocol/ClientProtocol.java | 32 +
.../hadoop/ozone/client/rest/RestClient.java| 41 ++
.../hadoop/ozone/client/rpc/RpcClient.java | 41 ++
.../ozone/client/TestHddsClientUtils.java | 2 +-
.../java/org/apache/hadoop/ozone/OmUtils.java | 17 +-
.../apache/hadoop/ozone/om/OMConfigKeys.java| 13 +
.../ozone/om/protocol/OzoneManagerProtocol.java | 2 +-
.../protocol/OzoneManagerSecurityProtocol.java | 67 +++
...neManagerProtocolClientSideTranslatorPB.java | 77 +++
.../om/protocolPB/OzoneManagerProtocolPB.java | 3 +
.../hadoop/ozone/protocolPB/OMPBHelper.java | 38 ++
.../security/OzoneDelegationTokenSelector.java | 3 +-
.../ozone/security/OzoneSecretManager.java | 598 +++
.../hadoop/ozone/security/OzoneSecretStore.java | 250
.../ozone/security/OzoneSecurityException.java | 104
.../src/main/proto/OzoneManagerProtocol.proto | 19 +
.../ozone/security/TestOzoneSecretManager.java | 216 +++
.../hadoop/ozone/TestSecureOzoneCluster.java| 309 --
.../apache/hadoop/ozone/om/OzoneManager.java| 300 --
...neManagerProtocolServerSideTranslatorPB.java | 59 ++
23 files changed, 2150 insertions(+), 79 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/hadoop/blob/59767be1/hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/OzoneConfigKeys.java
--
diff --git
a/hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/OzoneConfigKeys.java
b/hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/OzoneConfigKeys.java
index ac97422..742fe3a 100644
---
a/hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/OzoneConfigKeys.java
+++
b/hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/OzoneConfigKeys.java
@@ -332,6 +332,10 @@ public final class OzoneConfigKeys {
public static final String OZONE_CONTAINER_COPY_WORKDIR =
"hdds.datanode.replication.work.dir";
+ public static final String OZONE_MAX_KEY_LEN =
+ "ozone.max.key.len";
+ public static final int OZONE_MAX_KEY_LEN_DEFAULT = 1024 * 1024;
+
/**
* Config properties to set client side checksum properties.
*/
http://git-wip-us.apache.org/repos/asf/hadoop/blob/59767be1/hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/OzoneConsts.java
--
diff --git
a/hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/OzoneConsts.java
b/hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/OzoneConsts.java
index 096baee..e7c354c 100644
--- a/hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/OzoneConsts.java
+++ b/hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/OzoneConsts.java
@@ -99,6 +99,7 @@ public final class OzoneConsts {
public static final String DN_CONTAINER_DB = "-dn-"+ CONTAINER_DB_SUFFIX;
public static final String DELETED_BLOCK_DB = "deletedBlock.db";
public static final String OM_DB_NAME = "om.db";
+ public static final String OZONE_MANAGER_TOKEN_DB_NAME = "om-token.db";
public static final String STORAGE_DIR_CHUNKS = "chunks";
http://git-wip-us.apache.org/repos/asf/hadoop/blob/59767be1/hadoop-hdds/common/src/main/resources/ozone-default.xml
--
diff --git a/hadoop-hdds/common/src/main/resources/ozone-default.xml
b/hadoop-hdds/common/src/main/resources/ozone-default.xml
index 5fb594b..766f31d 100644
--- a/hadoop-hdds/common/src/main/resources/ozone-default.xml
+++ b/hadoop-hdds/common/src/main/resources/ozone-default.xml
@@ -984,6 +984,15 @@
every principal specified in the keytab file.
+
+ozone.max.key.len
+1048576
+OZONE, SECURITY
+
+ Maximum length of private key in Ozone. Used in Ozone delegation and
+ block tokens.
+
+
@@ -1492,4 +1501,28 @@
Name of file which stores public key generated for SCM CA.
+
+ozone.manager.delegation.remover.scan.interval
+360
+
+ Time interval after
[48/50] [abbrv] hadoop git commit: HDDS-696. Bootstrap genesis SCM(CA) with self-signed certificate. Contributed by Anu Engineer.
http://git-wip-us.apache.org/repos/asf/hadoop/blob/87f51d23/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/security/x509/keys/HDDSKeyGenerator.java
--
diff --git
a/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/security/x509/keys/HDDSKeyGenerator.java
b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/security/x509/keys/HDDSKeyGenerator.java
index 459dce7..ded50f9 100644
---
a/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/security/x509/keys/HDDSKeyGenerator.java
+++
b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/security/x509/keys/HDDSKeyGenerator.java
@@ -28,7 +28,9 @@ import java.security.KeyPairGenerator;
import java.security.NoSuchAlgorithmException;
import java.security.NoSuchProviderException;
-/** A class to generate Key Pair for use with Certificates. */
+/**
+ * A class to generate Key Pair for use with Certificates.
+ */
public class HDDSKeyGenerator {
private static final Logger LOG =
LoggerFactory.getLogger(HDDSKeyGenerator.class);
@@ -44,7 +46,17 @@ public class HDDSKeyGenerator {
}
/**
+ * Constructor that takes a SecurityConfig as the Argument.
+ *
+ * @param config - SecurityConfig
+ */
+ public HDDSKeyGenerator(SecurityConfig config) {
+this.securityConfig = config;
+ }
+
+ /**
* Returns the Security config used for this object.
+ *
* @return SecurityConfig
*/
public SecurityConfig getSecurityConfig() {
@@ -55,10 +67,10 @@ public class HDDSKeyGenerator {
* Use Config to generate key.
*
* @return KeyPair
- * @throws NoSuchProviderException - On Error, due to missing Java
- * dependencies.
+ * @throws NoSuchProviderException - On Error, due to missing Java
+ * dependencies.
* @throws NoSuchAlgorithmException - On Error, due to missing Java
- * dependencies.
+ * dependencies.
*/
public KeyPair generateKey() throws NoSuchProviderException,
NoSuchAlgorithmException {
@@ -71,10 +83,10 @@ public class HDDSKeyGenerator {
*
* @param size - int, valid key sizes.
* @return KeyPair
- * @throws NoSuchProviderException - On Error, due to missing Java
- * dependencies.
+ * @throws NoSuchProviderException - On Error, due to missing Java
+ * dependencies.
* @throws NoSuchAlgorithmException - On Error, due to missing Java
- * dependencies.
+ * dependencies.
*/
public KeyPair generateKey(int size) throws
NoSuchProviderException, NoSuchAlgorithmException {
@@ -89,10 +101,10 @@ public class HDDSKeyGenerator {
* @param algorithm - Algorithm to use
* @param provider - Security provider.
* @return KeyPair.
- * @throws NoSuchProviderException - On Error, due to missing Java
- * dependencies.
+ * @throws NoSuchProviderException - On Error, due to missing Java
+ * dependencies.
* @throws NoSuchAlgorithmException - On Error, due to missing Java
- * dependencies.
+ * dependencies.
*/
public KeyPair generateKey(int size, String algorithm, String provider)
throws NoSuchProviderException, NoSuchAlgorithmException {
http://git-wip-us.apache.org/repos/asf/hadoop/blob/87f51d23/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/security/x509/keys/HDDSKeyPEMWriter.java
--
diff --git
a/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/security/x509/keys/HDDSKeyPEMWriter.java
b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/security/x509/keys/HDDSKeyPEMWriter.java
deleted file mode 100644
index 95be1c4..000
---
a/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/security/x509/keys/HDDSKeyPEMWriter.java
+++ /dev/null
@@ -1,255 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- *
- */
-package org.apache.hadoop.hdds.security.x509.keys;
-
-import com.google.common.annotations.VisibleForTesting;
-import com.google.common.base.Preconditions;
-import
[08/50] [abbrv] hadoop git commit: YARN-8948. PlacementRule interface should be for all YarnSchedulers. Contributed by Bibin A Chundatt.
YARN-8948. PlacementRule interface should be for all YarnSchedulers.
Contributed by Bibin A Chundatt.
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/a68d766e
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/a68d766e
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/a68d766e
Branch: refs/heads/HDDS-4
Commit: a68d766e876631d7ee2e1a6504d4120ba628d178
Parents: c1d24f8
Author: bibinchundatt
Authored: Thu Nov 29 21:43:34 2018 +0530
Committer: bibinchundatt
Committed: Thu Nov 29 21:43:34 2018 +0530
--
.../placement/AppNameMappingPlacementRule.java | 12 ++--
.../server/resourcemanager/placement/PlacementRule.java | 4 ++--
.../placement/UserGroupMappingPlacementRule.java| 11 ++-
3 files changed, 22 insertions(+), 5 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/hadoop/blob/a68d766e/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/AppNameMappingPlacementRule.java
--
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/AppNameMappingPlacementRule.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/AppNameMappingPlacementRule.java
index 2debade..7a46962 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/AppNameMappingPlacementRule.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/AppNameMappingPlacementRule.java
@@ -20,11 +20,12 @@ package
org.apache.hadoop.yarn.server.resourcemanager.placement;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
-import org.apache.hadoop.yarn.api.records.ApplicationId;
import org.apache.hadoop.yarn.api.records.ApplicationSubmissionContext;
import org.apache.hadoop.yarn.conf.YarnConfiguration;
import org.apache.hadoop.yarn.exceptions.YarnException;
+import
org.apache.hadoop.yarn.server.resourcemanager.scheduler.ResourceScheduler;
import
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CSQueue;
+import
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler;
import
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerConfiguration;
import
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerContext;
import
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager;
@@ -61,8 +62,15 @@ public class AppNameMappingPlacementRule extends
PlacementRule {
}
@Override
- public boolean initialize(CapacitySchedulerContext schedulerContext)
+ public boolean initialize(ResourceScheduler scheduler)
throws IOException {
+if (!(scheduler instanceof CapacityScheduler)) {
+ throw new IOException(
+ "AppNameMappingPlacementRule can be configured only for "
+ + "CapacityScheduler");
+}
+CapacitySchedulerContext schedulerContext =
+(CapacitySchedulerContext) scheduler;
CapacitySchedulerConfiguration conf = schedulerContext.getConfiguration();
boolean overrideWithQueueMappings = conf.getOverrideWithQueueMappings();
LOG.info(
http://git-wip-us.apache.org/repos/asf/hadoop/blob/a68d766e/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/PlacementRule.java
--
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/PlacementRule.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/PlacementRule.java
index 21ab32a..0f3d43c 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/PlacementRule.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/PlacementRule.java
@@ -22,7 +22,7 @@ import java.io.IOException;
import
[02/50] [abbrv] hadoop git commit: YARN-8974. Improve the assertion message in TestGPUResourceHandler. (Zhankun Tang via wangda)
YARN-8974. Improve the assertion message in TestGPUResourceHandler. (Zhankun
Tang via wangda)
Change-Id: I4eb58e9d251d5f54e7feffc4fbb813b4f5ae4b1b
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/8ebeda98
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/8ebeda98
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/8ebeda98
Branch: refs/heads/HDDS-4
Commit: 8ebeda98a9d3eac45598a33bae3d62e3ebb92cad
Parents: 9ed8756
Author: Wangda Tan
Authored: Wed Nov 28 14:36:30 2018 -0800
Committer: Wangda Tan
Committed: Wed Nov 28 14:36:30 2018 -0800
--
.../linux/resources/gpu/TestGpuResourceHandler.java| 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
--
http://git-wip-us.apache.org/repos/asf/hadoop/blob/8ebeda98/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/gpu/TestGpuResourceHandler.java
--
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/gpu/TestGpuResourceHandler.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/gpu/TestGpuResourceHandler.java
index 10e5cd1..18785e1 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/gpu/TestGpuResourceHandler.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/gpu/TestGpuResourceHandler.java
@@ -465,7 +465,7 @@ public class TestGpuResourceHandler {
caughtException = true;
}
Assert.assertTrue(
-"Should fail since requested device Id is not in allowed list",
+"Should fail since requested device Id is already assigned",
caughtException);
// Make sure internal state not changed.
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
[01/50] [abbrv] hadoop git commit: YARN-9061. Improve the GPU/FPGA module log message of container-executor. (Zhankun Tang via wangda) [Forced Update!]
Repository: hadoop
Updated Branches:
refs/heads/HDDS-4 278d4b9b7 -> 187bbbe68 (forced update)
YARN-9061. Improve the GPU/FPGA module log message of container-executor.
(Zhankun Tang via wangda)
Change-Id: Iece9b47438357077a53984a820d4d6423f480518
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/9ed87567
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/9ed87567
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/9ed87567
Branch: refs/heads/HDDS-4
Commit: 9ed87567ad0f1c26a263ce6d8fba56d066260c5c
Parents: 579ef4b
Author: Wangda Tan
Authored: Wed Nov 28 14:31:31 2018 -0800
Committer: Wangda Tan
Committed: Wed Nov 28 14:31:31 2018 -0800
--
.../native/container-executor/impl/modules/fpga/fpga-module.c | 5 +++--
.../native/container-executor/impl/modules/gpu/gpu-module.c | 5 +++--
2 files changed, 6 insertions(+), 4 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/hadoop/blob/9ed87567/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/modules/fpga/fpga-module.c
--
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/modules/fpga/fpga-module.c
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/modules/fpga/fpga-module.c
index c1a2f83..e947d7c 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/modules/fpga/fpga-module.c
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/modules/fpga/fpga-module.c
@@ -141,7 +141,7 @@ void reload_fpga_configuration() {
/*
* Format of FPGA request commandline:
*
- * c-e fpga --excluded_fpgas 0,1,3 --container_id container_x_y
+ * c-e --module-fpga --excluded_fpgas 0,1,3 --container_id container_x_y
*/
int handle_fpga_request(update_cgroups_parameters_function func,
const char* module_name, int module_argc, char** module_argv) {
@@ -213,7 +213,8 @@ int handle_fpga_request(update_cgroups_parameters_function
func,
if (!minor_devices) {
// Minor devices is null, skip following call.
- fprintf(ERRORFILE, "is not specified, skip cgroups call.\n");
+ fprintf(ERRORFILE,
+ "--excluded-fpgas is not specified, skip cgroups call.\n");
goto cleanup;
}
http://git-wip-us.apache.org/repos/asf/hadoop/blob/9ed87567/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/modules/gpu/gpu-module.c
--
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/modules/gpu/gpu-module.c
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/modules/gpu/gpu-module.c
index 1a1b164..7522338 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/modules/gpu/gpu-module.c
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/modules/gpu/gpu-module.c
@@ -141,7 +141,7 @@ void reload_gpu_configuration() {
/*
* Format of GPU request commandline:
*
- * c-e gpu --excluded_gpus 0,1,3 --container_id container_x_y
+ * c-e --module-gpu --excluded_gpus 0,1,3 --container_id container_x_y
*/
int handle_gpu_request(update_cgroups_parameters_func func,
const char* module_name, int module_argc, char** module_argv) {
@@ -213,7 +213,8 @@ int handle_gpu_request(update_cgroups_parameters_func func,
if (!minor_devices) {
// Minor devices is null, skip following call.
- fprintf(ERRORFILE, "is not specified, skip cgroups call.\n");
+ fprintf(ERRORFILE,
+ "--excluded_gpus is not specified, skip cgroups call.\n");
goto cleanup;
}
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
[40/50] [abbrv] hadoop git commit: HDDS-760. Add asf license to TestCertificateSignRequest. Contributed by Ajay Kumar.
HDDS-760. Add asf license to TestCertificateSignRequest. Contributed by Ajay Kumar. Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/cd470bfd Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/cd470bfd Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/cd470bfd Branch: refs/heads/HDDS-4 Commit: cd470bfd27c878783666ebb151941ae4c2eed812 Parents: d0be865 Author: Ajay Kumar Authored: Tue Oct 30 09:10:21 2018 -0700 Committer: Xiaoyu Yao Committed: Thu Nov 29 11:57:47 2018 -0800 -- .../certificates/TestCertificateSignRequest.java | 17 + 1 file changed, 17 insertions(+) -- http://git-wip-us.apache.org/repos/asf/hadoop/blob/cd470bfd/hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/security/x509/certificates/TestCertificateSignRequest.java -- diff --git a/hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/security/x509/certificates/TestCertificateSignRequest.java b/hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/security/x509/certificates/TestCertificateSignRequest.java index a9285df..0b9ef31 100644 --- a/hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/security/x509/certificates/TestCertificateSignRequest.java +++ b/hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/security/x509/certificates/TestCertificateSignRequest.java @@ -1,3 +1,20 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ package org.apache.hadoop.hdds.security.x509.certificates; import org.apache.hadoop.hdds.conf.OzoneConfiguration; - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
[14/50] [abbrv] hadoop git commit: HDFS-14095. EC: Track Erasure Coding commands in DFS statistics. Contributed by Ayush Saxena.
HDFS-14095. EC: Track Erasure Coding commands in DFS statistics. Contributed by
Ayush Saxena.
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/f5347368
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/f5347368
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/f5347368
Branch: refs/heads/HDDS-4
Commit: f534736867eed962899615ca1b7eb68bcf591d17
Parents: d0edd37
Author: Brahma Reddy Battula
Authored: Fri Nov 30 00:18:27 2018 +0530
Committer: Brahma Reddy Battula
Committed: Fri Nov 30 00:18:27 2018 +0530
--
.../hadoop/hdfs/DFSOpsCountStatistics.java | 9 +++
.../hadoop/hdfs/DistributedFileSystem.java | 18 ++
.../hadoop/hdfs/TestDistributedFileSystem.java | 63 +++-
3 files changed, 89 insertions(+), 1 deletion(-)
--
http://git-wip-us.apache.org/repos/asf/hadoop/blob/f5347368/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSOpsCountStatistics.java
--
diff --git
a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSOpsCountStatistics.java
b/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSOpsCountStatistics.java
index 3dcf13b..b9852ba 100644
---
a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSOpsCountStatistics.java
+++
b/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSOpsCountStatistics.java
@@ -41,6 +41,7 @@ public class DFSOpsCountStatistics extends StorageStatistics {
/** This is for counting distributed file system operations. */
public enum OpType {
+ADD_EC_POLICY("op_add_ec_policy"),
ALLOW_SNAPSHOT("op_allow_snapshot"),
APPEND(CommonStatisticNames.OP_APPEND),
CONCAT("op_concat"),
@@ -51,10 +52,15 @@ public class DFSOpsCountStatistics extends
StorageStatistics {
CREATE_SYM_LINK("op_create_symlink"),
DELETE(CommonStatisticNames.OP_DELETE),
DELETE_SNAPSHOT("op_delete_snapshot"),
+DISABLE_EC_POLICY("op_disable_ec_policy"),
DISALLOW_SNAPSHOT("op_disallow_snapshot"),
+ENABLE_EC_POLICY("op_enable_ec_policy"),
EXISTS(CommonStatisticNames.OP_EXISTS),
GET_BYTES_WITH_FUTURE_GS("op_get_bytes_with_future_generation_stamps"),
GET_CONTENT_SUMMARY(CommonStatisticNames.OP_GET_CONTENT_SUMMARY),
+GET_EC_CODECS("op_get_ec_codecs"),
+GET_EC_POLICY("op_get_ec_policy"),
+GET_EC_POLICIES("op_get_ec_policies"),
GET_FILE_BLOCK_LOCATIONS("op_get_file_block_locations"),
GET_FILE_CHECKSUM(CommonStatisticNames.OP_GET_FILE_CHECKSUM),
GET_FILE_LINK_STATUS("op_get_file_link_status"),
@@ -76,11 +82,13 @@ public class DFSOpsCountStatistics extends
StorageStatistics {
REMOVE_ACL(CommonStatisticNames.OP_REMOVE_ACL),
REMOVE_ACL_ENTRIES(CommonStatisticNames.OP_REMOVE_ACL_ENTRIES),
REMOVE_DEFAULT_ACL(CommonStatisticNames.OP_REMOVE_DEFAULT_ACL),
+REMOVE_EC_POLICY("op_remove_ec_policy"),
REMOVE_XATTR("op_remove_xattr"),
RENAME(CommonStatisticNames.OP_RENAME),
RENAME_SNAPSHOT("op_rename_snapshot"),
RESOLVE_LINK("op_resolve_link"),
SET_ACL(CommonStatisticNames.OP_SET_ACL),
+SET_EC_POLICY("op_set_ec_policy"),
SET_OWNER(CommonStatisticNames.OP_SET_OWNER),
SET_PERMISSION(CommonStatisticNames.OP_SET_PERMISSION),
SET_REPLICATION("op_set_replication"),
@@ -90,6 +98,7 @@ public class DFSOpsCountStatistics extends StorageStatistics {
GET_SNAPSHOT_DIFF("op_get_snapshot_diff"),
GET_SNAPSHOTTABLE_DIRECTORY_LIST("op_get_snapshottable_directory_list"),
TRUNCATE(CommonStatisticNames.OP_TRUNCATE),
+UNSET_EC_POLICY("op_unset_ec_policy"),
UNSET_STORAGE_POLICY("op_unset_storage_policy");
private static final Map SYMBOL_MAP =
http://git-wip-us.apache.org/repos/asf/hadoop/blob/f5347368/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java
--
diff --git
a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java
b/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java
index ca1546c..7dd02bd 100644
---
a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java
+++
b/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java
@@ -2845,6 +2845,8 @@ public class DistributedFileSystem extends FileSystem
*/
public void setErasureCodingPolicy(final Path path,
final String ecPolicyName) throws IOException {
+statistics.incrementWriteOps(1);
+
[28/50] [abbrv] hadoop git commit: HDDS-548. Create a Self-Signed Certificate. Contributed by Anu Engineer.
HDDS-548. Create a Self-Signed Certificate. Contributed by Anu Engineer.
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/87acc150
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/87acc150
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/87acc150
Branch: refs/heads/HDDS-4
Commit: 87acc1508a4bafc8cd3d77ffa684bf9871f52698
Parents: d1c6ff7
Author: Ajay Kumar
Authored: Fri Sep 28 06:52:56 2018 -0700
Committer: Xiaoyu Yao
Committed: Thu Nov 29 11:57:46 2018 -0800
--
.../org/apache/hadoop/hdds/HddsConfigKeys.java | 16 ++
.../hdds/security/x509/HDDSKeyGenerator.java| 99 ---
.../hdds/security/x509/HDDSKeyPEMWriter.java| 254 --
.../hdds/security/x509/SecurityConfig.java | 105 +---
.../certificates/SelfSignedCertificate.java | 212 +++
.../x509/certificates/package-info.java | 22 ++
.../x509/exceptions/CertificateException.java | 63 +
.../x509/exceptions/SCMSecurityException.java | 64 +
.../security/x509/exceptions/package-info.java | 23 ++
.../security/x509/keys/HDDSKeyGenerator.java| 106
.../security/x509/keys/HDDSKeyPEMWriter.java| 255 ++
.../hdds/security/x509/keys/package-info.java | 23 ++
.../security/x509/TestHDDSKeyGenerator.java | 81 --
.../security/x509/TestHDDSKeyPEMWriter.java | 213 ---
.../x509/certificates/TestRootCertificate.java | 258 +++
.../x509/certificates/package-info.java | 22 ++
.../x509/keys/TestHDDSKeyGenerator.java | 87 +++
.../x509/keys/TestHDDSKeyPEMWriter.java | 216
.../hdds/security/x509/keys/package-info.java | 22 ++
.../hadoop/hdds/security/x509/package-info.java | 22 ++
20 files changed, 1484 insertions(+), 679 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/hadoop/blob/87acc150/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/HddsConfigKeys.java
--
diff --git
a/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/HddsConfigKeys.java
b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/HddsConfigKeys.java
index 2a1404a..ef1194b 100644
---
a/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/HddsConfigKeys.java
+++
b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/HddsConfigKeys.java
@@ -131,4 +131,20 @@ public final class HddsConfigKeys {
public static final String HDDS_PUBLIC_KEY_FILE_NAME = "hdds.public.key.file"
+ ".name";
public static final String HDDS_PUBLIC_KEY_FILE_NAME_DEFAULT = "public.pem";
+
+ /**
+ * Maximum duration of certificates issued by SCM including Self-Signed
Roots.
+ * The formats accepted are based on the ISO-8601 duration format
PnDTnHnMn.nS
+ * Default value is 5 years and written as P1865D.
+ */
+ public static final String HDDS_X509_MAX_DURATION = "hdds.x509.max.duration";
+ // Limit Certificate duration to a max value of 5 years.
+ public static final String HDDS_X509_MAX_DURATION_DEFAULT= "P1865D";
+
+ public static final String HDDS_X509_SIGNATURE_ALGO =
+ "hdds.x509.signature.algorithm";
+ public static final String HDDS_X509_SIGNATURE_ALGO_DEFAULT =
"SHA256withRSA";
+
+
+
}
http://git-wip-us.apache.org/repos/asf/hadoop/blob/87acc150/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/security/x509/HDDSKeyGenerator.java
--
diff --git
a/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/security/x509/HDDSKeyGenerator.java
b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/security/x509/HDDSKeyGenerator.java
deleted file mode 100644
index cb411b2..000
---
a/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/security/x509/HDDSKeyGenerator.java
+++ /dev/null
@@ -1,99 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- *
- */
-package org.apache.hadoop.hdds.security.x509;
-
-import
[25/50] [abbrv] hadoop git commit: Fix merge conflicts
Fix merge conflicts
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/b802cb5b
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/b802cb5b
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/b802cb5b
Branch: refs/heads/HDDS-4
Commit: b802cb5b039172111a815f9ba8c8415f2d402fb8
Parents: ed10fa6
Author: Xiaoyu Yao
Authored: Tue Jul 31 18:17:29 2018 -0700
Committer: Xiaoyu Yao
Committed: Thu Nov 29 11:57:46 2018 -0800
--
.../conf/TestConfigurationFieldsBase.java | 2 -
.../apache/hadoop/hdds/scm/ScmConfigKeys.java | 10 ++-
.../apache/hadoop/ozone/OzoneConfigKeys.java| 1 +
.../common/src/main/resources/ozone-default.xml | 21 ++---
.../scm/server/StorageContainerManager.java | 9 ++-
.../StorageContainerManagerHttpServer.java | 4 +-
.../src/test/compose/compose-secure/.env| 2 +-
.../test/compose/compose-secure/docker-config | 55 ++---
.../apache/hadoop/ozone/ksm/KSMConfigKeys.java | 84
.../apache/hadoop/ozone/MiniOzoneCluster.java | 3 +-
.../hadoop/ozone/MiniOzoneClusterImpl.java | 5 +-
.../hadoop/ozone/TestSecureOzoneCluster.java| 22 ++---
12 files changed, 87 insertions(+), 131 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/hadoop/blob/b802cb5b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/TestConfigurationFieldsBase.java
--
diff --git
a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/TestConfigurationFieldsBase.java
b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/TestConfigurationFieldsBase.java
index bce1cd5..152159b 100644
---
a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/TestConfigurationFieldsBase.java
+++
b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/TestConfigurationFieldsBase.java
@@ -436,8 +436,6 @@ public abstract class TestConfigurationFieldsBase {
// Create XML key/value map
LOG_XML.debug("Reading XML property files\n");
xmlKeyValueMap = extractPropertiesFromXml(xmlFilename);
-// Remove hadoop property set in ozone-default.xml
-xmlKeyValueMap.remove("hadoop.custom.tags");
LOG_XML.debug("\n=\n");
// Create default configuration variable key/value map
http://git-wip-us.apache.org/repos/asf/hadoop/blob/b802cb5b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/ScmConfigKeys.java
--
diff --git
a/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/ScmConfigKeys.java
b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/ScmConfigKeys.java
index 376e6db..cb6672f 100644
---
a/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/ScmConfigKeys.java
+++
b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/ScmConfigKeys.java
@@ -313,10 +313,12 @@ public final class ScmConfigKeys {
public static final String HDDS_SCM_WATCHER_TIMEOUT_DEFAULT =
"10m";
- public static final String SCM_WEB_AUTHENTICATION_KERBEROS_PRINCIPAL_KEY =
- "ozone.scm.web.authentication.kerberos.principal";
- public static final String SCM_WEB_AUTHENTICATION_KERBEROS_KEYTAB_FILE_KEY =
- "ozone.scm.web.authentication.kerberos.keytab";
+ public static final String
+ HDDS_SCM_WEB_AUTHENTICATION_KERBEROS_PRINCIPAL_KEY =
+ "hdds.scm.web.authentication.kerberos.principal";
+ public static final String
+ HDDS_SCM_WEB_AUTHENTICATION_KERBEROS_KEYTAB_FILE_KEY =
+ "hdds.scm.web.authentication.kerberos.keytab";
/**
* Never constructed.
*/
http://git-wip-us.apache.org/repos/asf/hadoop/blob/b802cb5b/hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/OzoneConfigKeys.java
--
diff --git
a/hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/OzoneConfigKeys.java
b/hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/OzoneConfigKeys.java
index 578a983..b47113b 100644
---
a/hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/OzoneConfigKeys.java
+++
b/hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/OzoneConfigKeys.java
@@ -334,6 +334,7 @@ public final class OzoneConfigKeys {
public static final String OZONE_CONTAINER_COPY_WORKDIR =
"hdds.datanode.replication.work.dir";
+
/**
* Config properties to set client side checksum properties.
*/
http://git-wip-us.apache.org/repos/asf/hadoop/blob/b802cb5b/hadoop-hdds/common/src/main/resources/ozone-default.xml
--
diff --git a/hadoop-hdds/common/src/main/resources/ozone-default.xml
[07/50] [abbrv] hadoop git commit: HDFS-13713. Add specification of Multipart Upload API to FS specification, with contract tests.
HDFS-13713. Add specification of Multipart Upload API to FS specification, with
contract tests.
Contributed by Ewan Higgs and Steve Loughran.
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/c1d24f84
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/c1d24f84
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/c1d24f84
Branch: refs/heads/HDDS-4
Commit: c1d24f848345f6d34a2ac2d570d49e9787a0df6a
Parents: b71cc7f
Author: Ewan Higgs
Authored: Thu Nov 29 15:11:07 2018 +
Committer: Steve Loughran
Committed: Thu Nov 29 15:12:17 2018 +
--
.../hadoop/fs/FileSystemMultipartUploader.java | 36 +-
.../org/apache/hadoop/fs/MultipartUploader.java | 88 ++-
.../hadoop/fs/MultipartUploaderFactory.java | 7 +
.../src/site/markdown/filesystem/index.md | 1 +
.../markdown/filesystem/multipartuploader.md| 235
.../AbstractContractMultipartUploaderTest.java | 565 +++
.../TestLocalFSContractMultipartUploader.java | 10 +
.../hdfs/TestHDFSContractMultipartUploader.java | 15 +
.../hadoop/fs/s3a/S3AMultipartUploader.java | 31 +-
.../s3a/ITestS3AContractMultipartUploader.java | 64 ++-
.../fs/s3a/TestS3AMultipartUploaderSupport.java | 2 +-
11 files changed, 876 insertions(+), 178 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/hadoop/blob/c1d24f84/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystemMultipartUploader.java
--
diff --git
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystemMultipartUploader.java
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystemMultipartUploader.java
index 94c7861..b77c244 100644
---
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystemMultipartUploader.java
+++
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystemMultipartUploader.java
@@ -19,21 +19,23 @@ package org.apache.hadoop.fs;
import java.io.IOException;
import java.io.InputStream;
import java.nio.ByteBuffer;
+import java.util.ArrayList;
import java.util.Comparator;
import java.util.List;
+import java.util.Map;
+import java.util.UUID;
import java.util.stream.Collectors;
import com.google.common.base.Charsets;
-import com.google.common.base.Preconditions;
import org.apache.commons.compress.utils.IOUtils;
-import org.apache.commons.lang3.tuple.Pair;
import org.apache.hadoop.classification.InterfaceAudience;
import org.apache.hadoop.classification.InterfaceStability;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.permission.FsPermission;
import static org.apache.hadoop.fs.Path.mergePaths;
+import static org.apache.hadoop.io.IOUtils.cleanupWithLogger;
/**
* A MultipartUploader that uses the basic FileSystem commands.
@@ -70,7 +72,8 @@ public class FileSystemMultipartUploader extends
MultipartUploader {
public PartHandle putPart(Path filePath, InputStream inputStream,
int partNumber, UploadHandle uploadId, long lengthInBytes)
throws IOException {
-
+checkPutArguments(filePath, inputStream, partNumber, uploadId,
+lengthInBytes);
byte[] uploadIdByteArray = uploadId.toByteArray();
checkUploadId(uploadIdByteArray);
Path collectorPath = new Path(new String(uploadIdByteArray, 0,
@@ -82,16 +85,17 @@ public class FileSystemMultipartUploader extends
MultipartUploader {
fs.createFile(partPath).build()) {
IOUtils.copy(inputStream, fsDataOutputStream, 4096);
} finally {
- org.apache.hadoop.io.IOUtils.cleanupWithLogger(LOG, inputStream);
+ cleanupWithLogger(LOG, inputStream);
}
return BBPartHandle.from(ByteBuffer.wrap(
partPath.toString().getBytes(Charsets.UTF_8)));
}
private Path createCollectorPath(Path filePath) {
+String uuid = UUID.randomUUID().toString();
return mergePaths(filePath.getParent(),
mergePaths(new Path(filePath.getName().split("\\.")[0]),
-mergePaths(new Path("_multipart"),
+mergePaths(new Path("_multipart_" + uuid),
new Path(Path.SEPARATOR;
}
@@ -110,21 +114,16 @@ public class FileSystemMultipartUploader extends
MultipartUploader {
@Override
@SuppressWarnings("deprecation") // rename w/ OVERWRITE
- public PathHandle complete(Path filePath,
- List> handles, UploadHandle multipartUploadId)
- throws IOException {
+ public PathHandle complete(Path filePath, Map handleMap,
+ UploadHandle multipartUploadId) throws IOException {
checkUploadId(multipartUploadId.toByteArray());
-if (handles.isEmpty()) {
- throw new IOException("Empty upload");
-}
-// If
[41/50] [abbrv] hadoop git commit: HDDS-753. SCM security protocol server is not starting. Contributed by Ajay Kumar.
HDDS-753. SCM security protocol server is not starting. Contributed by Ajay
Kumar.
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/9128c665
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/9128c665
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/9128c665
Branch: refs/heads/HDDS-4
Commit: 9128c665ccdf26eb3152d6903b98e779585acb2d
Parents: cd470bf
Author: Ajay Kumar
Authored: Thu Nov 1 15:48:52 2018 -0700
Committer: Xiaoyu Yao
Committed: Thu Nov 29 11:57:47 2018 -0800
--
.../scm/server/SCMSecurityProtocolServer.java | 4 +-
.../server/TestSCMSecurityProtocolServer.java | 60
.../hadoop/ozone/TestSecureOzoneCluster.java| 3 +-
3 files changed, 65 insertions(+), 2 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/hadoop/blob/9128c665/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/SCMSecurityProtocolServer.java
--
diff --git
a/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/SCMSecurityProtocolServer.java
b/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/SCMSecurityProtocolServer.java
index e810c54..ab29c1e 100644
---
a/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/SCMSecurityProtocolServer.java
+++
b/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/SCMSecurityProtocolServer.java
@@ -28,7 +28,7 @@ import
org.apache.hadoop.hdds.protocolPB.SCMSecurityProtocolServerSideTranslator
import org.apache.hadoop.hdds.scm.HddsServerUtil;
import org.apache.hadoop.hdds.scm.ScmConfigKeys;
import org.apache.hadoop.hdds.protocol.SCMSecurityProtocol;
-import org.apache.hadoop.io.IOUtils;
+import org.apache.hadoop.ipc.ProtobufRpcEngine;
import org.apache.hadoop.ipc.RPC;
import org.apache.hadoop.security.KerberosInfo;
import org.slf4j.Logger;
@@ -60,6 +60,8 @@ public class SCMSecurityProtocolServer implements
SCMSecurityProtocol {
rpcAddress = HddsServerUtil
.getScmSecurityInetAddress(conf);
// SCM security service RPC service.
+RPC.setProtocolEngine(conf, SCMSecurityProtocolPB.class,
+ProtobufRpcEngine.class);
BlockingService secureProtoPbService =
SCMSecurityProtocolProtos.SCMSecurityProtocolService
.newReflectiveBlockingService(
http://git-wip-us.apache.org/repos/asf/hadoop/blob/9128c665/hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/server/TestSCMSecurityProtocolServer.java
--
diff --git
a/hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/server/TestSCMSecurityProtocolServer.java
b/hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/server/TestSCMSecurityProtocolServer.java
new file mode 100644
index 000..8e7d84c
--- /dev/null
+++
b/hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/server/TestSCMSecurityProtocolServer.java
@@ -0,0 +1,60 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations
under
+ * the License.
+ */
+package org.apache.hadoop.hdds.scm.server;
+
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+import org.junit.After;
+import org.junit.Before;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.Timeout;
+
+/**
+ * Test class for {@link SCMSecurityProtocolServer}.
+ * */
+public class TestSCMSecurityProtocolServer {
+ private SCMSecurityProtocolServer securityProtocolServer;
+ private OzoneConfiguration config;
+
+ @Rule
+ public Timeout timeout = new Timeout(1000 * 20);
+
+ @Before
+ public void setUp() throws Exception {
+config = new OzoneConfiguration();
+securityProtocolServer = new SCMSecurityProtocolServer(config, null);
+ }
+
+ @After
+ public void tearDown() throws Exception {
+if (securityProtocolServer != null) {
+ securityProtocolServer.stop();
+ securityProtocolServer = null;
+}
+config = null;
+ }
+
+ @Test
+ public void testStart() {
+
[35/50] [abbrv] hadoop git commit: HDDS-8. Add OzoneManager Delegation Token support. Contributed by Ajay Kumar.
http://git-wip-us.apache.org/repos/asf/hadoop/blob/59767be1/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/TestSecureOzoneCluster.java
--
diff --git
a/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/TestSecureOzoneCluster.java
b/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/TestSecureOzoneCluster.java
index 904e597..c1695f1 100644
---
a/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/TestSecureOzoneCluster.java
+++
b/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/TestSecureOzoneCluster.java
@@ -17,16 +17,23 @@
*/
package org.apache.hadoop.ozone;
+import static org.apache.hadoop.hdds.HddsConfigKeys.OZONE_METADATA_DIRS;
import static org.apache.hadoop.ozone.OzoneConfigKeys.OZONE_ADMINISTRATORS;
+import static org.apache.hadoop.ozone.OzoneConfigKeys.OZONE_ENABLED;
import static
org.apache.hadoop.ozone.OzoneConfigKeys.OZONE_SECURITY_ENABLED_KEY;
+import static org.slf4j.event.Level.INFO;
import java.io.File;
import java.io.IOException;
+import java.net.InetAddress;
import java.nio.file.Path;
import java.nio.file.Paths;
+import java.security.KeyPair;
+import java.security.PrivilegedExceptionAction;
import java.util.Properties;
import java.util.UUID;
import java.util.concurrent.Callable;
+import org.apache.commons.io.FileUtils;
import org.apache.hadoop.classification.InterfaceAudience;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.CommonConfigurationKeysPublic;
@@ -35,14 +42,29 @@ import org.apache.hadoop.hdds.scm.ScmConfigKeys;
import org.apache.hadoop.hdds.scm.ScmInfo;
import org.apache.hadoop.hdds.scm.server.SCMStorage;
import org.apache.hadoop.hdds.scm.server.StorageContainerManager;
+import
org.apache.hadoop.hdds.security.x509.certificate.client.CertificateClient;
+import org.apache.hadoop.hdds.security.x509.keys.HDDSKeyGenerator;
+import org.apache.hadoop.hdds.security.x509.keys.HDDSKeyPEMWriter;
+import org.apache.hadoop.io.Text;
+import org.apache.hadoop.ipc.Client;
+import org.apache.hadoop.ipc.RPC;
+import org.apache.hadoop.ipc.RemoteException;
+import org.apache.hadoop.ipc.Server;
+import org.apache.hadoop.metrics2.lib.DefaultMetricsSystem;
import org.apache.hadoop.minikdc.MiniKdc;
+import org.apache.hadoop.net.NetUtils;
import org.apache.hadoop.ozone.om.OMConfigKeys;
import org.apache.hadoop.ozone.om.OMStorage;
import org.apache.hadoop.ozone.om.OzoneManager;
+import
org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB;
+import org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolPB;
+import org.apache.hadoop.ozone.security.OzoneTokenIdentifier;
import org.apache.hadoop.security.KerberosAuthException;
+import org.apache.hadoop.security.SaslRpcServer.AuthMethod;
import org.apache.hadoop.security.UserGroupInformation;
+import org.apache.hadoop.security.UserGroupInformation.AuthenticationMethod;
import
org.apache.hadoop.security.authentication.client.AuthenticationException;
-import org.apache.hadoop.security.authentication.util.KerberosUtil;
+import org.apache.hadoop.security.token.Token;
import org.apache.hadoop.test.GenericTestUtils;
import org.apache.hadoop.test.GenericTestUtils.LogCapturer;
import org.apache.hadoop.test.LambdaTestUtils;
@@ -52,6 +74,7 @@ import org.junit.Before;
import org.junit.Rule;
import org.junit.Test;
import org.junit.rules.Timeout;
+import org.mockito.Mockito;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -61,6 +84,8 @@ import org.slf4j.LoggerFactory;
@InterfaceAudience.Private
public final class TestSecureOzoneCluster {
+ private static final String TEST_USER = "testUgiUser";
+ private static final int CLIENT_TIMEOUT = 2 * 1000;
private Logger LOGGER = LoggerFactory
.getLogger(TestSecureOzoneCluster.class);
@@ -81,14 +106,24 @@ public final class TestSecureOzoneCluster {
private static String clusterId;
private static String scmId;
private static String omId;
+ private OzoneManagerProtocolClientSideTranslatorPB omClient;
+ private KeyPair keyPair;
+ private Path metaDirPath;
@Before
public void init() {
try {
conf = new OzoneConfiguration();
+ conf.set(ScmConfigKeys.OZONE_SCM_CLIENT_ADDRESS_KEY, "localhost");
+ DefaultMetricsSystem.setMiniClusterMode(true);
+ final String path = GenericTestUtils
+ .getTempPath(UUID.randomUUID().toString());
+ metaDirPath = Paths.get(path, "om-meta");
+ conf.set(OZONE_METADATA_DIRS, metaDirPath.toString());
startMiniKdc();
setSecureConfig(conf);
createCredentialsInKDC(conf, miniKdc);
+ generateKeyPair(conf);
} catch (IOException e) {
LOGGER.error("Failed to initialize TestSecureOzoneCluster", e);
} catch (Exception e) {
@@ -106,6 +141,10 @@ public final class TestSecureOzoneCluster {
if (om != null) {
om.stop();
[19/50] [abbrv] hadoop git commit: Revert "Bad merge with 996a627b289947af3894bf83e7b63ec702a665cd"
Revert "Bad merge with 996a627b289947af3894bf83e7b63ec702a665cd" This reverts commit 996a627b289947af3894bf83e7b63ec702a665cd. Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/074b8f4e Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/074b8f4e Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/074b8f4e Branch: refs/heads/HDDS-4 Commit: 074b8f4ed52a9defaedaff788c517d7b3f3dd024 Parents: fcd705a Author: Xiaoyu Yao Authored: Tue May 15 16:56:24 2018 -0700 Committer: Xiaoyu Yao Committed: Thu Nov 29 11:56:49 2018 -0800 -- hadoop-hdds/common/src/main/resources/ozone-default.xml | 4 1 file changed, 4 deletions(-) -- http://git-wip-us.apache.org/repos/asf/hadoop/blob/074b8f4e/hadoop-hdds/common/src/main/resources/ozone-default.xml -- diff --git a/hadoop-hdds/common/src/main/resources/ozone-default.xml b/hadoop-hdds/common/src/main/resources/ozone-default.xml index 4d79e8c..d3e352b 100644 --- a/hadoop-hdds/common/src/main/resources/ozone-default.xml +++ b/hadoop-hdds/common/src/main/resources/ozone-default.xml @@ -198,7 +198,6 @@ -<<< HEAD dfs.ratis.client.request.timeout.duration 3s OZONE, RATIS, MANAGEMENT @@ -256,9 +255,6 @@ hdds.container.report.interval -=== -ozone.container.report.interval ->>> HDDS-6. Enable SCM kerberos auth. Contributed by Ajay Kumar. 6ms OZONE, CONTAINER, MANAGEMENT Time interval of the datanode to send container report. Each - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
[46/50] [abbrv] hadoop git commit: Fix HDDS-4 after HDDS-751.
Fix HDDS-4 after HDDS-751.
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/3bee23d2
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/3bee23d2
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/3bee23d2
Branch: refs/heads/HDDS-4
Commit: 3bee23d24f6d90592e715be35fa6213a2723e0a6
Parents: 18c9db9
Author: Xiaoyu Yao
Authored: Wed Nov 21 08:38:30 2018 -0800
Committer: Xiaoyu Yao
Committed: Thu Nov 29 11:57:48 2018 -0800
--
.../main/java/org/apache/hadoop/hdds/scm/HddsServerUtil.java | 6 --
1 file changed, 4 insertions(+), 2 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/hadoop/blob/3bee23d2/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/hdds/scm/HddsServerUtil.java
--
diff --git
a/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/hdds/scm/HddsServerUtil.java
b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/hdds/scm/HddsServerUtil.java
index e355d0a..1bb9972 100644
---
a/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/hdds/scm/HddsServerUtil.java
+++
b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/hdds/scm/HddsServerUtil.java
@@ -176,9 +176,11 @@ public final class HddsServerUtil {
ScmConfigKeys.OZONE_SCM_SECURITY_SERVICE_ADDRESS_KEY);
return NetUtils.createSocketAddr(
-host.or(ScmConfigKeys.OZONE_SCM_SECURITY_SERVICE_BIND_HOST_DEFAULT) +
+host.orElse(ScmConfigKeys
+.OZONE_SCM_SECURITY_SERVICE_BIND_HOST_DEFAULT) +
":" + port
-.or(conf.getInt(ScmConfigKeys.OZONE_SCM_SECURITY_SERVICE_PORT_KEY,
+.orElse(conf.getInt(ScmConfigKeys
+.OZONE_SCM_SECURITY_SERVICE_PORT_KEY,
ScmConfigKeys.OZONE_SCM_SECURITY_SERVICE_PORT_DEFAULT)));
}
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
[47/50] [abbrv] hadoop git commit: Fix HDDS-4 after HDDS-759
Fix HDDS-4 after HDDS-759
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/18c9db98
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/18c9db98
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/18c9db98
Branch: refs/heads/HDDS-4
Commit: 18c9db980f8d3bcd01e3a9fa0f59861e2689
Parents: 96bd574
Author: Xiaoyu Yao
Authored: Wed Nov 21 08:21:44 2018 -0800
Committer: Xiaoyu Yao
Committed: Thu Nov 29 11:57:48 2018 -0800
--
.../java/org/apache/hadoop/hdds/security/x509/SecurityConfig.java | 2 +-
.../security/x509/certificates/TestCertificateSignRequest.java | 2 +-
.../hdds/security/x509/certificates/TestRootCertificate.java | 2 +-
.../hadoop/hdds/security/x509/keys/TestHDDSKeyGenerator.java | 2 +-
4 files changed, 4 insertions(+), 4 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/hadoop/blob/18c9db98/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/security/x509/SecurityConfig.java
--
diff --git
a/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/security/x509/SecurityConfig.java
b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/security/x509/SecurityConfig.java
index 97627ca..2826e55 100644
---
a/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/security/x509/SecurityConfig.java
+++
b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/security/x509/SecurityConfig.java
@@ -50,7 +50,7 @@ import static
org.apache.hadoop.hdds.HddsConfigKeys.HDDS_X509_MAX_DURATION;
import static
org.apache.hadoop.hdds.HddsConfigKeys.HDDS_X509_MAX_DURATION_DEFAULT;
import static org.apache.hadoop.hdds.HddsConfigKeys.HDDS_X509_SIGNATURE_ALGO;
import static
org.apache.hadoop.hdds.HddsConfigKeys.HDDS_X509_SIGNATURE_ALGO_DEFAULT;
-import static org.apache.hadoop.ozone.OzoneConfigKeys.OZONE_METADATA_DIRS;
+import static org.apache.hadoop.hdds.HddsConfigKeys.OZONE_METADATA_DIRS;
/**
* A class that deals with all Security related configs in HDDS.
http://git-wip-us.apache.org/repos/asf/hadoop/blob/18c9db98/hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/security/x509/certificates/TestCertificateSignRequest.java
--
diff --git
a/hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/security/x509/certificates/TestCertificateSignRequest.java
b/hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/security/x509/certificates/TestCertificateSignRequest.java
index e8de1ea..9328c50 100644
---
a/hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/security/x509/certificates/TestCertificateSignRequest.java
+++
b/hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/security/x509/certificates/TestCertificateSignRequest.java
@@ -43,7 +43,7 @@ import java.security.NoSuchAlgorithmException;
import java.security.NoSuchProviderException;
import java.util.UUID;
-import static org.apache.hadoop.ozone.OzoneConfigKeys.OZONE_METADATA_DIRS;
+import static org.apache.hadoop.hdds.HddsConfigKeys.OZONE_METADATA_DIRS;
public class TestCertificateSignRequest {
http://git-wip-us.apache.org/repos/asf/hadoop/blob/18c9db98/hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/security/x509/certificates/TestRootCertificate.java
--
diff --git
a/hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/security/x509/certificates/TestRootCertificate.java
b/hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/security/x509/certificates/TestRootCertificate.java
index 16245a6..e581dc8 100644
---
a/hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/security/x509/certificates/TestRootCertificate.java
+++
b/hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/security/x509/certificates/TestRootCertificate.java
@@ -46,7 +46,7 @@ import java.time.Instant;
import java.util.Date;
import java.util.UUID;
-import static org.apache.hadoop.ozone.OzoneConfigKeys.OZONE_METADATA_DIRS;
+import static org.apache.hadoop.hdds.HddsConfigKeys.OZONE_METADATA_DIRS;
/**
* Test Class for Root Certificate generation.
http://git-wip-us.apache.org/repos/asf/hadoop/blob/18c9db98/hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/security/x509/keys/TestHDDSKeyGenerator.java
--
diff --git
a/hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/security/x509/keys/TestHDDSKeyGenerator.java
b/hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/security/x509/keys/TestHDDSKeyGenerator.java
index f9541a2..08761f4 100644
---
a/hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/security/x509/keys/TestHDDSKeyGenerator.java
+++
[24/50] [abbrv] hadoop git commit: HDDS-684. Fix HDDS-4 branch after HDDS-490 and HADOOP-15832. Contributed by Xiaoyu Yao.
HDDS-684. Fix HDDS-4 branch after HDDS-490 and HADOOP-15832. Contributed by
Xiaoyu Yao.
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/72468986
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/72468986
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/72468986
Branch: refs/heads/HDDS-4
Commit: 724689864b6723a2ada9ccb925f9b828c19f305b
Parents: 950a473
Author: Xiaoyu Yao
Authored: Wed Oct 24 15:46:34 2018 -0700
Committer: Xiaoyu Yao
Committed: Thu Nov 29 11:57:46 2018 -0800
--
hadoop-ozone/dist/dev-support/bin/dist-layout-stitching | 2 +-
.../dist/src/main/compose/ozonesecure/docker-config | 1 +
.../ozonesecure/docker-image/runner/scripts/starter.sh | 4 ++--
.../java/org/apache/hadoop/ozone/om/OzoneManager.java | 12 ++--
4 files changed, 10 insertions(+), 9 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/hadoop/blob/72468986/hadoop-ozone/dist/dev-support/bin/dist-layout-stitching
--
diff --git a/hadoop-ozone/dist/dev-support/bin/dist-layout-stitching
b/hadoop-ozone/dist/dev-support/bin/dist-layout-stitching
index e736065..2a4f0ad 100755
--- a/hadoop-ozone/dist/dev-support/bin/dist-layout-stitching
+++ b/hadoop-ozone/dist/dev-support/bin/dist-layout-stitching
@@ -111,5 +111,5 @@ run cp
"${ROOT}/hadoop-ozone/objectstore-service/target/hadoop-ozone-objectstore
cp -r "${ROOT}/hadoop-hdds/docs/target/classes/docs" ./
#Copy docker compose files
-run cp -p -r "${ROOT}/hadoop-ozone/dist/src/main/compose" .
+run cp -p -R "${ROOT}/hadoop-ozone/dist/src/main/compose" .
run cp -p -r "${ROOT}/hadoop-ozone/dist/src/main/smoketest" .
http://git-wip-us.apache.org/repos/asf/hadoop/blob/72468986/hadoop-ozone/dist/src/main/compose/ozonesecure/docker-config
--
diff --git a/hadoop-ozone/dist/src/main/compose/ozonesecure/docker-config
b/hadoop-ozone/dist/src/main/compose/ozonesecure/docker-config
index 704dc7b..36f05ae 100644
--- a/hadoop-ozone/dist/src/main/compose/ozonesecure/docker-config
+++ b/hadoop-ozone/dist/src/main/compose/ozonesecure/docker-config
@@ -37,6 +37,7 @@
HDFS-SITE.XML_dfs.datanode.kerberos.principal=dn/_h...@example.com
HDFS-SITE.XML_dfs.datanode.keytab.file=/etc/security/keytabs/dn.keytab
HDFS-SITE.XML_dfs.web.authentication.kerberos.principal=HTTP/_h...@example.com
HDFS-SITE.XML_dfs.web.authentication.kerberos.keytab=/etc/security/keytabs/HTTP.keytab
+OZONE-SITE.XML_hdds.datanode.dir=/data/hdds
HDFS-SITE.XML_dfs.datanode.address=0.0.0.0:1019
HDFS-SITE.XML_dfs.datanode.http.address=0.0.0.0:1012
CORE-SITE.XML_dfs.data.transfer.protection=authentication
http://git-wip-us.apache.org/repos/asf/hadoop/blob/72468986/hadoop-ozone/dist/src/main/compose/ozonesecure/docker-image/runner/scripts/starter.sh
--
diff --git
a/hadoop-ozone/dist/src/main/compose/ozonesecure/docker-image/runner/scripts/starter.sh
b/hadoop-ozone/dist/src/main/compose/ozonesecure/docker-image/runner/scripts/starter.sh
index 04cd49d..eec7ce9 100755
---
a/hadoop-ozone/dist/src/main/compose/ozonesecure/docker-image/runner/scripts/starter.sh
+++
b/hadoop-ozone/dist/src/main/compose/ozonesecure/docker-image/runner/scripts/starter.sh
@@ -82,7 +82,7 @@ fi
if [ -n "$ENSURE_SCM_INITIALIZED" ]; then
if [ ! -f "$ENSURE_SCM_INITIALIZED" ]; then
-/opt/hadoop/bin/ozone scm -init
+/opt/hadoop/bin/ozone scm --init
fi
fi
@@ -92,7 +92,7 @@ if [ -n "$ENSURE_OM_INITIALIZED" ]; then
# Could be removed after HDFS-13203
echo "Waiting 15 seconds for SCM startup"
sleep 15
-/opt/hadoop/bin/ozone om -createObjectStore
+/opt/hadoop/bin/ozone om --init
fi
fi
http://git-wip-us.apache.org/repos/asf/hadoop/blob/72468986/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java
--
diff --git
a/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java
b/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java
index 7fc6aee..a5cec20 100644
---
a/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java
+++
b/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java
@@ -259,17 +259,17 @@ public final class OzoneManager extends
ServiceRuntimeInfoImpl
/**
- * Login KSM service user if security and Kerberos are enabled.
+ * Login OM service user if security and Kerberos are enabled.
*
* @param conf
* @throws IOException, AuthenticationException
*/
- private static void loginKSMUser(OzoneConfiguration conf)
+
[10/50] [abbrv] hadoop git commit: HDDS-808. Simplify OMAction and DNAction classes used for AuditLogging. Contributed by Dinesh Chitlangia.
HDDS-808. Simplify OMAction and DNAction classes used for AuditLogging.
Contributed by Dinesh Chitlangia.
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/184cced5
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/184cced5
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/184cced5
Branch: refs/heads/HDDS-4
Commit: 184cced513c5599d7b33c9124692fbcd2e6d338e
Parents: 07142f5
Author: Ajay Kumar
Authored: Thu Nov 29 08:35:02 2018 -0800
Committer: Ajay Kumar
Committed: Thu Nov 29 08:35:20 2018 -0800
--
.../org/apache/hadoop/ozone/audit/DNAction.java | 44 +++-
.../apache/hadoop/ozone/audit/DummyAction.java | 36 ++---
.../org/apache/hadoop/ozone/audit/OMAction.java | 54 +---
3 files changed, 58 insertions(+), 76 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/hadoop/blob/184cced5/hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/audit/DNAction.java
--
diff --git
a/hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/audit/DNAction.java
b/hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/audit/DNAction.java
index ce34c46..1c87f2b 100644
---
a/hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/audit/DNAction.java
+++
b/hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/audit/DNAction.java
@@ -21,34 +21,28 @@ package org.apache.hadoop.ozone.audit;
*/
public enum DNAction implements AuditAction {
- CREATE_CONTAINER("CREATE_CONTAINER"),
- READ_CONTAINER("READ_CONTAINER"),
- UPDATE_CONTAINER("UPDATE_CONTAINER"),
- DELETE_CONTAINER("DELETE_CONTAINER"),
- LIST_CONTAINER("LIST_CONTAINER"),
- PUT_BLOCK("PUT_BLOCK"),
- GET_BLOCK("GET_BLOCK"),
- DELETE_BLOCK("DELETE_BLOCK"),
- LIST_BLOCK("LIST_BLOCK"),
- READ_CHUNK("READ_CHUNK"),
- DELETE_CHUNK("DELETE_CHUNK"),
- WRITE_CHUNK("WRITE_CHUNK"),
- LIST_CHUNK("LIST_CHUNK"),
- COMPACT_CHUNK("COMPACT_CHUNK"),
- PUT_SMALL_FILE("PUT_SMALL_FILE"),
- GET_SMALL_FILE("GET_SMALL_FILE"),
- CLOSE_CONTAINER("CLOSE_CONTAINER"),
- GET_COMMITTED_BLOCK_LENGTH("GET_COMMITTED_BLOCK_LENGTH");
-
- private String action;
-
- DNAction(String action) {
-this.action = action;
- }
+ CREATE_CONTAINER,
+ READ_CONTAINER,
+ UPDATE_CONTAINER,
+ DELETE_CONTAINER,
+ LIST_CONTAINER,
+ PUT_BLOCK,
+ GET_BLOCK,
+ DELETE_BLOCK,
+ LIST_BLOCK,
+ READ_CHUNK,
+ DELETE_CHUNK,
+ WRITE_CHUNK,
+ LIST_CHUNK,
+ COMPACT_CHUNK,
+ PUT_SMALL_FILE,
+ GET_SMALL_FILE,
+ CLOSE_CONTAINER,
+ GET_COMMITTED_BLOCK_LENGTH;
@Override
public String getAction() {
-return this.action;
+return this.toString();
}
}
http://git-wip-us.apache.org/repos/asf/hadoop/blob/184cced5/hadoop-hdds/common/src/test/java/org/apache/hadoop/ozone/audit/DummyAction.java
--
diff --git
a/hadoop-hdds/common/src/test/java/org/apache/hadoop/ozone/audit/DummyAction.java
b/hadoop-hdds/common/src/test/java/org/apache/hadoop/ozone/audit/DummyAction.java
index 76cd39a..d2da3e6 100644
---
a/hadoop-hdds/common/src/test/java/org/apache/hadoop/ozone/audit/DummyAction.java
+++
b/hadoop-hdds/common/src/test/java/org/apache/hadoop/ozone/audit/DummyAction.java
@@ -22,30 +22,24 @@ package org.apache.hadoop.ozone.audit;
*/
public enum DummyAction implements AuditAction {
- CREATE_VOLUME("CREATE_VOLUME"),
- CREATE_BUCKET("CREATE_BUCKET"),
- CREATE_KEY("CREATE_KEY"),
- READ_VOLUME("READ_VOLUME"),
- READ_BUCKET("READ_BUCKET"),
- READ_KEY("READ_BUCKET"),
- UPDATE_VOLUME("UPDATE_VOLUME"),
- UPDATE_BUCKET("UPDATE_BUCKET"),
- UPDATE_KEY("UPDATE_KEY"),
- DELETE_VOLUME("DELETE_VOLUME"),
- DELETE_BUCKET("DELETE_BUCKET"),
- DELETE_KEY("DELETE_KEY"),
- SET_OWNER("SET_OWNER"),
- SET_QUOTA("SET_QUOTA");
-
- private final String action;
-
- DummyAction(String action) {
-this.action = action;
- }
+ CREATE_VOLUME,
+ CREATE_BUCKET,
+ CREATE_KEY,
+ READ_VOLUME,
+ READ_BUCKET,
+ READ_KEY,
+ UPDATE_VOLUME,
+ UPDATE_BUCKET,
+ UPDATE_KEY,
+ DELETE_VOLUME,
+ DELETE_BUCKET,
+ DELETE_KEY,
+ SET_OWNER,
+ SET_QUOTA;
@Override
public String getAction() {
-return this.action;
+return this.toString();
}
}
http://git-wip-us.apache.org/repos/asf/hadoop/blob/184cced5/hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/audit/OMAction.java
--
diff --git
a/hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/audit/OMAction.java
b/hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/audit/OMAction.java
index 1d4d646..8794014 100644
---
a/hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/audit/OMAction.java
+++
[20/50] [abbrv] hadoop git commit: HDDS-7. Enable kerberos auth for Ozone client in hadoop rpc. Contributed by Ajay Kumar.
HDDS-7. Enable kerberos auth for Ozone client in hadoop rpc. Contributed by
Ajay Kumar.
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/5c821937
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/5c821937
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/5c821937
Branch: refs/heads/HDDS-4
Commit: 5c821937a4972c9a6001c18e18b7a58891198011
Parents: 074b8f4
Author: Xiaoyu Yao
Authored: Fri May 18 13:09:17 2018 -0700
Committer: Xiaoyu Yao
Committed: Thu Nov 29 11:56:49 2018 -0800
--
.../src/test/compose/compose-secure/.env| 17
.../compose/compose-secure/docker-compose.yaml | 66 ++
.../test/compose/compose-secure/docker-config | 66 ++
.../acceptance/ozone-secure.robot | 95
.../hadoop/ozone/client/rest/RestClient.java| 4 +-
.../hadoop/ozone/client/rpc/RpcClient.java | 6 +-
6 files changed, 248 insertions(+), 6 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/hadoop/blob/5c821937/hadoop-ozone/acceptance-test/src/test/compose/compose-secure/.env
--
diff --git a/hadoop-ozone/acceptance-test/src/test/compose/compose-secure/.env
b/hadoop-ozone/acceptance-test/src/test/compose/compose-secure/.env
new file mode 100644
index 000..3254735
--- /dev/null
+++ b/hadoop-ozone/acceptance-test/src/test/compose/compose-secure/.env
@@ -0,0 +1,17 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+OZONEDIR=../../../hadoop-dist/target/ozone
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/hadoop/blob/5c821937/hadoop-ozone/acceptance-test/src/test/compose/compose-secure/docker-compose.yaml
--
diff --git
a/hadoop-ozone/acceptance-test/src/test/compose/compose-secure/docker-compose.yaml
b/hadoop-ozone/acceptance-test/src/test/compose/compose-secure/docker-compose.yaml
new file mode 100644
index 000..2661163
--- /dev/null
+++
b/hadoop-ozone/acceptance-test/src/test/compose/compose-secure/docker-compose.yaml
@@ -0,0 +1,66 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+version: "3"
+services:
+ ozone.kdc:
+ image: ahadoop/kdc:v1
+ namenode:
+ image: ahadoop/ozone:v1
+ hostname: namenode
+ volumes:
+ - ${OZONEDIR}:/opt/hadoop
+ ports:
+ - 9000:9000
+ environment:
+ ENSURE_NAMENODE_DIR: /data/namenode
+ env_file:
+ - ./docker-config
+ command: ["/opt/hadoop/bin/hdfs","namenode"]
+ datanode:
+ image: ahadoop/ozone:v1
+ hostname: datanode
+ volumes:
+- ${OZONEDIR}:/opt/hadoop
+ ports:
+- 9874
+ env_file:
+- ./docker-config
+ command: ["/opt/hadoop/bin/ozone","datanode"]
+ ksm:
+ image: ahadoop/ozone:v1
+ hostname: ksm
+ volumes:
+ - ${OZONEDIR}:/opt/hadoop
+ ports:
+ - 9874:9874
+ environment:
+ ENSURE_KSM_INITIALIZED: /data/metadata/ksm/current/VERSION
+ env_file:
+ - ./docker-config
+ command: ["/opt/hadoop/bin/ozone","ksm"]
+ scm:
+ image: ahadoop/ozone:v1
+ hostname: scm
+ volumes:
+ - ${OZONEDIR}:/opt/hadoop
+ ports:
+ - 9876:9876
[45/50] [abbrv] hadoop git commit: HDDS-873. Fix TestSecureOzoneContainer NPE after HDDS-837. Contributed by Xiaoyu Yao.
HDDS-873. Fix TestSecureOzoneContainer NPE after HDDS-837. Contributed by
Xiaoyu Yao.
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/9b9b8e41
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/9b9b8e41
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/9b9b8e41
Branch: refs/heads/HDDS-4
Commit: 9b9b8e410f0d056ec3bae64da1f40013bbefde8b
Parents: 3bee23d
Author: Ajay Kumar
Authored: Mon Nov 26 15:49:01 2018 -0800
Committer: Xiaoyu Yao
Committed: Thu Nov 29 11:57:48 2018 -0800
--
.../container/ozoneimpl/TestSecureOzoneContainer.java | 14 +-
1 file changed, 13 insertions(+), 1 deletion(-)
--
http://git-wip-us.apache.org/repos/asf/hadoop/blob/9b9b8e41/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/container/ozoneimpl/TestSecureOzoneContainer.java
--
diff --git
a/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/container/ozoneimpl/TestSecureOzoneContainer.java
b/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/container/ozoneimpl/TestSecureOzoneContainer.java
index 02d5e28..2224300 100644
---
a/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/container/ozoneimpl/TestSecureOzoneContainer.java
+++
b/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/container/ozoneimpl/TestSecureOzoneContainer.java
@@ -32,6 +32,8 @@ import org.apache.hadoop.hdds.scm.TestUtils;
import org.apache.hadoop.hdds.scm.XceiverClientGrpc;
import org.apache.hadoop.hdds.scm.XceiverClientSpi;
import org.apache.hadoop.hdds.scm.pipeline.Pipeline;
+import
org.apache.hadoop.ozone.container.common.statemachine.DatanodeStateMachine;
+import org.apache.hadoop.ozone.container.common.statemachine.StateContext;
import org.apache.hadoop.security.SecurityUtil;
import org.apache.hadoop.security.UserGroupInformation;
import org.apache.hadoop.security.token.Token;
@@ -45,6 +47,7 @@ import org.junit.rules.TemporaryFolder;
import org.junit.rules.Timeout;
import org.junit.runner.RunWith;
import org.junit.runners.Parameterized;
+import org.mockito.Mockito;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -133,7 +136,7 @@ public class TestSecureOzoneContainer {
OzoneConfigKeys.DFS_CONTAINER_IPC_RANDOM_PORT, false);
DatanodeDetails dn = TestUtils.randomDatanodeDetails();
- container = new OzoneContainer(dn, conf, null);
+ container = new OzoneContainer(dn, conf, getContext(dn));
//Setting scmId, as we start manually ozone container.
container.getDispatcher().setScmId(UUID.randomUUID().toString());
container.start();
@@ -206,4 +209,13 @@ public class TestSecureOzoneContainer {
Assert.assertNotNull(response);
Assert.assertTrue(request.getTraceID().equals(response.getTraceID()));
}
+
+ private StateContext getContext(DatanodeDetails datanodeDetails) {
+DatanodeStateMachine stateMachine = Mockito.mock(
+DatanodeStateMachine.class);
+StateContext context = Mockito.mock(StateContext.class);
+
Mockito.when(stateMachine.getDatanodeDetails()).thenReturn(datanodeDetails);
+Mockito.when(context.getParent()).thenReturn(stateMachine);
+return context;
+ }
}
\ No newline at end of file
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
[29/50] [abbrv] hadoop git commit: HDDS-566. Move OzoneSecure docker-compose after HDDS-447. Contributed by Xiaoyu Yao.
HDDS-566. Move OzoneSecure docker-compose after HDDS-447. Contributed by Xiaoyu
Yao.
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/585c3448
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/585c3448
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/585c3448
Branch: refs/heads/HDDS-4
Commit: 585c3448684cdae3c9f6c9c175c23038dfef80fc
Parents: eddbe99
Author: Ajay Kumar
Authored: Tue Oct 2 10:07:35 2018 -0700
Committer: Xiaoyu Yao
Committed: Thu Nov 29 11:57:46 2018 -0800
--
hadoop-dist/src/main/compose/ozonesecure/.env | 18 ---
.../compose/ozonesecure/docker-compose.yaml | 57 ---
.../src/main/compose/ozonesecure/docker-config | 103 -
.../ozonesecure/docker-image/runner/Dockerfile | 39 -
.../ozonesecure/docker-image/runner/build.sh| 26
.../docker-image/runner/scripts/envtoconf.py| 115 --
.../docker-image/runner/scripts/krb5.conf | 38 -
.../docker-image/runner/scripts/starter.sh | 100 -
.../runner/scripts/transformation.py| 150 ---
.../dist/src/main/compose/ozonesecure/.env | 18 +++
.../compose/ozonesecure/docker-compose.yaml | 57 +++
.../src/main/compose/ozonesecure/docker-config | 103 +
.../ozonesecure/docker-image/runner/Dockerfile | 39 +
.../ozonesecure/docker-image/runner/build.sh| 26
.../docker-image/runner/scripts/envtoconf.py| 115 ++
.../docker-image/runner/scripts/krb5.conf | 38 +
.../docker-image/runner/scripts/starter.sh | 100 +
.../runner/scripts/transformation.py| 150 +++
18 files changed, 646 insertions(+), 646 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/hadoop/blob/585c3448/hadoop-dist/src/main/compose/ozonesecure/.env
--
diff --git a/hadoop-dist/src/main/compose/ozonesecure/.env
b/hadoop-dist/src/main/compose/ozonesecure/.env
deleted file mode 100644
index a494004..000
--- a/hadoop-dist/src/main/compose/ozonesecure/.env
+++ /dev/null
@@ -1,18 +0,0 @@
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-HDDS_VERSION=${hdds.version}
-SRC_VOLUME=../../
http://git-wip-us.apache.org/repos/asf/hadoop/blob/585c3448/hadoop-dist/src/main/compose/ozonesecure/docker-compose.yaml
--
diff --git a/hadoop-dist/src/main/compose/ozonesecure/docker-compose.yaml
b/hadoop-dist/src/main/compose/ozonesecure/docker-compose.yaml
deleted file mode 100644
index 42ab05e..000
--- a/hadoop-dist/src/main/compose/ozonesecure/docker-compose.yaml
+++ /dev/null
@@ -1,57 +0,0 @@
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-version: "3"
-services:
- kdc:
- image: ahadoop/kdc:v1
- hostname: kdc
- volumes:
- - $SRC_VOLUME:/opt/hadoop
- datanode:
- image: ahadoop/runner:latest
- volumes:
-- $SRC_VOLUME:/opt/hadoop
- hostname: datanode
- ports:
-- 9864
- command: ["/opt/hadoop/bin/ozone","datanode"]
- env_file:
-- ./docker-config
- ozoneManager:
- image: ahadoop/runner:latest
- hostname: om
- volumes:
- -
[22/50] [abbrv] hadoop git commit: HDDS-100. SCM CA: generate public/private key pair for SCM/OM/DNs. Contributed by Ajay Kumar.
HDDS-100. SCM CA: generate public/private key pair for SCM/OM/DNs. Contributed
by Ajay Kumar.
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/ed10fa6f
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/ed10fa6f
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/ed10fa6f
Branch: refs/heads/HDDS-4
Commit: ed10fa6fb160f210bcf424e151f2eeead34bf184
Parents: c8af727
Author: Xiaoyu Yao
Authored: Fri Jun 8 08:33:58 2018 -0700
Committer: Xiaoyu Yao
Committed: Thu Nov 29 11:57:45 2018 -0800
--
hadoop-hdds/common/pom.xml | 6 +-
.../org/apache/hadoop/hdds/HddsConfigKeys.java | 19 ++
.../hdds/security/x509/HDDSKeyGenerator.java| 99
.../hdds/security/x509/HDDSKeyPEMWriter.java| 254 +++
.../hdds/security/x509/SecurityConfig.java | 190 ++
.../hadoop/hdds/security/x509/package-info.java | 25 ++
.../common/src/main/resources/ozone-default.xml | 42 ++-
.../security/x509/TestHDDSKeyGenerator.java | 81 ++
.../security/x509/TestHDDSKeyPEMWriter.java | 213
.../ozone/TestOzoneConfigurationFields.java | 6 +
10 files changed, 933 insertions(+), 2 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/hadoop/blob/ed10fa6f/hadoop-hdds/common/pom.xml
--
diff --git a/hadoop-hdds/common/pom.xml b/hadoop-hdds/common/pom.xml
index 0345012..175500a 100644
--- a/hadoop-hdds/common/pom.xml
+++ b/hadoop-hdds/common/pom.xml
@@ -81,7 +81,6 @@ http://maven.apache.org/xsd/maven-4.0.0.xsd;>
rocksdbjni
5.14.2
-
org.apache.hadoop
hadoop-common
@@ -110,6 +109,11 @@ http://maven.apache.org/xsd/maven-4.0.0.xsd;>
2.6.0
+
+ org.bouncycastle
+ bcprov-jdk15on
+ 1.49
+
http://git-wip-us.apache.org/repos/asf/hadoop/blob/ed10fa6f/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/HddsConfigKeys.java
--
diff --git
a/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/HddsConfigKeys.java
b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/HddsConfigKeys.java
index f16503e..2a1404a 100644
---
a/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/HddsConfigKeys.java
+++
b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/HddsConfigKeys.java
@@ -112,4 +112,23 @@ public final class HddsConfigKeys {
public static final String HDDS_PROMETHEUS_ENABLED =
"hdds.prometheus.endpoint.enabled";
+
+ public static final String HDDS_KEY_LEN = "hdds.key.len";
+ public static final int HDDS_DEFAULT_KEY_LEN = 2048;
+ public static final String HDDS_KEY_ALGORITHM = "hdds.key.algo";
+ public static final String HDDS_DEFAULT_KEY_ALGORITHM = "RSA";
+ public static final String HDDS_SECURITY_PROVIDER = "hdds.security.provider";
+ public static final String HDDS_DEFAULT_SECURITY_PROVIDER = "BC";
+ public static final String HDDS_KEY_DIR_NAME = "hdds.key.dir.name";
+ public static final String HDDS_KEY_DIR_NAME_DEFAULT = "keys";
+
+ // TODO : Talk to StorageIO classes and see if they can return a secure
+ // storage location for each node.
+ public static final String HDDS_METADATA_DIR_NAME = "hdds.metadata.dir";
+ public static final String HDDS_PRIVATE_KEY_FILE_NAME =
+ "hdds.priv.key.file.name";
+ public static final String HDDS_PRIVATE_KEY_FILE_NAME_DEFAULT =
"private.pem";
+ public static final String HDDS_PUBLIC_KEY_FILE_NAME = "hdds.public.key.file"
+ + ".name";
+ public static final String HDDS_PUBLIC_KEY_FILE_NAME_DEFAULT = "public.pem";
}
http://git-wip-us.apache.org/repos/asf/hadoop/blob/ed10fa6f/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/security/x509/HDDSKeyGenerator.java
--
diff --git
a/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/security/x509/HDDSKeyGenerator.java
b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/security/x509/HDDSKeyGenerator.java
new file mode 100644
index 000..cb411b2
--- /dev/null
+++
b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/security/x509/HDDSKeyGenerator.java
@@ -0,0 +1,99 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by
[32/50] [abbrv] hadoop git commit: HDDS-588. SelfSignedCertificate#generateCertificate should sign the certificate the configured security provider. Contributed by Xiaoyu Yao.
HDDS-588. SelfSignedCertificate#generateCertificate should sign the certificate
the configured security provider. Contributed by Xiaoyu Yao.
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/602aa807
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/602aa807
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/602aa807
Branch: refs/heads/HDDS-4
Commit: 602aa807483464ed3b601ace2a6c379719dff653
Parents: 812c07e
Author: Ajay Kumar
Authored: Tue Oct 9 00:28:01 2018 -0700
Committer: Xiaoyu Yao
Committed: Thu Nov 29 11:57:46 2018 -0800
--
.../hdds/security/x509/certificates/SelfSignedCertificate.java | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/hadoop/blob/602aa807/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/security/x509/certificates/SelfSignedCertificate.java
--
diff --git
a/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/security/x509/certificates/SelfSignedCertificate.java
b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/security/x509/certificates/SelfSignedCertificate.java
index fef7ac3..f221246 100644
---
a/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/security/x509/certificates/SelfSignedCertificate.java
+++
b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/security/x509/certificates/SelfSignedCertificate.java
@@ -103,8 +103,8 @@ public final class SelfSignedCertificate {
ContentSigner contentSigner =
-new JcaContentSignerBuilder(
-config.getSignatureAlgo()).build(key.getPrivate());
+new JcaContentSignerBuilder(config.getSignatureAlgo())
+.setProvider(config.getProvider()).build(key.getPrivate());
// Please note: Since this is a root certificate we use "ONE" as the
// serial number. Also note that skip enforcing locale or UTC. We are
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
[21/50] [abbrv] hadoop git commit: HDDS-70. Fix config names for secure ksm and scm. Contributed by Ajay Kumar.
HDDS-70. Fix config names for secure ksm and scm. Contributed by Ajay Kumar.
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/c8af727f
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/c8af727f
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/c8af727f
Branch: refs/heads/HDDS-4
Commit: c8af727fa5efb58e9bfaeb2816d7d63c53e0bd62
Parents: 5c82193
Author: Xiaoyu Yao
Authored: Tue May 22 13:32:28 2018 -0700
Committer: Xiaoyu Yao
Committed: Thu Nov 29 11:56:50 2018 -0800
--
.../apache/hadoop/hdds/scm/ScmConfigKeys.java | 6 +-
.../scm/protocol/ScmBlockLocationProtocol.java | 2 +-
.../StorageContainerLocationProtocol.java | 3 +-
.../protocolPB/ScmBlockLocationProtocolPB.java | 4 +-
.../StorageContainerLocationProtocolPB.java | 2 +-
.../apache/hadoop/ozone/OzoneConfigKeys.java| 1 -
.../common/src/main/resources/ozone-default.xml | 31 +---
.../StorageContainerDatanodeProtocol.java | 2 +-
.../StorageContainerDatanodeProtocolPB.java | 2 +-
.../scm/server/StorageContainerManager.java | 18 ++---
.../compose/compose-secure/docker-compose.yaml | 6 +-
.../test/compose/compose-secure/docker-config | 12 +--
.../acceptance/ozone-secure.robot | 12 +--
.../ozone/client/protocol/ClientProtocol.java | 2 +-
.../apache/hadoop/ozone/ksm/KSMConfigKeys.java | 84
.../ozone/om/protocol/OzoneManagerProtocol.java | 4 +-
.../hadoop/ozone/TestSecureOzoneCluster.java| 21 +++--
.../apache/hadoop/ozone/om/OzoneManager.java| 4 +-
18 files changed, 151 insertions(+), 65 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/hadoop/blob/c8af727f/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/ScmConfigKeys.java
--
diff --git
a/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/ScmConfigKeys.java
b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/ScmConfigKeys.java
index e18fe91..376e6db 100644
---
a/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/ScmConfigKeys.java
+++
b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/ScmConfigKeys.java
@@ -192,9 +192,9 @@ public final class ScmConfigKeys {
"ozone.scm.http-address";
public static final String OZONE_SCM_HTTPS_ADDRESS_KEY =
"ozone.scm.https-address";
- public static final String OZONE_SCM_KERBEROS_KEYTAB_FILE_KEY =
- "ozone.scm.kerberos.keytab.file";
- public static final String OZONE_SCM_KERBEROS_PRINCIPAL_KEY =
"ozone.scm.kerberos.principal";
+ public static final String HDDS_SCM_KERBEROS_KEYTAB_FILE_KEY =
+ "hdds.scm.kerberos.keytab.file";
+ public static final String HDDS_SCM_KERBEROS_PRINCIPAL_KEY =
"hdds.scm.kerberos.principal";
public static final String OZONE_SCM_HTTP_BIND_HOST_DEFAULT = "0.0.0.0";
public static final int OZONE_SCM_HTTP_BIND_PORT_DEFAULT = 9876;
public static final int OZONE_SCM_HTTPS_BIND_PORT_DEFAULT = 9877;
http://git-wip-us.apache.org/repos/asf/hadoop/blob/c8af727f/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/protocol/ScmBlockLocationProtocol.java
--
diff --git
a/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/protocol/ScmBlockLocationProtocol.java
b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/protocol/ScmBlockLocationProtocol.java
index e17f1c2..2d46ae0 100644
---
a/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/protocol/ScmBlockLocationProtocol.java
+++
b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/protocol/ScmBlockLocationProtocol.java
@@ -33,7 +33,7 @@ import java.util.List;
* ScmBlockLocationProtocol is used by an HDFS node to find the set of nodes
* to read/write a block.
*/
-@KerberosInfo(serverPrincipal = ScmConfigKeys.OZONE_SCM_KERBEROS_PRINCIPAL_KEY)
+@KerberosInfo(serverPrincipal = ScmConfigKeys.HDDS_SCM_KERBEROS_PRINCIPAL_KEY)
public interface ScmBlockLocationProtocol {
/**
http://git-wip-us.apache.org/repos/asf/hadoop/blob/c8af727f/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/protocol/StorageContainerLocationProtocol.java
--
diff --git
a/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/protocol/StorageContainerLocationProtocol.java
b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/protocol/StorageContainerLocationProtocol.java
index 5bc2521..e21bc53 100644
---
a/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/protocol/StorageContainerLocationProtocol.java
+++
b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/protocol/StorageContainerLocationProtocol.java
@@
[27/50] [abbrv] hadoop git commit: HDDS-548. Create a Self-Signed Certificate. Contributed by Anu Engineer.
http://git-wip-us.apache.org/repos/asf/hadoop/blob/87acc150/hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/security/x509/keys/TestHDDSKeyPEMWriter.java
--
diff --git
a/hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/security/x509/keys/TestHDDSKeyPEMWriter.java
b/hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/security/x509/keys/TestHDDSKeyPEMWriter.java
new file mode 100644
index 000..db5d430
--- /dev/null
+++
b/hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/security/x509/keys/TestHDDSKeyPEMWriter.java
@@ -0,0 +1,216 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.hadoop.hdds.security.x509.keys;
+
+import static org.apache.hadoop.hdds.HddsConfigKeys.HDDS_METADATA_DIR_NAME;
+
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.nio.file.attribute.PosixFilePermission;
+import java.security.KeyFactory;
+import java.security.KeyPair;
+import java.security.NoSuchAlgorithmException;
+import java.security.NoSuchProviderException;
+import java.security.PrivateKey;
+import java.security.PublicKey;
+import java.security.spec.InvalidKeySpecException;
+import java.security.spec.PKCS8EncodedKeySpec;
+import java.security.spec.X509EncodedKeySpec;
+import java.util.Set;
+import org.apache.commons.codec.binary.Base64;
+import org.apache.commons.io.FileUtils;
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+import org.apache.hadoop.hdds.security.x509.SecurityConfig;
+import org.apache.hadoop.test.LambdaTestUtils;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+
+/**
+ * Test class for HDDS pem writer.
+ */
+public class TestHDDSKeyPEMWriter {
+
+ @Rule
+ public TemporaryFolder temporaryFolder = new TemporaryFolder();
+ private OzoneConfiguration configuration;
+ private HDDSKeyGenerator keyGenerator;
+ private String prefix;
+
+ @Before
+ public void init() throws IOException {
+configuration = new OzoneConfiguration();
+prefix = temporaryFolder.newFolder().toString();
+configuration.set(HDDS_METADATA_DIR_NAME, prefix);
+keyGenerator = new HDDSKeyGenerator(configuration);
+ }
+
+ /**
+ * Assert basic things like we are able to create a file, and the names are
+ * in expected format etc.
+ *
+ * @throws NoSuchProviderException - On Error, due to missing Java
+ * dependencies.
+ * @throws NoSuchAlgorithmException - On Error, due to missing Java
+ * dependencies.
+ * @throws IOException - On I/O failure.
+ */
+ @Test
+ public void testWriteKey()
+ throws NoSuchProviderException, NoSuchAlgorithmException,
+ IOException, InvalidKeySpecException {
+KeyPair keys = keyGenerator.generateKey();
+HDDSKeyPEMWriter pemWriter = new HDDSKeyPEMWriter(configuration);
+pemWriter.writeKey(keys);
+
+// Assert that locations have been created.
+Path keyLocation = pemWriter.getSecurityConfig().getKeyLocation();
+Assert.assertTrue(keyLocation.toFile().exists());
+
+// Assert that locations are created in the locations that we specified
+// using the Config.
+Assert.assertTrue(keyLocation.toString().startsWith(prefix));
+Path privateKeyPath = Paths.get(keyLocation.toString(),
+pemWriter.getSecurityConfig().getPrivateKeyFileName());
+Assert.assertTrue(privateKeyPath.toFile().exists());
+Path publicKeyPath = Paths.get(keyLocation.toString(),
+pemWriter.getSecurityConfig().getPublicKeyFileName());
+Assert.assertTrue(publicKeyPath.toFile().exists());
+
+// Read the private key and test if the expected String in the PEM file
+// format exists.
+byte[] privateKey = Files.readAllBytes(privateKeyPath);
+String privateKeydata = new String(privateKey, StandardCharsets.UTF_8);
+Assert.assertTrue(privateKeydata.contains("PRIVATE KEY"));
+
+// Read the public key and test if the expected String in the PEM file
+// format exists.
+byte[] publicKey = Files.readAllBytes(publicKeyPath);
+
[44/50] [abbrv] hadoop git commit: HDDS-778. Add an interface for CA and Clients for Certificate operations Contributed by Anu Engineer.
HDDS-778. Add an interface for CA and Clients for Certificate operations
Contributed by Anu Engineer.
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/4770e9de
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/4770e9de
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/4770e9de
Branch: refs/heads/HDDS-4
Commit: 4770e9dea8199753962c6517bff96fd39fbfd826
Parents: 8bbc95e
Author: Anu Engineer
Authored: Thu Nov 8 09:54:27 2018 -0800
Committer: Xiaoyu Yao
Committed: Thu Nov 29 11:57:47 2018 -0800
--
.../authority/CertificateServer.java| 99
.../certificate/authority/package-info.java | 22 +++
.../certificate/client/CertificateClient.java | 159 +++
.../x509/certificate/client/package-info.java | 22 +++
4 files changed, 302 insertions(+)
--
http://git-wip-us.apache.org/repos/asf/hadoop/blob/4770e9de/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/security/x509/certificate/authority/CertificateServer.java
--
diff --git
a/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/security/x509/certificate/authority/CertificateServer.java
b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/security/x509/certificate/authority/CertificateServer.java
new file mode 100644
index 000..9332e5b
--- /dev/null
+++
b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/security/x509/certificate/authority/CertificateServer.java
@@ -0,0 +1,99 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.hadoop.hdds.security.x509.certificate.authority;
+
+import
org.apache.hadoop.hdds.security.x509.certificates.CertificateSignRequest;
+import org.apache.hadoop.hdds.security.x509.exceptions.SCMSecurityException;
+import org.apache.hadoop.hdds.security.x509.SecurityConfig;
+import org.bouncycastle.cert.X509CertificateHolder;
+
+import java.security.cert.X509Certificate;
+import java.util.concurrent.Future;
+
+/**
+ * Interface for Certificate Authority. This can be extended to talk to
external
+ * CAs later or HSMs later.
+ */
+public interface CertificateServer {
+ /**
+ * Initialize the Certificate Authority.
+ *
+ * @param securityConfig - Security Configuration.
+ * @param type - The Type of CertificateServer we are creating, we make this
+ * explicit so that when we read code it is visible to the users.
+ * @throws SCMSecurityException - Throws if the init fails.
+ */
+ void init(SecurityConfig securityConfig, CAType type)
+ throws SCMSecurityException;
+
+ /**
+ * Returns the CA Certificate for this CA.
+ *
+ * @return X509CertificateHolder - Certificate for this CA.
+ * @throws SCMSecurityException -- usually thrown if this CA is not
+ * initialized.
+ */
+ X509CertificateHolder getCACertificate()
+ throws SCMSecurityException;
+
+ /**
+ * Request a Certificate based on Certificate Signing Request.
+ *
+ * @param csr - Certificate Signing Request.
+ * @return A future that will have this certificate when this request is
+ * approved.
+ * @throws SCMSecurityException - on Error.
+ */
+ Future requestCertificate(CertificateSignRequest csr,
+ CertificateApprover approver) throws SCMSecurityException;
+
+ /**
+ * Revokes a Certificate issued by this CertificateServer.
+ *
+ * @param certificate - Certificate to revoke
+ * @param approver - Approval process to follow.
+ * @return Future that tells us what happened.
+ * @throws SCMSecurityException - on Error.
+ */
+ Future revokeCertificate(X509Certificate certificate,
+ CertificateApprover approver) throws SCMSecurityException;
+
+ /**
+ * TODO : CRL, OCSP etc. Later. This is the start of a CertificateServer
+ * framework.
+ */
+
+ /**
+ * Approval Types for a certificate request.
+ */
+ enum CertificateApprover {
+KERBEROS_TRUSTED, /* The Request came from a DN using Kerberos Identity*/
+MANUAL, /* Wait
[17/50] [abbrv] hadoop git commit: HDDS-5. Enable OzoneManager kerberos auth. Contributed by Ajay Kumar.
HDDS-5. Enable OzoneManager kerberos auth. Contributed by Ajay Kumar.
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/2119be45
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/2119be45
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/2119be45
Branch: refs/heads/HDDS-4
Commit: 2119be450b6516fe79278870effd13d542fb3d35
Parents: aa7ca15
Author: Xiaoyu Yao
Authored: Mon May 14 09:36:57 2018 -0700
Committer: Xiaoyu Yao
Committed: Thu Nov 29 11:56:49 2018 -0800
--
.../common/src/main/resources/ozone-default.xml | 32 +++-
.../apache/hadoop/ozone/om/OMConfigKeys.java| 9 +
.../ozone/om/protocol/OzoneManagerProtocol.java | 6 +
.../om/protocolPB/OzoneManagerProtocolPB.java | 4 +
.../hadoop/ozone/MiniOzoneClusterImpl.java | 3 +-
.../hadoop/ozone/TestSecureOzoneCluster.java| 168 +++
.../apache/hadoop/ozone/om/OzoneManager.java| 69 +++-
.../hadoop/ozone/om/OzoneManagerHttpServer.java | 5 +-
8 files changed, 246 insertions(+), 50 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/hadoop/blob/2119be45/hadoop-hdds/common/src/main/resources/ozone-default.xml
--
diff --git a/hadoop-hdds/common/src/main/resources/ozone-default.xml
b/hadoop-hdds/common/src/main/resources/ozone-default.xml
index a51443c..d3e352b 100644
--- a/hadoop-hdds/common/src/main/resources/ozone-default.xml
+++ b/hadoop-hdds/common/src/main/resources/ozone-default.xml
@@ -1400,7 +1400,23 @@
ozone.scm.kerberos.principal
OZONE, SECURITY
-The SCM service principal. Ex
scm/_h...@realm.tld.
+The SCM service principal. Ex
scm/_h...@realm.com
+
+
+
+ozone.om.kerberos.keytab.file
+
+ OZONE, SECURITY
+ The keytab file used by KSM daemon to login as its
+ service principal. The principal name is configured with
+ hdds.ksm.kerberos.principal.
+
+
+
+ozone.om.kerberos.principal
+
+ OZONE, SECURITY
+The KSM service principal. Ex
ksm/_h...@realm.com
@@ -1412,4 +1428,18 @@
/etc/security/keytabs/HTTP.keytab
+
+ozone.om.http.kerberos.principal
+HTTP/_h...@example.com
+
+ KSM http server kerberos principal.
+
+
+
+ozone.om.http.kerberos.keytab.file
+/etc/security/keytabs/HTTP.keytab
+
+ KSM http server kerberos keytab.
+
+
http://git-wip-us.apache.org/repos/asf/hadoop/blob/2119be45/hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/OMConfigKeys.java
--
diff --git
a/hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/OMConfigKeys.java
b/hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/OMConfigKeys.java
index 739c75e..0119eb5 100644
---
a/hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/OMConfigKeys.java
+++
b/hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/OMConfigKeys.java
@@ -86,4 +86,13 @@ public final class OMConfigKeys {
public static final String OZONE_OM_METRICS_SAVE_INTERVAL =
"ozone.om.save.metrics.interval";
public static final String OZONE_OM_METRICS_SAVE_INTERVAL_DEFAULT = "5m";
+
+ public static final String OZONE_OM_KERBEROS_KEYTAB_FILE_KEY = "ozone.om."
+ + "kerberos.keytab.file";
+ public static final String OZONE_OM_KERBEROS_PRINCIPAL_KEY = "ozone.om"
+ + ".kerberos.principal";
+ public static final String OZONE_OM_WEB_AUTHENTICATION_KERBEROS_KEYTAB_FILE =
+ "ozone.om.http.kerberos.keytab.file";
+ public static final String OZONE_OM_WEB_AUTHENTICATION_KERBEROS_PRINCIPAL_KEY
+ = "ozone.om.http.kerberos.principal";
}
http://git-wip-us.apache.org/repos/asf/hadoop/blob/2119be45/hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/protocol/OzoneManagerProtocol.java
--
diff --git
a/hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/protocol/OzoneManagerProtocol.java
b/hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/protocol/OzoneManagerProtocol.java
index e4cce65..2a4e864 100644
---
a/hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/protocol/OzoneManagerProtocol.java
+++
b/hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/protocol/OzoneManagerProtocol.java
@@ -17,6 +17,7 @@
*/
package org.apache.hadoop.ozone.om.protocol;
+import org.apache.hadoop.ozone.om.OMConfigKeys;
import org.apache.hadoop.ozone.om.helpers.OmBucketArgs;
import org.apache.hadoop.ozone.om.helpers.OmBucketInfo;
import org.apache.hadoop.ozone.om.helpers.OmKeyArgs;
@@ -25,14 +26,19 @@ import org.apache.hadoop.ozone.om.helpers.OmKeyLocationInfo;
import
[38/50] [abbrv] hadoop git commit: HDDS-9. Add GRPC protocol interceptors for Ozone Block Token. Contributed by Xiaoyu Yao.
HDDS-9. Add GRPC protocol interceptors for Ozone Block Token. Contributed by
Xiaoyu Yao.
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/96bd574d
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/96bd574d
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/96bd574d
Branch: refs/heads/HDDS-4
Commit: 96bd574dc9104ce017c889dea93792ac00900f01
Parents: 59767be
Author: Xiaoyu Yao
Authored: Tue Nov 20 20:21:08 2018 -0800
Committer: Xiaoyu Yao
Committed: Thu Nov 29 11:57:47 2018 -0800
--
.../hdds/scm/ClientCredentialInterceptor.java | 65
.../hadoop/hdds/scm/XceiverClientGrpc.java | 64 +++-
.../org/apache/hadoop/hdds/HddsConfigKeys.java | 5 +-
.../exception/SCMSecurityException.java | 52 +++
.../hdds/security/exception/package-info.java | 23 ++
.../security/token/BlockTokenException.java | 53
.../hdds/security/token/BlockTokenVerifier.java | 113 +++
.../token/OzoneBlockTokenIdentifier.java| 199
.../security/token/OzoneBlockTokenSelector.java | 55
.../hdds/security/token/TokenVerifier.java | 38 +++
.../hdds/security/token/package-info.java | 22 ++
.../hdds/security/x509/SecurityConfig.java | 11 +
.../authority/CertificateServer.java| 2 +-
.../certificate/client/CertificateClient.java | 11 +
.../certificates/CertificateSignRequest.java| 2 +-
.../certificates/SelfSignedCertificate.java | 2 +-
.../x509/exceptions/CertificateException.java | 14 +-
.../x509/exceptions/SCMSecurityException.java | 64
.../org/apache/hadoop/ozone/OzoneConsts.java| 14 +
hadoop-hdds/common/src/main/proto/hdds.proto| 2 +-
.../token/TestOzoneBlockTokenIdentifier.java| 313 +++
.../hdds/security/token/package-info.java | 22 ++
.../TestCertificateSignRequest.java | 2 +-
.../x509/certificates/TestRootCertificate.java | 2 +-
.../server/ServerCredentialInterceptor.java | 74 +
.../transport/server/XceiverServerGrpc.java | 23 +-
.../security/OzoneBlockTokenIdentifier.java | 178 ---
.../ozone/security/OzoneBlockTokenSelector.java | 55
.../ozoneimpl/TestSecureOzoneContainer.java | 209 +
.../security/TestOzoneBlockTokenIdentifier.java | 255 ---
30 files changed, 1363 insertions(+), 581 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/hadoop/blob/96bd574d/hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/ClientCredentialInterceptor.java
--
diff --git
a/hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/ClientCredentialInterceptor.java
b/hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/ClientCredentialInterceptor.java
new file mode 100644
index 000..7a15808
--- /dev/null
+++
b/hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/ClientCredentialInterceptor.java
@@ -0,0 +1,65 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hdds.scm;
+
+import org.apache.ratis.thirdparty.io.grpc.CallOptions;
+import org.apache.ratis.thirdparty.io.grpc.Channel;
+import org.apache.ratis.thirdparty.io.grpc.ClientCall;
+import org.apache.ratis.thirdparty.io.grpc.ClientInterceptor;
+import org.apache.ratis.thirdparty.io.grpc.ForwardingClientCall;
+import org.apache.ratis.thirdparty.io.grpc.Metadata;
+import org.apache.ratis.thirdparty.io.grpc.MethodDescriptor;
+
+import static org.apache.hadoop.ozone.OzoneConsts.OBT_METADATA_KEY;
+import static org.apache.hadoop.ozone.OzoneConsts.USER_METADATA_KEY;
+
+/**
+ * GRPC client interceptor for ozone block token.
+ */
+public class ClientCredentialInterceptor implements ClientInterceptor {
+
+ private final String user;
+ private final String token;
+
+ public ClientCredentialInterceptor(String user, String token) {
+this.user = user;
+this.token = token;
+ }
+
+ @Override
+ public ClientCall
[39/50] [abbrv] hadoop git commit: HDDS-103. SCM CA: Add new security protocol for SCM to expose security related functions. Contributed by Ajay Kumar.
HDDS-103. SCM CA: Add new security protocol for SCM to expose security related
functions. Contributed by Ajay Kumar.
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/d0be8650
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/d0be8650
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/d0be8650
Branch: refs/heads/HDDS-4
Commit: d0be86504421594ed70080ad6bc4faba52c8e437
Parents: 1842ced
Author: Ajay Kumar
Authored: Sun Oct 28 22:44:41 2018 -0700
Committer: Xiaoyu Yao
Committed: Thu Nov 29 11:57:47 2018 -0800
--
hadoop-hdds/common/pom.xml | 1 +
.../hdds/protocol/SCMSecurityProtocol.java | 44 +++
...MSecurityProtocolClientSideTranslatorPB.java | 99 +++
.../hdds/protocolPB/SCMSecurityProtocolPB.java | 35 ++
...MSecurityProtocolServerSideTranslatorPB.java | 66 ++
.../hadoop/hdds/protocolPB/package-info.java| 22
.../apache/hadoop/hdds/scm/ScmConfigKeys.java | 16 +++
.../apache/hadoop/ozone/OzoneSecurityUtil.java | 43 +++
.../src/main/proto/SCMSecurityProtocol.proto| 66 ++
.../apache/hadoop/hdds/scm/HddsServerUtil.java | 22
.../hdds/scm/server/SCMBlockProtocolServer.java | 2 +-
.../scm/server/SCMSecurityProtocolServer.java | 121 +++
.../scm/server/StorageContainerManager.java | 26 +++-
13 files changed, 558 insertions(+), 5 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/hadoop/blob/d0be8650/hadoop-hdds/common/pom.xml
--
diff --git a/hadoop-hdds/common/pom.xml b/hadoop-hdds/common/pom.xml
index 061158c..25774f8 100644
--- a/hadoop-hdds/common/pom.xml
+++ b/hadoop-hdds/common/pom.xml
@@ -239,6 +239,7 @@ http://maven.apache.org/xsd/maven-4.0.0.xsd;>
StorageContainerLocationProtocol.proto
hdds.proto
ScmBlockLocationProtocol.proto
+ SCMSecurityProtocol.proto
http://git-wip-us.apache.org/repos/asf/hadoop/blob/d0be8650/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/protocol/SCMSecurityProtocol.java
--
diff --git
a/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/protocol/SCMSecurityProtocol.java
b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/protocol/SCMSecurityProtocol.java
new file mode 100644
index 000..f0ae41c
--- /dev/null
+++
b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/protocol/SCMSecurityProtocol.java
@@ -0,0 +1,44 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations
under
+ * the License.
+ */
+package org.apache.hadoop.hdds.protocol;
+
+import java.io.IOException;
+import org.apache.hadoop.classification.InterfaceAudience;
+import org.apache.hadoop.hdds.protocol.proto.HddsProtos.DatanodeDetailsProto;
+import org.apache.hadoop.hdds.scm.ScmConfigKeys;
+import org.apache.hadoop.security.KerberosInfo;
+
+/**
+ * The protocol used to perform security related operations with SCM.
+ */
+@KerberosInfo(
+serverPrincipal = ScmConfigKeys.HDDS_SCM_KERBEROS_PRINCIPAL_KEY)
+@InterfaceAudience.Private
+public interface SCMSecurityProtocol {
+
+ /**
+ * Get SCM signed certificate for DataNode.
+ *
+ * @param dataNodeDetails - DataNode Details.
+ * @param certSignReq - Certificate signing request.
+ * @return byte[] - SCM signed certificate.
+ */
+ String getDataNodeCertificate(
+ DatanodeDetailsProto dataNodeDetails,
+ String certSignReq) throws IOException;
+
+}
http://git-wip-us.apache.org/repos/asf/hadoop/blob/d0be8650/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/protocolPB/SCMSecurityProtocolClientSideTranslatorPB.java
--
diff --git
a/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/protocolPB/SCMSecurityProtocolClientSideTranslatorPB.java
[18/50] [abbrv] hadoop git commit: HDDS-6. Enable SCM kerberos auth. Contributed by Ajay Kumar.
HDDS-6. Enable SCM kerberos auth. Contributed by Ajay Kumar.
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/aa7ca153
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/aa7ca153
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/aa7ca153
Branch: refs/heads/HDDS-4
Commit: aa7ca153d8a9c9c9381f29f9ed739a56416db015
Parents: ae5fbdd
Author: Xiaoyu Yao
Authored: Wed May 9 15:56:03 2018 -0700
Committer: Xiaoyu Yao
Committed: Thu Nov 29 11:56:49 2018 -0800
--
.../authentication/util/KerberosUtil.java | 2 +-
.../conf/TestConfigurationFieldsBase.java | 2 +
.../java/org/apache/hadoop/hdds/HddsUtils.java | 13 +-
.../apache/hadoop/hdds/scm/ScmConfigKeys.java | 9 +-
.../scm/protocol/ScmBlockLocationProtocol.java | 3 +
.../StorageContainerLocationProtocol.java | 4 +
.../protocolPB/ScmBlockLocationProtocolPB.java | 6 +
.../StorageContainerLocationProtocolPB.java | 4 +
.../apache/hadoop/ozone/OzoneConfigKeys.java| 4 +
.../common/src/main/resources/ozone-default.xml | 26 ++-
.../StorageContainerDatanodeProtocol.java | 4 +
.../StorageContainerDatanodeProtocolPB.java | 6 +
.../scm/server/StorageContainerManager.java | 51 -
.../StorageContainerManagerHttpServer.java | 5 +-
.../hadoop/hdds/scm/block/TestBlockManager.java | 3 +-
.../ozone/client/protocol/ClientProtocol.java | 3 +
hadoop-ozone/common/src/main/bin/start-ozone.sh | 16 +-
hadoop-ozone/common/src/main/bin/stop-ozone.sh | 13 +-
hadoop-ozone/integration-test/pom.xml | 6 +
.../TestContainerStateManagerIntegration.java | 5 +-
.../apache/hadoop/ozone/MiniOzoneCluster.java | 4 +-
.../hadoop/ozone/MiniOzoneClusterImpl.java | 21 +-
.../hadoop/ozone/TestSecureOzoneCluster.java| 205 +++
.../ozone/TestStorageContainerManager.java | 8 +-
24 files changed, 375 insertions(+), 48 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/hadoop/blob/aa7ca153/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/util/KerberosUtil.java
--
diff --git
a/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/util/KerberosUtil.java
b/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/util/KerberosUtil.java
index c011045..4459928 100644
---
a/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/util/KerberosUtil.java
+++
b/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/util/KerberosUtil.java
@@ -167,7 +167,7 @@ public class KerberosUtil {
}
/* Return fqdn of the current host */
- static String getLocalHostName() throws UnknownHostException {
+ public static String getLocalHostName() throws UnknownHostException {
return InetAddress.getLocalHost().getCanonicalHostName();
}
http://git-wip-us.apache.org/repos/asf/hadoop/blob/aa7ca153/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/TestConfigurationFieldsBase.java
--
diff --git
a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/TestConfigurationFieldsBase.java
b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/TestConfigurationFieldsBase.java
index 152159b..bce1cd5 100644
---
a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/TestConfigurationFieldsBase.java
+++
b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/TestConfigurationFieldsBase.java
@@ -436,6 +436,8 @@ public abstract class TestConfigurationFieldsBase {
// Create XML key/value map
LOG_XML.debug("Reading XML property files\n");
xmlKeyValueMap = extractPropertiesFromXml(xmlFilename);
+// Remove hadoop property set in ozone-default.xml
+xmlKeyValueMap.remove("hadoop.custom.tags");
LOG_XML.debug("\n=\n");
// Create default configuration variable key/value map
http://git-wip-us.apache.org/repos/asf/hadoop/blob/aa7ca153/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/HddsUtils.java
--
diff --git
a/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/HddsUtils.java
b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/HddsUtils.java
index 18637af..fd6a0e3 100644
--- a/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/HddsUtils.java
+++ b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/HddsUtils.java
@@ -278,18 +278,7 @@ public final class HddsUtils {
}
public static boolean
[23/50] [abbrv] hadoop git commit: HDDS-101. SCM CA: generate CSR for SCM CA clients. Contributed by Xiaoyu Yao.
HDDS-101. SCM CA: generate CSR for SCM CA clients.
Contributed by Xiaoyu Yao.
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/1842ced1
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/1842ced1
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/1842ced1
Branch: refs/heads/HDDS-4
Commit: 1842ced1268acdf59accb54aec664043bf07da73
Parents: 7246898
Author: Anu Engineer
Authored: Fri Oct 26 17:57:21 2018 -0700
Committer: Xiaoyu Yao
Committed: Thu Nov 29 11:57:46 2018 -0800
--
.../certificates/CertificateSignRequest.java| 245 +
.../security/x509/keys/HDDSKeyGenerator.java| 2 +-
.../hdds/security/x509/keys/SecurityUtil.java | 79 ++
.../TestCertificateSignRequest.java | 268 +++
4 files changed, 593 insertions(+), 1 deletion(-)
--
http://git-wip-us.apache.org/repos/asf/hadoop/blob/1842ced1/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/security/x509/certificates/CertificateSignRequest.java
--
diff --git
a/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/security/x509/certificates/CertificateSignRequest.java
b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/security/x509/certificates/CertificateSignRequest.java
new file mode 100644
index 000..2e1f9df
--- /dev/null
+++
b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/security/x509/certificates/CertificateSignRequest.java
@@ -0,0 +1,245 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+package org.apache.hadoop.hdds.security.x509.certificates;
+
+import com.google.common.base.Preconditions;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hdds.security.x509.SecurityConfig;
+import org.apache.hadoop.hdds.security.x509.exceptions.CertificateException;
+import org.apache.hadoop.hdds.security.x509.exceptions.SCMSecurityException;
+import org.apache.hadoop.hdds.security.x509.keys.SecurityUtil;
+import org.apache.logging.log4j.util.Strings;
+import org.bouncycastle.asn1.DEROctetString;
+import org.bouncycastle.asn1.pkcs.PKCSObjectIdentifiers;
+import org.bouncycastle.asn1.x500.X500Name;
+import org.bouncycastle.asn1.x509.BasicConstraints;
+import org.bouncycastle.asn1.x509.Extension;
+import org.bouncycastle.asn1.x509.Extensions;
+import org.bouncycastle.asn1.x509.GeneralName;
+import org.bouncycastle.asn1.x509.GeneralNames;
+import org.bouncycastle.asn1.x509.KeyUsage;
+import org.bouncycastle.operator.ContentSigner;
+import org.bouncycastle.operator.OperatorCreationException;
+import org.bouncycastle.operator.jcajce.JcaContentSignerBuilder;
+import org.bouncycastle.pkcs.PKCS10CertificationRequest;
+import org.bouncycastle.pkcs.PKCS10CertificationRequestBuilder;
+import org.bouncycastle.pkcs.jcajce.JcaPKCS10CertificationRequestBuilder;
+
+import java.io.IOException;
+import java.security.KeyPair;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Optional;
+
+/**
+ * A certificate sign request object that wraps operations to build a
+ * PKCS10CertificationRequest to CA.
+ */
+public final class CertificateSignRequest {
+ private final KeyPair keyPair;
+ private final SecurityConfig config;
+ private final Extensions extensions;
+ private String subject;
+ private String clusterID;
+ private String scmID;
+
+ /**
+ * Private Ctor for CSR.
+ *
+ * @param subject - Subject
+ * @param scmID - SCM ID
+ * @param clusterID - Cluster ID
+ * @param keyPair - KeyPair
+ * @param config - SCM Config
+ * @param extensions - CSR extensions
+ */
+ private CertificateSignRequest(String subject, String scmID, String
clusterID,
+ KeyPair keyPair, SecurityConfig config, Extensions extensions) {
+this.subject = subject;
+this.clusterID = clusterID;
+this.scmID = scmID;
+this.keyPair = keyPair;
+this.config = config;
+this.extensions = extensions;
+ }
+
+ private PKCS10CertificationRequest generateCSR() throws
+
[34/50] [abbrv] hadoop git commit: HDDS-591. Adding ASF license header to kadm5.acl. Contributed by Ajay Kumar.
HDDS-591. Adding ASF license header to kadm5.acl. Contributed by Ajay Kumar. Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/b4c0280d Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/b4c0280d Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/b4c0280d Branch: refs/heads/HDDS-4 Commit: b4c0280d1126eca2190dc40988695f7746429899 Parents: 602aa80 Author: Xiaoyu Yao Authored: Wed Oct 10 10:01:01 2018 -0700 Committer: Xiaoyu Yao Committed: Thu Nov 29 11:57:46 2018 -0800 -- .../docker-image/docker-krb5/Dockerfile-krb5 | 1 + .../docker-image/docker-krb5/kadm5.acl | 19 +++ 2 files changed, 20 insertions(+) -- http://git-wip-us.apache.org/repos/asf/hadoop/blob/b4c0280d/hadoop-ozone/dist/src/main/compose/ozonesecure/docker-image/docker-krb5/Dockerfile-krb5 -- diff --git a/hadoop-ozone/dist/src/main/compose/ozonesecure/docker-image/docker-krb5/Dockerfile-krb5 b/hadoop-ozone/dist/src/main/compose/ozonesecure/docker-image/docker-krb5/Dockerfile-krb5 index b5b931d..14532d4 100644 --- a/hadoop-ozone/dist/src/main/compose/ozonesecure/docker-image/docker-krb5/Dockerfile-krb5 +++ b/hadoop-ozone/dist/src/main/compose/ozonesecure/docker-image/docker-krb5/Dockerfile-krb5 @@ -28,6 +28,7 @@ RUN kdb5_util create -s -P Welcome1 RUN kadmin.local -q "addprinc -randkey admin/ad...@example.com" RUN kadmin.local -q "ktadd -k /tmp/admin.keytab admin/ad...@example.com" ADD launcher.sh . +RUN chmod +x /opt/launcher.sh RUN mkdir -p /data ENTRYPOINT ["/usr/local/bin/dumb-init", "--", "/opt/launcher.sh"] http://git-wip-us.apache.org/repos/asf/hadoop/blob/b4c0280d/hadoop-ozone/dist/src/main/compose/ozonesecure/docker-image/docker-krb5/kadm5.acl -- diff --git a/hadoop-ozone/dist/src/main/compose/ozonesecure/docker-image/docker-krb5/kadm5.acl b/hadoop-ozone/dist/src/main/compose/ozonesecure/docker-image/docker-krb5/kadm5.acl index 8fe9f69..f0cd660 100644 --- a/hadoop-ozone/dist/src/main/compose/ozonesecure/docker-image/docker-krb5/kadm5.acl +++ b/hadoop-ozone/dist/src/main/compose/ozonesecure/docker-image/docker-krb5/kadm5.acl @@ -1 +1,20 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +# + */ad...@example.com x - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
[30/50] [abbrv] hadoop git commit: HDDS-547. Fix secure docker and configs. Contributed by Xiaoyu Yao.
HDDS-547. Fix secure docker and configs. Contributed by Xiaoyu Yao.
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/eddbe997
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/eddbe997
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/eddbe997
Branch: refs/heads/HDDS-4
Commit: eddbe997d86c54c21763b45de8d7f4a89a97dfa2
Parents: 87acc15
Author: Ajay Kumar
Authored: Mon Oct 1 11:03:27 2018 -0700
Committer: Xiaoyu Yao
Committed: Thu Nov 29 11:57:46 2018 -0800
--
hadoop-dist/src/main/compose/ozonesecure/.env | 18 +++
.../compose/ozonesecure/docker-compose.yaml | 57 +++
.../src/main/compose/ozonesecure/docker-config | 103 +
.../ozonesecure/docker-image/runner/Dockerfile | 39 +
.../ozonesecure/docker-image/runner/build.sh| 26
.../docker-image/runner/scripts/envtoconf.py| 115 ++
.../docker-image/runner/scripts/krb5.conf | 38 +
.../docker-image/runner/scripts/starter.sh | 100 +
.../runner/scripts/transformation.py| 150 +++
hadoop-hdds/common/pom.xml | 6 +
.../apache/hadoop/hdds/scm/ScmConfigKeys.java | 8 +-
.../apache/hadoop/ozone/OzoneConfigKeys.java| 3 -
.../common/src/main/resources/ozone-default.xml | 6 +-
.../hadoop/ozone/HddsDatanodeService.java | 30
.../StorageContainerManagerHttpServer.java | 4 +-
.../src/test/compose/compose-secure/.env| 17 ---
.../compose/compose-secure/docker-compose.yaml | 66
.../test/compose/compose-secure/docker-config | 99
.../apache/hadoop/ozone/om/OMConfigKeys.java| 4 +-
.../hadoop/ozone/TestSecureOzoneCluster.java| 11 +-
.../hadoop/ozone/om/OzoneManagerHttpServer.java | 4 +-
21 files changed, 701 insertions(+), 203 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/hadoop/blob/eddbe997/hadoop-dist/src/main/compose/ozonesecure/.env
--
diff --git a/hadoop-dist/src/main/compose/ozonesecure/.env
b/hadoop-dist/src/main/compose/ozonesecure/.env
new file mode 100644
index 000..a494004
--- /dev/null
+++ b/hadoop-dist/src/main/compose/ozonesecure/.env
@@ -0,0 +1,18 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+HDDS_VERSION=${hdds.version}
+SRC_VOLUME=../../
http://git-wip-us.apache.org/repos/asf/hadoop/blob/eddbe997/hadoop-dist/src/main/compose/ozonesecure/docker-compose.yaml
--
diff --git a/hadoop-dist/src/main/compose/ozonesecure/docker-compose.yaml
b/hadoop-dist/src/main/compose/ozonesecure/docker-compose.yaml
new file mode 100644
index 000..42ab05e
--- /dev/null
+++ b/hadoop-dist/src/main/compose/ozonesecure/docker-compose.yaml
@@ -0,0 +1,57 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+version: "3"
+services:
+ kdc:
+ image: ahadoop/kdc:v1
+ hostname: kdc
+ volumes:
+ - $SRC_VOLUME:/opt/hadoop
+ datanode:
+ image: ahadoop/runner:latest
+ volumes:
+- $SRC_VOLUME:/opt/hadoop
+ hostname: datanode
+ ports:
+- 9864
+ command: ["/opt/hadoop/bin/ozone","datanode"]
+ env_file:
+- ./docker-config
+ ozoneManager:
+
[50/50] [abbrv] hadoop git commit: HDDS-804. Block token: Add secret token manager. Contributed by Ajay Kumar.
HDDS-804. Block token: Add secret token manager. Contributed by Ajay Kumar.
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/187bbbe6
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/187bbbe6
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/187bbbe6
Branch: refs/heads/HDDS-4
Commit: 187bbbe68cc87729f327e5ed614474d1269b8d85
Parents: 87f51d2
Author: Ajay Kumar
Authored: Thu Nov 29 08:00:41 2018 -0800
Committer: Xiaoyu Yao
Committed: Thu Nov 29 11:58:55 2018 -0800
--
.../hdds/security/x509/SecurityConfig.java | 9 +
.../security/OzoneBlockTokenSecretManager.java | 191 +++
.../OzoneDelegationTokenSecretManager.java | 455 +
.../ozone/security/OzoneSecretManager.java | 498 ---
.../TestOzoneBlockTokenSecretManager.java | 146 ++
.../TestOzoneDelegationTokenSecretManager.java | 218
.../ozone/security/TestOzoneSecretManager.java | 216
.../apache/hadoop/ozone/om/OzoneManager.java| 23 +-
.../security/TestOzoneManagerBlockToken.java| 251 ++
9 files changed, 1371 insertions(+), 636 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/hadoop/blob/187bbbe6/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/security/x509/SecurityConfig.java
--
diff --git
a/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/security/x509/SecurityConfig.java
b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/security/x509/SecurityConfig.java
index ee20a21..b38ee7c 100644
---
a/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/security/x509/SecurityConfig.java
+++
b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/security/x509/SecurityConfig.java
@@ -21,6 +21,7 @@ package org.apache.hadoop.hdds.security.x509;
import com.google.common.base.Preconditions;
import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.ozone.OzoneConfigKeys;
import org.bouncycastle.jce.provider.BouncyCastleProvider;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -77,6 +78,7 @@ public class SecurityConfig {
private final Duration certDuration;
private final String x509SignatureAlgo;
private final Boolean grpcBlockTokenEnabled;
+ private final int getMaxKeyLength;
private final String certificateDir;
private final String certificateFileName;
@@ -88,6 +90,9 @@ public class SecurityConfig {
public SecurityConfig(Configuration configuration) {
Preconditions.checkNotNull(configuration, "Configuration cannot be null");
this.configuration = configuration;
+this.getMaxKeyLength = configuration.getInt(
+OzoneConfigKeys.OZONE_MAX_KEY_LEN,
+OzoneConfigKeys.OZONE_MAX_KEY_LEN_DEFAULT);
this.size = this.configuration.getInt(HDDS_KEY_LEN, HDDS_DEFAULT_KEY_LEN);
this.keyAlgo = this.configuration.get(HDDS_KEY_ALGORITHM,
HDDS_DEFAULT_KEY_ALGORITHM);
@@ -289,4 +294,8 @@ public class SecurityConfig {
throw new SecurityException("Unknown security provider:" + provider);
}
}
+
+ public int getMaxKeyLength() {
+return this.getMaxKeyLength;
+ }
}
http://git-wip-us.apache.org/repos/asf/hadoop/blob/187bbbe6/hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/security/OzoneBlockTokenSecretManager.java
--
diff --git
a/hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/security/OzoneBlockTokenSecretManager.java
b/hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/security/OzoneBlockTokenSecretManager.java
new file mode 100644
index 000..3b833cb
--- /dev/null
+++
b/hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/security/OzoneBlockTokenSecretManager.java
@@ -0,0 +1,191 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations
under
+ * the License.
+ */
+
+package org.apache.hadoop.ozone.security;
+
+import org.apache.hadoop.classification.InterfaceAudience;
+import
[43/50] [abbrv] hadoop git commit: HDDS-836. Add TokenIdentifier Ozone for delegation token and block token. Contributed by Ajay Kumar.
HDDS-836. Add TokenIdentifier Ozone for delegation token and block token.
Contributed by Ajay Kumar.
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/9c0baf0e
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/9c0baf0e
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/9c0baf0e
Branch: refs/heads/HDDS-4
Commit: 9c0baf0ea30a0dc38778c6030eca1468ed7e40ae
Parents: 4770e9d
Author: Xiaoyu Yao
Authored: Wed Nov 14 14:26:33 2018 -0800
Committer: Xiaoyu Yao
Committed: Thu Nov 29 11:57:47 2018 -0800
--
.../hdds/security/x509/keys/SecurityUtil.java | 59
hadoop-hdds/common/src/main/proto/hdds.proto| 24 ++
.../security/OzoneBlockTokenIdentifier.java | 178 +++
.../ozone/security/OzoneBlockTokenSelector.java | 55
.../security/OzoneDelegationTokenSelector.java | 52
.../hadoop/ozone/security/OzoneSecretKey.java | 195
.../ozone/security/OzoneTokenIdentifier.java| 217 ++
.../hadoop/ozone/security/package-info.java | 21 ++
.../src/main/proto/OzoneManagerProtocol.proto | 20 ++
.../security/TestOzoneBlockTokenIdentifier.java | 255
.../security/TestOzoneTokenIdentifier.java | 300 +++
.../hadoop/ozone/security/package-info.java | 21 ++
12 files changed, 1397 insertions(+)
--
http://git-wip-us.apache.org/repos/asf/hadoop/blob/9c0baf0e/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/security/x509/keys/SecurityUtil.java
--
diff --git
a/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/security/x509/keys/SecurityUtil.java
b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/security/x509/keys/SecurityUtil.java
index 2ca8825..6147d3a 100644
---
a/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/security/x509/keys/SecurityUtil.java
+++
b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/security/x509/keys/SecurityUtil.java
@@ -18,6 +18,15 @@
*/
package org.apache.hadoop.hdds.security.x509.keys;
+import java.security.KeyFactory;
+import java.security.NoSuchAlgorithmException;
+import java.security.NoSuchProviderException;
+import java.security.PrivateKey;
+import java.security.PublicKey;
+import java.security.spec.InvalidKeySpecException;
+import java.security.spec.PKCS8EncodedKeySpec;
+import java.security.spec.X509EncodedKeySpec;
+import org.apache.hadoop.hdds.security.x509.SecurityConfig;
import org.apache.hadoop.hdds.security.x509.exceptions.CertificateException;
import org.bouncycastle.asn1.ASN1ObjectIdentifier;
import org.bouncycastle.asn1.ASN1Sequence;
@@ -76,4 +85,54 @@ public final class SecurityUtil {
}
throw new CertificateException("No PKCS#9 extension found in CSR");
}
+
+ /*
+ * Returns private key created from encoded key.
+ * @return private key if successful else returns null.
+ */
+ public static PrivateKey getPrivateKey(byte[] encodedKey,
+ SecurityConfig secureConfig) {
+PrivateKey pvtKey = null;
+if (encodedKey == null || encodedKey.length == 0) {
+ return null;
+}
+
+try {
+ KeyFactory kf = null;
+
+ kf = KeyFactory.getInstance(secureConfig.getKeyAlgo(),
+ secureConfig.getProvider());
+ pvtKey = kf.generatePrivate(new PKCS8EncodedKeySpec(encodedKey));
+
+} catch (NoSuchAlgorithmException | InvalidKeySpecException |
+NoSuchProviderException e) {
+ return null;
+}
+return pvtKey;
+ }
+
+ /*
+ * Returns public key created from encoded key.
+ * @return public key if successful else returns null.
+ */
+ public static PublicKey getPublicKey(byte[] encodedKey,
+ SecurityConfig secureConfig) {
+PublicKey key = null;
+if (encodedKey == null || encodedKey.length == 0) {
+ return null;
+}
+
+try {
+ KeyFactory kf = null;
+ kf = KeyFactory.getInstance(secureConfig.getKeyAlgo(),
+ secureConfig.getProvider());
+ key = kf.generatePublic(new X509EncodedKeySpec(encodedKey));
+
+} catch (NoSuchAlgorithmException | InvalidKeySpecException |
+NoSuchProviderException e) {
+ return null;
+}
+return key;
+ }
+
}
http://git-wip-us.apache.org/repos/asf/hadoop/blob/9c0baf0e/hadoop-hdds/common/src/main/proto/hdds.proto
--
diff --git a/hadoop-hdds/common/src/main/proto/hdds.proto
b/hadoop-hdds/common/src/main/proto/hdds.proto
index cf3d6d4..8598fbf 100644
--- a/hadoop-hdds/common/src/main/proto/hdds.proto
+++ b/hadoop-hdds/common/src/main/proto/hdds.proto
@@ -189,6 +189,30 @@ message ContainerBlockID {
required int64 localID = 2;
}
+
+/**
+ * Information for the Hdds block token.
+ * When adding further fields, make sure
[12/50] [abbrv] hadoop git commit: HDDS-850. ReadStateMachineData hits OverlappingFileLockException in ContainerStateMachine. Contributed by Shashikant Banerjee.
HDDS-850. ReadStateMachineData hits OverlappingFileLockException in
ContainerStateMachine. Contributed by Shashikant Banerjee.
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/5e102f9a
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/5e102f9a
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/5e102f9a
Branch: refs/heads/HDDS-4
Commit: 5e102f9aa54d3057ef5f0755d45428f22a24990b
Parents: 7eb0d3a
Author: Shashikant Banerjee
Authored: Thu Nov 29 22:20:08 2018 +0530
Committer: Shashikant Banerjee
Committed: Thu Nov 29 22:20:08 2018 +0530
--
.../apache/hadoop/hdds/scm/ScmConfigKeys.java | 8 ++
.../apache/hadoop/ozone/OzoneConfigKeys.java| 9 ++
.../main/proto/DatanodeContainerProtocol.proto | 1 +
.../common/src/main/resources/ozone-default.xml | 8 ++
.../server/ratis/ContainerStateMachine.java | 134 +++
.../server/ratis/XceiverServerRatis.java| 14 +-
.../container/keyvalue/KeyValueHandler.java | 7 +-
.../keyvalue/impl/ChunkManagerImpl.java | 11 +-
.../keyvalue/interfaces/ChunkManager.java | 5 +-
.../keyvalue/TestChunkManagerImpl.java | 6 +-
.../common/impl/TestContainerPersistence.java | 11 +-
11 files changed, 143 insertions(+), 71 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/hadoop/blob/5e102f9a/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/ScmConfigKeys.java
--
diff --git
a/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/ScmConfigKeys.java
b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/ScmConfigKeys.java
index 6733b8e..062b101 100644
---
a/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/ScmConfigKeys.java
+++
b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/ScmConfigKeys.java
@@ -93,6 +93,14 @@ public final class ScmConfigKeys {
public static final String DFS_CONTAINER_RATIS_LOG_QUEUE_SIZE =
"dfs.container.ratis.log.queue.size";
public static final int DFS_CONTAINER_RATIS_LOG_QUEUE_SIZE_DEFAULT = 128;
+
+ // expiry interval stateMachineData cache entry inside containerStateMachine
+ public static final String
+ DFS_CONTAINER_RATIS_STATEMACHINEDATA_CACHE_EXPIRY_INTERVAL =
+ "dfs.container.ratis.statemachine.cache.expiry.interval";
+ public static final String
+ DFS_CONTAINER_RATIS_STATEMACHINEDATA_CACHE_EXPIRY_INTERVAL_DEFAULT =
+ "10s";
public static final String DFS_RATIS_CLIENT_REQUEST_TIMEOUT_DURATION_KEY =
"dfs.ratis.client.request.timeout.duration";
public static final TimeDuration
http://git-wip-us.apache.org/repos/asf/hadoop/blob/5e102f9a/hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/OzoneConfigKeys.java
--
diff --git
a/hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/OzoneConfigKeys.java
b/hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/OzoneConfigKeys.java
index 879f773..df233f7 100644
---
a/hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/OzoneConfigKeys.java
+++
b/hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/OzoneConfigKeys.java
@@ -249,6 +249,15 @@ public final class OzoneConfigKeys {
DFS_CONTAINER_RATIS_STATEMACHINEDATA_SYNC_TIMEOUT_DEFAULT =
ScmConfigKeys.DFS_CONTAINER_RATIS_STATEMACHINEDATA_SYNC_TIMEOUT_DEFAULT;
+ public static final String
+ DFS_CONTAINER_RATIS_STATEMACHINEDATA_CACHE_EXPIRY_INTERVAL =
+ ScmConfigKeys.
+ DFS_CONTAINER_RATIS_STATEMACHINEDATA_CACHE_EXPIRY_INTERVAL;
+ public static final String
+ DFS_CONTAINER_RATIS_STATEMACHINEDATA_CACHE_EXPIRY_INTERVAL_DEFAULT =
+ ScmConfigKeys.
+ DFS_CONTAINER_RATIS_STATEMACHINEDATA_CACHE_EXPIRY_INTERVAL_DEFAULT;
+
public static final String DFS_CONTAINER_RATIS_DATANODE_STORAGE_DIR =
"dfs.container.ratis.datanode.storage.dir";
public static final String DFS_RATIS_CLIENT_REQUEST_TIMEOUT_DURATION_KEY =
http://git-wip-us.apache.org/repos/asf/hadoop/blob/5e102f9a/hadoop-hdds/common/src/main/proto/DatanodeContainerProtocol.proto
--
diff --git a/hadoop-hdds/common/src/main/proto/DatanodeContainerProtocol.proto
b/hadoop-hdds/common/src/main/proto/DatanodeContainerProtocol.proto
index 3695b6b..5237af8 100644
--- a/hadoop-hdds/common/src/main/proto/DatanodeContainerProtocol.proto
+++ b/hadoop-hdds/common/src/main/proto/DatanodeContainerProtocol.proto
@@ -392,6 +392,7 @@ message WriteChunkResponseProto {
message ReadChunkRequestProto {
required DatanodeBlockID blockID = 1;
required ChunkInfo chunkData = 2;
+ optional bool readFromTmpFile = 3 [default = false];
}
message
[11/50] [abbrv] hadoop git commit: HADOOP-14927. ITestS3GuardTool failures in testDestroyNoBucket(). Contributed by Gabor Bota.
HADOOP-14927. ITestS3GuardTool failures in testDestroyNoBucket(). Contributed
by Gabor Bota.
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/7eb0d3a3
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/7eb0d3a3
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/7eb0d3a3
Branch: refs/heads/HDDS-4
Commit: 7eb0d3a32435da110dc9e6004dba8c5c9b082c35
Parents: 184cced
Author: Sean Mackrory
Authored: Wed Nov 28 16:57:12 2018 -0700
Committer: Sean Mackrory
Committed: Thu Nov 29 09:36:39 2018 -0700
--
.../hadoop/fs/s3a/s3guard/S3GuardTool.java | 38
1 file changed, 24 insertions(+), 14 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/hadoop/blob/7eb0d3a3/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/S3GuardTool.java
--
diff --git
a/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/S3GuardTool.java
b/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/S3GuardTool.java
index 1316121..aea57a6 100644
---
a/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/S3GuardTool.java
+++
b/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/S3GuardTool.java
@@ -218,6 +218,27 @@ public abstract class S3GuardTool extends Configured
implements Tool {
format.addOptionWithValue(SECONDS_FLAG);
}
+ protected void checkMetadataStoreUri(List paths) throws IOException {
+// be sure that path is provided in params, so there's no IOoBE
+String s3Path = "";
+if(!paths.isEmpty()) {
+ s3Path = paths.get(0);
+}
+
+// Check if DynamoDB url is set from arguments.
+String metadataStoreUri = getCommandFormat().getOptValue(META_FLAG);
+if(metadataStoreUri == null || metadataStoreUri.isEmpty()) {
+ // If not set, check if filesystem is guarded by creating an
+ // S3AFileSystem and check if hasMetadataStore is true
+ try (S3AFileSystem s3AFileSystem = (S3AFileSystem)
+ S3AFileSystem.newInstance(toUri(s3Path), getConf())){
+Preconditions.checkState(s3AFileSystem.hasMetadataStore(),
+"The S3 bucket is unguarded. " + getName()
++ " can not be used on an unguarded bucket.");
+ }
+}
+ }
+
/**
* Parse metadata store from command line option or HDFS configuration.
*
@@ -500,20 +521,7 @@ public abstract class S3GuardTool extends Configured
implements Tool {
public int run(String[] args, PrintStream out) throws Exception {
List paths = parseArgs(args);
Map options = new HashMap<>();
- String s3Path = paths.get(0);
-
- // Check if DynamoDB url is set from arguments.
- String metadataStoreUri = getCommandFormat().getOptValue(META_FLAG);
- if(metadataStoreUri == null || metadataStoreUri.isEmpty()) {
-// If not set, check if filesystem is guarded by creating an
-// S3AFileSystem and check if hasMetadataStore is true
-try (S3AFileSystem s3AFileSystem = (S3AFileSystem)
-S3AFileSystem.newInstance(toUri(s3Path), getConf())){
- Preconditions.checkState(s3AFileSystem.hasMetadataStore(),
- "The S3 bucket is unguarded. " + getName()
- + " can not be used on an unguarded bucket.");
-}
- }
+ checkMetadataStoreUri(paths);
String readCap = getCommandFormat().getOptValue(READ_FLAG);
if (StringUtils.isNotEmpty(readCap)) {
@@ -590,6 +598,8 @@ public abstract class S3GuardTool extends Configured
implements Tool {
throw e;
}
+ checkMetadataStoreUri(paths);
+
try {
initMetadataStore(false);
} catch (FileNotFoundException e) {
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
[15/50] [abbrv] hadoop git commit: HDDS-877. Ensure correct surefire version for Ozone test. Contributed by Xiaoyu Yao.
HDDS-877. Ensure correct surefire version for Ozone test. Contributed by Xiaoyu
Yao.
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/ae5fbdd9
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/ae5fbdd9
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/ae5fbdd9
Branch: refs/heads/HDDS-4
Commit: ae5fbdd9ed6ef09b588637f2eadd7a04e8382289
Parents: f534736
Author: Xiaoyu Yao
Authored: Thu Nov 29 11:37:36 2018 -0800
Committer: Xiaoyu Yao
Committed: Thu Nov 29 11:37:36 2018 -0800
--
hadoop-hdds/pom.xml | 1 +
hadoop-ozone/pom.xml | 3 +--
2 files changed, 2 insertions(+), 2 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/hadoop/blob/ae5fbdd9/hadoop-hdds/pom.xml
--
diff --git a/hadoop-hdds/pom.xml b/hadoop-hdds/pom.xml
index 869ecbf..5537b3a 100644
--- a/hadoop-hdds/pom.xml
+++ b/hadoop-hdds/pom.xml
@@ -53,6 +53,7 @@ http://maven.apache.org/xsd/maven-4.0.0.xsd;>
0.5.1
1.5.0.Final
+3.0.0-M1
http://git-wip-us.apache.org/repos/asf/hadoop/blob/ae5fbdd9/hadoop-ozone/pom.xml
--
diff --git a/hadoop-ozone/pom.xml b/hadoop-ozone/pom.xml
index 39c65d5..4c13bd6 100644
--- a/hadoop-ozone/pom.xml
+++ b/hadoop-ozone/pom.xml
@@ -37,8 +37,7 @@ http://maven.apache.org/xsd/maven-4.0.0.xsd;>
1.60
Badlands
${ozone.version}
-
-
+3.0.0-M1
common
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
[49/50] [abbrv] hadoop git commit: HDDS-696. Bootstrap genesis SCM(CA) with self-signed certificate. Contributed by Anu Engineer.
HDDS-696. Bootstrap genesis SCM(CA) with self-signed certificate.
Contributed by Anu Engineer.
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/87f51d23
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/87f51d23
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/87f51d23
Branch: refs/heads/HDDS-4
Commit: 87f51d23d9899e46ac31058b726b272f1a7880ba
Parents: 9b9b8e4
Author: Anu Engineer
Authored: Tue Nov 27 15:02:07 2018 -0800
Committer: Xiaoyu Yao
Committed: Thu Nov 29 11:58:54 2018 -0800
--
.../org/apache/hadoop/hdds/HddsConfigKeys.java | 57 ++-
.../hdds/security/x509/SecurityConfig.java | 84 -
.../authority/CertificateServer.java| 11 +-
.../certificate/authority/DefaultCAServer.java | 373 +++
.../certificate/client/CertificateClient.java | 18 +-
.../certificate/utils/CertificateCodec.java | 280 ++
.../x509/certificate/utils/package-info.java| 22 ++
.../certificates/CertificateSignRequest.java| 245
.../certificates/SelfSignedCertificate.java | 212 ---
.../x509/certificates/package-info.java | 22 --
.../utils/CertificateSignRequest.java | 245
.../utils/SelfSignedCertificate.java| 238
.../x509/certificates/utils/package-info.java | 22 ++
.../security/x509/keys/HDDSKeyGenerator.java| 32 +-
.../security/x509/keys/HDDSKeyPEMWriter.java| 255 -
.../hdds/security/x509/keys/KeyCodec.java | 337 +
.../apache/hadoop/ozone/common/StorageInfo.java | 2 +-
.../authority/TestDefaultCAServer.java | 118 ++
.../certificate/authority/package-info.java | 22 ++
.../certificate/utils/TestCertificateCodec.java | 218 +++
.../TestCertificateSignRequest.java | 19 +-
.../x509/certificates/TestRootCertificate.java | 46 +--
.../x509/certificates/package-info.java | 4 +-
.../x509/keys/TestHDDSKeyPEMWriter.java | 216 ---
.../hdds/security/x509/keys/TestKeyCodec.java | 216 +++
.../hadoop/utils/db/TestDBStoreBuilder.java | 5 +-
.../hadoop/ozone/TestSecureOzoneCluster.java| 15 +-
27 files changed, 2267 insertions(+), 1067 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/hadoop/blob/87f51d23/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/HddsConfigKeys.java
--
diff --git
a/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/HddsConfigKeys.java
b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/HddsConfigKeys.java
index 553f4aa..a02152d 100644
---
a/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/HddsConfigKeys.java
+++
b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/HddsConfigKeys.java
@@ -1,19 +1,18 @@
/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations
under
+ * the License.
*/
package org.apache.hadoop.hdds;
@@ -25,61 +24,45 @@ import org.apache.hadoop.utils.db.DBProfile;
*/
public final class HddsConfigKeys {
- /**
- * Do not instantiate.
- */
- private HddsConfigKeys() {
- }
-
public static final String HDDS_HEARTBEAT_INTERVAL =
"hdds.heartbeat.interval";
public static final String HDDS_HEARTBEAT_INTERVAL_DEFAULT =
"30s";
-
public static final String
[31/50] [abbrv] hadoop git commit: HDDS-10. Add kdc docker image for secure ozone cluster. Contributed by Ajay Kumar.
HDDS-10. Add kdc docker image for secure ozone cluster. Contributed by Ajay Kumar. Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/812c07ee Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/812c07ee Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/812c07ee Branch: refs/heads/HDDS-4 Commit: 812c07ee4ffe110953d0d6b4ff3c1cb0287d569b Parents: 585c344 Author: Xiaoyu Yao Authored: Thu Oct 4 13:20:09 2018 -0700 Committer: Xiaoyu Yao Committed: Thu Nov 29 11:57:46 2018 -0800 -- .../dist/src/main/compose/ozonesecure/README.md | 22 + .../compose/ozonesecure/docker-compose.yaml | 94 .../docker-image/docker-krb5/Dockerfile-krb5| 33 +++ .../docker-image/docker-krb5/README.md | 34 +++ .../docker-image/docker-krb5/kadm5.acl | 1 + .../docker-image/docker-krb5/krb5.conf | 40 + .../docker-image/docker-krb5/launcher.sh| 25 ++ 7 files changed, 210 insertions(+), 39 deletions(-) -- http://git-wip-us.apache.org/repos/asf/hadoop/blob/812c07ee/hadoop-ozone/dist/src/main/compose/ozonesecure/README.md -- diff --git a/hadoop-ozone/dist/src/main/compose/ozonesecure/README.md b/hadoop-ozone/dist/src/main/compose/ozonesecure/README.md new file mode 100644 index 000..0ce9a0a --- /dev/null +++ b/hadoop-ozone/dist/src/main/compose/ozonesecure/README.md @@ -0,0 +1,22 @@ + +# Experimental UNSECURE krb5 Kerberos container. + +Only for development. Not for production. + + Dockerfile for KDC: +* ./docker-image/docker-krb5/Dockerfile-krb5 + + Dockerfile for SCM,OM and DataNode: +* ./docker-image/runner/Dockerfile \ No newline at end of file http://git-wip-us.apache.org/repos/asf/hadoop/blob/812c07ee/hadoop-ozone/dist/src/main/compose/ozonesecure/docker-compose.yaml -- diff --git a/hadoop-ozone/dist/src/main/compose/ozonesecure/docker-compose.yaml b/hadoop-ozone/dist/src/main/compose/ozonesecure/docker-compose.yaml index 42ab05e..fab5ba9 100644 --- a/hadoop-ozone/dist/src/main/compose/ozonesecure/docker-compose.yaml +++ b/hadoop-ozone/dist/src/main/compose/ozonesecure/docker-compose.yaml @@ -16,42 +16,58 @@ version: "3" services: - kdc: - image: ahadoop/kdc:v1 - hostname: kdc - volumes: - - $SRC_VOLUME:/opt/hadoop - datanode: - image: ahadoop/runner:latest - volumes: -- $SRC_VOLUME:/opt/hadoop - hostname: datanode - ports: -- 9864 - command: ["/opt/hadoop/bin/ozone","datanode"] - env_file: -- ./docker-config - ozoneManager: - image: ahadoop/runner:latest - hostname: om - volumes: - - $SRC_VOLUME:/opt/hadoop - ports: - - 9874:9874 - environment: - ENSURE_OM_INITIALIZED: /data/metadata/ozoneManager/current/VERSION - env_file: - - ./docker-config - command: ["/opt/hadoop/bin/ozone","om"] - scm: - image: ahadoop/runner:latest - hostname: scm - volumes: - - $SRC_VOLUME:/opt/hadoop - ports: - - 9876:9876 - env_file: - - ./docker-config - environment: - ENSURE_SCM_INITIALIZED: /data/metadata/scm/current/VERSION - command: ["/opt/hadoop/bin/ozone","scm"] + kdc: +build: + context: docker-image/docker-krb5 + dockerfile: Dockerfile-krb5 + args: +buildno: 1 +hostname: kdc +volumes: +- $SRC_VOLUME:/opt/hadoop + datanode: +build: + context: docker-image/runner + dockerfile: Dockerfile + args: +buildno: 1 +volumes: +- $SRC_VOLUME:/opt/hadoop +hostname: datanode +ports: +- 9864 +command: ["/opt/hadoop/bin/ozone","datanode"] +env_file: +- docker-config + om: +build: + context: docker-image/runner + dockerfile: Dockerfile + args: +buildno: 1 +hostname: om +volumes: +- $SRC_VOLUME:/opt/hadoop +ports: +- 9874:9874 +environment: + ENSURE_OM_INITIALIZED: /data/metadata/om/current/VERSION +env_file: +- docker-config +command: ["/opt/hadoop/bin/ozone","om"] + scm: +build: + context: docker-image/runner + dockerfile: Dockerfile + args: +buildno: 1 +hostname: scm +volumes: +- $SRC_VOLUME:/opt/hadoop +ports: +- 9876:9876 +env_file: +- docker-config +environment: + ENSURE_SCM_INITIALIZED: /data/metadata/scm/current/VERSION +command: ["/opt/hadoop/bin/ozone","scm"] http://git-wip-us.apache.org/repos/asf/hadoop/blob/812c07ee/hadoop-ozone/dist/src/main/compose/ozonesecure/docker-image/docker-krb5/Dockerfile-krb5
[26/50] [abbrv] hadoop git commit: HDDS-546. Resolve bouncy castle dependency for hadoop-hdds-common. Contributed by Ajay Kumar.
HDDS-546. Resolve bouncy castle dependency for hadoop-hdds-common. Contributed by Ajay Kumar. Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/d1c6ff7e Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/d1c6ff7e Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/d1c6ff7e Branch: refs/heads/HDDS-4 Commit: d1c6ff7ea3f4f02df0d766860187a391975475d1 Parents: b802cb5 Author: Ajay Kumar Authored: Tue Sep 25 14:19:14 2018 -0700 Committer: Xiaoyu Yao Committed: Thu Nov 29 11:57:46 2018 -0800 -- hadoop-hdds/common/pom.xml | 6 +- 1 file changed, 5 insertions(+), 1 deletion(-) -- http://git-wip-us.apache.org/repos/asf/hadoop/blob/d1c6ff7e/hadoop-hdds/common/pom.xml -- diff --git a/hadoop-hdds/common/pom.xml b/hadoop-hdds/common/pom.xml index 175500a..15ae307 100644 --- a/hadoop-hdds/common/pom.xml +++ b/hadoop-hdds/common/pom.xml @@ -59,6 +59,10 @@ http://maven.apache.org/xsd/maven-4.0.0.xsd;> io.dropwizard.metrics metrics-core + + org.bouncycastle + bcprov-jdk15on + @@ -112,7 +116,7 @@ http://maven.apache.org/xsd/maven-4.0.0.xsd;> org.bouncycastle bcprov-jdk15on - 1.49 + 1.54 - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
[33/50] [abbrv] hadoop git commit: HDDS-704. Fix the Dependency convergence issue on HDDS-4. Contributed by Xiaoyu Yao.
HDDS-704. Fix the Dependency convergence issue on HDDS-4. Contributed by Xiaoyu Yao. Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/950a4733 Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/950a4733 Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/950a4733 Branch: refs/heads/HDDS-4 Commit: 950a4733ac86a0284ec46a1060ea3862f15181d3 Parents: b4c0280 Author: Xiaoyu Yao Authored: Fri Oct 19 21:09:51 2018 -0700 Committer: Xiaoyu Yao Committed: Thu Nov 29 11:57:46 2018 -0800 -- hadoop-hdds/common/pom.xml | 7 --- 1 file changed, 7 deletions(-) -- http://git-wip-us.apache.org/repos/asf/hadoop/blob/950a4733/hadoop-hdds/common/pom.xml -- diff --git a/hadoop-hdds/common/pom.xml b/hadoop-hdds/common/pom.xml index 30dcc58..061158c 100644 --- a/hadoop-hdds/common/pom.xml +++ b/hadoop-hdds/common/pom.xml @@ -112,13 +112,6 @@ http://maven.apache.org/xsd/maven-4.0.0.xsd;> commons-pool2 2.6.0 - - - org.bouncycastle - bcprov-jdk15on - 1.54 - - org.bouncycastle bcpkix-jdk15on - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
[09/50] [abbrv] hadoop git commit: YARN-9069. Fix SchedulerInfo#getSchedulerType for custom schedulers. Contributed by Bilwa S T.
YARN-9069. Fix SchedulerInfo#getSchedulerType for custom schedulers.
Contributed by Bilwa S T.
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/07142f54
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/07142f54
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/07142f54
Branch: refs/heads/HDDS-4
Commit: 07142f54a8c7f70857e99c041f3a2a5189c809b5
Parents: a68d766
Author: bibinchundatt
Authored: Thu Nov 29 22:02:59 2018 +0530
Committer: bibinchundatt
Committed: Thu Nov 29 22:02:59 2018 +0530
--
.../yarn/server/resourcemanager/webapp/dao/SchedulerInfo.java | 2 ++
1 file changed, 2 insertions(+)
--
http://git-wip-us.apache.org/repos/asf/hadoop/blob/07142f54/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/SchedulerInfo.java
--
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/SchedulerInfo.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/SchedulerInfo.java
index 163f707..ede0d15 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/SchedulerInfo.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/SchedulerInfo.java
@@ -54,6 +54,8 @@ public class SchedulerInfo {
this.schedulerName = "Fair Scheduler";
} else if (rs instanceof FifoScheduler) {
this.schedulerName = "Fifo Scheduler";
+} else {
+ this.schedulerName = rs.getClass().getSimpleName();
}
this.minAllocResource = new
ResourceInfo(rs.getMinimumResourceCapability());
this.maxAllocResource = new
ResourceInfo(rs.getMaximumResourceCapability());
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
[42/50] [abbrv] hadoop git commit: HDDS-592. Fix ozone-secure.robot test. Contributed by Ajay Kumar.
HDDS-592. Fix ozone-secure.robot test. Contributed by Ajay Kumar.
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/8bbc95ee
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/8bbc95ee
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/8bbc95ee
Branch: refs/heads/HDDS-4
Commit: 8bbc95ee536caae21f28f9513fe624955aa122a0
Parents: 9128c66
Author: Xiaoyu Yao
Authored: Tue Nov 6 16:53:04 2018 -0800
Committer: Xiaoyu Yao
Committed: Thu Nov 29 11:57:47 2018 -0800
--
.../acceptance/ozone-secure.robot | 95
.../dist/src/main/compose/ozonesecure/.env | 1 -
.../compose/ozonesecure/docker-compose.yaml | 22 ++--
.../ozonesecure/docker-image/runner/Dockerfile | 4 +-
.../main/smoketest/security/ozone-secure.robot | 111 +++
hadoop-ozone/dist/src/main/smoketest/test.sh| 2 +
6 files changed, 126 insertions(+), 109 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/hadoop/blob/8bbc95ee/hadoop-ozone/acceptance-test/src/test/robotframework/acceptance/ozone-secure.robot
--
diff --git
a/hadoop-ozone/acceptance-test/src/test/robotframework/acceptance/ozone-secure.robot
b/hadoop-ozone/acceptance-test/src/test/robotframework/acceptance/ozone-secure.robot
deleted file mode 100644
index 7fc1088..000
---
a/hadoop-ozone/acceptance-test/src/test/robotframework/acceptance/ozone-secure.robot
+++ /dev/null
@@ -1,95 +0,0 @@
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-*** Settings ***
-Documentation Smoke test to start cluster with docker-compose
environments.
-Library OperatingSystem
-Suite Setup Startup Ozone Cluster
-Suite Teardown Teardown Ozone Cluster
-
-*** Variables ***
-${COMMON_REST_HEADER} -H "x-ozone-user: bilbo" -H "x-ozone-version: v1" -H
"Date: Mon, 26 Jun 2017 04:23:30 GMT" -H "Authorization:OZONE root"
-${version}
-
-*** Test Cases ***
-
-Daemons are running
-Is daemon running om
-Is daemon running scm
-Is daemon running datanode
-Is daemon running ozone.kdc
-
-Check if datanode is connected to the scm
-Wait Until Keyword Succeeds 3min5secHave healthy datanodes 1
-
-Test rest interface
-${result} = Execute on 0 datanodecurl -i -X POST
${COMMON_RESTHEADER} "http://localhost:9880/volume1;
-Should contain ${result} 201 Created
-${result} = Execute on 0 datanodecurl -i -X POST
${COMMON_RESTHEADER} "http://localhost:9880/volume1/bucket1;
-Should contain ${result} 201 Created
-${result} = Execute on 0 datanodecurl -i -X DELETE
${COMMON_RESTHEADER} "http://localhost:9880/volume1/bucket1;
-Should contain ${result} 200 OK
-${result} = Execute on 0 datanodecurl -i -X DELETE
${COMMON_RESTHEADER} "http://localhost:9880/volume1;
-Should contain ${result} 200 OK
-
-Test ozone cli
-${result} = Execute on 1 datanodeozone oz -createVolume
o3://om/hive -user bilbo -quota 100TB -root
-Should contain ${result} Client cannot
authenticate via
-# Authenticate testuser
-Execute on 0 datanodekinit -k
testuser/datan...@example.com -t /etc/security/keytabs/testuser.keytab
-Execute on 0 datanodeozone oz -createVolume
o3://om/hive -user bilbo -quota 100TB -root
-${result} = Execute on 0 datanodeozone oz -listVolume
o3://om/ -user bilbo | grep -Ev 'Removed|WARN|DEBUG|ERROR|INFO|TRACE' | jq -r
'.[] | select(.volumeName=="hive")'
-Should contain ${result} createdOn
-Execute on 0 datanodeozone oz -updateVolume
o3://om/hive -user bill -quota 10TB
-${result} = Execute on 0
[37/50] [abbrv] hadoop git commit: HDDS-9. Add GRPC protocol interceptors for Ozone Block Token. Contributed by Xiaoyu Yao.
http://git-wip-us.apache.org/repos/asf/hadoop/blob/96bd574d/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/container/ozoneimpl/TestSecureOzoneContainer.java
--
diff --git
a/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/container/ozoneimpl/TestSecureOzoneContainer.java
b/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/container/ozoneimpl/TestSecureOzoneContainer.java
new file mode 100644
index 000..02d5e28
--- /dev/null
+++
b/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/container/ozoneimpl/TestSecureOzoneContainer.java
@@ -0,0 +1,209 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.container.ozoneimpl;
+
+import org.apache.hadoop.hdds.HddsConfigKeys;
+import org.apache.hadoop.hdds.protocol.DatanodeDetails;
+import org.apache.hadoop.hdds.protocol.datanode.proto.ContainerProtos;
+import org.apache.hadoop.hdds.protocol.proto.HddsProtos;
+import org.apache.hadoop.hdds.security.exception.SCMSecurityException;
+import org.apache.hadoop.hdds.security.token.OzoneBlockTokenIdentifier;
+import org.apache.hadoop.hdds.security.x509.SecurityConfig;
+import org.apache.hadoop.ozone.OzoneConfigKeys;
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+import org.apache.hadoop.ozone.container.ContainerTestHelper;
+import org.apache.hadoop.hdds.scm.TestUtils;
+import org.apache.hadoop.hdds.scm.XceiverClientGrpc;
+import org.apache.hadoop.hdds.scm.XceiverClientSpi;
+import org.apache.hadoop.hdds.scm.pipeline.Pipeline;
+import org.apache.hadoop.security.SecurityUtil;
+import org.apache.hadoop.security.UserGroupInformation;
+import org.apache.hadoop.security.token.Token;
+import org.apache.hadoop.test.GenericTestUtils;
+import org.apache.hadoop.util.Time;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+import org.junit.rules.Timeout;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.net.InetSocketAddress;
+import java.security.PrivilegedAction;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.EnumSet;
+import java.util.UUID;
+
+import static org.apache.hadoop.hdds.HddsConfigKeys.OZONE_METADATA_DIRS;
+import static org.apache.hadoop.hdds.scm.ScmConfigKeys.HDDS_DATANODE_DIR_KEY;
+import static org.junit.Assert.assertTrue;
+import static org.junit.Assert.fail;
+
+/**
+ * Tests ozone containers via secure grpc/netty.
+ */
+@RunWith(Parameterized.class)
+public class TestSecureOzoneContainer {
+ private static final Logger LOG = LoggerFactory.getLogger(
+ TestSecureOzoneContainer.class);
+ /**
+ * Set the timeout for every test.
+ */
+ @Rule
+ public Timeout testTimeout = new Timeout(30);
+
+ @Rule
+ public TemporaryFolder tempFolder = new TemporaryFolder();
+
+ private OzoneConfiguration conf;
+ private SecurityConfig secConfig;
+ private Boolean requireBlockToken;
+ private Boolean hasBlockToken;
+ private Boolean blockTokeExpired;
+
+
+ public TestSecureOzoneContainer(Boolean requireBlockToken,
+ Boolean hasBlockToken, Boolean blockTokenExpired) {
+this.requireBlockToken = requireBlockToken;
+this.hasBlockToken = hasBlockToken;
+this.blockTokeExpired = blockTokenExpired;
+ }
+
+ @Parameterized.Parameters
+ public static Collection blockTokenOptions() {
+return Arrays.asList(new Object[][] {
+{true, true, false},
+{true, true, true},
+{true, false, false},
+{false, true, false},
+{false, false, false}});
+ }
+
+ @Before
+ public void setup() throws IOException{
+conf = new OzoneConfiguration();
+String ozoneMetaPath =
+GenericTestUtils.getTempPath("ozoneMeta");
+conf.set(OZONE_METADATA_DIRS, ozoneMetaPath);
+
+secConfig = new SecurityConfig(conf);
+
+ }
+
+ @Test
+ public void testCreateOzoneContainer() throws Exception {
+LOG.info("Test case: requireBlockToken: {} hasBlockToken: {} " +
[06/50] [abbrv] hadoop git commit: HDDS-642. Add chill mode exit condition for pipeline availability. Contributed by Yiqun Lin.
HDDS-642. Add chill mode exit condition for pipeline availability. Contributed
by Yiqun Lin.
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/b71cc7f3
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/b71cc7f3
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/b71cc7f3
Branch: refs/heads/HDDS-4
Commit: b71cc7f33edbbf6a98d1efb330f1c748b5dd6e75
Parents: efc4d91
Author: Ajay Kumar
Authored: Wed Nov 28 17:45:46 2018 -0800
Committer: Ajay Kumar
Committed: Wed Nov 28 17:47:57 2018 -0800
--
.../org/apache/hadoop/hdds/HddsConfigKeys.java | 5 +
.../common/src/main/resources/ozone-default.xml | 9 ++
.../scm/chillmode/PipelineChillModeRule.java| 108 +++
.../hdds/scm/chillmode/SCMChillModeManager.java | 19 +++-
.../scm/server/StorageContainerManager.java | 5 +-
.../scm/chillmode/TestSCMChillModeManager.java | 81 --
6 files changed, 213 insertions(+), 14 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/hadoop/blob/b71cc7f3/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/HddsConfigKeys.java
--
diff --git
a/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/HddsConfigKeys.java
b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/HddsConfigKeys.java
index 2d28a5b..f16503e 100644
---
a/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/HddsConfigKeys.java
+++
b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/HddsConfigKeys.java
@@ -87,6 +87,11 @@ public final class HddsConfigKeys {
"hdds.scm.chillmode.min.datanode";
public static final int HDDS_SCM_CHILLMODE_MIN_DATANODE_DEFAULT = 1;
+ public static final String HDDS_SCM_CHILLMODE_PIPELINE_AVAILABILITY_CHECK =
+ "hdds.scm.chillmode.pipeline-availability.check";
+ public static final boolean
+ HDDS_SCM_CHILLMODE_PIPELINE_AVAILABILITY_CHECK_DEFAULT = false;
+
// % of containers which should have at least one reported replica
// before SCM comes out of chill mode.
public static final String HDDS_SCM_CHILLMODE_THRESHOLD_PCT =
http://git-wip-us.apache.org/repos/asf/hadoop/blob/b71cc7f3/hadoop-hdds/common/src/main/resources/ozone-default.xml
--
diff --git a/hadoop-hdds/common/src/main/resources/ozone-default.xml
b/hadoop-hdds/common/src/main/resources/ozone-default.xml
index 9f3d7e1..aa22b2b 100644
--- a/hadoop-hdds/common/src/main/resources/ozone-default.xml
+++ b/hadoop-hdds/common/src/main/resources/ozone-default.xml
@@ -1232,6 +1232,15 @@
+hdds.scm.chillmode.pipeline-availability.check
+false
+HDDS,SCM,OPERATION
+
+ Boolean value to enable pipeline availability check during SCM chill
mode.
+
+
+
+
hdds.container.action.max.limit
20
DATANODE
http://git-wip-us.apache.org/repos/asf/hadoop/blob/b71cc7f3/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/chillmode/PipelineChillModeRule.java
--
diff --git
a/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/chillmode/PipelineChillModeRule.java
b/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/chillmode/PipelineChillModeRule.java
new file mode 100644
index 000..f9a6e59
--- /dev/null
+++
b/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/chillmode/PipelineChillModeRule.java
@@ -0,0 +1,108 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hdds.scm.chillmode;
+
+import java.util.concurrent.atomic.AtomicBoolean;
+
+import
org.apache.hadoop.hdds.protocol.proto.StorageContainerDatanodeProtocolProtos.PipelineReport;
+import
org.apache.hadoop.hdds.protocol.proto.StorageContainerDatanodeProtocolProtos.PipelineReportsProto;
+import org.apache.hadoop.hdds.scm.pipeline.Pipeline;
+import org.apache.hadoop.hdds.scm.pipeline.PipelineID;
+import
[03/50] [abbrv] hadoop git commit: YARN-8975. [Submarine] Use predefined Charset object StandardCharsets.UTF_8 instead of String UTF-8. (Zhankun Tang via wangda)
YARN-8975. [Submarine] Use predefined Charset object StandardCharsets.UTF_8
instead of String UTF-8. (Zhankun Tang via wangda)
Change-Id: If6c7904aa17895e543cfca245264249eb7328bdc
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/89764392
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/89764392
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/89764392
Branch: refs/heads/HDDS-4
Commit: 897643928c534062d45d00a36b95fd99b4f6
Parents: 8ebeda9
Author: Wangda Tan
Authored: Wed Nov 28 14:39:06 2018 -0800
Committer: Wangda Tan
Committed: Wed Nov 28 14:39:06 2018 -0800
--
.../submarine/runtimes/yarnservice/YarnServiceJobSubmitter.java | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
--
http://git-wip-us.apache.org/repos/asf/hadoop/blob/89764392/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-submarine/src/main/java/org/apache/hadoop/yarn/submarine/runtimes/yarnservice/YarnServiceJobSubmitter.java
--
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-submarine/src/main/java/org/apache/hadoop/yarn/submarine/runtimes/yarnservice/YarnServiceJobSubmitter.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-submarine/src/main/java/org/apache/hadoop/yarn/submarine/runtimes/yarnservice/YarnServiceJobSubmitter.java
index b58ad77..2e84c96 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-submarine/src/main/java/org/apache/hadoop/yarn/submarine/runtimes/yarnservice/YarnServiceJobSubmitter.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-submarine/src/main/java/org/apache/hadoop/yarn/submarine/runtimes/yarnservice/YarnServiceJobSubmitter.java
@@ -49,6 +49,7 @@ import java.io.IOException;
import java.io.OutputStreamWriter;
import java.io.PrintWriter;
import java.io.Writer;
+import java.nio.charset.StandardCharsets;
import java.util.HashMap;
import java.util.HashSet;
import java.util.List;
@@ -218,7 +219,8 @@ public class YarnServiceJobSubmitter implements
JobSubmitter {
private String generateCommandLaunchScript(RunJobParameters parameters,
TaskType taskType, Component comp) throws IOException {
File file = File.createTempFile(taskType.name() + "-launch-script", ".sh");
-Writer w = new OutputStreamWriter(new FileOutputStream(file), "UTF-8");
+Writer w = new OutputStreamWriter(new FileOutputStream(file),
+StandardCharsets.UTF_8);
PrintWriter pw = new PrintWriter(w);
try {
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
[05/50] [abbrv] hadoop git commit: YARN-9067. YARN Resource Manager is running OOM because of leak of Configuration Object. Contributed by Eric Yang.
YARN-9067. YARN Resource Manager is running OOM because of leak of
Configuration Object. Contributed by Eric Yang.
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/efc4d91c
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/efc4d91c
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/efc4d91c
Branch: refs/heads/HDDS-4
Commit: efc4d91cbeab8a13f6d61cb0e56443adb2d77559
Parents: fe7dab8
Author: Weiwei Yang
Authored: Thu Nov 29 09:34:14 2018 +0800
Committer: Weiwei Yang
Committed: Thu Nov 29 09:34:14 2018 +0800
--
.../hadoop/yarn/service/webapp/ApiServer.java | 209 +++
.../hadoop/yarn/service/ServiceClientTest.java | 2 +-
.../yarn/service/client/ServiceClient.java | 1 +
3 files changed, 126 insertions(+), 86 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/hadoop/blob/efc4d91c/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-api/src/main/java/org/apache/hadoop/yarn/service/webapp/ApiServer.java
--
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-api/src/main/java/org/apache/hadoop/yarn/service/webapp/ApiServer.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-api/src/main/java/org/apache/hadoop/yarn/service/webapp/ApiServer.java
index db831ba..88aeefd 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-api/src/main/java/org/apache/hadoop/yarn/service/webapp/ApiServer.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-api/src/main/java/org/apache/hadoop/yarn/service/webapp/ApiServer.java
@@ -118,10 +118,13 @@ public class ApiServer {
@Override
public Void run() throws YarnException, IOException {
ServiceClient sc = getServiceClient();
-sc.init(YARN_CONFIG);
-sc.start();
-sc.actionBuild(service);
-sc.close();
+try {
+ sc.init(YARN_CONFIG);
+ sc.start();
+ sc.actionBuild(service);
+} finally {
+ sc.close();
+}
return null;
}
});
@@ -133,11 +136,14 @@ public class ApiServer {
@Override
public ApplicationId run() throws IOException, YarnException {
ServiceClient sc = getServiceClient();
-sc.init(YARN_CONFIG);
-sc.start();
-ApplicationId applicationId = sc.actionCreate(service);
-sc.close();
-return applicationId;
+try {
+ sc.init(YARN_CONFIG);
+ sc.start();
+ ApplicationId applicationId = sc.actionCreate(service);
+ return applicationId;
+} finally {
+ sc.close();
+}
}
});
serviceStatus.setDiagnostics("Application ID: " + applicationId);
@@ -245,29 +251,32 @@ public class ApiServer {
public Integer run() throws Exception {
int result = 0;
ServiceClient sc = getServiceClient();
-sc.init(YARN_CONFIG);
-sc.start();
-Exception stopException = null;
try {
- result = sc.actionStop(appName, destroy);
- if (result == EXIT_SUCCESS) {
-LOG.info("Successfully stopped service {}", appName);
- }
-} catch (Exception e) {
- LOG.info("Got exception stopping service", e);
- stopException = e;
-}
-if (destroy) {
- result = sc.actionDestroy(appName);
- if (result == EXIT_SUCCESS) {
-LOG.info("Successfully deleted service {}", appName);
+ sc.init(YARN_CONFIG);
+ sc.start();
+ Exception stopException = null;
+ try {
+result = sc.actionStop(appName, destroy);
+if (result == EXIT_SUCCESS) {
+ LOG.info("Successfully stopped service {}", appName);
+}
+ } catch (Exception e) {
+LOG.info("Got exception stopping service", e);
+stopException = e;
}
-} else {
- if (stopException != null) {
-throw stopException;
+ if (destroy) {
+result = sc.actionDestroy(appName);
+if (result == EXIT_SUCCESS) {
+ LOG.info("Successfully deleted service {}", appName);
+}
+ } else {
+if (stopException != null) {
+ throw stopException;
+
[04/50] [abbrv] hadoop git commit: YARN-8989. [YARN-8851] Move DockerCommandPlugin volume related APIs' invocation from DockerLinuxContainerRuntime#prepareContainer to #launchContainer. (Zhankun Tang
YARN-8989. [YARN-8851] Move DockerCommandPlugin volume related APIs' invocation
from DockerLinuxContainerRuntime#prepareContainer to #launchContainer. (Zhankun
Tang via wangda)
Change-Id: Ia6d532c687168448416dfdf46f0ac34bff20e6ca
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/fe7dab8e
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/fe7dab8e
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/fe7dab8e
Branch: refs/heads/HDDS-4
Commit: fe7dab8ef55f08cf18c2d62c782c1ab8930a5a15
Parents: 8976439
Author: Wangda Tan
Authored: Wed Nov 28 14:55:16 2018 -0800
Committer: Wangda Tan
Committed: Wed Nov 28 15:03:06 2018 -0800
--
.../runtime/DockerLinuxContainerRuntime.java| 44
.../runtime/TestDockerContainerRuntime.java | 15 ---
2 files changed, 24 insertions(+), 35 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/hadoop/blob/fe7dab8e/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/runtime/DockerLinuxContainerRuntime.java
--
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/runtime/DockerLinuxContainerRuntime.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/runtime/DockerLinuxContainerRuntime.java
index 15ff0ff..225bc19 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/runtime/DockerLinuxContainerRuntime.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/runtime/DockerLinuxContainerRuntime.java
@@ -456,32 +456,6 @@ public class DockerLinuxContainerRuntime implements
LinuxContainerRuntime {
@Override
public void prepareContainer(ContainerRuntimeContext ctx)
throws ContainerExecutionException {
-Container container = ctx.getContainer();
-
-// Create volumes when needed.
-if (nmContext != null
-&& nmContext.getResourcePluginManager().getNameToPlugins() != null) {
- for (ResourcePlugin plugin : nmContext.getResourcePluginManager()
- .getNameToPlugins().values()) {
-DockerCommandPlugin dockerCommandPlugin =
-plugin.getDockerCommandPluginInstance();
-if (dockerCommandPlugin != null) {
- DockerVolumeCommand dockerVolumeCommand =
- dockerCommandPlugin.getCreateDockerVolumeCommand(
- ctx.getContainer());
- if (dockerVolumeCommand != null) {
-runDockerVolumeCommand(dockerVolumeCommand, container);
-
-// After volume created, run inspect to make sure volume properly
-// created.
-if (dockerVolumeCommand.getSubCommand().equals(
-DockerVolumeCommand.VOLUME_CREATE_SUB_COMMAND)) {
- checkDockerVolumeCreated(dockerVolumeCommand, container);
-}
- }
-}
- }
-}
}
private void checkDockerVolumeCreated(
@@ -1034,14 +1008,30 @@ public class DockerLinuxContainerRuntime implements
LinuxContainerRuntime {
}
}
-// use plugins to update docker run command.
+// use plugins to create volume and update docker run command.
if (nmContext != null
&& nmContext.getResourcePluginManager().getNameToPlugins() != null) {
for (ResourcePlugin plugin : nmContext.getResourcePluginManager()
.getNameToPlugins().values()) {
DockerCommandPlugin dockerCommandPlugin =
plugin.getDockerCommandPluginInstance();
+
if (dockerCommandPlugin != null) {
+ // Create volumes when needed.
+ DockerVolumeCommand dockerVolumeCommand =
+ dockerCommandPlugin.getCreateDockerVolumeCommand(
+ ctx.getContainer());
+ if (dockerVolumeCommand != null) {
+runDockerVolumeCommand(dockerVolumeCommand, container);
+
+// After volume created, run inspect to make sure volume properly
+// created.
+if (dockerVolumeCommand.getSubCommand().equals(
+DockerVolumeCommand.VOLUME_CREATE_SUB_COMMAND)) {
+ checkDockerVolumeCreated(dockerVolumeCommand, container);
+}
+ }
+ // Update cmd
dockerCommandPlugin.updateDockerRunCommand(runCommand, container);
}
}
hadoop git commit: HDDS-877. Ensure correct surefire version for Ozone test. Contributed by Xiaoyu Yao.
Repository: hadoop
Updated Branches:
refs/heads/trunk f53473686 -> ae5fbdd9e
HDDS-877. Ensure correct surefire version for Ozone test. Contributed by Xiaoyu
Yao.
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/ae5fbdd9
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/ae5fbdd9
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/ae5fbdd9
Branch: refs/heads/trunk
Commit: ae5fbdd9ed6ef09b588637f2eadd7a04e8382289
Parents: f534736
Author: Xiaoyu Yao
Authored: Thu Nov 29 11:37:36 2018 -0800
Committer: Xiaoyu Yao
Committed: Thu Nov 29 11:37:36 2018 -0800
--
hadoop-hdds/pom.xml | 1 +
hadoop-ozone/pom.xml | 3 +--
2 files changed, 2 insertions(+), 2 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/hadoop/blob/ae5fbdd9/hadoop-hdds/pom.xml
--
diff --git a/hadoop-hdds/pom.xml b/hadoop-hdds/pom.xml
index 869ecbf..5537b3a 100644
--- a/hadoop-hdds/pom.xml
+++ b/hadoop-hdds/pom.xml
@@ -53,6 +53,7 @@ http://maven.apache.org/xsd/maven-4.0.0.xsd;>
0.5.1
1.5.0.Final
+3.0.0-M1
http://git-wip-us.apache.org/repos/asf/hadoop/blob/ae5fbdd9/hadoop-ozone/pom.xml
--
diff --git a/hadoop-ozone/pom.xml b/hadoop-ozone/pom.xml
index 39c65d5..4c13bd6 100644
--- a/hadoop-ozone/pom.xml
+++ b/hadoop-ozone/pom.xml
@@ -37,8 +37,7 @@ http://maven.apache.org/xsd/maven-4.0.0.xsd;>
1.60
Badlands
${ozone.version}
-
-
+3.0.0-M1
common
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
[2/2] hadoop git commit: HDFS-14095. EC: Track Erasure Coding commands in DFS statistics. Contributed by Ayush Saxena.
HDFS-14095. EC: Track Erasure Coding commands in DFS statistics. Contributed by
Ayush Saxena.
(cherry picked from commit f534736867eed962899615ca1b7eb68bcf591d17)
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/e2fa9e8c
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/e2fa9e8c
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/e2fa9e8c
Branch: refs/heads/branch-3.2
Commit: e2fa9e8cddb95789e210e0400a38a676242de968
Parents: a8f67ad
Author: Brahma Reddy Battula
Authored: Fri Nov 30 00:18:27 2018 +0530
Committer: Brahma Reddy Battula
Committed: Fri Nov 30 00:28:04 2018 +0530
--
.../hadoop/hdfs/DFSOpsCountStatistics.java | 9 +++
.../hadoop/hdfs/DistributedFileSystem.java | 18 ++
.../hadoop/hdfs/TestDistributedFileSystem.java | 63 +++-
3 files changed, 89 insertions(+), 1 deletion(-)
--
http://git-wip-us.apache.org/repos/asf/hadoop/blob/e2fa9e8c/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSOpsCountStatistics.java
--
diff --git
a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSOpsCountStatistics.java
b/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSOpsCountStatistics.java
index 3dcf13b..b9852ba 100644
---
a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSOpsCountStatistics.java
+++
b/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSOpsCountStatistics.java
@@ -41,6 +41,7 @@ public class DFSOpsCountStatistics extends StorageStatistics {
/** This is for counting distributed file system operations. */
public enum OpType {
+ADD_EC_POLICY("op_add_ec_policy"),
ALLOW_SNAPSHOT("op_allow_snapshot"),
APPEND(CommonStatisticNames.OP_APPEND),
CONCAT("op_concat"),
@@ -51,10 +52,15 @@ public class DFSOpsCountStatistics extends
StorageStatistics {
CREATE_SYM_LINK("op_create_symlink"),
DELETE(CommonStatisticNames.OP_DELETE),
DELETE_SNAPSHOT("op_delete_snapshot"),
+DISABLE_EC_POLICY("op_disable_ec_policy"),
DISALLOW_SNAPSHOT("op_disallow_snapshot"),
+ENABLE_EC_POLICY("op_enable_ec_policy"),
EXISTS(CommonStatisticNames.OP_EXISTS),
GET_BYTES_WITH_FUTURE_GS("op_get_bytes_with_future_generation_stamps"),
GET_CONTENT_SUMMARY(CommonStatisticNames.OP_GET_CONTENT_SUMMARY),
+GET_EC_CODECS("op_get_ec_codecs"),
+GET_EC_POLICY("op_get_ec_policy"),
+GET_EC_POLICIES("op_get_ec_policies"),
GET_FILE_BLOCK_LOCATIONS("op_get_file_block_locations"),
GET_FILE_CHECKSUM(CommonStatisticNames.OP_GET_FILE_CHECKSUM),
GET_FILE_LINK_STATUS("op_get_file_link_status"),
@@ -76,11 +82,13 @@ public class DFSOpsCountStatistics extends
StorageStatistics {
REMOVE_ACL(CommonStatisticNames.OP_REMOVE_ACL),
REMOVE_ACL_ENTRIES(CommonStatisticNames.OP_REMOVE_ACL_ENTRIES),
REMOVE_DEFAULT_ACL(CommonStatisticNames.OP_REMOVE_DEFAULT_ACL),
+REMOVE_EC_POLICY("op_remove_ec_policy"),
REMOVE_XATTR("op_remove_xattr"),
RENAME(CommonStatisticNames.OP_RENAME),
RENAME_SNAPSHOT("op_rename_snapshot"),
RESOLVE_LINK("op_resolve_link"),
SET_ACL(CommonStatisticNames.OP_SET_ACL),
+SET_EC_POLICY("op_set_ec_policy"),
SET_OWNER(CommonStatisticNames.OP_SET_OWNER),
SET_PERMISSION(CommonStatisticNames.OP_SET_PERMISSION),
SET_REPLICATION("op_set_replication"),
@@ -90,6 +98,7 @@ public class DFSOpsCountStatistics extends StorageStatistics {
GET_SNAPSHOT_DIFF("op_get_snapshot_diff"),
GET_SNAPSHOTTABLE_DIRECTORY_LIST("op_get_snapshottable_directory_list"),
TRUNCATE(CommonStatisticNames.OP_TRUNCATE),
+UNSET_EC_POLICY("op_unset_ec_policy"),
UNSET_STORAGE_POLICY("op_unset_storage_policy");
private static final Map SYMBOL_MAP =
http://git-wip-us.apache.org/repos/asf/hadoop/blob/e2fa9e8c/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java
--
diff --git
a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java
b/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java
index ca1546c..7dd02bd 100644
---
a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java
+++
b/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java
@@ -2845,6 +2845,8 @@ public class DistributedFileSystem extends FileSystem
*/
public void setErasureCodingPolicy(final Path path,
final String ecPolicyName) throws IOException {
+
[1/2] hadoop git commit: HDFS-14095. EC: Track Erasure Coding commands in DFS statistics. Contributed by Ayush Saxena.
Repository: hadoop
Updated Branches:
refs/heads/branch-3.1 1b731de94 -> 0b83c95ff
refs/heads/branch-3.2 a8f67ad7c -> e2fa9e8cd
HDFS-14095. EC: Track Erasure Coding commands in DFS statistics. Contributed by
Ayush Saxena.
(cherry picked from commit f534736867eed962899615ca1b7eb68bcf591d17)
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/0b83c95f
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/0b83c95f
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/0b83c95f
Branch: refs/heads/branch-3.1
Commit: 0b83c95ff43e043af58150bf7e23bf091f6d7fe7
Parents: 1b731de
Author: Brahma Reddy Battula
Authored: Fri Nov 30 00:18:27 2018 +0530
Committer: Brahma Reddy Battula
Committed: Fri Nov 30 00:27:37 2018 +0530
--
.../hadoop/hdfs/DFSOpsCountStatistics.java | 9 +++
.../hadoop/hdfs/DistributedFileSystem.java | 18 ++
.../hadoop/hdfs/TestDistributedFileSystem.java | 63 +++-
3 files changed, 89 insertions(+), 1 deletion(-)
--
http://git-wip-us.apache.org/repos/asf/hadoop/blob/0b83c95f/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSOpsCountStatistics.java
--
diff --git
a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSOpsCountStatistics.java
b/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSOpsCountStatistics.java
index 3dcf13b..b9852ba 100644
---
a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSOpsCountStatistics.java
+++
b/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSOpsCountStatistics.java
@@ -41,6 +41,7 @@ public class DFSOpsCountStatistics extends StorageStatistics {
/** This is for counting distributed file system operations. */
public enum OpType {
+ADD_EC_POLICY("op_add_ec_policy"),
ALLOW_SNAPSHOT("op_allow_snapshot"),
APPEND(CommonStatisticNames.OP_APPEND),
CONCAT("op_concat"),
@@ -51,10 +52,15 @@ public class DFSOpsCountStatistics extends
StorageStatistics {
CREATE_SYM_LINK("op_create_symlink"),
DELETE(CommonStatisticNames.OP_DELETE),
DELETE_SNAPSHOT("op_delete_snapshot"),
+DISABLE_EC_POLICY("op_disable_ec_policy"),
DISALLOW_SNAPSHOT("op_disallow_snapshot"),
+ENABLE_EC_POLICY("op_enable_ec_policy"),
EXISTS(CommonStatisticNames.OP_EXISTS),
GET_BYTES_WITH_FUTURE_GS("op_get_bytes_with_future_generation_stamps"),
GET_CONTENT_SUMMARY(CommonStatisticNames.OP_GET_CONTENT_SUMMARY),
+GET_EC_CODECS("op_get_ec_codecs"),
+GET_EC_POLICY("op_get_ec_policy"),
+GET_EC_POLICIES("op_get_ec_policies"),
GET_FILE_BLOCK_LOCATIONS("op_get_file_block_locations"),
GET_FILE_CHECKSUM(CommonStatisticNames.OP_GET_FILE_CHECKSUM),
GET_FILE_LINK_STATUS("op_get_file_link_status"),
@@ -76,11 +82,13 @@ public class DFSOpsCountStatistics extends
StorageStatistics {
REMOVE_ACL(CommonStatisticNames.OP_REMOVE_ACL),
REMOVE_ACL_ENTRIES(CommonStatisticNames.OP_REMOVE_ACL_ENTRIES),
REMOVE_DEFAULT_ACL(CommonStatisticNames.OP_REMOVE_DEFAULT_ACL),
+REMOVE_EC_POLICY("op_remove_ec_policy"),
REMOVE_XATTR("op_remove_xattr"),
RENAME(CommonStatisticNames.OP_RENAME),
RENAME_SNAPSHOT("op_rename_snapshot"),
RESOLVE_LINK("op_resolve_link"),
SET_ACL(CommonStatisticNames.OP_SET_ACL),
+SET_EC_POLICY("op_set_ec_policy"),
SET_OWNER(CommonStatisticNames.OP_SET_OWNER),
SET_PERMISSION(CommonStatisticNames.OP_SET_PERMISSION),
SET_REPLICATION("op_set_replication"),
@@ -90,6 +98,7 @@ public class DFSOpsCountStatistics extends StorageStatistics {
GET_SNAPSHOT_DIFF("op_get_snapshot_diff"),
GET_SNAPSHOTTABLE_DIRECTORY_LIST("op_get_snapshottable_directory_list"),
TRUNCATE(CommonStatisticNames.OP_TRUNCATE),
+UNSET_EC_POLICY("op_unset_ec_policy"),
UNSET_STORAGE_POLICY("op_unset_storage_policy");
private static final Map SYMBOL_MAP =
http://git-wip-us.apache.org/repos/asf/hadoop/blob/0b83c95f/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java
--
diff --git
a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java
b/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java
index 65d211c..3553428 100644
---
a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java
+++
b/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java
@@ -2846,6 +2846,8 @@ public class DistributedFileSystem extends
[1/2] hadoop git commit: HDFS-14095. EC: Track Erasure Coding commands in DFS statistics. Contributed by Ayush Saxena.
Repository: hadoop
Updated Branches:
refs/heads/branch-3.0 065a1e72f -> 689555004
refs/heads/trunk d0edd3726 -> f53473686
HDFS-14095. EC: Track Erasure Coding commands in DFS statistics. Contributed by
Ayush Saxena.
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/f5347368
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/f5347368
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/f5347368
Branch: refs/heads/trunk
Commit: f534736867eed962899615ca1b7eb68bcf591d17
Parents: d0edd37
Author: Brahma Reddy Battula
Authored: Fri Nov 30 00:18:27 2018 +0530
Committer: Brahma Reddy Battula
Committed: Fri Nov 30 00:18:27 2018 +0530
--
.../hadoop/hdfs/DFSOpsCountStatistics.java | 9 +++
.../hadoop/hdfs/DistributedFileSystem.java | 18 ++
.../hadoop/hdfs/TestDistributedFileSystem.java | 63 +++-
3 files changed, 89 insertions(+), 1 deletion(-)
--
http://git-wip-us.apache.org/repos/asf/hadoop/blob/f5347368/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSOpsCountStatistics.java
--
diff --git
a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSOpsCountStatistics.java
b/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSOpsCountStatistics.java
index 3dcf13b..b9852ba 100644
---
a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSOpsCountStatistics.java
+++
b/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSOpsCountStatistics.java
@@ -41,6 +41,7 @@ public class DFSOpsCountStatistics extends StorageStatistics {
/** This is for counting distributed file system operations. */
public enum OpType {
+ADD_EC_POLICY("op_add_ec_policy"),
ALLOW_SNAPSHOT("op_allow_snapshot"),
APPEND(CommonStatisticNames.OP_APPEND),
CONCAT("op_concat"),
@@ -51,10 +52,15 @@ public class DFSOpsCountStatistics extends
StorageStatistics {
CREATE_SYM_LINK("op_create_symlink"),
DELETE(CommonStatisticNames.OP_DELETE),
DELETE_SNAPSHOT("op_delete_snapshot"),
+DISABLE_EC_POLICY("op_disable_ec_policy"),
DISALLOW_SNAPSHOT("op_disallow_snapshot"),
+ENABLE_EC_POLICY("op_enable_ec_policy"),
EXISTS(CommonStatisticNames.OP_EXISTS),
GET_BYTES_WITH_FUTURE_GS("op_get_bytes_with_future_generation_stamps"),
GET_CONTENT_SUMMARY(CommonStatisticNames.OP_GET_CONTENT_SUMMARY),
+GET_EC_CODECS("op_get_ec_codecs"),
+GET_EC_POLICY("op_get_ec_policy"),
+GET_EC_POLICIES("op_get_ec_policies"),
GET_FILE_BLOCK_LOCATIONS("op_get_file_block_locations"),
GET_FILE_CHECKSUM(CommonStatisticNames.OP_GET_FILE_CHECKSUM),
GET_FILE_LINK_STATUS("op_get_file_link_status"),
@@ -76,11 +82,13 @@ public class DFSOpsCountStatistics extends
StorageStatistics {
REMOVE_ACL(CommonStatisticNames.OP_REMOVE_ACL),
REMOVE_ACL_ENTRIES(CommonStatisticNames.OP_REMOVE_ACL_ENTRIES),
REMOVE_DEFAULT_ACL(CommonStatisticNames.OP_REMOVE_DEFAULT_ACL),
+REMOVE_EC_POLICY("op_remove_ec_policy"),
REMOVE_XATTR("op_remove_xattr"),
RENAME(CommonStatisticNames.OP_RENAME),
RENAME_SNAPSHOT("op_rename_snapshot"),
RESOLVE_LINK("op_resolve_link"),
SET_ACL(CommonStatisticNames.OP_SET_ACL),
+SET_EC_POLICY("op_set_ec_policy"),
SET_OWNER(CommonStatisticNames.OP_SET_OWNER),
SET_PERMISSION(CommonStatisticNames.OP_SET_PERMISSION),
SET_REPLICATION("op_set_replication"),
@@ -90,6 +98,7 @@ public class DFSOpsCountStatistics extends StorageStatistics {
GET_SNAPSHOT_DIFF("op_get_snapshot_diff"),
GET_SNAPSHOTTABLE_DIRECTORY_LIST("op_get_snapshottable_directory_list"),
TRUNCATE(CommonStatisticNames.OP_TRUNCATE),
+UNSET_EC_POLICY("op_unset_ec_policy"),
UNSET_STORAGE_POLICY("op_unset_storage_policy");
private static final Map SYMBOL_MAP =
http://git-wip-us.apache.org/repos/asf/hadoop/blob/f5347368/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java
--
diff --git
a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java
b/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java
index ca1546c..7dd02bd 100644
---
a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java
+++
b/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java
@@ -2845,6 +2845,8 @@ public class DistributedFileSystem extends FileSystem
*/
public void setErasureCodingPolicy(final Path path,
final
[2/2] hadoop git commit: HDFS-14095. EC: Track Erasure Coding commands in DFS statistics. Contributed by Ayush Saxena.
HDFS-14095. EC: Track Erasure Coding commands in DFS statistics. Contributed by
Ayush Saxena.
(cherry picked from commit f534736867eed962899615ca1b7eb68bcf591d17)
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/68955500
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/68955500
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/68955500
Branch: refs/heads/branch-3.0
Commit: 6895550043ea75d54600673102b36c1bfabb69d5
Parents: 065a1e7
Author: Brahma Reddy Battula
Authored: Fri Nov 30 00:18:27 2018 +0530
Committer: Brahma Reddy Battula
Committed: Fri Nov 30 00:21:07 2018 +0530
--
.../hadoop/hdfs/DFSOpsCountStatistics.java | 9 +++
.../hadoop/hdfs/DistributedFileSystem.java | 18 ++
.../hadoop/hdfs/TestDistributedFileSystem.java | 63 +++-
3 files changed, 89 insertions(+), 1 deletion(-)
--
http://git-wip-us.apache.org/repos/asf/hadoop/blob/68955500/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSOpsCountStatistics.java
--
diff --git
a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSOpsCountStatistics.java
b/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSOpsCountStatistics.java
index 3dcf13b..b9852ba 100644
---
a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSOpsCountStatistics.java
+++
b/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSOpsCountStatistics.java
@@ -41,6 +41,7 @@ public class DFSOpsCountStatistics extends StorageStatistics {
/** This is for counting distributed file system operations. */
public enum OpType {
+ADD_EC_POLICY("op_add_ec_policy"),
ALLOW_SNAPSHOT("op_allow_snapshot"),
APPEND(CommonStatisticNames.OP_APPEND),
CONCAT("op_concat"),
@@ -51,10 +52,15 @@ public class DFSOpsCountStatistics extends
StorageStatistics {
CREATE_SYM_LINK("op_create_symlink"),
DELETE(CommonStatisticNames.OP_DELETE),
DELETE_SNAPSHOT("op_delete_snapshot"),
+DISABLE_EC_POLICY("op_disable_ec_policy"),
DISALLOW_SNAPSHOT("op_disallow_snapshot"),
+ENABLE_EC_POLICY("op_enable_ec_policy"),
EXISTS(CommonStatisticNames.OP_EXISTS),
GET_BYTES_WITH_FUTURE_GS("op_get_bytes_with_future_generation_stamps"),
GET_CONTENT_SUMMARY(CommonStatisticNames.OP_GET_CONTENT_SUMMARY),
+GET_EC_CODECS("op_get_ec_codecs"),
+GET_EC_POLICY("op_get_ec_policy"),
+GET_EC_POLICIES("op_get_ec_policies"),
GET_FILE_BLOCK_LOCATIONS("op_get_file_block_locations"),
GET_FILE_CHECKSUM(CommonStatisticNames.OP_GET_FILE_CHECKSUM),
GET_FILE_LINK_STATUS("op_get_file_link_status"),
@@ -76,11 +82,13 @@ public class DFSOpsCountStatistics extends
StorageStatistics {
REMOVE_ACL(CommonStatisticNames.OP_REMOVE_ACL),
REMOVE_ACL_ENTRIES(CommonStatisticNames.OP_REMOVE_ACL_ENTRIES),
REMOVE_DEFAULT_ACL(CommonStatisticNames.OP_REMOVE_DEFAULT_ACL),
+REMOVE_EC_POLICY("op_remove_ec_policy"),
REMOVE_XATTR("op_remove_xattr"),
RENAME(CommonStatisticNames.OP_RENAME),
RENAME_SNAPSHOT("op_rename_snapshot"),
RESOLVE_LINK("op_resolve_link"),
SET_ACL(CommonStatisticNames.OP_SET_ACL),
+SET_EC_POLICY("op_set_ec_policy"),
SET_OWNER(CommonStatisticNames.OP_SET_OWNER),
SET_PERMISSION(CommonStatisticNames.OP_SET_PERMISSION),
SET_REPLICATION("op_set_replication"),
@@ -90,6 +98,7 @@ public class DFSOpsCountStatistics extends StorageStatistics {
GET_SNAPSHOT_DIFF("op_get_snapshot_diff"),
GET_SNAPSHOTTABLE_DIRECTORY_LIST("op_get_snapshottable_directory_list"),
TRUNCATE(CommonStatisticNames.OP_TRUNCATE),
+UNSET_EC_POLICY("op_unset_ec_policy"),
UNSET_STORAGE_POLICY("op_unset_storage_policy");
private static final Map SYMBOL_MAP =
http://git-wip-us.apache.org/repos/asf/hadoop/blob/68955500/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java
--
diff --git
a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java
b/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java
index 5f63f81..4bf73fb 100644
---
a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java
+++
b/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java
@@ -2628,6 +2628,8 @@ public class DistributedFileSystem extends FileSystem
*/
public void setErasureCodingPolicy(final Path path,
final String ecPolicyName) throws IOException {
+
hadoop git commit: YARN-9067. Fixed Resource Manager resource leak via YARN service. Contributed by Eric Yang
Repository: hadoop
Updated Branches:
refs/heads/branch-3.1 65e0d6ff4 -> 1b731de94
YARN-9067. Fixed Resource Manager resource leak via YARN service.
Contributed by Eric Yang
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/1b731de9
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/1b731de9
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/1b731de9
Branch: refs/heads/branch-3.1
Commit: 1b731de94403c05743f698b6fee3dbaf1a2540a3
Parents: 65e0d6f
Author: Eric Yang
Authored: Thu Nov 29 13:50:06 2018 -0500
Committer: Eric Yang
Committed: Thu Nov 29 13:51:39 2018 -0500
--
.../hadoop/yarn/service/webapp/ApiServer.java | 194 +++
.../hadoop/yarn/service/ServiceClientTest.java | 2 +-
.../yarn/service/client/ServiceClient.java | 1 +
3 files changed, 117 insertions(+), 80 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/hadoop/blob/1b731de9/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-api/src/main/java/org/apache/hadoop/yarn/service/webapp/ApiServer.java
--
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-api/src/main/java/org/apache/hadoop/yarn/service/webapp/ApiServer.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-api/src/main/java/org/apache/hadoop/yarn/service/webapp/ApiServer.java
index c4e3317..51ad00a 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-api/src/main/java/org/apache/hadoop/yarn/service/webapp/ApiServer.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-api/src/main/java/org/apache/hadoop/yarn/service/webapp/ApiServer.java
@@ -117,10 +117,13 @@ public class ApiServer {
@Override
public Void run() throws YarnException, IOException {
ServiceClient sc = getServiceClient();
-sc.init(YARN_CONFIG);
-sc.start();
-sc.actionBuild(service);
-sc.close();
+try {
+ sc.init(YARN_CONFIG);
+ sc.start();
+ sc.actionBuild(service);
+} finally {
+ sc.close();
+}
return null;
}
});
@@ -132,11 +135,14 @@ public class ApiServer {
@Override
public ApplicationId run() throws IOException, YarnException {
ServiceClient sc = getServiceClient();
-sc.init(YARN_CONFIG);
-sc.start();
-ApplicationId applicationId = sc.actionCreate(service);
-sc.close();
-return applicationId;
+try {
+ sc.init(YARN_CONFIG);
+ sc.start();
+ ApplicationId applicationId = sc.actionCreate(service);
+ return applicationId;
+} finally {
+ sc.close();
+}
}
});
serviceStatus.setDiagnostics("Application ID: " + applicationId);
@@ -244,29 +250,32 @@ public class ApiServer {
public Integer run() throws Exception {
int result = 0;
ServiceClient sc = getServiceClient();
-sc.init(YARN_CONFIG);
-sc.start();
-Exception stopException = null;
try {
- result = sc.actionStop(appName, destroy);
- if (result == EXIT_SUCCESS) {
-LOG.info("Successfully stopped service {}", appName);
- }
-} catch (Exception e) {
- LOG.info("Got exception stopping service", e);
- stopException = e;
-}
-if (destroy) {
- result = sc.actionDestroy(appName);
- if (result == EXIT_SUCCESS) {
-LOG.info("Successfully deleted service {}", appName);
+ sc.init(YARN_CONFIG);
+ sc.start();
+ Exception stopException = null;
+ try {
+result = sc.actionStop(appName, destroy);
+if (result == EXIT_SUCCESS) {
+ LOG.info("Successfully stopped service {}", appName);
+}
+ } catch (Exception e) {
+LOG.info("Got exception stopping service", e);
+stopException = e;
}
-} else {
- if (stopException != null) {
-throw stopException;
+ if (destroy) {
+result = sc.actionDestroy(appName);
+if (result == EXIT_SUCCESS) {
+ LOG.info("Successfully deleted service {}", appName);
+}
+ } else {
+if
svn commit: r1847745 - /hadoop/common/site/main/author/src/documentation/content/xdocs/who.xml
Author: xiao Date: Thu Nov 29 18:20:01 2018 New Revision: 1847745 URL: http://svn.apache.org/viewvc?rev=1847745=rev Log: update xiao Modified: hadoop/common/site/main/author/src/documentation/content/xdocs/who.xml Modified: hadoop/common/site/main/author/src/documentation/content/xdocs/who.xml URL: http://svn.apache.org/viewvc/hadoop/common/site/main/author/src/documentation/content/xdocs/who.xml?rev=1847745=1847744=1847745=diff == --- hadoop/common/site/main/author/src/documentation/content/xdocs/who.xml (original) +++ hadoop/common/site/main/author/src/documentation/content/xdocs/who.xml Thu Nov 29 18:20:01 2018 @@ -626,8 +626,8 @@ xiao Xiao Chen - Cloudera - HDFS + Netflix + -8 @@ -1719,8 +1719,8 @@ xiao Xiao Chen - Cloudera - HDFS + Netflix + -8 - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
[1/4] hadoop git commit: HADOOP-15959. Revert "HADOOP-12751. While using kerberos Hadoop incorrectly assumes names with '@' to be non-simple"
Repository: hadoop
Updated Branches:
refs/heads/branch-3.0 b42c2679a -> 065a1e72f
refs/heads/branch-3.1 a7d3f22b4 -> 65e0d6ff4
refs/heads/branch-3.2 183ec39c4 -> 1a448565a
refs/heads/trunk 5e102f9aa -> d0edd3726
HADOOP-15959. Revert "HADOOP-12751. While using kerberos Hadoop incorrectly
assumes names with '@' to be non-simple"
This reverts commit 829a2e4d271f05afb209ddc834cd4a0e85492eda.
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/d0edd372
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/d0edd372
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/d0edd372
Branch: refs/heads/trunk
Commit: d0edd37269bb40290b409d583bcf3b70897c13e0
Parents: 5e102f9
Author: Steve Loughran
Authored: Thu Nov 29 17:52:11 2018 +
Committer: Steve Loughran
Committed: Thu Nov 29 17:52:11 2018 +
--
.../authentication/util/KerberosName.java | 9 ++--
.../TestKerberosAuthenticationHandler.java | 7 ++-
.../authentication/util/TestKerberosName.java | 17 ++--
.../java/org/apache/hadoop/security/KDiag.java | 46 +---
.../src/site/markdown/SecureMode.md | 6 ---
.../org/apache/hadoop/security/TestKDiag.java | 16 ---
.../security/TestUserGroupInformation.java | 27
7 files changed, 33 insertions(+), 95 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/hadoop/blob/d0edd372/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/util/KerberosName.java
--
diff --git
a/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/util/KerberosName.java
b/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/util/KerberosName.java
index 4e7ee3c..287bb13 100644
---
a/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/util/KerberosName.java
+++
b/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/util/KerberosName.java
@@ -324,8 +324,8 @@ public class KerberosName {
}
}
if (result != null && nonSimplePattern.matcher(result).find()) {
-LOG.info("Non-simple name {} after auth_to_local rule {}",
-result, this);
+throw new NoMatchingRule("Non-simple name " + result +
+ " after auth_to_local rule " + this);
}
if (toLowerCase && result != null) {
result = result.toLowerCase(Locale.ENGLISH);
@@ -378,7 +378,7 @@ public class KerberosName {
/**
* Get the translation of the principal name into an operating system
* user name.
- * @return the user name
+ * @return the short name
* @throws IOException throws if something is wrong with the rules
*/
public String getShortName() throws IOException {
@@ -398,8 +398,7 @@ public class KerberosName {
return result;
}
}
-LOG.info("No auth_to_local rules applied to {}", this);
-return toString();
+throw new NoMatchingRule("No rules applied to " + toString());
}
/**
http://git-wip-us.apache.org/repos/asf/hadoop/blob/d0edd372/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/server/TestKerberosAuthenticationHandler.java
--
diff --git
a/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/server/TestKerberosAuthenticationHandler.java
b/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/server/TestKerberosAuthenticationHandler.java
index e672391..8b4bc15 100644
---
a/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/server/TestKerberosAuthenticationHandler.java
+++
b/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/server/TestKerberosAuthenticationHandler.java
@@ -108,7 +108,12 @@ public class TestKerberosAuthenticationHandler
kn = new KerberosName("bar@BAR");
Assert.assertEquals("bar", kn.getShortName());
kn = new KerberosName("bar@FOO");
-Assert.assertEquals("bar@FOO", kn.getShortName());
+try {
+ kn.getShortName();
+ Assert.fail();
+}
+catch (Exception ex) {
+}
}
@Test(timeout=6)
http://git-wip-us.apache.org/repos/asf/hadoop/blob/d0edd372/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/util/TestKerberosName.java
--
diff --git
[4/4] hadoop git commit: HADOOP-15959. Revert "HADOOP-12751. While using kerberos Hadoop incorrectly assumes names with '@' to be non-simple"
HADOOP-15959. Revert "HADOOP-12751. While using kerberos Hadoop incorrectly
assumes names with '@' to be non-simple"
This reverts commit 829a2e4d271f05afb209ddc834cd4a0e85492eda.
(cherry picked from commit d0edd37269bb40290b409d583bcf3b70897c13e0)
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/065a1e72
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/065a1e72
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/065a1e72
Branch: refs/heads/branch-3.0
Commit: 065a1e72f268766a76af43103fcf39532a7873e6
Parents: b42c267
Author: Steve Loughran
Authored: Thu Nov 29 17:58:15 2018 +
Committer: Steve Loughran
Committed: Thu Nov 29 17:58:15 2018 +
--
.../authentication/util/KerberosName.java | 9 ++--
.../TestKerberosAuthenticationHandler.java | 7 ++-
.../authentication/util/TestKerberosName.java | 17 ++--
.../java/org/apache/hadoop/security/KDiag.java | 46 +---
.../src/site/markdown/SecureMode.md | 6 ---
.../org/apache/hadoop/security/TestKDiag.java | 16 ---
.../security/TestUserGroupInformation.java | 27
7 files changed, 33 insertions(+), 95 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/hadoop/blob/065a1e72/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/util/KerberosName.java
--
diff --git
a/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/util/KerberosName.java
b/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/util/KerberosName.java
index 4e7ee3c..287bb13 100644
---
a/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/util/KerberosName.java
+++
b/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/util/KerberosName.java
@@ -324,8 +324,8 @@ public class KerberosName {
}
}
if (result != null && nonSimplePattern.matcher(result).find()) {
-LOG.info("Non-simple name {} after auth_to_local rule {}",
-result, this);
+throw new NoMatchingRule("Non-simple name " + result +
+ " after auth_to_local rule " + this);
}
if (toLowerCase && result != null) {
result = result.toLowerCase(Locale.ENGLISH);
@@ -378,7 +378,7 @@ public class KerberosName {
/**
* Get the translation of the principal name into an operating system
* user name.
- * @return the user name
+ * @return the short name
* @throws IOException throws if something is wrong with the rules
*/
public String getShortName() throws IOException {
@@ -398,8 +398,7 @@ public class KerberosName {
return result;
}
}
-LOG.info("No auth_to_local rules applied to {}", this);
-return toString();
+throw new NoMatchingRule("No rules applied to " + toString());
}
/**
http://git-wip-us.apache.org/repos/asf/hadoop/blob/065a1e72/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/server/TestKerberosAuthenticationHandler.java
--
diff --git
a/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/server/TestKerberosAuthenticationHandler.java
b/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/server/TestKerberosAuthenticationHandler.java
index e672391..8b4bc15 100644
---
a/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/server/TestKerberosAuthenticationHandler.java
+++
b/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/server/TestKerberosAuthenticationHandler.java
@@ -108,7 +108,12 @@ public class TestKerberosAuthenticationHandler
kn = new KerberosName("bar@BAR");
Assert.assertEquals("bar", kn.getShortName());
kn = new KerberosName("bar@FOO");
-Assert.assertEquals("bar@FOO", kn.getShortName());
+try {
+ kn.getShortName();
+ Assert.fail();
+}
+catch (Exception ex) {
+}
}
@Test(timeout=6)
http://git-wip-us.apache.org/repos/asf/hadoop/blob/065a1e72/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/util/TestKerberosName.java
--
diff --git
a/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/util/TestKerberosName.java
b/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/util/TestKerberosName.java
index c584fce..2db0df4
[3/4] hadoop git commit: HADOOP-15959. Revert "HADOOP-12751. While using kerberos Hadoop incorrectly assumes names with '@' to be non-simple"
HADOOP-15959. Revert "HADOOP-12751. While using kerberos Hadoop incorrectly
assumes names with '@' to be non-simple"
This reverts commit 829a2e4d271f05afb209ddc834cd4a0e85492eda.
(cherry picked from commit d0edd37269bb40290b409d583bcf3b70897c13e0)
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/65e0d6ff
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/65e0d6ff
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/65e0d6ff
Branch: refs/heads/branch-3.1
Commit: 65e0d6ff46c4d891784d85c72abf9eff38d0d77b
Parents: a7d3f22
Author: Steve Loughran
Authored: Thu Nov 29 17:57:24 2018 +
Committer: Steve Loughran
Committed: Thu Nov 29 17:57:24 2018 +
--
.../authentication/util/KerberosName.java | 9 ++--
.../TestKerberosAuthenticationHandler.java | 7 ++-
.../authentication/util/TestKerberosName.java | 17 ++--
.../java/org/apache/hadoop/security/KDiag.java | 46 +---
.../src/site/markdown/SecureMode.md | 6 ---
.../org/apache/hadoop/security/TestKDiag.java | 16 ---
.../security/TestUserGroupInformation.java | 27
7 files changed, 33 insertions(+), 95 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/hadoop/blob/65e0d6ff/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/util/KerberosName.java
--
diff --git
a/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/util/KerberosName.java
b/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/util/KerberosName.java
index 4e7ee3c..287bb13 100644
---
a/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/util/KerberosName.java
+++
b/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/util/KerberosName.java
@@ -324,8 +324,8 @@ public class KerberosName {
}
}
if (result != null && nonSimplePattern.matcher(result).find()) {
-LOG.info("Non-simple name {} after auth_to_local rule {}",
-result, this);
+throw new NoMatchingRule("Non-simple name " + result +
+ " after auth_to_local rule " + this);
}
if (toLowerCase && result != null) {
result = result.toLowerCase(Locale.ENGLISH);
@@ -378,7 +378,7 @@ public class KerberosName {
/**
* Get the translation of the principal name into an operating system
* user name.
- * @return the user name
+ * @return the short name
* @throws IOException throws if something is wrong with the rules
*/
public String getShortName() throws IOException {
@@ -398,8 +398,7 @@ public class KerberosName {
return result;
}
}
-LOG.info("No auth_to_local rules applied to {}", this);
-return toString();
+throw new NoMatchingRule("No rules applied to " + toString());
}
/**
http://git-wip-us.apache.org/repos/asf/hadoop/blob/65e0d6ff/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/server/TestKerberosAuthenticationHandler.java
--
diff --git
a/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/server/TestKerberosAuthenticationHandler.java
b/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/server/TestKerberosAuthenticationHandler.java
index e672391..8b4bc15 100644
---
a/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/server/TestKerberosAuthenticationHandler.java
+++
b/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/server/TestKerberosAuthenticationHandler.java
@@ -108,7 +108,12 @@ public class TestKerberosAuthenticationHandler
kn = new KerberosName("bar@BAR");
Assert.assertEquals("bar", kn.getShortName());
kn = new KerberosName("bar@FOO");
-Assert.assertEquals("bar@FOO", kn.getShortName());
+try {
+ kn.getShortName();
+ Assert.fail();
+}
+catch (Exception ex) {
+}
}
@Test(timeout=6)
http://git-wip-us.apache.org/repos/asf/hadoop/blob/65e0d6ff/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/util/TestKerberosName.java
--
diff --git
a/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/util/TestKerberosName.java
b/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/util/TestKerberosName.java
index c584fce..2db0df4
[2/4] hadoop git commit: HADOOP-15959. Revert "HADOOP-12751. While using kerberos Hadoop incorrectly assumes names with '@' to be non-simple"
HADOOP-15959. Revert "HADOOP-12751. While using kerberos Hadoop incorrectly
assumes names with '@' to be non-simple"
This reverts commit 829a2e4d271f05afb209ddc834cd4a0e85492eda.
(cherry picked from commit d0edd37269bb40290b409d583bcf3b70897c13e0)
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/1a448565
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/1a448565
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/1a448565
Branch: refs/heads/branch-3.2
Commit: 1a448565a85e13572795696ed5e5c6643de62431
Parents: 183ec39
Author: Steve Loughran
Authored: Thu Nov 29 17:53:23 2018 +
Committer: Steve Loughran
Committed: Thu Nov 29 17:53:23 2018 +
--
.../authentication/util/KerberosName.java | 9 ++--
.../TestKerberosAuthenticationHandler.java | 7 ++-
.../authentication/util/TestKerberosName.java | 17 ++--
.../java/org/apache/hadoop/security/KDiag.java | 46 +---
.../src/site/markdown/SecureMode.md | 6 ---
.../org/apache/hadoop/security/TestKDiag.java | 16 ---
.../security/TestUserGroupInformation.java | 27
7 files changed, 33 insertions(+), 95 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/hadoop/blob/1a448565/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/util/KerberosName.java
--
diff --git
a/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/util/KerberosName.java
b/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/util/KerberosName.java
index 4e7ee3c..287bb13 100644
---
a/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/util/KerberosName.java
+++
b/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/util/KerberosName.java
@@ -324,8 +324,8 @@ public class KerberosName {
}
}
if (result != null && nonSimplePattern.matcher(result).find()) {
-LOG.info("Non-simple name {} after auth_to_local rule {}",
-result, this);
+throw new NoMatchingRule("Non-simple name " + result +
+ " after auth_to_local rule " + this);
}
if (toLowerCase && result != null) {
result = result.toLowerCase(Locale.ENGLISH);
@@ -378,7 +378,7 @@ public class KerberosName {
/**
* Get the translation of the principal name into an operating system
* user name.
- * @return the user name
+ * @return the short name
* @throws IOException throws if something is wrong with the rules
*/
public String getShortName() throws IOException {
@@ -398,8 +398,7 @@ public class KerberosName {
return result;
}
}
-LOG.info("No auth_to_local rules applied to {}", this);
-return toString();
+throw new NoMatchingRule("No rules applied to " + toString());
}
/**
http://git-wip-us.apache.org/repos/asf/hadoop/blob/1a448565/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/server/TestKerberosAuthenticationHandler.java
--
diff --git
a/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/server/TestKerberosAuthenticationHandler.java
b/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/server/TestKerberosAuthenticationHandler.java
index e672391..8b4bc15 100644
---
a/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/server/TestKerberosAuthenticationHandler.java
+++
b/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/server/TestKerberosAuthenticationHandler.java
@@ -108,7 +108,12 @@ public class TestKerberosAuthenticationHandler
kn = new KerberosName("bar@BAR");
Assert.assertEquals("bar", kn.getShortName());
kn = new KerberosName("bar@FOO");
-Assert.assertEquals("bar@FOO", kn.getShortName());
+try {
+ kn.getShortName();
+ Assert.fail();
+}
+catch (Exception ex) {
+}
}
@Test(timeout=6)
http://git-wip-us.apache.org/repos/asf/hadoop/blob/1a448565/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/util/TestKerberosName.java
--
diff --git
a/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/util/TestKerberosName.java
b/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/util/TestKerberosName.java
index c584fce..2db0df4
hadoop git commit: HADOOP-15932. Oozie unable to create sharelib in s3a filesystem.
Repository: hadoop
Updated Branches:
refs/heads/branch-3.1 8be2d16b9 -> a7d3f22b4
HADOOP-15932. Oozie unable to create sharelib in s3a filesystem.
Contributed by Steve Loughran.
(cherry picked from commit 4c106fca0ca91536e288f11052568406a0b84300)
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/a7d3f22b
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/a7d3f22b
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/a7d3f22b
Branch: refs/heads/branch-3.1
Commit: a7d3f22b4ffee3015c36e5c8f74115f416ef99a7
Parents: 8be2d16
Author: Steve Loughran
Authored: Thu Nov 29 17:56:29 2018 +
Committer: Steve Loughran
Committed: Thu Nov 29 17:56:29 2018 +
--
.../java/org/apache/hadoop/fs/s3a/S3AFileSystem.java | 8 +++-
.../hadoop/fs/s3a/ITestS3ACopyFromLocalFile.java | 15 +++
2 files changed, 18 insertions(+), 5 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/hadoop/blob/a7d3f22b/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
--
diff --git
a/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
b/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
index 4b0c208..db5b88b 100644
---
a/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
+++
b/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
@@ -2336,7 +2336,10 @@ public class S3AFileSystem extends FileSystem implements
StreamCapabilities {
@Override
public void copyFromLocalFile(boolean delSrc, boolean overwrite, Path src,
Path dst) throws IOException {
-innerCopyFromLocalFile(delSrc, overwrite, src, dst);
+entryPoint(INVOCATION_COPY_FROM_LOCAL_FILE);
+LOG.debug("Copying local file from {} to {}", src, dst);
+//innerCopyFromLocalFile(delSrc, overwrite, src, dst);
+super.copyFromLocalFile(delSrc, overwrite, src, dst);
}
/**
@@ -2346,6 +2349,9 @@ public class S3AFileSystem extends FileSystem implements
StreamCapabilities {
* This version doesn't need to create a temporary file to calculate the md5.
* Sadly this doesn't seem to be used by the shell cp :(
*
+ * HADOOP-15932: this method has been unwired from
+ * {@link #copyFromLocalFile(boolean, boolean, Path, Path)} until
+ * it is extended to list and copy whole directories.
* delSrc indicates if the source should be removed
* @param delSrc whether to delete the src
* @param overwrite whether to overwrite an existing file
http://git-wip-us.apache.org/repos/asf/hadoop/blob/a7d3f22b/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3ACopyFromLocalFile.java
--
diff --git
a/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3ACopyFromLocalFile.java
b/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3ACopyFromLocalFile.java
index 7dc286d..668e129 100644
---
a/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3ACopyFromLocalFile.java
+++
b/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3ACopyFromLocalFile.java
@@ -22,23 +22,27 @@ import java.io.File;
import java.io.FileNotFoundException;
import java.io.IOException;
import java.nio.charset.Charset;
+import java.nio.charset.StandardCharsets;
+import org.junit.Ignore;
import org.junit.Test;
-import org.apache.commons.io.Charsets;
import org.apache.commons.io.FileUtils;
import org.apache.commons.io.IOUtils;
import org.apache.hadoop.fs.FileAlreadyExistsException;
import org.apache.hadoop.fs.FileStatus;
import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.PathExistsException;
import static org.apache.hadoop.test.LambdaTestUtils.intercept;
/**
* Test {@link S3AFileSystem#copyFromLocalFile(boolean, boolean, Path, Path)}.
+ * Some of the tests have been disabled pending a fix for HADOOP-15932 and
+ * recursive directory copying; the test cases themselves may be obsolete.
*/
public class ITestS3ACopyFromLocalFile extends AbstractS3ATestBase {
- private static final Charset ASCII = Charsets.US_ASCII;
+ private static final Charset ASCII = StandardCharsets.US_ASCII;
private File file;
@@ -80,7 +84,8 @@ public class ITestS3ACopyFromLocalFile extends
AbstractS3ATestBase {
public void testCopyFileNoOverwrite() throws Throwable {
file = createTempFile("hello");
Path dest = upload(file, true);
-intercept(FileAlreadyExistsException.class,
+// HADOOP-15932: the exception type changes here
+intercept(PathExistsException.class,
() -> upload(file, false));
}
@@ -95,6 +100,7 @@ public class
hadoop git commit: HDDS-850. ReadStateMachineData hits OverlappingFileLockException in ContainerStateMachine. Contributed by Shashikant Banerjee.
Repository: hadoop
Updated Branches:
refs/heads/trunk 7eb0d3a32 -> 5e102f9aa
HDDS-850. ReadStateMachineData hits OverlappingFileLockException in
ContainerStateMachine. Contributed by Shashikant Banerjee.
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/5e102f9a
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/5e102f9a
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/5e102f9a
Branch: refs/heads/trunk
Commit: 5e102f9aa54d3057ef5f0755d45428f22a24990b
Parents: 7eb0d3a
Author: Shashikant Banerjee
Authored: Thu Nov 29 22:20:08 2018 +0530
Committer: Shashikant Banerjee
Committed: Thu Nov 29 22:20:08 2018 +0530
--
.../apache/hadoop/hdds/scm/ScmConfigKeys.java | 8 ++
.../apache/hadoop/ozone/OzoneConfigKeys.java| 9 ++
.../main/proto/DatanodeContainerProtocol.proto | 1 +
.../common/src/main/resources/ozone-default.xml | 8 ++
.../server/ratis/ContainerStateMachine.java | 134 +++
.../server/ratis/XceiverServerRatis.java| 14 +-
.../container/keyvalue/KeyValueHandler.java | 7 +-
.../keyvalue/impl/ChunkManagerImpl.java | 11 +-
.../keyvalue/interfaces/ChunkManager.java | 5 +-
.../keyvalue/TestChunkManagerImpl.java | 6 +-
.../common/impl/TestContainerPersistence.java | 11 +-
11 files changed, 143 insertions(+), 71 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/hadoop/blob/5e102f9a/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/ScmConfigKeys.java
--
diff --git
a/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/ScmConfigKeys.java
b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/ScmConfigKeys.java
index 6733b8e..062b101 100644
---
a/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/ScmConfigKeys.java
+++
b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/ScmConfigKeys.java
@@ -93,6 +93,14 @@ public final class ScmConfigKeys {
public static final String DFS_CONTAINER_RATIS_LOG_QUEUE_SIZE =
"dfs.container.ratis.log.queue.size";
public static final int DFS_CONTAINER_RATIS_LOG_QUEUE_SIZE_DEFAULT = 128;
+
+ // expiry interval stateMachineData cache entry inside containerStateMachine
+ public static final String
+ DFS_CONTAINER_RATIS_STATEMACHINEDATA_CACHE_EXPIRY_INTERVAL =
+ "dfs.container.ratis.statemachine.cache.expiry.interval";
+ public static final String
+ DFS_CONTAINER_RATIS_STATEMACHINEDATA_CACHE_EXPIRY_INTERVAL_DEFAULT =
+ "10s";
public static final String DFS_RATIS_CLIENT_REQUEST_TIMEOUT_DURATION_KEY =
"dfs.ratis.client.request.timeout.duration";
public static final TimeDuration
http://git-wip-us.apache.org/repos/asf/hadoop/blob/5e102f9a/hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/OzoneConfigKeys.java
--
diff --git
a/hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/OzoneConfigKeys.java
b/hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/OzoneConfigKeys.java
index 879f773..df233f7 100644
---
a/hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/OzoneConfigKeys.java
+++
b/hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/OzoneConfigKeys.java
@@ -249,6 +249,15 @@ public final class OzoneConfigKeys {
DFS_CONTAINER_RATIS_STATEMACHINEDATA_SYNC_TIMEOUT_DEFAULT =
ScmConfigKeys.DFS_CONTAINER_RATIS_STATEMACHINEDATA_SYNC_TIMEOUT_DEFAULT;
+ public static final String
+ DFS_CONTAINER_RATIS_STATEMACHINEDATA_CACHE_EXPIRY_INTERVAL =
+ ScmConfigKeys.
+ DFS_CONTAINER_RATIS_STATEMACHINEDATA_CACHE_EXPIRY_INTERVAL;
+ public static final String
+ DFS_CONTAINER_RATIS_STATEMACHINEDATA_CACHE_EXPIRY_INTERVAL_DEFAULT =
+ ScmConfigKeys.
+ DFS_CONTAINER_RATIS_STATEMACHINEDATA_CACHE_EXPIRY_INTERVAL_DEFAULT;
+
public static final String DFS_CONTAINER_RATIS_DATANODE_STORAGE_DIR =
"dfs.container.ratis.datanode.storage.dir";
public static final String DFS_RATIS_CLIENT_REQUEST_TIMEOUT_DURATION_KEY =
http://git-wip-us.apache.org/repos/asf/hadoop/blob/5e102f9a/hadoop-hdds/common/src/main/proto/DatanodeContainerProtocol.proto
--
diff --git a/hadoop-hdds/common/src/main/proto/DatanodeContainerProtocol.proto
b/hadoop-hdds/common/src/main/proto/DatanodeContainerProtocol.proto
index 3695b6b..5237af8 100644
--- a/hadoop-hdds/common/src/main/proto/DatanodeContainerProtocol.proto
+++ b/hadoop-hdds/common/src/main/proto/DatanodeContainerProtocol.proto
@@ -392,6 +392,7 @@ message WriteChunkResponseProto {
message ReadChunkRequestProto {
required DatanodeBlockID blockID = 1;
required ChunkInfo chunkData
hadoop git commit: YARN-9069. Fix SchedulerInfo#getSchedulerType for custom schedulers. Contributed by Bilwa S T.
Repository: hadoop
Updated Branches:
refs/heads/branch-3.2 ee9deb6e9 -> 183ec39c4
YARN-9069. Fix SchedulerInfo#getSchedulerType for custom schedulers.
Contributed by Bilwa S T.
(cherry picked from commit 07142f54a8c7f70857e99c041f3a2a5189c809b5)
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/183ec39c
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/183ec39c
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/183ec39c
Branch: refs/heads/branch-3.2
Commit: 183ec39c4bb3a132a8207fedfedd65f399529150
Parents: ee9deb6
Author: bibinchundatt
Authored: Thu Nov 29 22:02:59 2018 +0530
Committer: bibinchundatt
Committed: Thu Nov 29 22:16:32 2018 +0530
--
.../yarn/server/resourcemanager/webapp/dao/SchedulerInfo.java | 2 ++
1 file changed, 2 insertions(+)
--
http://git-wip-us.apache.org/repos/asf/hadoop/blob/183ec39c/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/SchedulerInfo.java
--
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/SchedulerInfo.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/SchedulerInfo.java
index 163f707..ede0d15 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/SchedulerInfo.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/SchedulerInfo.java
@@ -54,6 +54,8 @@ public class SchedulerInfo {
this.schedulerName = "Fair Scheduler";
} else if (rs instanceof FifoScheduler) {
this.schedulerName = "Fifo Scheduler";
+} else {
+ this.schedulerName = rs.getClass().getSimpleName();
}
this.minAllocResource = new
ResourceInfo(rs.getMinimumResourceCapability());
this.maxAllocResource = new
ResourceInfo(rs.getMaximumResourceCapability());
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
hadoop git commit: YARN-9069. Fix SchedulerInfo#getSchedulerType for custom schedulers. Contributed by Bilwa S T.
Repository: hadoop
Updated Branches:
refs/heads/branch-3.0 4c238b50d -> b42c2679a
YARN-9069. Fix SchedulerInfo#getSchedulerType for custom schedulers.
Contributed by Bilwa S T.
(cherry picked from commit 07142f54a8c7f70857e99c041f3a2a5189c809b5)
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/b42c2679
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/b42c2679
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/b42c2679
Branch: refs/heads/branch-3.0
Commit: b42c2679ad24f0a5f9a48490b002351db7b7a6a1
Parents: 4c238b5
Author: bibinchundatt
Authored: Thu Nov 29 22:02:59 2018 +0530
Committer: bibinchundatt
Committed: Thu Nov 29 22:08:48 2018 +0530
--
.../yarn/server/resourcemanager/webapp/dao/SchedulerInfo.java | 2 ++
1 file changed, 2 insertions(+)
--
http://git-wip-us.apache.org/repos/asf/hadoop/blob/b42c2679/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/SchedulerInfo.java
--
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/SchedulerInfo.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/SchedulerInfo.java
index 81491b1..ae6b3d3 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/SchedulerInfo.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/SchedulerInfo.java
@@ -53,6 +53,8 @@ public class SchedulerInfo {
this.schedulerName = "Fair Scheduler";
} else if (rs instanceof FifoScheduler) {
this.schedulerName = "Fifo Scheduler";
+} else {
+ this.schedulerName = rs.getClass().getSimpleName();
}
this.minAllocResource = new
ResourceInfo(rs.getMinimumResourceCapability());
this.maxAllocResource = new
ResourceInfo(rs.getMaximumResourceCapability());
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
hadoop git commit: YARN-9069. Fix SchedulerInfo#getSchedulerType for custom schedulers. Contributed by Bilwa S T.
Repository: hadoop
Updated Branches:
refs/heads/branch-3.1 e7fa638fe -> 8be2d16b9
YARN-9069. Fix SchedulerInfo#getSchedulerType for custom schedulers.
Contributed by Bilwa S T.
(cherry picked from commit 07142f54a8c7f70857e99c041f3a2a5189c809b5)
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/8be2d16b
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/8be2d16b
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/8be2d16b
Branch: refs/heads/branch-3.1
Commit: 8be2d16b940929653f9ee507e637df92ded1fa65
Parents: e7fa638
Author: bibinchundatt
Authored: Thu Nov 29 22:02:59 2018 +0530
Committer: bibinchundatt
Committed: Thu Nov 29 22:08:35 2018 +0530
--
.../yarn/server/resourcemanager/webapp/dao/SchedulerInfo.java | 2 ++
1 file changed, 2 insertions(+)
--
http://git-wip-us.apache.org/repos/asf/hadoop/blob/8be2d16b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/SchedulerInfo.java
--
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/SchedulerInfo.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/SchedulerInfo.java
index 81491b1..ae6b3d3 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/SchedulerInfo.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/SchedulerInfo.java
@@ -53,6 +53,8 @@ public class SchedulerInfo {
this.schedulerName = "Fair Scheduler";
} else if (rs instanceof FifoScheduler) {
this.schedulerName = "Fifo Scheduler";
+} else {
+ this.schedulerName = rs.getClass().getSimpleName();
}
this.minAllocResource = new
ResourceInfo(rs.getMinimumResourceCapability());
this.maxAllocResource = new
ResourceInfo(rs.getMaximumResourceCapability());
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
hadoop git commit: HADOOP-14927. ITestS3GuardTool failures in testDestroyNoBucket(). Contributed by Gabor Bota.
Repository: hadoop
Updated Branches:
refs/heads/trunk 184cced51 -> 7eb0d3a32
HADOOP-14927. ITestS3GuardTool failures in testDestroyNoBucket(). Contributed
by Gabor Bota.
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/7eb0d3a3
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/7eb0d3a3
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/7eb0d3a3
Branch: refs/heads/trunk
Commit: 7eb0d3a32435da110dc9e6004dba8c5c9b082c35
Parents: 184cced
Author: Sean Mackrory
Authored: Wed Nov 28 16:57:12 2018 -0700
Committer: Sean Mackrory
Committed: Thu Nov 29 09:36:39 2018 -0700
--
.../hadoop/fs/s3a/s3guard/S3GuardTool.java | 38
1 file changed, 24 insertions(+), 14 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/hadoop/blob/7eb0d3a3/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/S3GuardTool.java
--
diff --git
a/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/S3GuardTool.java
b/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/S3GuardTool.java
index 1316121..aea57a6 100644
---
a/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/S3GuardTool.java
+++
b/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/S3GuardTool.java
@@ -218,6 +218,27 @@ public abstract class S3GuardTool extends Configured
implements Tool {
format.addOptionWithValue(SECONDS_FLAG);
}
+ protected void checkMetadataStoreUri(List paths) throws IOException {
+// be sure that path is provided in params, so there's no IOoBE
+String s3Path = "";
+if(!paths.isEmpty()) {
+ s3Path = paths.get(0);
+}
+
+// Check if DynamoDB url is set from arguments.
+String metadataStoreUri = getCommandFormat().getOptValue(META_FLAG);
+if(metadataStoreUri == null || metadataStoreUri.isEmpty()) {
+ // If not set, check if filesystem is guarded by creating an
+ // S3AFileSystem and check if hasMetadataStore is true
+ try (S3AFileSystem s3AFileSystem = (S3AFileSystem)
+ S3AFileSystem.newInstance(toUri(s3Path), getConf())){
+Preconditions.checkState(s3AFileSystem.hasMetadataStore(),
+"The S3 bucket is unguarded. " + getName()
++ " can not be used on an unguarded bucket.");
+ }
+}
+ }
+
/**
* Parse metadata store from command line option or HDFS configuration.
*
@@ -500,20 +521,7 @@ public abstract class S3GuardTool extends Configured
implements Tool {
public int run(String[] args, PrintStream out) throws Exception {
List paths = parseArgs(args);
Map options = new HashMap<>();
- String s3Path = paths.get(0);
-
- // Check if DynamoDB url is set from arguments.
- String metadataStoreUri = getCommandFormat().getOptValue(META_FLAG);
- if(metadataStoreUri == null || metadataStoreUri.isEmpty()) {
-// If not set, check if filesystem is guarded by creating an
-// S3AFileSystem and check if hasMetadataStore is true
-try (S3AFileSystem s3AFileSystem = (S3AFileSystem)
-S3AFileSystem.newInstance(toUri(s3Path), getConf())){
- Preconditions.checkState(s3AFileSystem.hasMetadataStore(),
- "The S3 bucket is unguarded. " + getName()
- + " can not be used on an unguarded bucket.");
-}
- }
+ checkMetadataStoreUri(paths);
String readCap = getCommandFormat().getOptValue(READ_FLAG);
if (StringUtils.isNotEmpty(readCap)) {
@@ -590,6 +598,8 @@ public abstract class S3GuardTool extends Configured
implements Tool {
throw e;
}
+ checkMetadataStoreUri(paths);
+
try {
initMetadataStore(false);
} catch (FileNotFoundException e) {
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
hadoop git commit: HDDS-808. Simplify OMAction and DNAction classes used for AuditLogging. Contributed by Dinesh Chitlangia.
Repository: hadoop
Updated Branches:
refs/heads/trunk 07142f54a -> 184cced51
HDDS-808. Simplify OMAction and DNAction classes used for AuditLogging.
Contributed by Dinesh Chitlangia.
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/184cced5
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/184cced5
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/184cced5
Branch: refs/heads/trunk
Commit: 184cced513c5599d7b33c9124692fbcd2e6d338e
Parents: 07142f5
Author: Ajay Kumar
Authored: Thu Nov 29 08:35:02 2018 -0800
Committer: Ajay Kumar
Committed: Thu Nov 29 08:35:20 2018 -0800
--
.../org/apache/hadoop/ozone/audit/DNAction.java | 44 +++-
.../apache/hadoop/ozone/audit/DummyAction.java | 36 ++---
.../org/apache/hadoop/ozone/audit/OMAction.java | 54 +---
3 files changed, 58 insertions(+), 76 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/hadoop/blob/184cced5/hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/audit/DNAction.java
--
diff --git
a/hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/audit/DNAction.java
b/hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/audit/DNAction.java
index ce34c46..1c87f2b 100644
---
a/hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/audit/DNAction.java
+++
b/hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/audit/DNAction.java
@@ -21,34 +21,28 @@ package org.apache.hadoop.ozone.audit;
*/
public enum DNAction implements AuditAction {
- CREATE_CONTAINER("CREATE_CONTAINER"),
- READ_CONTAINER("READ_CONTAINER"),
- UPDATE_CONTAINER("UPDATE_CONTAINER"),
- DELETE_CONTAINER("DELETE_CONTAINER"),
- LIST_CONTAINER("LIST_CONTAINER"),
- PUT_BLOCK("PUT_BLOCK"),
- GET_BLOCK("GET_BLOCK"),
- DELETE_BLOCK("DELETE_BLOCK"),
- LIST_BLOCK("LIST_BLOCK"),
- READ_CHUNK("READ_CHUNK"),
- DELETE_CHUNK("DELETE_CHUNK"),
- WRITE_CHUNK("WRITE_CHUNK"),
- LIST_CHUNK("LIST_CHUNK"),
- COMPACT_CHUNK("COMPACT_CHUNK"),
- PUT_SMALL_FILE("PUT_SMALL_FILE"),
- GET_SMALL_FILE("GET_SMALL_FILE"),
- CLOSE_CONTAINER("CLOSE_CONTAINER"),
- GET_COMMITTED_BLOCK_LENGTH("GET_COMMITTED_BLOCK_LENGTH");
-
- private String action;
-
- DNAction(String action) {
-this.action = action;
- }
+ CREATE_CONTAINER,
+ READ_CONTAINER,
+ UPDATE_CONTAINER,
+ DELETE_CONTAINER,
+ LIST_CONTAINER,
+ PUT_BLOCK,
+ GET_BLOCK,
+ DELETE_BLOCK,
+ LIST_BLOCK,
+ READ_CHUNK,
+ DELETE_CHUNK,
+ WRITE_CHUNK,
+ LIST_CHUNK,
+ COMPACT_CHUNK,
+ PUT_SMALL_FILE,
+ GET_SMALL_FILE,
+ CLOSE_CONTAINER,
+ GET_COMMITTED_BLOCK_LENGTH;
@Override
public String getAction() {
-return this.action;
+return this.toString();
}
}
http://git-wip-us.apache.org/repos/asf/hadoop/blob/184cced5/hadoop-hdds/common/src/test/java/org/apache/hadoop/ozone/audit/DummyAction.java
--
diff --git
a/hadoop-hdds/common/src/test/java/org/apache/hadoop/ozone/audit/DummyAction.java
b/hadoop-hdds/common/src/test/java/org/apache/hadoop/ozone/audit/DummyAction.java
index 76cd39a..d2da3e6 100644
---
a/hadoop-hdds/common/src/test/java/org/apache/hadoop/ozone/audit/DummyAction.java
+++
b/hadoop-hdds/common/src/test/java/org/apache/hadoop/ozone/audit/DummyAction.java
@@ -22,30 +22,24 @@ package org.apache.hadoop.ozone.audit;
*/
public enum DummyAction implements AuditAction {
- CREATE_VOLUME("CREATE_VOLUME"),
- CREATE_BUCKET("CREATE_BUCKET"),
- CREATE_KEY("CREATE_KEY"),
- READ_VOLUME("READ_VOLUME"),
- READ_BUCKET("READ_BUCKET"),
- READ_KEY("READ_BUCKET"),
- UPDATE_VOLUME("UPDATE_VOLUME"),
- UPDATE_BUCKET("UPDATE_BUCKET"),
- UPDATE_KEY("UPDATE_KEY"),
- DELETE_VOLUME("DELETE_VOLUME"),
- DELETE_BUCKET("DELETE_BUCKET"),
- DELETE_KEY("DELETE_KEY"),
- SET_OWNER("SET_OWNER"),
- SET_QUOTA("SET_QUOTA");
-
- private final String action;
-
- DummyAction(String action) {
-this.action = action;
- }
+ CREATE_VOLUME,
+ CREATE_BUCKET,
+ CREATE_KEY,
+ READ_VOLUME,
+ READ_BUCKET,
+ READ_KEY,
+ UPDATE_VOLUME,
+ UPDATE_BUCKET,
+ UPDATE_KEY,
+ DELETE_VOLUME,
+ DELETE_BUCKET,
+ DELETE_KEY,
+ SET_OWNER,
+ SET_QUOTA;
@Override
public String getAction() {
-return this.action;
+return this.toString();
}
}
http://git-wip-us.apache.org/repos/asf/hadoop/blob/184cced5/hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/audit/OMAction.java
--
diff --git
a/hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/audit/OMAction.java
b/hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/audit/OMAction.java
index 1d4d646..8794014 100644
---
hadoop git commit: YARN-9069. Fix SchedulerInfo#getSchedulerType for custom schedulers. Contributed by Bilwa S T.
Repository: hadoop
Updated Branches:
refs/heads/trunk a68d766e8 -> 07142f54a
YARN-9069. Fix SchedulerInfo#getSchedulerType for custom schedulers.
Contributed by Bilwa S T.
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/07142f54
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/07142f54
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/07142f54
Branch: refs/heads/trunk
Commit: 07142f54a8c7f70857e99c041f3a2a5189c809b5
Parents: a68d766
Author: bibinchundatt
Authored: Thu Nov 29 22:02:59 2018 +0530
Committer: bibinchundatt
Committed: Thu Nov 29 22:02:59 2018 +0530
--
.../yarn/server/resourcemanager/webapp/dao/SchedulerInfo.java | 2 ++
1 file changed, 2 insertions(+)
--
http://git-wip-us.apache.org/repos/asf/hadoop/blob/07142f54/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/SchedulerInfo.java
--
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/SchedulerInfo.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/SchedulerInfo.java
index 163f707..ede0d15 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/SchedulerInfo.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/SchedulerInfo.java
@@ -54,6 +54,8 @@ public class SchedulerInfo {
this.schedulerName = "Fair Scheduler";
} else if (rs instanceof FifoScheduler) {
this.schedulerName = "Fifo Scheduler";
+} else {
+ this.schedulerName = rs.getClass().getSimpleName();
}
this.minAllocResource = new
ResourceInfo(rs.getMinimumResourceCapability());
this.maxAllocResource = new
ResourceInfo(rs.getMaximumResourceCapability());
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
hadoop git commit: YARN-8948. PlacementRule interface should be for all YarnSchedulers. Contributed by Bibin A Chundatt.
Repository: hadoop
Updated Branches:
refs/heads/trunk c1d24f848 -> a68d766e8
YARN-8948. PlacementRule interface should be for all YarnSchedulers.
Contributed by Bibin A Chundatt.
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/a68d766e
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/a68d766e
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/a68d766e
Branch: refs/heads/trunk
Commit: a68d766e876631d7ee2e1a6504d4120ba628d178
Parents: c1d24f8
Author: bibinchundatt
Authored: Thu Nov 29 21:43:34 2018 +0530
Committer: bibinchundatt
Committed: Thu Nov 29 21:43:34 2018 +0530
--
.../placement/AppNameMappingPlacementRule.java | 12 ++--
.../server/resourcemanager/placement/PlacementRule.java | 4 ++--
.../placement/UserGroupMappingPlacementRule.java| 11 ++-
3 files changed, 22 insertions(+), 5 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/hadoop/blob/a68d766e/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/AppNameMappingPlacementRule.java
--
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/AppNameMappingPlacementRule.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/AppNameMappingPlacementRule.java
index 2debade..7a46962 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/AppNameMappingPlacementRule.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/AppNameMappingPlacementRule.java
@@ -20,11 +20,12 @@ package
org.apache.hadoop.yarn.server.resourcemanager.placement;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
-import org.apache.hadoop.yarn.api.records.ApplicationId;
import org.apache.hadoop.yarn.api.records.ApplicationSubmissionContext;
import org.apache.hadoop.yarn.conf.YarnConfiguration;
import org.apache.hadoop.yarn.exceptions.YarnException;
+import
org.apache.hadoop.yarn.server.resourcemanager.scheduler.ResourceScheduler;
import
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CSQueue;
+import
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler;
import
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerConfiguration;
import
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerContext;
import
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager;
@@ -61,8 +62,15 @@ public class AppNameMappingPlacementRule extends
PlacementRule {
}
@Override
- public boolean initialize(CapacitySchedulerContext schedulerContext)
+ public boolean initialize(ResourceScheduler scheduler)
throws IOException {
+if (!(scheduler instanceof CapacityScheduler)) {
+ throw new IOException(
+ "AppNameMappingPlacementRule can be configured only for "
+ + "CapacityScheduler");
+}
+CapacitySchedulerContext schedulerContext =
+(CapacitySchedulerContext) scheduler;
CapacitySchedulerConfiguration conf = schedulerContext.getConfiguration();
boolean overrideWithQueueMappings = conf.getOverrideWithQueueMappings();
LOG.info(
http://git-wip-us.apache.org/repos/asf/hadoop/blob/a68d766e/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/PlacementRule.java
--
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/PlacementRule.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/PlacementRule.java
index 21ab32a..0f3d43c 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/PlacementRule.java
+++
[2/2] hadoop git commit: HDDS-804. Block token: Add secret token manager. Contributed by Ajay Kumar.
HDDS-804. Block token: Add secret token manager. Contributed by Ajay Kumar.
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/278d4b9b
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/278d4b9b
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/278d4b9b
Branch: refs/heads/HDDS-4
Commit: 278d4b9b7bb2cd52d8708870602be292be189359
Parents: 1d3d40b
Author: Ajay Kumar
Authored: Thu Nov 29 08:00:41 2018 -0800
Committer: Ajay Kumar
Committed: Thu Nov 29 08:00:41 2018 -0800
--
.../hdds/security/x509/SecurityConfig.java | 9 +
.../security/OzoneBlockTokenSecretManager.java | 191 +++
.../OzoneDelegationTokenSecretManager.java | 455 +
.../ozone/security/OzoneSecretManager.java | 498 ---
.../TestOzoneBlockTokenSecretManager.java | 146 ++
.../TestOzoneDelegationTokenSecretManager.java | 218
.../ozone/security/TestOzoneSecretManager.java | 216
.../apache/hadoop/ozone/om/OzoneManager.java| 23 +-
.../security/TestOzoneManagerBlockToken.java| 251 ++
9 files changed, 1371 insertions(+), 636 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/hadoop/blob/278d4b9b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/security/x509/SecurityConfig.java
--
diff --git
a/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/security/x509/SecurityConfig.java
b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/security/x509/SecurityConfig.java
index ee20a21..b38ee7c 100644
---
a/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/security/x509/SecurityConfig.java
+++
b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/security/x509/SecurityConfig.java
@@ -21,6 +21,7 @@ package org.apache.hadoop.hdds.security.x509;
import com.google.common.base.Preconditions;
import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.ozone.OzoneConfigKeys;
import org.bouncycastle.jce.provider.BouncyCastleProvider;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -77,6 +78,7 @@ public class SecurityConfig {
private final Duration certDuration;
private final String x509SignatureAlgo;
private final Boolean grpcBlockTokenEnabled;
+ private final int getMaxKeyLength;
private final String certificateDir;
private final String certificateFileName;
@@ -88,6 +90,9 @@ public class SecurityConfig {
public SecurityConfig(Configuration configuration) {
Preconditions.checkNotNull(configuration, "Configuration cannot be null");
this.configuration = configuration;
+this.getMaxKeyLength = configuration.getInt(
+OzoneConfigKeys.OZONE_MAX_KEY_LEN,
+OzoneConfigKeys.OZONE_MAX_KEY_LEN_DEFAULT);
this.size = this.configuration.getInt(HDDS_KEY_LEN, HDDS_DEFAULT_KEY_LEN);
this.keyAlgo = this.configuration.get(HDDS_KEY_ALGORITHM,
HDDS_DEFAULT_KEY_ALGORITHM);
@@ -289,4 +294,8 @@ public class SecurityConfig {
throw new SecurityException("Unknown security provider:" + provider);
}
}
+
+ public int getMaxKeyLength() {
+return this.getMaxKeyLength;
+ }
}
http://git-wip-us.apache.org/repos/asf/hadoop/blob/278d4b9b/hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/security/OzoneBlockTokenSecretManager.java
--
diff --git
a/hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/security/OzoneBlockTokenSecretManager.java
b/hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/security/OzoneBlockTokenSecretManager.java
new file mode 100644
index 000..3b833cb
--- /dev/null
+++
b/hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/security/OzoneBlockTokenSecretManager.java
@@ -0,0 +1,191 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations
under
+ * the License.
+ */
+
+package org.apache.hadoop.ozone.security;
+
+import org.apache.hadoop.classification.InterfaceAudience;
+import
[1/2] [hadoop] Git Push Summary
Repository: hadoop Updated Branches: refs/heads/HDDS-4 1d3d40b9c -> 278d4b9b7 - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
