[jira] [Commented] (HADOOP-15967) KMS Benchmark Tool
[ https://issues.apache.org/jira/browse/HADOOP-15967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16772443#comment-16772443 ] Hudson commented on HADOOP-15967: - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #16001 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/16001/]) HADOOP-15967. KMS Benchmark Tool. Contributed by George Huang. (weichiu: rev 0525d85d57763a0078bdaf9b08d36909f3c6ae2e) * (add) hadoop-common-project/hadoop-kms/src/test/java/org/apache/hadoop/crypto/key/kms/server/KMSBenchmark.java > KMS Benchmark Tool > -- > > Key: HADOOP-15967 > URL: https://issues.apache.org/jira/browse/HADOOP-15967 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Wei-Chiu Chuang >Assignee: George Huang >Priority: Major > Fix For: 3.3.0 > > Attachments: HADOOP-15967.001.patch, HADOOP-15967.002.patch, > HADOOP-15967.003.patch > > > We've been working on several pieces of KMS improvement work. One thing > that's missing so far is a good benchmark tool for KMS, similar to > NNThroughputBenchmark. > Some requirements I have in mind: > # it should be a standalone benchmark tool, requiring only KMS and a > benchmark client. No NameNode or DataNode should be involved. > # specify the type of KMS request sent by client. E.g., generate_eek, > decrypt_eek, reencrypt_eek > # optionally specify number of threads sending KMS requests. > File this jira to gather more requirements. Thoughts? [~knanasi] [~xyao] > [~daryn] -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15967) KMS Benchmark Tool
[ https://issues.apache.org/jira/browse/HADOOP-15967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16772320#comment-16772320 ] Wei-Chiu Chuang commented on HADOOP-15967: -- +1 will commit soon > KMS Benchmark Tool > -- > > Key: HADOOP-15967 > URL: https://issues.apache.org/jira/browse/HADOOP-15967 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Wei-Chiu Chuang >Assignee: George Huang >Priority: Major > Attachments: HADOOP-15967.001.patch, HADOOP-15967.002.patch, > HADOOP-15967.003.patch > > > We've been working on several pieces of KMS improvement work. One thing > that's missing so far is a good benchmark tool for KMS, similar to > NNThroughputBenchmark. > Some requirements I have in mind: > # it should be a standalone benchmark tool, requiring only KMS and a > benchmark client. No NameNode or DataNode should be involved. > # specify the type of KMS request sent by client. E.g., generate_eek, > decrypt_eek, reencrypt_eek > # optionally specify number of threads sending KMS requests. > File this jira to gather more requirements. Thoughts? [~knanasi] [~xyao] > [~daryn] -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15967) KMS Benchmark Tool
[ https://issues.apache.org/jira/browse/HADOOP-15967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16770016#comment-16770016 ] Hadoop QA commented on HADOOP-15967: | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 54s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 1 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m 47s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m 23s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 25s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 34s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 12m 30s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 39s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 28s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 21s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m 47s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 14m 47s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 25s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 35s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 11m 31s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 47s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 28s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m 11s{color} | {color:green} hadoop-kms in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 39s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 83m 9s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f | | JIRA Issue | HADOOP-15967 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12958956/HADOOP-15967.003.patch | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 4469e16186a8 4.4.0-138-generic #164~14.04.1-Ubuntu SMP Fri Oct 5 08:56:16 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/patchprocess/precommit/personality/provided.sh | | git revision | trunk / dde0ab5 | | maven | version: Apache Maven 3.3.9 | | Default Java | 1.8.0_191 | | findbugs | v3.1.0-RC1 | | Test Results | https://builds.apache.org/job/PreCommit-HADOOP-Build/15927/testReport/ | | Max. process+thread count | 339 (vs. ulimit of 1) | | modules | C: hadoop-common-project/hadoop-kms U: hadoop-common-project/hadoop-kms | | Console output | https://builds.apache.org/job/PreCommit-HADOOP-Build/15927/console | | Powered by | Apache Yetus 0.8.0 http://yetus.apache.org | This message was automatically generated. > KMS Benchmark Tool > -- > > Key: HADOOP-15967 > URL: https://issues.apache.org/jira/browse/HADOOP-15967 >
[jira] [Commented] (HADOOP-15967) KMS Benchmark Tool
[ https://issues.apache.org/jira/browse/HADOOP-15967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16769995#comment-16769995 ] Hadoop QA commented on HADOOP-15967: | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 55s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 1 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m 13s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m 32s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 24s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 34s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 12m 26s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 37s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 26s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 21s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m 45s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 14m 45s{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 0m 26s{color} | {color:orange} hadoop-common-project/hadoop-kms: The patch generated 3 new + 0 unchanged - 0 fixed = 3 total (was 0) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 34s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 11m 32s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 45s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 27s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m 9s{color} | {color:green} hadoop-kms in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 41s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 83m 39s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f | | JIRA Issue | HADOOP-15967 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12958952/HADOOP-15967.002.patch | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux c5fa0da822de 4.4.0-138-generic #164~14.04.1-Ubuntu SMP Fri Oct 5 08:56:16 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/patchprocess/precommit/personality/provided.sh | | git revision | trunk / dde0ab5 | | maven | version: Apache Maven 3.3.9 | | Default Java | 1.8.0_191 | | findbugs | v3.1.0-RC1 | | checkstyle | https://builds.apache.org/job/PreCommit-HADOOP-Build/15926/artifact/out/diff-checkstyle-hadoop-common-project_hadoop-kms.txt | | Test Results | https://builds.apache.org/job/PreCommit-HADOOP-Build/15926/testReport/ | | Max. process+thread count | 294 (vs. ulimit of 1) | | modules | C: hadoop-common-project/hadoop-kms U: hadoop-common-project/hadoop-kms | | Console output | https://builds.apache.org/job/PreCommit-HADOOP-Build/15926/console | | Powered by | Apache Yetus 0.8.0
[jira] [Commented] (HADOOP-15967) KMS Benchmark Tool
[ https://issues.apache.org/jira/browse/HADOOP-15967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16769973#comment-16769973 ] George Huang commented on HADOOP-15967: --- Thanks [~jojochuang]! Newer patch uploaded addressed comments and check-style issues. > KMS Benchmark Tool > -- > > Key: HADOOP-15967 > URL: https://issues.apache.org/jira/browse/HADOOP-15967 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Wei-Chiu Chuang >Assignee: George Huang >Priority: Major > Attachments: HADOOP-15967.001.patch, HADOOP-15967.002.patch > > > We've been working on several pieces of KMS improvement work. One thing > that's missing so far is a good benchmark tool for KMS, similar to > NNThroughputBenchmark. > Some requirements I have in mind: > # it should be a standalone benchmark tool, requiring only KMS and a > benchmark client. No NameNode or DataNode should be involved. > # specify the type of KMS request sent by client. E.g., generate_eek, > decrypt_eek, reencrypt_eek > # optionally specify number of threads sending KMS requests. > File this jira to gather more requirements. Thoughts? [~knanasi] [~xyao] > [~daryn] -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15967) KMS Benchmark Tool
[ https://issues.apache.org/jira/browse/HADOOP-15967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16769329#comment-16769329 ] Wei-Chiu Chuang commented on HADOOP-15967: -- Thanks for making the patch! [~ghuangups] Overall looks good. I've also ran the benchmark and it works well. Some nits I found in the code: {code:java} * For usage, please see http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/Benchmarking.html#KMSBenchmark;>the documentation. {code} There is no doc yet. We should file another Jira to add the doc. {code:java} LOG.warn("encryption key already exists: ", encryptionKeyName); {code} Missed a {} {code:java} LOG.warn("encryption key already exists: {}", encryptionKeyName); {code} {code:java} final String HADOOP_SECURITY_KEY_PROVIDER_PATH = "hadoop.security.key.provider.path"; {code} Could you use {{CommonConfigurationKeysPublic.HADOOP_SECURITY_KEY_PROVIDER_PATH}} instead? {code:java} if (keyProvider == null) { return null; } {code} I think we want to throw an exception saying key provider is not configured. Otherwise a null keyProvider will result in an NPE when accessed later, and that would be hard to troubleshoot. > KMS Benchmark Tool > -- > > Key: HADOOP-15967 > URL: https://issues.apache.org/jira/browse/HADOOP-15967 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Wei-Chiu Chuang >Assignee: George Huang >Priority: Major > Attachments: HADOOP-15967.001.patch > > > We've been working on several pieces of KMS improvement work. One thing > that's missing so far is a good benchmark tool for KMS, similar to > NNThroughputBenchmark. > Some requirements I have in mind: > # it should be a standalone benchmark tool, requiring only KMS and a > benchmark client. No NameNode or DataNode should be involved. > # specify the type of KMS request sent by client. E.g., generate_eek, > decrypt_eek, reencrypt_eek > # optionally specify number of threads sending KMS requests. > File this jira to gather more requirements. Thoughts? [~knanasi] [~xyao] > [~daryn] -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15967) KMS Benchmark Tool
[ https://issues.apache.org/jira/browse/HADOOP-15967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16768753#comment-16768753 ] Hadoop QA commented on HADOOP-15967: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 22s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 1 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 21m 11s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 16m 35s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 25s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 35s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 12m 26s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 39s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 28s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 21s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m 12s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 15m 12s{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 0m 25s{color} | {color:orange} hadoop-common-project/hadoop-kms: The patch generated 36 new + 0 unchanged - 0 fixed = 36 total (was 0) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 36s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s{color} | {color:red} The patch has 13 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 11m 37s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 46s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 27s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m 20s{color} | {color:green} hadoop-kms in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 43s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 87m 59s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f | | JIRA Issue | HADOOP-15967 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12958782/HADOOP-15967.001.patch | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 27759e550ec3 3.13.0-153-generic #203-Ubuntu SMP Thu Jun 14 08:52:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/patchprocess/precommit/personality/provided.sh | | git revision | trunk / 64f28f9 | | maven | version: Apache Maven 3.3.9 | | Default Java | 1.8.0_191 | | findbugs | v3.1.0-RC1 | | checkstyle | https://builds.apache.org/job/PreCommit-HADOOP-Build/15924/artifact/out/diff-checkstyle-hadoop-common-project_hadoop-kms.txt | | whitespace | https://builds.apache.org/job/PreCommit-HADOOP-Build/15924/artifact/out/whitespace-eol.txt | | Test Results | https://builds.apache.org/job/PreCommit-HADOOP-Build/15924/testReport/ | | Max. process+thread count | 305 (vs. ulimit of 1) | | modules | C: