[jira] [Commented] (HADOOP-13597) Switch KMS from Tomcat to Jetty
[ https://issues.apache.org/jira/browse/HADOOP-13597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15803202#comment-15803202 ] Hudson commented on HADOOP-13597: - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11078 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/11078/]) HADOOP-13597. Switch KMS from Tomcat to Jetty. Contributed by John (xiao: rev 5d182949badb2eb80393de7ba3838102d006488b) * (add) hadoop-common-project/hadoop-kms/src/site/configuration.xsl * (delete) hadoop-common-project/hadoop-kms/src/main/tomcat/ssl-server.xml.conf * (edit) hadoop-common-project/hadoop-kms/src/main/conf/kms-log4j.properties * (delete) hadoop-common-project/hadoop-kms/src/main/tomcat/logging.properties * (delete) hadoop-common-project/hadoop-kms/src/main/tomcat/ROOT/index.html * (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/http/HttpServer2.java * (edit) hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSWebApp.java * (add) hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSWebServer.java * (add) hadoop-common-project/hadoop-kms/src/main/resources/webapps/static/index.html * (edit) hadoop-common-project/hadoop-kms/src/test/java/org/apache/hadoop/crypto/key/kms/server/MiniKMS.java * (edit) hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/server/AuthenticationFilter.java * (edit) hadoop-common-project/hadoop-kms/src/main/sbin/kms.sh * (edit) hadoop-common-project/hadoop-kms/src/main/conf/kms-site.xml * (add) hadoop-common-project/hadoop-kms/src/main/resources/kms-default.xml * (edit) hadoop-common-project/hadoop-kms/src/main/conf/kms-env.sh * (delete) hadoop-common-project/hadoop-kms/src/main/libexec/kms-config.sh * (delete) hadoop-common-project/hadoop-kms/src/main/tomcat/server.xml * (edit) hadoop-common-project/hadoop-kms/dev-support/findbugsExcludeFile.xml * (delete) hadoop-common-project/hadoop-kms/src/main/tomcat/ROOT/WEB-INF/web.xml * (edit) hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSConfiguration.java * (edit) hadoop-common-project/hadoop-common/src/site/markdown/CommandsManual.md * (delete) hadoop-common-project/hadoop-kms/src/main/webapp/WEB-INF/web.xml * (edit) hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh * (add) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/ConfigurationWithLogging.java * (add) hadoop-common-project/hadoop-kms/src/main/resources/webapps/kms/WEB-INF/web.xml * (add) hadoop-common-project/hadoop-common/src/test/scripts/hadoop_using_envvar.bats * (add) hadoop-common-project/hadoop-kms/src/main/libexec/shellprofile.d/hadoop-kms.sh * (add) hadoop-common-project/hadoop-common/src/test/scripts/hadoop_mkdir.bats * (edit) hadoop-common-project/hadoop-kms/src/site/markdown/index.md.vm * (delete) hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSJMXServlet.java * (edit) hadoop-common-project/hadoop-kms/pom.xml * (edit) hadoop-assemblies/src/main/resources/assemblies/hadoop-kms-dist.xml * (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/ssl/SSLFactory.java > Switch KMS from Tomcat to Jetty > --- > > Key: HADOOP-13597 > URL: https://issues.apache.org/jira/browse/HADOOP-13597 > Project: Hadoop Common > Issue Type: New Feature > Components: kms >Affects Versions: 2.6.0 >Reporter: John Zhuge >Assignee: John Zhuge > Fix For: 3.0.0-alpha2 > > Attachments: HADOOP-13597.001.patch, HADOOP-13597.002.patch, > HADOOP-13597.003.patch, HADOOP-13597.004.patch, HADOOP-13597.005.patch, > HADOOP-13597.006.patch, HADOOP-13597.007.patch > > > The Tomcat 6 we are using will reach EOL at the end of 2017. While there are > other good options, I would propose switching to {{Jetty 9}} for the > following reasons: > * Easier migration. Both Tomcat and Jetty are based on {{Servlet > Containers}}, so we don't have change client code that much. It would require > more work to switch to {{JAX-RS}}. > * Well established. > * Good performance and scalability. > Other alternatives: > * Jersey + Grizzly > * Tomcat 8 > Your opinions will be greatly appreciated. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13597) Switch KMS from Tomcat to Jetty
[ https://issues.apache.org/jira/browse/HADOOP-13597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15803175#comment-15803175 ] John Zhuge commented on HADOOP-13597: - File HADOOP-13955 to follow up. Will add relnotes. Thanks a lot [~xiaochen] for the review and commit. Thanks [~aw], [~jojochuang], [~andrew.wang], and [~ste...@apache.org] for reviews and discussions. > Switch KMS from Tomcat to Jetty > --- > > Key: HADOOP-13597 > URL: https://issues.apache.org/jira/browse/HADOOP-13597 > Project: Hadoop Common > Issue Type: New Feature > Components: kms >Affects Versions: 2.6.0 >Reporter: John Zhuge >Assignee: John Zhuge > Fix For: 3.0.0-alpha2 > > Attachments: HADOOP-13597.001.patch, HADOOP-13597.002.patch, > HADOOP-13597.003.patch, HADOOP-13597.004.patch, HADOOP-13597.005.patch, > HADOOP-13597.006.patch, HADOOP-13597.007.patch > > > The Tomcat 6 we are using will reach EOL at the end of 2017. While there are > other good options, I would propose switching to {{Jetty 9}} for the > following reasons: > * Easier migration. Both Tomcat and Jetty are based on {{Servlet > Containers}}, so we don't have change client code that much. It would require > more work to switch to {{JAX-RS}}. > * Well established. > * Good performance and scalability. > Other alternatives: > * Jersey + Grizzly > * Tomcat 8 > Your opinions will be greatly appreciated. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13597) Switch KMS from Tomcat to Jetty
[ https://issues.apache.org/jira/browse/HADOOP-13597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15803116#comment-15803116 ] Xiao Chen commented on HADOOP-13597: Had a final pass of the patch, +1. Verified the built documentation looks good. Verified no perf regression on my local Max, in a simple non-ssl test, using apache benchmark: {{ab -n 1 -c 10 -T "application/json" -p postfile "http://IP:9600/kms/v1/keyversion/k1%408/_eek?eek_op=decrypt=whatever"}} {noformat:title=before} This is ApacheBench, Version 2.3 <$Revision: 1748469 $> Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/ Licensed to The Apache Software Foundation, http://www.apache.org/ Benchmarking 172.16.3.181 (be patient) Completed 1000 requests Completed 2000 requests Completed 3000 requests Completed 4000 requests Completed 5000 requests Completed 6000 requests Completed 7000 requests Completed 8000 requests Completed 9000 requests Completed 1 requests Finished 1 requests Server Software:Apache-Coyote/1.1 Server Hostname:172.16.3.181 Server Port:9600 Document Path: /kms/v1/keyversion/k1%408/_eek?eek_op=decrypt=whatever Document Length:84 bytes Concurrency Level: 10 Time taken for tests: 1.881 seconds Complete requests: 1 Failed requests:0 Total transferred: 354 bytes Total body sent:294 HTML transferred: 84 bytes Requests per second:5316.11 [#/sec] (mean) Time per request: 1.881 [ms] (mean) Time per request: 0.188 [ms] (mean, across all concurrent requests) Transfer rate: 1837.79 [Kbytes/sec] received 1526.30 kb/s sent 3364.10 kb/s total Connection Times (ms) min mean[+/-sd] median max Connect:00 0.3 0 17 Processing: 11 1.5 1 49 Waiting:01 1.4 1 49 Total: 12 1.5 2 49 Percentage of the requests served within a certain time (ms) 50% 2 66% 2 75% 2 80% 2 90% 3 95% 3 98% 5 99% 7 100% 49 (longest request) {noformat} {noformat:title=after} xiao-MBP:Downloads xiao$ ab -n 5000 -c 10 -T "application/json" -p postfile "http://172.16.3.181:9600/kms/v1/keyversion/k1%408/_eek?eek_op=decrypt=whatever; This is ApacheBench, Version 2.3 <$Revision: 1748469 $> Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/ Licensed to The Apache Software Foundation, http://www.apache.org/ Benchmarking 172.16.3.181 (be patient) Completed 500 requests Completed 1000 requests Completed 1500 requests Completed 2000 requests Completed 2500 requests Completed 3000 requests Completed 3500 requests Completed 4000 requests Completed 4500 requests Completed 5000 requests Finished 5000 requests Server Software:Jetty(9.3.11.v20160721) Server Hostname:172.16.3.181 Server Port:9600 Document Path: /kms/v1/keyversion/k1%408/_eek?eek_op=decrypt=whatever Document Length:84 bytes Concurrency Level: 10 Time taken for tests: 0.949 seconds Complete requests: 5000 Failed requests:0 Total transferred: 2315000 bytes Total body sent:1475000 HTML transferred: 42 bytes Requests per second:5267.81 [#/sec] (mean) Time per request: 1.898 [ms] (mean) Time per request: 0.190 [ms] (mean, across all concurrent requests) Transfer rate: 2381.83 [Kbytes/sec] received 1517.58 kb/s sent 3899.41 kb/s total Connection Times (ms) min mean[+/-sd] median max Connect:01 0.3 0 3 Processing: 11 0.8 1 12 Waiting:01 0.7 1 11 Total: 12 0.9 2 12 ERROR: The median and mean for the initial connection time are more than twice the standard deviation apart. These results are NOT reliable. Percentage of the requests served within a certain time (ms) 50% 2 66% 2 75% 2 80% 2 90% 3 95% 3 98% 4 99% 4 100% 12 (longest request) {noformat} (Tried 1 but seems to hang around 6k for a while, guessing it's because of some changed defaults etc. IMO as long as the latency is at the same level, we can always tune the threads etc as needed.) Committing this. Great work here [~jzhuge]! Do you mind file a follow-on to fix the javac? > Switch KMS from Tomcat to Jetty > --- > > Key: HADOOP-13597 > URL: https://issues.apache.org/jira/browse/HADOOP-13597 > Project: Hadoop Common > Issue Type: New Feature > Components: kms >Affects Versions: 2.6.0 >Reporter: John Zhuge >Assignee: John Zhuge >
[jira] [Commented] (HADOOP-13597) Switch KMS from Tomcat to Jetty
[ https://issues.apache.org/jira/browse/HADOOP-13597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15797873#comment-15797873 ] Hadoop QA commented on HADOOP-13597: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 13s{color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 3 new or modified test files. {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 47s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 12m 31s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m 40s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 37s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 23s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m 31s{color} | {color:green} trunk passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Skipped patched modules with no Java source: hadoop-assemblies {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 40s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 10s{color} | {color:green} trunk passed {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 20s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 32s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m 23s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} javac {color} | {color:red} 9m 23s{color} | {color:red} root generated 4 new + 690 unchanged - 0 fixed = 694 total (was 690) {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 1m 42s{color} | {color:orange} root: The patch generated 1 new + 98 unchanged - 6 fixed = 99 total (was 104) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m 9s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m 51s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} shellcheck {color} | {color:green} 0m 16s{color} | {color:green} The patch generated 0 new + 566 unchanged - 9 fixed = 566 total (was 575) {color} | | {color:green}+1{color} | {color:green} shelldocs {color} | {color:green} 0m 23s{color} | {color:green} The patch generated 0 new + 372 unchanged - 4 fixed = 372 total (was 376) {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 7s{color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Skipped patched modules with no Java source: hadoop-assemblies {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 25s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 38s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 27s{color} | {color:green} hadoop-assemblies in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m 47s{color} | {color:green} hadoop-auth in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 10m 14s{color} | {color:green} hadoop-common in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 28s{color} | {color:green} hadoop-kms in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 46s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} |
[jira] [Commented] (HADOOP-13597) Switch KMS from Tomcat to Jetty
[ https://issues.apache.org/jira/browse/HADOOP-13597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15797457#comment-15797457 ] Xiao Chen commented on HADOOP-13597: Thanks [~jojochuang] for the review. Good catch, searching for tomcat I can also see {{SecureMode.md}} has a mention of tomcat, maybe better changed in the httpfs jira though (since that goes in later). Also 1 reference of tomcat in {{hadoop-common-project/hadoop-kms/dev-support/findbugsExcludeFile.xml}}, can be updated here. +1 pending those 2 nits though. Since it's a fairly big and impactful patch, will let it float a few days in case other watchers have comments. Plan to commit by Thursday if no objections. > Switch KMS from Tomcat to Jetty > --- > > Key: HADOOP-13597 > URL: https://issues.apache.org/jira/browse/HADOOP-13597 > Project: Hadoop Common > Issue Type: New Feature > Components: kms >Affects Versions: 2.6.0 >Reporter: John Zhuge >Assignee: John Zhuge > Attachments: HADOOP-13597.001.patch, HADOOP-13597.002.patch, > HADOOP-13597.003.patch, HADOOP-13597.004.patch, HADOOP-13597.005.patch, > HADOOP-13597.006.patch > > > The Tomcat 6 we are using will reach EOL at the end of 2017. While there are > other good options, I would propose switching to {{Jetty 9}} for the > following reasons: > * Easier migration. Both Tomcat and Jetty are based on {{Servlet > Containers}}, so we don't have change client code that much. It would require > more work to switch to {{JAX-RS}}. > * Well established. > * Good performance and scalability. > Other alternatives: > * Jersey + Grizzly > * Tomcat 8 > Your opinions will be greatly appreciated. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13597) Switch KMS from Tomcat to Jetty
[ https://issues.apache.org/jira/browse/HADOOP-13597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15795365#comment-15795365 ] Wei-Chiu Chuang commented on HADOOP-13597: -- Thanks [~jzhuge] and [~xiaochen] for driving forward this far. I think the patch is good to me. There's one nit in doc CommandsManual.md though: {quote} However, the command does not support KMS server, because its web interface is based on Tomcat, which does not support the servlet. {quote} This statement will no longer be valid after this change. > Switch KMS from Tomcat to Jetty > --- > > Key: HADOOP-13597 > URL: https://issues.apache.org/jira/browse/HADOOP-13597 > Project: Hadoop Common > Issue Type: New Feature > Components: kms >Affects Versions: 2.6.0 >Reporter: John Zhuge >Assignee: John Zhuge > Attachments: HADOOP-13597.001.patch, HADOOP-13597.002.patch, > HADOOP-13597.003.patch, HADOOP-13597.004.patch, HADOOP-13597.005.patch, > HADOOP-13597.006.patch > > > The Tomcat 6 we are using will reach EOL at the end of 2017. While there are > other good options, I would propose switching to {{Jetty 9}} for the > following reasons: > * Easier migration. Both Tomcat and Jetty are based on {{Servlet > Containers}}, so we don't have change client code that much. It would require > more work to switch to {{JAX-RS}}. > * Well established. > * Good performance and scalability. > Other alternatives: > * Jersey + Grizzly > * Tomcat 8 > Your opinions will be greatly appreciated. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13597) Switch KMS from Tomcat to Jetty
[ https://issues.apache.org/jira/browse/HADOOP-13597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15771401#comment-15771401 ] Xiao Chen commented on HADOOP-13597: Thanks [~jzhuge] for revving. I think the patch is generally ready. Considering the holiday season, I think we should wait awhile for the feedback from others. I'll go ahead and review the httpfs one first and check back later next week. Hopefully we can get this in right into the new year, if not earlier. :) > Switch KMS from Tomcat to Jetty > --- > > Key: HADOOP-13597 > URL: https://issues.apache.org/jira/browse/HADOOP-13597 > Project: Hadoop Common > Issue Type: New Feature > Components: kms >Affects Versions: 2.6.0 >Reporter: John Zhuge >Assignee: John Zhuge > Attachments: HADOOP-13597.001.patch, HADOOP-13597.002.patch, > HADOOP-13597.003.patch, HADOOP-13597.004.patch, HADOOP-13597.005.patch, > HADOOP-13597.006.patch > > > The Tomcat 6 we are using will reach EOL at the end of 2017. While there are > other good options, I would propose switching to {{Jetty 9}} for the > following reasons: > * Easier migration. Both Tomcat and Jetty are based on {{Servlet > Containers}}, so we don't have change client code that much. It would require > more work to switch to {{JAX-RS}}. > * Well established. > * Good performance and scalability. > Other alternatives: > * Jersey + Grizzly > * Tomcat 8 > Your opinions will be greatly appreciated. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13597) Switch KMS from Tomcat to Jetty
[ https://issues.apache.org/jira/browse/HADOOP-13597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15771334#comment-15771334 ] John Zhuge commented on HADOOP-13597: - Puzzled by the "ConfigurationWithLogging.java:0:: Missing package-info.java file." checkstyle warning. Didn't see the issue in previous patches. Javac warnings of deprecation can be ignored. Do not want to touch too much code in this patch. > Switch KMS from Tomcat to Jetty > --- > > Key: HADOOP-13597 > URL: https://issues.apache.org/jira/browse/HADOOP-13597 > Project: Hadoop Common > Issue Type: New Feature > Components: kms >Affects Versions: 2.6.0 >Reporter: John Zhuge >Assignee: John Zhuge > Attachments: HADOOP-13597.001.patch, HADOOP-13597.002.patch, > HADOOP-13597.003.patch, HADOOP-13597.004.patch, HADOOP-13597.005.patch, > HADOOP-13597.006.patch > > > The Tomcat 6 we are using will reach EOL at the end of 2017. While there are > other good options, I would propose switching to {{Jetty 9}} for the > following reasons: > * Easier migration. Both Tomcat and Jetty are based on {{Servlet > Containers}}, so we don't have change client code that much. It would require > more work to switch to {{JAX-RS}}. > * Well established. > * Good performance and scalability. > Other alternatives: > * Jersey + Grizzly > * Tomcat 8 > Your opinions will be greatly appreciated. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13597) Switch KMS from Tomcat to Jetty
[ https://issues.apache.org/jira/browse/HADOOP-13597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15771287#comment-15771287 ] Hadoop QA commented on HADOOP-13597: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 15s{color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 3 new or modified test files. {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 18s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m 44s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m 13s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 41s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 36s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m 38s{color} | {color:green} trunk passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Skipped patched modules with no Java source: hadoop-assemblies {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 43s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 16s{color} | {color:green} trunk passed {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 19s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 36s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m 31s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} javac {color} | {color:red} 10m 31s{color} | {color:red} root generated 4 new + 690 unchanged - 0 fixed = 694 total (was 690) {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 1m 56s{color} | {color:orange} root: The patch generated 1 new + 98 unchanged - 6 fixed = 99 total (was 104) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m 35s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m 51s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} shellcheck {color} | {color:green} 0m 17s{color} | {color:green} The patch generated 0 new + 566 unchanged - 9 fixed = 566 total (was 575) {color} | | {color:green}+1{color} | {color:green} shelldocs {color} | {color:green} 0m 24s{color} | {color:green} The patch generated 0 new + 372 unchanged - 4 fixed = 372 total (was 376) {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 7s{color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Skipped patched modules with no Java source: hadoop-assemblies {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 41s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 38s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 29s{color} | {color:green} hadoop-assemblies in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m 53s{color} | {color:green} hadoop-auth in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 11m 2s{color} | {color:green} hadoop-common in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 37s{color} | {color:green} hadoop-kms in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 53s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} |
[jira] [Commented] (HADOOP-13597) Switch KMS from Tomcat to Jetty
[ https://issues.apache.org/jira/browse/HADOOP-13597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15765808#comment-15765808 ] Xiao Chen commented on HADOOP-13597: Thanks for the new rev [~jzhuge]. Feels pretty close to me. Seems all comments are addressed to me too. I only have the following book-keeping comments, other than that LGTM. - ConfigurationWithLogging.java: javadoc should say about redaction. - kms doc: we should still mention 9600 is the default port in {{KMS Configuration}}. - kms doc: the envvars (e.g. KMS_MAX_THREADS) needs documentation. - kms doc: {{/jmx, /logLevel, /conf, /stack, /logs, and /static/index.html}} needs documentation. - The linked 'breaks' HADOOP-13872 KMS JMX exception seem to be fixed by this. Could you confirm, and if so, close that one out? Don't see any option close to 'fixed by', but I think 'relates to' with a comment should be clear enough. It's a fairly big patch, appreciate if other reviewers can take a look. > Switch KMS from Tomcat to Jetty > --- > > Key: HADOOP-13597 > URL: https://issues.apache.org/jira/browse/HADOOP-13597 > Project: Hadoop Common > Issue Type: New Feature > Components: kms >Affects Versions: 2.6.0 >Reporter: John Zhuge >Assignee: John Zhuge > Attachments: HADOOP-13597.001.patch, HADOOP-13597.002.patch, > HADOOP-13597.003.patch, HADOOP-13597.004.patch, HADOOP-13597.005.patch > > > The Tomcat 6 we are using will reach EOL at the end of 2017. While there are > other good options, I would propose switching to {{Jetty 9}} for the > following reasons: > * Easier migration. Both Tomcat and Jetty are based on {{Servlet > Containers}}, so we don't have change client code that much. It would require > more work to switch to {{JAX-RS}}. > * Well established. > * Good performance and scalability. > Other alternatives: > * Jersey + Grizzly > * Tomcat 8 > Your opinions will be greatly appreciated. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13597) Switch KMS from Tomcat to Jetty
[ https://issues.apache.org/jira/browse/HADOOP-13597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15750007#comment-15750007 ] John Zhuge commented on HADOOP-13597: - I think it is fixed: {code} hadoop.kms.authentication.signer.secret.provider.zookeeper.auth.type none The Zookeeper authentication type, 'none' (default) or 'sasl' (Kerberos). {code} > Switch KMS from Tomcat to Jetty > --- > > Key: HADOOP-13597 > URL: https://issues.apache.org/jira/browse/HADOOP-13597 > Project: Hadoop Common > Issue Type: New Feature > Components: kms >Affects Versions: 2.6.0 >Reporter: John Zhuge >Assignee: John Zhuge > Attachments: HADOOP-13597.001.patch, HADOOP-13597.002.patch, > HADOOP-13597.003.patch, HADOOP-13597.004.patch > > > The Tomcat 6 we are using will reach EOL at the end of 2017. While there are > other good options, I would propose switching to {{Jetty 9}} for the > following reasons: > * Easier migration. Both Tomcat and Jetty are based on {{Servlet > Containers}}, so we don't have change client code that much. It would require > more work to switch to {{JAX-RS}}. > * Well established. > * Good performance and scalability. > Other alternatives: > * Jersey + Grizzly > * Tomcat 8 > Your opinions will be greatly appreciated. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13597) Switch KMS from Tomcat to Jetty
[ https://issues.apache.org/jira/browse/HADOOP-13597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15750004#comment-15750004 ] John Zhuge commented on HADOOP-13597: - bq. We should keep that unchanged. The kerberos default value also doesn't work, it should be sasl. [~xiaochen] I don't understand. Could you check Patch 004? Thanks. > Switch KMS from Tomcat to Jetty > --- > > Key: HADOOP-13597 > URL: https://issues.apache.org/jira/browse/HADOOP-13597 > Project: Hadoop Common > Issue Type: New Feature > Components: kms >Affects Versions: 2.6.0 >Reporter: John Zhuge >Assignee: John Zhuge > Attachments: HADOOP-13597.001.patch, HADOOP-13597.002.patch, > HADOOP-13597.003.patch, HADOOP-13597.004.patch > > > The Tomcat 6 we are using will reach EOL at the end of 2017. While there are > other good options, I would propose switching to {{Jetty 9}} for the > following reasons: > * Easier migration. Both Tomcat and Jetty are based on {{Servlet > Containers}}, so we don't have change client code that much. It would require > more work to switch to {{JAX-RS}}. > * Well established. > * Good performance and scalability. > Other alternatives: > * Jersey + Grizzly > * Tomcat 8 > Your opinions will be greatly appreciated. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13597) Switch KMS from Tomcat to Jetty
[ https://issues.apache.org/jira/browse/HADOOP-13597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15743171#comment-15743171 ] John Zhuge commented on HADOOP-13597: - Thanks for the review. Will remove hadoop_deprecate_envvar and document the deprecated envvars in Release Notes. Will switch hadoop_using_envvar to use hadoop_debug. And will skip passoword envvars. Will change other similar places to use hadoop_mkdir. > Switch KMS from Tomcat to Jetty > --- > > Key: HADOOP-13597 > URL: https://issues.apache.org/jira/browse/HADOOP-13597 > Project: Hadoop Common > Issue Type: New Feature > Components: kms >Affects Versions: 2.6.0 >Reporter: John Zhuge >Assignee: John Zhuge > Attachments: HADOOP-13597.001.patch, HADOOP-13597.002.patch, > HADOOP-13597.003.patch, HADOOP-13597.004.patch > > > The Tomcat 6 we are using will reach EOL at the end of 2017. While there are > other good options, I would propose switching to {{Jetty 9}} for the > following reasons: > * Easier migration. Both Tomcat and Jetty are based on {{Servlet > Containers}}, so we don't have change client code that much. It would require > more work to switch to {{JAX-RS}}. > * Well established. > * Good performance and scalability. > Other alternatives: > * Jersey + Grizzly > * Tomcat 8 > Your opinions will be greatly appreciated. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13597) Switch KMS from Tomcat to Jetty
[ https://issues.apache.org/jira/browse/HADOOP-13597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15742931#comment-15742931 ] Allen Wittenauer commented on HADOOP-13597: --- {code} + hadoop_deprecate_envvar CATALINA_OUT + hadoop_deprecate_envvar CATALINA_PID + hadoop_deprecate_envvar KMS_ADMIN_PORT + hadoop_deprecate_envvar KMS_CATALINA_HOME + hadoop_deprecate_envvar KMS_SSL_TRUSTSTORE_PASS {code} We don't do this anywhere in the scripts. Instead, this is documented in the release notes. It's just extra console noise otherwise. {code} + hadoop_using_envvar KMS_HOME {code} This doesn't appear to have actually be configurable by users. I don't see a reason to add it now. {code} + hadoop_using_envvar KMS_HTTP_PORT + hadoop_using_envvar KMS_LOG + hadoop_using_envvar KMS_MAX_HTTP_HEADER_SIZE + hadoop_using_envvar KMS_MAX_THREADS + hadoop_using_envvar KMS_SSL_ENABLED + hadoop_using_envvar KMS_SSL_KEYSTORE_FILE + hadoop_using_envvar KMS_SSL_KEYSTORE_PASS + hadoop_using_envvar KMS_TEMP {code} I know that branch-2 spit out a bunch of stuff, but it always felt wrong. Is this actually valuable to anyone that aren't developers? Would \-\-debug be a better usage here? It seems like a lot of noise on the console that's probably more appropriate for a log file {code} + hadoop_using_envvar KMS_SSL_KEYSTORE_PASS {code} (!) > Switch KMS from Tomcat to Jetty > --- > > Key: HADOOP-13597 > URL: https://issues.apache.org/jira/browse/HADOOP-13597 > Project: Hadoop Common > Issue Type: New Feature > Components: kms >Affects Versions: 2.6.0 >Reporter: John Zhuge >Assignee: John Zhuge > Attachments: HADOOP-13597.001.patch, HADOOP-13597.002.patch, > HADOOP-13597.003.patch, HADOOP-13597.004.patch > > > The Tomcat 6 we are using will reach EOL at the end of 2017. While there are > other good options, I would propose switching to {{Jetty 9}} for the > following reasons: > * Easier migration. Both Tomcat and Jetty are based on {{Servlet > Containers}}, so we don't have change client code that much. It would require > more work to switch to {{JAX-RS}}. > * Well established. > * Good performance and scalability. > Other alternatives: > * Jersey + Grizzly > * Tomcat 8 > Your opinions will be greatly appreciated. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13597) Switch KMS from Tomcat to Jetty
[ https://issues.apache.org/jira/browse/HADOOP-13597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15742930#comment-15742930 ] John Zhuge commented on HADOOP-13597: - Agree with your complexity concern. I intially created a separate API but decided to multiplex {{hadoop_deprecate_envvar}} because it is such a good name :) Any suggestion on the separate API? How about {{hadoop_retire_envvar}}? > Switch KMS from Tomcat to Jetty > --- > > Key: HADOOP-13597 > URL: https://issues.apache.org/jira/browse/HADOOP-13597 > Project: Hadoop Common > Issue Type: New Feature > Components: kms >Affects Versions: 2.6.0 >Reporter: John Zhuge >Assignee: John Zhuge > Attachments: HADOOP-13597.001.patch, HADOOP-13597.002.patch, > HADOOP-13597.003.patch, HADOOP-13597.004.patch > > > The Tomcat 6 we are using will reach EOL at the end of 2017. While there are > other good options, I would propose switching to {{Jetty 9}} for the > following reasons: > * Easier migration. Both Tomcat and Jetty are based on {{Servlet > Containers}}, so we don't have change client code that much. It would require > more work to switch to {{JAX-RS}}. > * Well established. > * Good performance and scalability. > Other alternatives: > * Jersey + Grizzly > * Tomcat 8 > Your opinions will be greatly appreciated. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13597) Switch KMS from Tomcat to Jetty
[ https://issues.apache.org/jira/browse/HADOOP-13597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15742886#comment-15742886 ] Allen Wittenauer commented on HADOOP-13597: --- OK, I misread oldvar as newvar in the patch file. I'm not a fan of the change since it just increases the complexity of the code and the run time when oldvar is in use. > Switch KMS from Tomcat to Jetty > --- > > Key: HADOOP-13597 > URL: https://issues.apache.org/jira/browse/HADOOP-13597 > Project: Hadoop Common > Issue Type: New Feature > Components: kms >Affects Versions: 2.6.0 >Reporter: John Zhuge >Assignee: John Zhuge > Attachments: HADOOP-13597.001.patch, HADOOP-13597.002.patch, > HADOOP-13597.003.patch, HADOOP-13597.004.patch > > > The Tomcat 6 we are using will reach EOL at the end of 2017. While there are > other good options, I would propose switching to {{Jetty 9}} for the > following reasons: > * Easier migration. Both Tomcat and Jetty are based on {{Servlet > Containers}}, so we don't have change client code that much. It would require > more work to switch to {{JAX-RS}}. > * Well established. > * Good performance and scalability. > Other alternatives: > * Jersey + Grizzly > * Tomcat 8 > Your opinions will be greatly appreciated. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13597) Switch KMS from Tomcat to Jetty
[ https://issues.apache.org/jira/browse/HADOOP-13597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15739405#comment-15739405 ] Hadoop QA commented on HADOOP-13597: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 12s{color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 4 new or modified test files. {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 17s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m 49s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m 34s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 38s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 27s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m 32s{color} | {color:green} trunk passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Skipped patched modules with no Java source: hadoop-assemblies {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 40s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 9s{color} | {color:green} trunk passed {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 20s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 32s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m 16s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} javac {color} | {color:red} 9m 16s{color} | {color:red} root generated 4 new + 709 unchanged - 0 fixed = 713 total (was 709) {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 1m 42s{color} | {color:orange} root: The patch generated 1 new + 98 unchanged - 6 fixed = 99 total (was 104) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 44s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m 50s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} shellcheck {color} | {color:green} 0m 14s{color} | {color:green} The patch generated 0 new + 568 unchanged - 8 fixed = 568 total (was 576) {color} | | {color:green}+1{color} | {color:green} shelldocs {color} | {color:green} 0m 22s{color} | {color:green} The patch generated 0 new + 372 unchanged - 4 fixed = 372 total (was 376) {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 6s{color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Skipped patched modules with no Java source: hadoop-assemblies {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 13s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 28s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 28s{color} | {color:green} hadoop-assemblies in the patch passed. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 3m 42s{color} | {color:red} hadoop-auth in the patch failed. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 8m 9s{color} | {color:red} hadoop-common in the patch failed. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 2m 8s{color} | {color:red} hadoop-kms in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 44s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} |
[jira] [Commented] (HADOOP-13597) Switch KMS from Tomcat to Jetty
[ https://issues.apache.org/jira/browse/HADOOP-13597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15739208#comment-15739208 ] John Zhuge commented on HADOOP-13597: - These are {{getPassword}} variations: - S3AUtils#lookupPassword(Configuration conf, String key, String defVal) - LdapGroupsMapping#getPassword(Configuration conf, String alias, String defaultPass) - FileBasedKeyStoresFactory#getPassword(Configuration conf, String alias, String defaultPass) - DFSUtil#getPassword(Configuration conf, String alias) - WebAppUtils#getPassword(Configuration conf, String alias) These are various features of these wrappers: - Return String type. They all do that. What was the reason for {{Configuration#getPassword}} to return {{char[]}}? {{char[]}} is hard to work with, especially with the possibility of null. - Default value. Some APIs do have default value, thus avoid null return. - Swallow IOE from {{Configuration#getPassword}} and return default value or null - Re-throw IOE from {{Configuration#getPassword}} with extra message. > Switch KMS from Tomcat to Jetty > --- > > Key: HADOOP-13597 > URL: https://issues.apache.org/jira/browse/HADOOP-13597 > Project: Hadoop Common > Issue Type: New Feature > Components: kms >Affects Versions: 2.6.0 >Reporter: John Zhuge >Assignee: John Zhuge > Attachments: HADOOP-13597.001.patch, HADOOP-13597.002.patch, > HADOOP-13597.003.patch > > > The Tomcat 6 we are using will reach EOL at the end of 2017. While there are > other good options, I would propose switching to {{Jetty 9}} for the > following reasons: > * Easier migration. Both Tomcat and Jetty are based on {{Servlet > Containers}}, so we don't have change client code that much. It would require > more work to switch to {{JAX-RS}}. > * Well established. > * Good performance and scalability. > Other alternatives: > * Jersey + Grizzly > * Tomcat 8 > Your opinions will be greatly appreciated. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13597) Switch KMS from Tomcat to Jetty
[ https://issues.apache.org/jira/browse/HADOOP-13597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15738918#comment-15738918 ] John Zhuge commented on HADOOP-13597: - Thanks a lot, Xiao ! bq. HttpServer2: Don't understand the skipSecretProvider. Could you explain and add some comments/javadocs? Is this about to the AuthenticationFilter and it's related secret providers startup tricks? Remove it. Don't recall why I added it. bq. HttpServer2: Possible to have createHttpsChannelConnector call createHttpChannelConnector first, to reduce some duplicate codes? The httpConfig.setSecureScheme(HTTPS_SCHEME); line seems reasonable to be inside the method too. Fixed. bq. I found the AccessLoggingConfiguration class naming confusing. Looking at the class javadoc didn't help much either - I only figured out until looking at the code usage. Can't find a good replacement in my vocabulary (appreciate if anyone else has better naming), but we should state in javadoc: 1 this a configuration object that logs each access. 2. it redacts sensitive information. Actually, maybe this is better to be a composites a Configuration rather than inherits? At least whoever uses it later don't have to figure out which method is an Override etc. (BTW, currently missing @Override annotations on all methods, and set seems to be missing a super.set.) Fixed. Is {{ConfigurationWithLogging}} a better name? This class is extremely useful during development. Now it has served its purpose, I am ok to remove it. bq. KMSWebServer: Nit - I think hadoop code mostly have static import on 1 line and ignore the 80-char rules. Fixed. bq. KMSWebServer: Totally theoretical, it may be good to also have the isAlive method, and probably add a waitActive-ish method in the MiniKMS, so interested tests can call that and reduce flaky tests due to start up race. Good idea. Can we create a follow-up JIRA to add it later when we modify the flaky tests? Otherwise, the new method is unused and untested. bq. Searching for tomcat, see several nitty references still: AuthenticationFiler's var name isInitializedByTomcat, CommandsManual.md and SecureMode.md, KMS's doc index.md.vm, and some code comments etc. TODO to update the docs. Investigating {{AuthenticationFiler#isInitializedByTomcat}}. bq. For the passwords, agree with Robert on supportability. However I also see similar code in DFSUtil (loadSslConfToHttpServerBuilder and getPassword). Was these copied over? We should at least move that to a common util, and avoid this level of duplication. This will probably leave us not having to change Configuration, but adding a wrapper util. Or per Wei-Chiu's suggestion, maybe not needed any more. Appreciate more javadocs here too, as to why such method is needed. I will start a separate thread. Quite a few duplicate code in HDFS, YARN, and AWS. bq. Didn't see answer to Allen's ask about unit tests. (Take a look at hadoop-common-project/hadoop-common/src/test/scripts if you're wondering how that's done). Fixed. bq. Nit: kms-site.xml is following other -site.xmls, to have a comment line "put site-specific...", which is good. Please follow them closer to have this line before the element. Fixed. > Switch KMS from Tomcat to Jetty > --- > > Key: HADOOP-13597 > URL: https://issues.apache.org/jira/browse/HADOOP-13597 > Project: Hadoop Common > Issue Type: New Feature > Components: kms >Affects Versions: 2.6.0 >Reporter: John Zhuge >Assignee: John Zhuge > Attachments: HADOOP-13597.001.patch, HADOOP-13597.002.patch, > HADOOP-13597.003.patch > > > The Tomcat 6 we are using will reach EOL at the end of 2017. While there are > other good options, I would propose switching to {{Jetty 9}} for the > following reasons: > * Easier migration. Both Tomcat and Jetty are based on {{Servlet > Containers}}, so we don't have change client code that much. It would require > more work to switch to {{JAX-RS}}. > * Well established. > * Good performance and scalability. > Other alternatives: > * Jersey + Grizzly > * Tomcat 8 > Your opinions will be greatly appreciated. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13597) Switch KMS from Tomcat to Jetty
[ https://issues.apache.org/jira/browse/HADOOP-13597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15737413#comment-15737413 ] Xiao Chen commented on HADOOP-13597: Thanks John for revving and others for reviewing. My comments below: - {{HttpServer2}}: Don't understand the {{skipSecretProvider}}. Could you explain and add some comments/javadocs? Is this about to the AuthenticationFilter and it's related secret providers startup tricks? - {{HttpServer2}}: Possible to have {{createHttpsChannelConnector}} call {{createHttpChannelConnector}} first, to reduce some duplicate codes? The {{httpConfig.setSecureScheme(HTTPS_SCHEME);}} line seems reasonable to be inside the method too. - I found the {{AccessLoggingConfiguration}} class naming confusing. Looking at the class javadoc didn't help much either - I only figured out until looking at the code usage. Can't find a good replacement in my vocabulary (appreciate if anyone else has better naming), but we should state in javadoc: 1 this a configuration object that logs each access. 2. it redacts sensitive information. Actually, maybe this is better to be a composites a {{Configuration}} rather than inherits? At least whoever uses it later don't have to figure out which method is an Override etc. (BTW, currently missing {{@Override}} annotations on all methods, and {{set}} seems to be missing a {{super.set}}.) - {{KMSWebServer}}: Nit - I think hadoop code mostly have static import on 1 line and ignore the 80-char rules. - {{KMSWebServer}}: Totally theoretical, it may be good to also have the {{isAlive}} method, and probably add a {{waitActive}}-ish method in the MiniKMS, so interested tests can call that and reduce flaky tests due to start up race. - Searching for {{tomcat}}, see several nitty references still: {{AuthenticationFiler}}'s var name {{isInitializedByTomcat}}, {{CommandsManual.md}} and {{SecureMode.md}}, KMS's doc {{index.md.vm}}, and some code comments etc. - For the passwords, agree with Robert on supportability. However I also see similar code in {{DFSUtil}} ({{loadSslConfToHttpServerBuilder}} and {{getPassword}}). Was these copied over? We should at least move that to a common util, and avoid this level of duplication. This will probably leave us not having to change {{Configuration}}, but adding a wrapper util. Or per Wei-Chiu's suggestion, maybe not needed any more. Appreciate more javadocs here too, as to why such method is needed. - Didn't see answer to Allen's ask about unit tests. (Take a look at hadoop-common-project/hadoop-common/src/test/scripts if you're wondering how that's done). - Nit: {{kms-site.xml}} is following other -site.xmls, to have a comment line "put site-specific...", which is good. Please follow them closer to have this line before the {{}} element. :) > Switch KMS from Tomcat to Jetty > --- > > Key: HADOOP-13597 > URL: https://issues.apache.org/jira/browse/HADOOP-13597 > Project: Hadoop Common > Issue Type: New Feature > Components: kms >Affects Versions: 2.6.0 >Reporter: John Zhuge >Assignee: John Zhuge > Attachments: HADOOP-13597.001.patch, HADOOP-13597.002.patch, > HADOOP-13597.003.patch > > > The Tomcat 6 we are using will reach EOL at the end of 2017. While there are > other good options, I would propose switching to {{Jetty 9}} for the > following reasons: > * Easier migration. Both Tomcat and Jetty are based on {{Servlet > Containers}}, so we don't have change client code that much. It would require > more work to switch to {{JAX-RS}}. > * Well established. > * Good performance and scalability. > Other alternatives: > * Jersey + Grizzly > * Tomcat 8 > Your opinions will be greatly appreciated. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13597) Switch KMS from Tomcat to Jetty
[ https://issues.apache.org/jira/browse/HADOOP-13597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15737300#comment-15737300 ] John Zhuge commented on HADOOP-13597: - Thanks Robert for the review! Fixed 2 and 3. 1. I will post a simplified {{Configuration#getPasswordString}} in the next patch. It may still return null though for several reasons: 1) to be consistent with getPassword; 2) some passwords restrieved are simply stored somewhere and may not get used at all, and they are accessed, NPE is an ok indicator; 3) HttpServer2/SSLFactory callers can handle null passwords. > Switch KMS from Tomcat to Jetty > --- > > Key: HADOOP-13597 > URL: https://issues.apache.org/jira/browse/HADOOP-13597 > Project: Hadoop Common > Issue Type: New Feature > Components: kms >Affects Versions: 2.6.0 >Reporter: John Zhuge >Assignee: John Zhuge > Attachments: HADOOP-13597.001.patch, HADOOP-13597.002.patch, > HADOOP-13597.003.patch > > > The Tomcat 6 we are using will reach EOL at the end of 2017. While there are > other good options, I would propose switching to {{Jetty 9}} for the > following reasons: > * Easier migration. Both Tomcat and Jetty are based on {{Servlet > Containers}}, so we don't have change client code that much. It would require > more work to switch to {{JAX-RS}}. > * Well established. > * Good performance and scalability. > Other alternatives: > * Jersey + Grizzly > * Tomcat 8 > Your opinions will be greatly appreciated. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13597) Switch KMS from Tomcat to Jetty
[ https://issues.apache.org/jira/browse/HADOOP-13597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15736470#comment-15736470 ] Robert Kanter commented on HADOOP-13597: A few additional comments: # I don't know if {{getPasswordString}} is a good idea. Won't that just make things confusing for users? They try to set a password in the config, but it ends up being null (probably NPE?) instead of throwing an IOE about not finding the password. The latter would be more clear what the problem is. # Should we mark {{HttpServer2#HTTP_MAX_THREADS}} as {{\@deprecated}}? # Not you're doing, but typo in {{HttpServer2}} comment: {{explicitly destroy the secrete provider}} > Switch KMS from Tomcat to Jetty > --- > > Key: HADOOP-13597 > URL: https://issues.apache.org/jira/browse/HADOOP-13597 > Project: Hadoop Common > Issue Type: New Feature > Components: kms >Affects Versions: 2.6.0 >Reporter: John Zhuge >Assignee: John Zhuge > Attachments: HADOOP-13597.001.patch, HADOOP-13597.002.patch, > HADOOP-13597.003.patch > > > The Tomcat 6 we are using will reach EOL at the end of 2017. While there are > other good options, I would propose switching to {{Jetty 9}} for the > following reasons: > * Easier migration. Both Tomcat and Jetty are based on {{Servlet > Containers}}, so we don't have change client code that much. It would require > more work to switch to {{JAX-RS}}. > * Well established. > * Good performance and scalability. > Other alternatives: > * Jersey + Grizzly > * Tomcat 8 > Your opinions will be greatly appreciated. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13597) Switch KMS from Tomcat to Jetty
[ https://issues.apache.org/jira/browse/HADOOP-13597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15733584#comment-15733584 ] John Zhuge commented on HADOOP-13597: - However, do realize that the following {{elif}} condition can be omitted and changed to {{else}}: {code} elif [[ -n "${oldval}" && -n "${newvar}" ]]; then {code} > Switch KMS from Tomcat to Jetty > --- > > Key: HADOOP-13597 > URL: https://issues.apache.org/jira/browse/HADOOP-13597 > Project: Hadoop Common > Issue Type: New Feature > Components: kms >Affects Versions: 2.6.0 >Reporter: John Zhuge >Assignee: John Zhuge > Attachments: HADOOP-13597.001.patch, HADOOP-13597.002.patch, > HADOOP-13597.003.patch > > > The Tomcat 6 we are using will reach EOL at the end of 2017. While there are > other good options, I would propose switching to {{Jetty 9}} for the > following reasons: > * Easier migration. Both Tomcat and Jetty are based on {{Servlet > Containers}}, so we don't have change client code that much. It would require > more work to switch to {{JAX-RS}}. > * Well established. > * Good performance and scalability. > Other alternatives: > * Jersey + Grizzly > * Tomcat 8 > Your opinions will be greatly appreciated. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13597) Switch KMS from Tomcat to Jetty
[ https://issues.apache.org/jira/browse/HADOOP-13597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15733571#comment-15733571 ] John Zhuge commented on HADOOP-13597: - [~aw] I tries to preserve exactly the same behavior as the existing {{hadoop_deprecate_envvar}} when {{newvar}} is not empty. Isn't {{-z}} the reverse of {{-n}}? Did I miss something? The current code: {code} function hadoop_deprecate_envvar { local oldvar=$1 local newvar=$2 local oldval=${!oldvar} local newval=${!newvar} if [[ -n "${oldval}" ]]; then <<< hadoop_error "WARNING: ${oldvar} has been replaced by ${newvar}. Using value of ${oldvar}." # shellcheck disable=SC2086 eval ${newvar}=\"${oldval}\" # shellcheck disable=SC2086 newval=${oldval} # shellcheck disable=SC2086 eval ${newvar}=\"${newval}\" fi } {code} > Switch KMS from Tomcat to Jetty > --- > > Key: HADOOP-13597 > URL: https://issues.apache.org/jira/browse/HADOOP-13597 > Project: Hadoop Common > Issue Type: New Feature > Components: kms >Affects Versions: 2.6.0 >Reporter: John Zhuge >Assignee: John Zhuge > Attachments: HADOOP-13597.001.patch, HADOOP-13597.002.patch, > HADOOP-13597.003.patch > > > The Tomcat 6 we are using will reach EOL at the end of 2017. While there are > other good options, I would propose switching to {{Jetty 9}} for the > following reasons: > * Easier migration. Both Tomcat and Jetty are based on {{Servlet > Containers}}, so we don't have change client code that much. It would require > more work to switch to {{JAX-RS}}. > * Well established. > * Good performance and scalability. > Other alternatives: > * Jersey + Grizzly > * Tomcat 8 > Your opinions will be greatly appreciated. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13597) Switch KMS from Tomcat to Jetty
[ https://issues.apache.org/jira/browse/HADOOP-13597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15732944#comment-15732944 ] Allen Wittenauer commented on HADOOP-13597: --- {code} + if [[ -z "${oldval}" ]]; then +return {code} This is an even worse side effect since it means that we don't promote old values into new ones when the vars have been renamed. (There are a lot of them!) Also, where are the unit tests for the bash functions you added? > Switch KMS from Tomcat to Jetty > --- > > Key: HADOOP-13597 > URL: https://issues.apache.org/jira/browse/HADOOP-13597 > Project: Hadoop Common > Issue Type: New Feature > Components: kms >Affects Versions: 2.6.0 >Reporter: John Zhuge >Assignee: John Zhuge > Attachments: HADOOP-13597.001.patch, HADOOP-13597.002.patch, > HADOOP-13597.003.patch > > > The Tomcat 6 we are using will reach EOL at the end of 2017. While there are > other good options, I would propose switching to {{Jetty 9}} for the > following reasons: > * Easier migration. Both Tomcat and Jetty are based on {{Servlet > Containers}}, so we don't have change client code that much. It would require > more work to switch to {{JAX-RS}}. > * Well established. > * Good performance and scalability. > Other alternatives: > * Jersey + Grizzly > * Tomcat 8 > Your opinions will be greatly appreciated. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13597) Switch KMS from Tomcat to Jetty
[ https://issues.apache.org/jira/browse/HADOOP-13597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15732830#comment-15732830 ] John Zhuge commented on HADOOP-13597: - Thanks [~jojochuang] for the review ! bq. There's one bit that might cause confusion in deployment. The fact that keystore password could come from either environment variable, from configuration file or credential files (via Configuration#getPassword) make me feel a bit uneasy. If the password comes from a credential file, it will also need to ProviderUtils.excludeIncompatibleCredentialProviders in order to trim credential files on HdfsFileSystems. Very good point! I will engage wider discussion. As a precaution, I could revert to the existing KMS approach which does not consult credential provider and file a separate JIRA to integrate with credential provider. bq. It seems the KMS server is not Kerberized. You might want to construct a HttpServer2 object with extra options to enable Kerberos: KMS uses {{KMSAuthenticationFilter}} specified in web.xml instead of the generic {{AuthenticationFilter}} by HttpServer2. bq. When KMSWebServer starts/stops, it should print corresponding message using StringUtils.startupShutdownMessage. This will make supporters' life easier. Will do. bq. I think you can not assume the admin user is user.name=kms when accessing the servlets such as jmx, loglevel, etc. Also, need to make sure access permission is guarded properly accessing these servlets. Don't think existing KMS supports admin user. Will add this feature. bq. I am not sure how existing KMS handles truststore and its password, but I think you might be missing something in the new KMS when handling truststore and its password. Truststore password is obsoleted by HADOOP-13864. bq. The keystore password comes from configuration key hadoop.security.keystore.java-keystore-provider.password-file. If I understand ConfigRedact correctly it doesn't seem to redact this specific configuration key to me. Could you double check? This key is for the password file, not password, so it does not have to be redacted. bq. In Configuration#getPasswordString, please print name if there's an IOException to log. Also, should it catch IOException and return null? If it looks for a password but is unable to, would it be easier to let the exception be thrown? It could be a troubleshooting nightmare I imagine. Good point. {{getPasswordString}} seems unnecessary. Remove it. > Switch KMS from Tomcat to Jetty > --- > > Key: HADOOP-13597 > URL: https://issues.apache.org/jira/browse/HADOOP-13597 > Project: Hadoop Common > Issue Type: New Feature > Components: kms >Affects Versions: 2.6.0 >Reporter: John Zhuge >Assignee: John Zhuge > Attachments: HADOOP-13597.001.patch, HADOOP-13597.002.patch, > HADOOP-13597.003.patch > > > The Tomcat 6 we are using will reach EOL at the end of 2017. While there are > other good options, I would propose switching to {{Jetty 9}} for the > following reasons: > * Easier migration. Both Tomcat and Jetty are based on {{Servlet > Containers}}, so we don't have change client code that much. It would require > more work to switch to {{JAX-RS}}. > * Well established. > * Good performance and scalability. > Other alternatives: > * Jersey + Grizzly > * Tomcat 8 > Your opinions will be greatly appreciated. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13597) Switch KMS from Tomcat to Jetty
[ https://issues.apache.org/jira/browse/HADOOP-13597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15731992#comment-15731992 ] Wei-Chiu Chuang commented on HADOOP-13597: -- Hi [~jzhuge] thanks for working on this big patch! * There's one bit that might cause confusion in deployment. The fact that keystore password could come from either environment variable, from configuration file or credential files (via Configuration#getPassword) make me feel a bit uneasy. If the password comes from a credential file, it will also need to {{ProviderUtils.excludeIncompatibleCredentialProviders}} in order to trim credential files on HdfsFileSystems. * It seems the KMS server is not Kerberized. You might want to construct a HttpServer2 object with extra options to enable Kerberos: {code} new HttpServer2.Builder().setSecurityEnabled(true) .setUsernameConfKey(PRINCIPAL) .setKeytabConfKey(KEYTAB); {code} * When KMSWebServer starts/stops, it should print corresponding message using {{StringUtils.startupShutdownMessage}}. This will make supporters' life easier. * I think you can not assume the admin user is user.name=kms when accessing the servlets such as jmx, loglevel, etc. Also, need to make sure access permission is guarded properly accessing these servlets. * I am not sure how existing KMS handles truststore and its password, but I think you might be missing something in the new KMS when handling truststore and its password. * The keystore password comes from configuration key {{hadoop.security.keystore.java-keystore-provider.password-file}}. If I understand ConfigRedact correctly it doesn't seem to redact this specific configuration key to me. Could you double check? * In {{Configuration#getPasswordString}}, please print {{name}} if there's an IOException to log. Also, should it catch IOException and return null? If it looks for a password but is unable to, would it be easier to let the exception be thrown? It could be a troubleshooting nightmare I imagine. > Switch KMS from Tomcat to Jetty > --- > > Key: HADOOP-13597 > URL: https://issues.apache.org/jira/browse/HADOOP-13597 > Project: Hadoop Common > Issue Type: New Feature > Components: kms >Affects Versions: 2.6.0 >Reporter: John Zhuge >Assignee: John Zhuge > Attachments: HADOOP-13597.001.patch, HADOOP-13597.002.patch, > HADOOP-13597.003.patch > > > The Tomcat 6 we are using will reach EOL at the end of 2017. While there are > other good options, I would propose switching to {{Jetty 9}} for the > following reasons: > * Easier migration. Both Tomcat and Jetty are based on {{Servlet > Containers}}, so we don't have change client code that much. It would require > more work to switch to {{JAX-RS}}. > * Well established. > * Good performance and scalability. > Other alternatives: > * Jersey + Grizzly > * Tomcat 8 > Your opinions will be greatly appreciated. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13597) Switch KMS from Tomcat to Jetty
[ https://issues.apache.org/jira/browse/HADOOP-13597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15731401#comment-15731401 ] Hadoop QA commented on HADOOP-13597: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 13s{color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 1 new or modified test files. {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 39s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m 53s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m 35s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 40s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 56s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m 8s{color} | {color:green} trunk passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Skipped patched modules with no Java source: hadoop-assemblies {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 5s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 44s{color} | {color:green} trunk passed {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 20s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 13s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m 17s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 9m 17s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 44s{color} | {color:green} root: The patch generated 0 new + 233 unchanged - 8 fixed = 233 total (was 241) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 10s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m 22s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} shellcheck {color} | {color:red} 0m 16s{color} | {color:red} The patch generated 1 new + 568 unchanged - 8 fixed = 569 total (was 576) {color} | | {color:green}+1{color} | {color:green} shelldocs {color} | {color:green} 0m 23s{color} | {color:green} The patch generated 0 new + 372 unchanged - 4 fixed = 372 total (was 376) {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 1s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 6s{color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Skipped patched modules with no Java source: hadoop-assemblies {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 30s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 58s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 28s{color} | {color:green} hadoop-assemblies in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m 50s{color} | {color:green} hadoop-common in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 27s{color} | {color:green} hadoop-kms in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 45s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 86m 13s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:a9ad5d6 | | JIRA Issue | HADOOP-13597 | | JIRA Patch URL |
[jira] [Commented] (HADOOP-13597) Switch KMS from Tomcat to Jetty
[ https://issues.apache.org/jira/browse/HADOOP-13597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15731221#comment-15731221 ] John Zhuge commented on HADOOP-13597: - My private branch for 003: https://github.com/jzhuge/hadoop/tree/HADOOP-13597.003 > Switch KMS from Tomcat to Jetty > --- > > Key: HADOOP-13597 > URL: https://issues.apache.org/jira/browse/HADOOP-13597 > Project: Hadoop Common > Issue Type: New Feature > Components: kms >Affects Versions: 2.6.0 >Reporter: John Zhuge >Assignee: John Zhuge > Attachments: HADOOP-13597.001.patch, HADOOP-13597.002.patch, > HADOOP-13597.003.patch > > > The Tomcat 6 we are using will reach EOL at the end of 2017. While there are > other good options, I would propose switching to {{Jetty 9}} for the > following reasons: > * Easier migration. Both Tomcat and Jetty are based on {{Servlet > Containers}}, so we don't have change client code that much. It would require > more work to switch to {{JAX-RS}}. > * Well established. > * Good performance and scalability. > Other alternatives: > * Jersey + Grizzly > * Tomcat 8 > Your opinions will be greatly appreciated. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13597) Switch KMS from Tomcat to Jetty
[ https://issues.apache.org/jira/browse/HADOOP-13597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15720576#comment-15720576 ] Allen Wittenauer commented on HADOOP-13597: --- Oh, I see what you were concerned about. Looking closer at the code, I'd change the all-in-one to actually do {code} hadoop_add_param HADOOP_KMS_OPTS "Dkms.config.dir=" ... hadoop_add_param HADOOP_KMS_OPTS "Dkms.log.dir=" ... {code} since that's much more likely the place where kms.config values get set. Then the whole scenario becomes a non-issue. There's also precedent for just doing something like: {code} HADOOP_KMS_OPTS=${HADOOP_KMS_OPTS:-"-Dfoo"} {code} (see hdfs-config.sh) but I'm less of a fan of that. (We should probably undo those and use the hadoop_add_param trick. It's cleaner, more practical, and does a better job of actually protecting the user from missing defaults.) Also, be aware of the rather severe consequences that {code} hadoop_deprecate_envvar KMS_CONFIG HADOOP_CONF_DIR hadoop_deprecate_envvar KMS_LOG HADOOP_LOG_DIR {code} has if KMS_CONFIG was never an analog to HADOOP_CONF_DIR in branch-2, I'd be tempted to just throw a manual warning about it rather than go though deprecate. > Switch KMS from Tomcat to Jetty > --- > > Key: HADOOP-13597 > URL: https://issues.apache.org/jira/browse/HADOOP-13597 > Project: Hadoop Common > Issue Type: New Feature > Components: kms >Affects Versions: 2.6.0 >Reporter: John Zhuge >Assignee: John Zhuge > Attachments: HADOOP-13597.001.patch, HADOOP-13597.002.patch > > > The Tomcat 6 we are using will reach EOL at the end of 2017. While there are > other good options, I would propose switching to {{Jetty 9}} for the > following reasons: > * Easier migration. Both Tomcat and Jetty are based on {{Servlet > Containers}}, so we don't have change client code that much. It would require > more work to switch to {{JAX-RS}}. > * Well established. > * Good performance and scalability. > Other alternatives: > * Jersey + Grizzly > * Tomcat 8 > Your opinions will be greatly appreciated. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13597) Switch KMS from Tomcat to Jetty
[ https://issues.apache.org/jira/browse/HADOOP-13597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15719516#comment-15719516 ] John Zhuge commented on HADOOP-13597: - Another approach: {code} class AccessLoggingConfiguration implements Configuration { LoggedConfigurationAccess(conf, log) { super(conf); this.log = log; } String get(String name) { String value = super.get(name); log.info(..., name, value); return value; } String getInt(String name, int defaultValue) { String value = super.getInt(name, defaultValue); log.info(..., name, value, defaultValue); return value; } } {code} > Switch KMS from Tomcat to Jetty > --- > > Key: HADOOP-13597 > URL: https://issues.apache.org/jira/browse/HADOOP-13597 > Project: Hadoop Common > Issue Type: New Feature > Components: kms >Affects Versions: 2.6.0 >Reporter: John Zhuge >Assignee: John Zhuge > Attachments: HADOOP-13597.001.patch, HADOOP-13597.002.patch > > > The Tomcat 6 we are using will reach EOL at the end of 2017. While there are > other good options, I would propose switching to {{Jetty 9}} for the > following reasons: > * Easier migration. Both Tomcat and Jetty are based on {{Servlet > Containers}}, so we don't have change client code that much. It would require > more work to switch to {{JAX-RS}}. > * Well established. > * Good performance and scalability. > Other alternatives: > * Jersey + Grizzly > * Tomcat 8 > Your opinions will be greatly appreciated. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13597) Switch KMS from Tomcat to Jetty
[ https://issues.apache.org/jira/browse/HADOOP-13597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15717586#comment-15717586 ] John Zhuge commented on HADOOP-13597: - Could have used [Decorator Pattern|https://en.wikipedia.org/wiki/Decorator_pattern] to design a wrapper to log configuration access: {code} interface ConfigurationAccess { String get(String name); String getInt(String name, int defaultValue); } class Configuration implements ConfigurationAccess { ... } class LoggedConfigurationAccess { LoggedConfigurationAccess(conf, log) { this.conf = conf; this.log = log; } String get(String name) { String value = conf.get(name); log.info(..., name, value); return value; } String getInt(String name, int defaultValue) { String value = conf.getInt(name, defaultValue); log.info(..., name, value, defaultValue); return value; } } {code} A little downsize though: {{LoggedConfigurationAccess#getInt}} will log 2 messages because {{Configuration#getInt}} calls {{LoggedConfigurationAccess#get}}. > Switch KMS from Tomcat to Jetty > --- > > Key: HADOOP-13597 > URL: https://issues.apache.org/jira/browse/HADOOP-13597 > Project: Hadoop Common > Issue Type: New Feature > Components: kms >Affects Versions: 2.6.0 >Reporter: John Zhuge >Assignee: John Zhuge > Attachments: HADOOP-13597.001.patch, HADOOP-13597.002.patch > > > The Tomcat 6 we are using will reach EOL at the end of 2017. While there are > other good options, I would propose switching to {{Jetty 9}} for the > following reasons: > * Easier migration. Both Tomcat and Jetty are based on {{Servlet > Containers}}, so we don't have change client code that much. It would require > more work to switch to {{JAX-RS}}. > * Well established. > * Good performance and scalability. > Other alternatives: > * Jersey + Grizzly > * Tomcat 8 > Your opinions will be greatly appreciated. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13597) Switch KMS from Tomcat to Jetty
[ https://issues.apache.org/jira/browse/HADOOP-13597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15716720#comment-15716720 ] John Zhuge commented on HADOOP-13597: - I was wrong. The test case {{HADOOP_KMS_OPTS=-Dkms.config.dir=.. bin/hadoop kms}} works in both approaches, though for different reasons: * All in one: {{hadoop_subcommand_kms}} adds {{-Dkms_config_dir}} first, then HADOOP_KMS_OPTS is appended to HADOOP_OPTS. The later {{-Dkms.config.dir}} on HADOOP_OPTS takes effect. * multiple handlers: HADOOP_KMS_OPTS is appended to HADOOP_OPTS first, then _kms_hadoop_finalize calls {{hadoop_add_param HADOOP_OPTS "-Dkms.config.dir="}} which checks the existence of string "-Dkms.config.dir=" and decides not to add param. > Switch KMS from Tomcat to Jetty > --- > > Key: HADOOP-13597 > URL: https://issues.apache.org/jira/browse/HADOOP-13597 > Project: Hadoop Common > Issue Type: New Feature > Components: kms >Affects Versions: 2.6.0 >Reporter: John Zhuge >Assignee: John Zhuge > Attachments: HADOOP-13597.001.patch, HADOOP-13597.002.patch > > > The Tomcat 6 we are using will reach EOL at the end of 2017. While there are > other good options, I would propose switching to {{Jetty 9}} for the > following reasons: > * Easier migration. Both Tomcat and Jetty are based on {{Servlet > Containers}}, so we don't have change client code that much. It would require > more work to switch to {{JAX-RS}}. > * Well established. > * Good performance and scalability. > Other alternatives: > * Jersey + Grizzly > * Tomcat 8 > Your opinions will be greatly appreciated. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13597) Switch KMS from Tomcat to Jetty
[ https://issues.apache.org/jira/browse/HADOOP-13597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15715747#comment-15715747 ] John Zhuge commented on HADOOP-13597: - {{hadoop_add_param HADOOP_OPTS "-Dkms.config.dir=" "-Dkms.config.dir=${HADOOP_CONF_DIR}"}} may be called too early. Since {{hadoop_subcommand_opts}} is called between {{hadoo_subcommand_kms}} and {{_kms_hadoop_finalize}}, in all-in-one approach, {{HADOOP_KMS_OPTS=-Dkms.config.dir=..}} would not have taken effect. > Switch KMS from Tomcat to Jetty > --- > > Key: HADOOP-13597 > URL: https://issues.apache.org/jira/browse/HADOOP-13597 > Project: Hadoop Common > Issue Type: New Feature > Components: kms >Affects Versions: 2.6.0 >Reporter: John Zhuge >Assignee: John Zhuge > Attachments: HADOOP-13597.001.patch, HADOOP-13597.002.patch > > > The Tomcat 6 we are using will reach EOL at the end of 2017. While there are > other good options, I would propose switching to {{Jetty 9}} for the > following reasons: > * Easier migration. Both Tomcat and Jetty are based on {{Servlet > Containers}}, so we don't have change client code that much. It would require > more work to switch to {{JAX-RS}}. > * Well established. > * Good performance and scalability. > Other alternatives: > * Jersey + Grizzly > * Tomcat 8 > Your opinions will be greatly appreciated. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13597) Switch KMS from Tomcat to Jetty
[ https://issues.apache.org/jira/browse/HADOOP-13597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15715710#comment-15715710 ] Allen Wittenauer commented on HADOOP-13597: --- I'm sort of surprised that the "all in one" doesn't work, other than -e should probably be -x. It's almost certainly safer and doesn't have nearly as many side effects. Anything in particular that is breaking that I could help with? > Switch KMS from Tomcat to Jetty > --- > > Key: HADOOP-13597 > URL: https://issues.apache.org/jira/browse/HADOOP-13597 > Project: Hadoop Common > Issue Type: New Feature > Components: kms >Affects Versions: 2.6.0 >Reporter: John Zhuge >Assignee: John Zhuge > Attachments: HADOOP-13597.001.patch, HADOOP-13597.002.patch > > > The Tomcat 6 we are using will reach EOL at the end of 2017. While there are > other good options, I would propose switching to {{Jetty 9}} for the > following reasons: > * Easier migration. Both Tomcat and Jetty are based on {{Servlet > Containers}}, so we don't have change client code that much. It would require > more work to switch to {{JAX-RS}}. > * Well established. > * Good performance and scalability. > Other alternatives: > * Jersey + Grizzly > * Tomcat 8 > Your opinions will be greatly appreciated. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13597) Switch KMS from Tomcat to Jetty
[ https://issues.apache.org/jira/browse/HADOOP-13597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15715569#comment-15715569 ] John Zhuge commented on HADOOP-13597: - [~aw] Found mistake in {{hadoop-kms.sh}} for Patch 002 where everything is in {{hadoop_subcommand_kms}}. Should be: {code} if [[ "${HADOOP_SHELL_EXECNAME}" = hadoop ]]; then hadoop_add_profile kms hadoop_add_subcommand "kms" "run KMS, the Key Management Server" fi function _kms_hadoop_init { # init variables } function hadoop_subcommand_kms { # Called by bin/hadoop to provide subcommand case statement if any HADOOP_SUBCMD_SUPPORTDAEMONIZATION=true HADOOP_CLASSNAME=org.apache.hadoop.crypto.key.kms.server.KMSHttpServer } function _kms_hadoop_finalize { # Called in finalize phase, all env vars are settled hadoop_add_param HADOOP_OPTS "-Dkms.config.dir=" \ "-Dkms.config.dir=${HADOOP_CONF_DIR}" hadoop_add_param HADOOP_OPTS "-Dkms.log.dir=" \ "-Dkms.log.dir=${HADOOP_LOG_DIR}" } {code} The 3 functions are called in this order: # _kms_hadoop_init # hadoop_subcommand_kms # _kmd_hadoop_finalize > Switch KMS from Tomcat to Jetty > --- > > Key: HADOOP-13597 > URL: https://issues.apache.org/jira/browse/HADOOP-13597 > Project: Hadoop Common > Issue Type: New Feature > Components: kms >Affects Versions: 2.6.0 >Reporter: John Zhuge >Assignee: John Zhuge > Attachments: HADOOP-13597.001.patch, HADOOP-13597.002.patch > > > The Tomcat 6 we are using will reach EOL at the end of 2017. While there are > other good options, I would propose switching to {{Jetty 9}} for the > following reasons: > * Easier migration. Both Tomcat and Jetty are based on {{Servlet > Containers}}, so we don't have change client code that much. It would require > more work to switch to {{JAX-RS}}. > * Well established. > * Good performance and scalability. > Other alternatives: > * Jersey + Grizzly > * Tomcat 8 > Your opinions will be greatly appreciated. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13597) Switch KMS from Tomcat to Jetty
[ https://issues.apache.org/jira/browse/HADOOP-13597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15715380#comment-15715380 ] John Zhuge commented on HADOOP-13597: - [Private branch for Patch 002|https://github.com/jzhuge/hadoop/tree/HADOOP-13597.002]. > Switch KMS from Tomcat to Jetty > --- > > Key: HADOOP-13597 > URL: https://issues.apache.org/jira/browse/HADOOP-13597 > Project: Hadoop Common > Issue Type: New Feature > Components: kms >Affects Versions: 2.6.0 >Reporter: John Zhuge >Assignee: John Zhuge > Attachments: HADOOP-13597.001.patch, HADOOP-13597.002.patch > > > The Tomcat 6 we are using will reach EOL at the end of 2017. While there are > other good options, I would propose switching to {{Jetty 9}} for the > following reasons: > * Easier migration. Both Tomcat and Jetty are based on {{Servlet > Containers}}, so we don't have change client code that much. It would require > more work to switch to {{JAX-RS}}. > * Well established. > * Good performance and scalability. > Other alternatives: > * Jersey + Grizzly > * Tomcat 8 > Your opinions will be greatly appreciated. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13597) Switch KMS from Tomcat to Jetty
[ https://issues.apache.org/jira/browse/HADOOP-13597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15714570#comment-15714570 ] Hadoop QA commented on HADOOP-13597: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 13s{color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 1 new or modified test files. {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 48s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 36s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m 0s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 45s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 1s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m 10s{color} | {color:green} trunk passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Skipped patched modules with no Java source: hadoop-assemblies {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 10s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 47s{color} | {color:green} trunk passed {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 21s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 22s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m 29s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 10m 29s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m 1s{color} | {color:green} root: The patch generated 0 new + 236 unchanged - 9 fixed = 236 total (was 245) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 19s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m 25s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} shellcheck {color} | {color:red} 0m 17s{color} | {color:red} The patch generated 5 new + 559 unchanged - 8 fixed = 564 total (was 567) {color} | | {color:green}+1{color} | {color:green} shelldocs {color} | {color:green} 0m 22s{color} | {color:green} The patch generated 0 new + 342 unchanged - 4 fixed = 342 total (was 346) {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 7s{color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Skipped patched modules with no Java source: hadoop-assemblies {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 42s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 11s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 28s{color} | {color:green} hadoop-assemblies in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m 38s{color} | {color:green} hadoop-common in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 31s{color} | {color:green} hadoop-kms in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 48s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 90m 6s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:a9ad5d6 | | JIRA Issue | HADOOP-13597 | | JIRA Patch URL |
[jira] [Commented] (HADOOP-13597) Switch KMS from Tomcat to Jetty
[ https://issues.apache.org/jira/browse/HADOOP-13597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15709923#comment-15709923 ] Xiao Chen commented on HADOOP-13597: Thanks John. Yes agreed. Let's keep this jira focused so new configs only. The old ones aren't public so can be changed in another jira. re. KMSHttpServer testing, I was thinking to make the most basic checks so if something breaks it will be obvious. If it's all covered by MiniKMS and can be easily figured out, fine by me then. > Switch KMS from Tomcat to Jetty > --- > > Key: HADOOP-13597 > URL: https://issues.apache.org/jira/browse/HADOOP-13597 > Project: Hadoop Common > Issue Type: New Feature > Components: kms >Affects Versions: 2.6.0 >Reporter: John Zhuge >Assignee: John Zhuge > Attachments: HADOOP-13597.001.patch > > > The Tomcat 6 we are using will reach EOL at the end of 2017. While there are > other good options, I would propose switching to {{Jetty 9}} for the > following reasons: > * Easier migration. Both Tomcat and Jetty are based on {{Servlet > Containers}}, so we don't have change client code that much. It would require > more work to switch to {{JAX-RS}}. > * Well established. > * Good performance and scalability. > Other alternatives: > * Jersey + Grizzly > * Tomcat 8 > Your opinions will be greatly appreciated. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13597) Switch KMS from Tomcat to Jetty
[ https://issues.apache.org/jira/browse/HADOOP-13597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15709900#comment-15709900 ] John Zhuge commented on HADOOP-13597: - HDFS/Common way of key naming makes sense, I will switch. Leave the old keys alone. > Switch KMS from Tomcat to Jetty > --- > > Key: HADOOP-13597 > URL: https://issues.apache.org/jira/browse/HADOOP-13597 > Project: Hadoop Common > Issue Type: New Feature > Components: kms >Affects Versions: 2.6.0 >Reporter: John Zhuge >Assignee: John Zhuge > Attachments: HADOOP-13597.001.patch > > > The Tomcat 6 we are using will reach EOL at the end of 2017. While there are > other good options, I would propose switching to {{Jetty 9}} for the > following reasons: > * Easier migration. Both Tomcat and Jetty are based on {{Servlet > Containers}}, so we don't have change client code that much. It would require > more work to switch to {{JAX-RS}}. > * Well established. > * Good performance and scalability. > Other alternatives: > * Jersey + Grizzly > * Tomcat 8 > Your opinions will be greatly appreciated. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13597) Switch KMS from Tomcat to Jetty
[ https://issues.apache.org/jira/browse/HADOOP-13597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15707922#comment-15707922 ] John Zhuge commented on HADOOP-13597: - Great comments! bq. Suggest to name the new config key names _KEY In the case, we will have mixed naming style in KMSConfiguration. Is that ok? The new properties are much fewer than old ones especially after I move SSL properties to ssl-server.xml. bq. We should add basic testing to KMSHttpServer KMSHttpServer is called by MiniKMS thus all its methods are exercised by KMS unit tests.Should I add tests to ensure legacy env variables are still supported? > Switch KMS from Tomcat to Jetty > --- > > Key: HADOOP-13597 > URL: https://issues.apache.org/jira/browse/HADOOP-13597 > Project: Hadoop Common > Issue Type: New Feature > Components: kms >Affects Versions: 2.6.0 >Reporter: John Zhuge >Assignee: John Zhuge > Attachments: HADOOP-13597.001.patch > > > The Tomcat 6 we are using will reach EOL at the end of 2017. While there are > other good options, I would propose switching to {{Jetty 9}} for the > following reasons: > * Easier migration. Both Tomcat and Jetty are based on {{Servlet > Containers}}, so we don't have change client code that much. It would require > more work to switch to {{JAX-RS}}. > * Well established. > * Good performance and scalability. > Other alternatives: > * Jersey + Grizzly > * Tomcat 8 > Your opinions will be greatly appreciated. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13597) Switch KMS from Tomcat to Jetty
[ https://issues.apache.org/jira/browse/HADOOP-13597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15707117#comment-15707117 ] Xiao Chen commented on HADOOP-13597: Thanks for working on this [~jzhuge]. Some review comments aside from your todos: - Suggest to name the new config key names _KEY, to be more consistent with HDFS/Common way. - Nice we're loading envvars for compatibility. It'll make upgrades less painful. OTOH, could you add some text in the {{Loaded env}} logs to indicate this is a deprecated way, and user should user configuration files instead? - {{kms-default.xml}} has some duplicate key names (hadoop.kms.authentication.signer.secret.provider.zookeeper.XXX). - This exists before your patch, but good to fix along: kms-default.xml has some escaped chars () in configuration. - {{hadoop.kms.authentication.signer.secret.provider.zookeeper.auth.type}} defaults to {{none}} before. We should keep that unchanged. The {{kerberos}} default value also doesn't work, it should be {{sasl}} - We should add basic testing to {{KMSHttpServer}} - Had a quick run, kms.log seems record every request/response. By default this shouldn't be logged. - After clean install kms.log appears to be at TRACE level. Suggest to make it INFO. > Switch KMS from Tomcat to Jetty > --- > > Key: HADOOP-13597 > URL: https://issues.apache.org/jira/browse/HADOOP-13597 > Project: Hadoop Common > Issue Type: New Feature > Components: kms >Affects Versions: 2.6.0 >Reporter: John Zhuge >Assignee: John Zhuge > Attachments: HADOOP-13597.001.patch > > > The Tomcat 6 we are using will reach EOL at the end of 2017. While there are > other good options, I would propose switching to {{Jetty 9}} for the > following reasons: > * Easier migration. Both Tomcat and Jetty are based on {{Servlet > Containers}}, so we don't have change client code that much. It would require > more work to switch to {{JAX-RS}}. > * Well established. > * Good performance and scalability. > Other alternatives: > * Jersey + Grizzly > * Tomcat 8 > Your opinions will be greatly appreciated. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13597) Switch KMS from Tomcat to Jetty
[ https://issues.apache.org/jira/browse/HADOOP-13597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15704315#comment-15704315 ] Hadoop QA commented on HADOOP-13597: | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 14s{color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 1s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 1 new or modified test files. {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 17s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m 51s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m 42s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 37s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 58s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m 10s{color} | {color:green} trunk passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Skipped patched modules with no Java source: hadoop-assemblies {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 6s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 45s{color} | {color:green} trunk passed {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 20s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 14s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m 21s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 9m 21s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 43s{color} | {color:green} root: The patch generated 0 new + 57 unchanged - 5 fixed = 57 total (was 62) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 11s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m 24s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} shellcheck {color} | {color:green} 0m 16s{color} | {color:green} The patch generated 0 new + 566 unchanged - 1 fixed = 566 total (was 567) {color} | | {color:green}+1{color} | {color:green} shelldocs {color} | {color:green} 0m 24s{color} | {color:green} The patch generated 0 new + 344 unchanged - 2 fixed = 344 total (was 346) {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 6s{color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Skipped patched modules with no Java source: hadoop-assemblies {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 30s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 59s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 27s{color} | {color:green} hadoop-assemblies in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 10m 51s{color} | {color:green} hadoop-common in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 29s{color} | {color:green} hadoop-kms in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 45s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 86m 17s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:a9ad5d6 | | JIRA Issue | HADOOP-13597 | | JIRA Patch
[jira] [Commented] (HADOOP-13597) Switch KMS from Tomcat to Jetty
[ https://issues.apache.org/jira/browse/HADOOP-13597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15704286#comment-15704286 ] John Zhuge commented on HADOOP-13597: - Map {{sbin/kms.sh}} to {{bin/hadoop kms}}: | kms.sh run | hadoop kms | | kms.sh start | hadoop kms --daemon start | | kms.sh stop | hadoop kms --daemon stop | | | hadoop kms --daemon status | > Switch KMS from Tomcat to Jetty > --- > > Key: HADOOP-13597 > URL: https://issues.apache.org/jira/browse/HADOOP-13597 > Project: Hadoop Common > Issue Type: New Feature > Components: kms >Affects Versions: 2.6.0 >Reporter: John Zhuge >Assignee: John Zhuge > Attachments: HADOOP-13597.001.patch > > > The Tomcat 6 we are using will reach EOL at the end of 2017. While there are > other good options, I would propose switching to {{Jetty 9}} for the > following reasons: > * Easier migration. Both Tomcat and Jetty are based on {{Servlet > Containers}}, so we don't have change client code that much. It would require > more work to switch to {{JAX-RS}}. > * Well established. > * Good performance and scalability. > Other alternatives: > * Jersey + Grizzly > * Tomcat 8 > Your opinions will be greatly appreciated. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13597) Switch KMS from Tomcat to Jetty
[ https://issues.apache.org/jira/browse/HADOOP-13597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15704255#comment-15704255 ] John Zhuge commented on HADOOP-13597: - [~aw] I was about to ping you on the rewrite, thanks for the quick feedback ! It seems more natural to move it into a sub-command of bin/hadoop. Also ok to move sbin/kms.sh to bin/kms, while I found it a little awkward for this kind of script where there is one single implicit subcommand, unlike hadoop/hdfs/yarn scripts. > Switch KMS from Tomcat to Jetty > --- > > Key: HADOOP-13597 > URL: https://issues.apache.org/jira/browse/HADOOP-13597 > Project: Hadoop Common > Issue Type: New Feature > Components: kms >Affects Versions: 2.6.0 >Reporter: John Zhuge >Assignee: John Zhuge > Attachments: HADOOP-13597.001.patch > > > The Tomcat 6 we are using will reach EOL at the end of 2017. While there are > other good options, I would propose switching to {{Jetty 9}} for the > following reasons: > * Easier migration. Both Tomcat and Jetty are based on {{Servlet > Containers}}, so we don't have change client code that much. It would require > more work to switch to {{JAX-RS}}. > * Well established. > * Good performance and scalability. > Other alternatives: > * Jersey + Grizzly > * Tomcat 8 > Your opinions will be greatly appreciated. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13597) Switch KMS from Tomcat to Jetty
[ https://issues.apache.org/jira/browse/HADOOP-13597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15704230#comment-15704230 ] Allen Wittenauer commented on HADOOP-13597: --- Hooray! This is really great. bq. Rewrite kms.sh to use Hadoop shell script framework I didn't have any specific feedback about this bit (quick pass; didn't see anything obvious). One of the big goals I had for the rewrite was to get sbin out of the direct path for administrators. With that in mind, I wonder if this is the time to fix kms to be less of an outlier. One choice would be integrate it into bin/hadoop. (probably via shell profile a la the bits in hadoop-tools). Another, less drastic option would be just to move sbin/kms.sh to bin/kms. In either case, sbin/kms.sh just becomes a wrapper. Anyway, food for thought. > Switch KMS from Tomcat to Jetty > --- > > Key: HADOOP-13597 > URL: https://issues.apache.org/jira/browse/HADOOP-13597 > Project: Hadoop Common > Issue Type: New Feature > Components: kms >Affects Versions: 2.6.0 >Reporter: John Zhuge >Assignee: John Zhuge > Attachments: HADOOP-13597.001.patch > > > The Tomcat 6 we are using will reach EOL at the end of 2017. While there are > other good options, I would propose switching to {{Jetty 9}} for the > following reasons: > * Easier migration. Both Tomcat and Jetty are based on {{Servlet > Containers}}, so we don't have change client code that much. It would require > more work to switch to {{JAX-RS}}. > * Well established. > * Good performance and scalability. > Other alternatives: > * Jersey + Grizzly > * Tomcat 8 > Your opinions will be greatly appreciated. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13597) Switch KMS from Tomcat to Jetty
[ https://issues.apache.org/jira/browse/HADOOP-13597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15703923#comment-15703923 ] John Zhuge commented on HADOOP-13597: - Agree with you, it is an incompatible. > Switch KMS from Tomcat to Jetty > --- > > Key: HADOOP-13597 > URL: https://issues.apache.org/jira/browse/HADOOP-13597 > Project: Hadoop Common > Issue Type: New Feature > Components: kms >Affects Versions: 2.6.0 >Reporter: John Zhuge >Assignee: John Zhuge > > The Tomcat 6 we are using will reach EOL at the end of 2017. While there are > other good options, I would propose switching to {{Jetty 9}} for the > following reasons: > * Easier migration. Both Tomcat and Jetty are based on {{Servlet > Containers}}, so we don't have change client code that much. It would require > more work to switch to {{JAX-RS}}. > * Well established. > * Good performance and scalability. > Other alternatives: > * Jersey + Grizzly > * Tomcat 8 > Your opinions will be greatly appreciated. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13597) Switch KMS from Tomcat to Jetty
[ https://issues.apache.org/jira/browse/HADOOP-13597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15702893#comment-15702893 ] Andrew Wang commented on HADOOP-13597: -- Hi John, could you respond to my earlier question about if this change is incompatible? If so, this is a blocker for 3.0.0-beta1, and we should mark it as such. bq. since MiniKMS is also based on embedded Jetty, it is possible to replace it with full KMS once KMS is switched to embedded Jetty? Seems fine to me. > Switch KMS from Tomcat to Jetty > --- > > Key: HADOOP-13597 > URL: https://issues.apache.org/jira/browse/HADOOP-13597 > Project: Hadoop Common > Issue Type: New Feature > Components: kms >Affects Versions: 2.6.0 >Reporter: John Zhuge >Assignee: John Zhuge > > The Tomcat 6 we are using will reach EOL at the end of 2017. While there are > other good options, I would propose switching to {{Jetty 9}} for the > following reasons: > * Easier migration. Both Tomcat and Jetty are based on {{Servlet > Containers}}, so we don't have change client code that much. It would require > more work to switch to {{JAX-RS}}. > * Well established. > * Good performance and scalability. > Other alternatives: > * Jersey + Grizzly > * Tomcat 8 > Your opinions will be greatly appreciated. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org