[jira] [Commented] (YARN-8448) AM HTTPS Support for AM communication with RMWeb proxy
[ https://issues.apache.org/jira/browse/YARN-8448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16657044#comment-16657044 ] Eric Yang commented on YARN-8448: - [~rkanter] This patch has a problem with clean up /tmp/test-container-executor directory after test-container-executor is ran. If test-container-executor and cetest are run indpendently, then test result will show successful. In pre-commit build, running test-container-executor follow by cetest, then precommit build reports failures. > AM HTTPS Support for AM communication with RMWeb proxy > -- > > Key: YARN-8448 > URL: https://issues.apache.org/jira/browse/YARN-8448 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Robert Kanter >Assignee: Robert Kanter >Priority: Major > Fix For: 3.3.0 > > Attachments: YARN-8448.001.patch, YARN-8448.002.patch, > YARN-8448.003.patch, YARN-8448.004.patch, YARN-8448.005.patch, > YARN-8448.006.patch, YARN-8448.007.patch, YARN-8448.008.patch, > YARN-8448.009.patch, YARN-8448.010.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8448) AM HTTPS Support for AM communication with RMWeb proxy
[ https://issues.apache.org/jira/browse/YARN-8448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16652431#comment-16652431 ] Hudson commented on YARN-8448: -- FAILURE: Integrated in Jenkins build Hadoop-trunk-Commit #15230 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/15230/]) YARN-8448. AM HTTPS Support for AM communication with RMWeb proxy. (haibochen: rev c2288ac45b748b4119442c46147ccc324926c340) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/LinuxContainerExecutor.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/test/java/org/apache/hadoop/yarn/server/webproxy/TestProxyCA.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/runtime/LinuxContainerRuntimeConstants.java * (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/Credentials.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DefaultContainerExecutor.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/main/java/org/apache/hadoop/yarn/server/webproxy/WebAppProxy.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/runtime/TestDockerContainerRuntime.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/main/java/org/apache/hadoop/yarn/server/webproxy/ProxyCA.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/TestContainerRelaunch.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMContext.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/runtime/DockerLinuxContainerRuntime.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestLinuxContainerExecutorWithMocks.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestDefaultContainerExecutor.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/util.h * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/security/TestProxyCAManager.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/TestContainerLaunch.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/security/ProxyCAManager.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/ApplicationConstants.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/runtime/DefaultLinuxContainerRuntime.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/main/java/org/apache/hadoop/yarn/server/webproxy/WebAppProxyServlet.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/executor/ContainerStartContext.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/main.c * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/security/AMSecretKeys.java * (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/ssl/KeyStoreTestUtil.java * (edit)
[jira] [Commented] (YARN-8448) AM HTTPS Support
[ https://issues.apache.org/jira/browse/YARN-8448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16652392#comment-16652392 ] Haibo Chen commented on YARN-8448: -- I ran the cestest locally and it did not fail for me either. +1 on the latest patch. Will check it in shortly. > AM HTTPS Support > > > Key: YARN-8448 > URL: https://issues.apache.org/jira/browse/YARN-8448 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Robert Kanter >Assignee: Robert Kanter >Priority: Major > Attachments: YARN-8448.001.patch, YARN-8448.002.patch, > YARN-8448.003.patch, YARN-8448.004.patch, YARN-8448.005.patch, > YARN-8448.006.patch, YARN-8448.007.patch, YARN-8448.008.patch, > YARN-8448.009.patch, YARN-8448.010.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8448) AM HTTPS Support
[
https://issues.apache.org/jira/browse/YARN-8448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16652181#comment-16652181
]
Robert Kanter commented on YARN-8448:
-
{{TestCapacityOverTimePolicy}} failure is unrelated. I'm not sure why cetest
failed (it doesn't give any details), and it passes on my machine.
> AM HTTPS Support
>
>
> Key: YARN-8448
> URL: https://issues.apache.org/jira/browse/YARN-8448
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Robert Kanter
>Assignee: Robert Kanter
>Priority: Major
> Attachments: YARN-8448.001.patch, YARN-8448.002.patch,
> YARN-8448.003.patch, YARN-8448.004.patch, YARN-8448.005.patch,
> YARN-8448.006.patch, YARN-8448.007.patch, YARN-8448.008.patch,
> YARN-8448.009.patch, YARN-8448.010.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8448) AM HTTPS Support
[
https://issues.apache.org/jira/browse/YARN-8448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16651016#comment-16651016
]
Hadoop QA commented on YARN-8448:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
22s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 11 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m
44s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 19m
35s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
35s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 5m
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
21m 55s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 8m
57s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 4m
37s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
20s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 16m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 16m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 16m
18s{color} | {color:green} root generated 0 new + 1317 unchanged - 10 fixed =
1317 total (was 1327) {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
3m 42s{color} | {color:orange} root: The patch generated 7 new + 624 unchanged
- 8 fixed = 631 total (was 632) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 6m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 0s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 9m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 4m
47s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m
15s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
6s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
36s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
40s{color} | {color:green} hadoop-yarn-server-common in the patch passed.
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 18m 54s{color}
| {color:red} hadoop-yarn-server-nodemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
1s{color} | {color:green} hadoop-yarn-server-web-proxy in the patch passed.
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 95m 19s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch
[jira] [Commented] (YARN-8448) AM HTTPS Support
[ https://issues.apache.org/jira/browse/YARN-8448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16650740#comment-16650740 ] Robert Kanter commented on YARN-8448: - Thanks [~haibochen] for another review. The license issue is due to HADOOP-15853. The 010 patch: - rebased on latest trunk - changed wording of config property descriptions - fixed relevant checkstyle warnings > AM HTTPS Support > > > Key: YARN-8448 > URL: https://issues.apache.org/jira/browse/YARN-8448 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Robert Kanter >Assignee: Robert Kanter >Priority: Major > Attachments: YARN-8448.001.patch, YARN-8448.002.patch, > YARN-8448.003.patch, YARN-8448.004.patch, YARN-8448.005.patch, > YARN-8448.006.patch, YARN-8448.007.patch, YARN-8448.008.patch, > YARN-8448.009.patch, YARN-8448.010.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8448) AM HTTPS Support
[
https://issues.apache.org/jira/browse/YARN-8448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16650587#comment-16650587
]
Haibo Chen commented on YARN-8448:
--
Thanks [~rkanter] for addressing the comments!
{quote}The only reason we also verify (one of the) certs in the custom
{{HostnameVerifier}} is because we need to determine if we should ignore the
hostname of the certificate, or if we should fallback to the default one, which
does check the hostname; this was a convinent way to check if it's one of our
certs vs a real cert.
{quote}
Makes sense.
In addition to the checkstyle/license issue, one other minor thing from me
about the comment in YarnConfiguration and yarn-default.xml, "allow connections
to AMs that ..." is probably more accurate to say than "accept HTTP connections
though for AMs".
Otherwise, +1 from my side.
> AM HTTPS Support
>
>
> Key: YARN-8448
> URL: https://issues.apache.org/jira/browse/YARN-8448
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Robert Kanter
>Assignee: Robert Kanter
>Priority: Major
> Attachments: YARN-8448.001.patch, YARN-8448.002.patch,
> YARN-8448.003.patch, YARN-8448.004.patch, YARN-8448.005.patch,
> YARN-8448.006.patch, YARN-8448.007.patch, YARN-8448.008.patch,
> YARN-8448.009.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8448) AM HTTPS Support
[
https://issues.apache.org/jira/browse/YARN-8448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16648742#comment-16648742
]
Hadoop QA commented on YARN-8448:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
26s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 11 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
52s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 5m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
21m 14s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 8m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 4m
27s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
18s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 14m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 14m
21s{color} | {color:green} root generated 0 new + 1317 unchanged - 10 fixed =
1317 total (was 1327) {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
3m 27s{color} | {color:orange} root: The patch generated 20 new + 625 unchanged
- 8 fixed = 645 total (was 633) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 5m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 25s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 10m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 4m
22s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m
52s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
53s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
31s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
26s{color} | {color:green} hadoop-yarn-server-common in the patch passed.
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 18m 39s{color}
| {color:red} hadoop-yarn-server-nodemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
8s{color} | {color:green} hadoop-yarn-server-web-proxy in the patch passed.
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 94m 53s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch
[jira] [Commented] (YARN-8448) AM HTTPS Support
[
https://issues.apache.org/jira/browse/YARN-8448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16648614#comment-16648614
]
Robert Kanter commented on YARN-8448:
-
Thanks for the feedback {{haibochen}}! Some comments:
{quote}In KeyStoreTestUtil.bytesToKeyStore(), should we use try clause for the
inputstream?
{quote}
This isn't actually neceessary because the inputstream is a
{{ByteArrayInputStream}}. It's read methods don't throw {{IOException}} because
it's not doing any real IO (so there's nothing to handle here), and it's
{{close}} method is empty (so it does nothing).
{quote}testLaunchContainerCopyFiles(boolean https) has a lot of if-statements
which I think justified having two different methods, each calling some utility
methods. Can you try to break it into two? Likewise for
testContainerLaunch(boolean https).
{quote}
I think trying to split these out into utility methods will actually be harder
to follow. While there are a number of if statements checking if it's using
HTTPS or not, each check only does a small thing. For instance, in
{{testLaunchContainerCopyFiles}}, the only real difference is that whether or
not we have the keystore and truststore, and so there's an if statement to
write those files, to add them to the {{ContainerStartContext}}, and to check
that they exist - the rest of the test is identical.
{quote}In the host verifier, does the peer certificates come in any order?
Right now the code assumes that the 1st one is always signed by the ca cert.
{quote}
I can't find any docs on the ordering, but it shouldn't matter anyways because
both certs are signed with the same key (the CA key). You can see that we use
the CA's public key to verify both certs in the custom {{X509TrustManager}}.
The only reason we also verify (one of the) certs in the custom
{{HostnameVerifier}} is because we need to determine if we should ignore the
hostname of the certificate, or if we should fallback to the default one, which
does check the hostname; this was a convinent way to check if it's one of our
certs vs a real cert.
{quote}KeyPairGenerator is created locally. Is there a security reason not to
reuse KeyPairGenerator?
{quote}
>From what I can tell, there's no security issue with reusing a
>{{KeyPairGenerator}}, but it's unclear if it's thread safe; so it's safest to
>assume it isn't. That seems to be what people suggest (see
>[here|https://stackoverflow.com/questions/25691151/is-keypairgenerator-generatekeypair-thread-safe]
> and
>[here|http://bouncy-castle.1462172.n4.nabble.com/is-key-generation-thread-safe-td4658456.html])
{quote}In the custom X509TrustManager, how would the defaultTrustManager verify
the identify of the AM?
{quote}
If we determine that the cert was issued by the RM ({{issuedByRM==true}}), then
at the end of the method, we check that the Subject is "CN=". That will
only match if the RM connected to the AM it thought it was connecting to.
The 009 patch:
- Rebased on the latest trunk
- Addressed the cc warning
- Moved the secret keys to a new class, {{AMSecretKeys}}, in the
{{hadoop-yarn-server-common}} module
- Updated the wording of the config property in {{YarnConfiguration}} and
{{yarn-default.xml}}
- Changed the default to NONE, as per our offline discussion. In summary, we
don't need to generate certificates in a default non-HTTPS environment. If the
user sets up HTTPS for Hadoop, they can also change the config to LENIENT or
STRICT to get the AM certificates.
- Moved {{KEYSTORE_FILE_LOCATION}}, {{KEYSTORE_PASSWORD}},
{{TRUSTSTORE_FILE_LOCATION}}, and {{TRUSTSTORE_PASSWORD}} to
{{ApplicationConstants}}, and added javadoc
- {{DefaultLinuxContainerRuntime}} and {{DockerLinuxContainerRuntime}} are now
more defensive about null-checking for _both_ the keystore and truststore (that
shouldn't happen, but it is safer to check both in case that changes in the
future for some reason)
- In the C code, updated {{get_container_keystore_file}} and
{{get_container_truststore_file}} to to say "am container keystore" and "am
container truststore"
- Put back the exit code to {{OUT_OF_MEMORY}} for the string concat; I had
misread this before
- Removed the unnecessary checks before freeing possible NULL pointers
- Renamed {{COULD_NOT_CREATE_KEYSTORE_FILE}} to
{{COULD_NOT_CREATE_KEYSTORE_COPY}} and {{COULD_NOT_CREATE_TRUSTSTORE_FILE}} to
{{COULD_NOT_CREATE_TRUSTSTORE_COPY}} because we're copying and it's more
consistent with {{COULD_NOT_CREATE_SCRIPT_COPY}}. Also renamed
{{COULD_NOT_CREATE_CREDENTIALS_FILE}} to {{COULD_NOT_CREATE_CREDENTIALS_COPY}}
for the same reason.
- Renamed {{logpath}} to {{container_log_path}} and {{logpathapp}} to
{{app_log_path}} in {{test_launch_container}}
- Added {{@VisibleForTesting}} to {{ProxyCA#getCaCert}} and
{{ProxyCA#getCaKeyPair}}
- Split up {{TestProxyCA#testCreateTrustManager}} and
{{TestProxyCA#testCreateHostnameVerifier}}
[jira] [Commented] (YARN-8448) AM HTTPS Support
[ https://issues.apache.org/jira/browse/YARN-8448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16648434#comment-16648434 ] Haibo Chen commented on YARN-8448: -- For the ProxyCA related changes, I have a few questions/comments. 1) In the host verifier, does the peer certificates come in any order? Right now the code assumes that the 1st one is always signed by the ca cert. 2) Add @VisibleForTesting to getCaCert and getCaKeyPair? 3) KeyPairGenerator is created locally. Is there a security reason not to reuse KeyPairGenerator? 4) In the custom X509TrustManager, how would the defaultTrustManager verify the identify of the AM? 5) testCreateTrustManager() seem to have a lot of cases. Failing one would cause the following ones not to be executed. Can we split it into a few separate methods? Likewise for testCreateHostnameVerifier. > AM HTTPS Support > > > Key: YARN-8448 > URL: https://issues.apache.org/jira/browse/YARN-8448 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Robert Kanter >Assignee: Robert Kanter >Priority: Major > Attachments: YARN-8448.001.patch, YARN-8448.002.patch, > YARN-8448.003.patch, YARN-8448.004.patch, YARN-8448.005.patch, > YARN-8448.006.patch, YARN-8448.007.patch, YARN-8448.008.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8448) AM HTTPS Support
[ https://issues.apache.org/jira/browse/YARN-8448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16648293#comment-16648293 ] Haibo Chen commented on YARN-8448: -- A few minor comments/questions about the c code changes. 1) In container-executor.c#get_container_keystore_file(), do you think if is more specific to say 'AM container keystore'? Similar question for get_container_truststore_file(). 2) in create_script_paths(), the error code when checking get_container_launcher_file() and such should kept as OUT_OF_MEMORY given they are just string concatenation. 3) Looks like we follow c99 standard, so freeing a NULL pointer is not a problem, so we can remove the if(https=1) check when freeing the related pointers. 4) Let's rename COULD_NOT_CREATE_KEYSTORE_FILE to COULD_NOT_CREATE_KEYSTORE_COPY and COULD_NOT_CREATE_TRUSTSTORE_FILE to COULD_NOT_CREATE_TRUSTSTORE_COPY, given the c code makes a copy. 5) In test_launch_container(), "logpath" => "container_log_path", " logpathapp" => "app_log_path" > AM HTTPS Support > > > Key: YARN-8448 > URL: https://issues.apache.org/jira/browse/YARN-8448 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Robert Kanter >Assignee: Robert Kanter >Priority: Major > Attachments: YARN-8448.001.patch, YARN-8448.002.patch, > YARN-8448.003.patch, YARN-8448.004.patch, YARN-8448.005.patch, > YARN-8448.006.patch, YARN-8448.007.patch, YARN-8448.008.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8448) AM HTTPS Support
[
https://issues.apache.org/jira/browse/YARN-8448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16647227#comment-16647227
]
Hadoop QA commented on YARN-8448:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
14s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 11 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
21s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 18m
1s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 5m
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
22m 4s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 8m
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 4m
9s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
20s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 17m
52s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} cc {color} | {color:red} 17m 52s{color} |
{color:red} root generated 1 new + 9 unchanged - 0 fixed = 10 total (was 9)
{color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 17m
52s{color} | {color:green} root generated 0 new + 1317 unchanged - 10 fixed =
1317 total (was 1327) {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
3m 48s{color} | {color:orange} root: The patch generated 9 new + 595 unchanged
- 8 fixed = 604 total (was 603) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 5m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 29s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 8m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 4m
32s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m
49s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
53s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
18s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 18m 34s{color}
| {color:red} hadoop-yarn-server-nodemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
55s{color} | {color:green} hadoop-yarn-server-web-proxy in the patch passed.
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 90m 53s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
41s{color} | {color:red} The
[jira] [Commented] (YARN-8448) AM HTTPS Support
[ https://issues.apache.org/jira/browse/YARN-8448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16647209#comment-16647209 ] Haibo Chen commented on YARN-8448: -- Thanks [~rkanter] for the patch update! Posting some of my comments while I am still finishing up looking at the c code changes and the ProxyCA code. 1) The YARN-specific secret keys should probably be moved to yarn-modules (hadoop-yarn-server-common seems a good place), instead of being added to hadoop-common. 2) In KeyStoreTestUtil.bytesToKeyStore(), should we use try clause for the inputstream? 3) In YarnConfiguration and yarn-default.xml, can we rephrase the comments of the new configuration? "Sets the policy the RM should use when enforcing HTTPS ...". => "Specifies what RM does to enforce HTTPS..." For 'LENIEN', RM would always generate the key/trust store regardless of what URL AM sends to RM, if the policy is LENIENT or STRICT. In fact, that happens before AM is even launched. It is probably more accurate to say something along the lines of "RM will generate and provide to AMs a keystore and truststore , which AMs are free to use to set up HTTPs in their tracking web server. The RM webproxy would always connect users to AMs even they use HTTP" Similarly for 'STRICT', "RM will always generate and provide a keystore and truststore for AMs. AMs are free to use the keystore and truststore to set up HTTPs in their tracking web server. However, RM webproxy would block users from accessing any AM web server that runs in HTTP." 4) How about we move "KEYSTORE_FILE_LOCATION", "KEYSTORE_PASSWORD", TRUSTSTORE_FILE_LOCATION and TRUSTSTORE_PASSWORD to ApplicationConstants? 5) In DefaultLinuxContainerRuntime and DockerLinuxContainerRuntime, can we do null-checking for both keystore and truststore to be more defensive? 6) testLaunchContainerCopyFiles(boolean https) has a lot of if-statements which I think justified having two different methods, each calling some utility methods. Can you try to break it into two? Likewise for testContainerLaunch(boolean https). > AM HTTPS Support > > > Key: YARN-8448 > URL: https://issues.apache.org/jira/browse/YARN-8448 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Robert Kanter >Assignee: Robert Kanter >Priority: Major > Attachments: YARN-8448.001.patch, YARN-8448.002.patch, > YARN-8448.003.patch, YARN-8448.004.patch, YARN-8448.005.patch, > YARN-8448.006.patch, YARN-8448.007.patch, YARN-8448.008.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8448) AM HTTPS Support
[ https://issues.apache.org/jira/browse/YARN-8448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16647210#comment-16647210 ] Haibo Chen commented on YARN-8448: -- I'll continue the review tomorrow and post remaining comments, if any. > AM HTTPS Support > > > Key: YARN-8448 > URL: https://issues.apache.org/jira/browse/YARN-8448 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Robert Kanter >Assignee: Robert Kanter >Priority: Major > Attachments: YARN-8448.001.patch, YARN-8448.002.patch, > YARN-8448.003.patch, YARN-8448.004.patch, YARN-8448.005.patch, > YARN-8448.006.patch, YARN-8448.007.patch, YARN-8448.008.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8448) AM HTTPS Support
[ https://issues.apache.org/jira/browse/YARN-8448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16647168#comment-16647168 ] Robert Kanter commented on YARN-8448: - I see, thanks for the details [~jlowe] - it's been a while since I've done much C coding. I'll be sure to fix this in the next update to the patch. > AM HTTPS Support > > > Key: YARN-8448 > URL: https://issues.apache.org/jira/browse/YARN-8448 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Robert Kanter >Assignee: Robert Kanter >Priority: Major > Attachments: YARN-8448.001.patch, YARN-8448.002.patch, > YARN-8448.003.patch, YARN-8448.004.patch, YARN-8448.005.patch, > YARN-8448.006.patch, YARN-8448.007.patch, YARN-8448.008.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8448) AM HTTPS Support
[ https://issues.apache.org/jira/browse/YARN-8448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16647131#comment-16647131 ] Jason Lowe commented on YARN-8448: -- bq. The cc warning isn't a problem. IMHO the warning should be fixed. If it's a function argument and not intended to be a format string then the code should be calling fwrite instead of fprintf. That should fix the warning and prevent a potential crash if someone ever accidentally passes an argument in the future that contains a format directive thinking the contents will not try to be interpreted. > AM HTTPS Support > > > Key: YARN-8448 > URL: https://issues.apache.org/jira/browse/YARN-8448 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Robert Kanter >Assignee: Robert Kanter >Priority: Major > Attachments: YARN-8448.001.patch, YARN-8448.002.patch, > YARN-8448.003.patch, YARN-8448.004.patch, YARN-8448.005.patch, > YARN-8448.006.patch, YARN-8448.007.patch, YARN-8448.008.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8448) AM HTTPS Support
[
https://issues.apache.org/jira/browse/YARN-8448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16646811#comment-16646811
]
Robert Kanter commented on YARN-8448:
-
- The cc warning isn't a problem. It's complaining that I'm using a variable
in fprintf, but that variable is always hardcoded in the calling function, so
it's effectively not a variable.
- {{TestIncreaseAllocationExpirer}} failure is unrelated
- I'm not sure why cetest failed (it doesn't give any details), and it passes
on my machine
The 008 patch:
- Rebased on latest trunk
- Fixes the relevant checkstyle warnings
> AM HTTPS Support
>
>
> Key: YARN-8448
> URL: https://issues.apache.org/jira/browse/YARN-8448
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Robert Kanter
>Assignee: Robert Kanter
>Priority: Major
> Attachments: YARN-8448.001.patch, YARN-8448.002.patch,
> YARN-8448.003.patch, YARN-8448.004.patch, YARN-8448.005.patch,
> YARN-8448.006.patch, YARN-8448.007.patch, YARN-8448.008.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8448) AM HTTPS Support
[
https://issues.apache.org/jira/browse/YARN-8448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16646106#comment-16646106
]
Hadoop QA commented on YARN-8448:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
23s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 11 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m
6s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 5m
3s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
20m 52s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 7m
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 4m
1s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
18s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 16m
9s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} cc {color} | {color:red} 16m 9s{color} |
{color:red} root generated 1 new + 9 unchanged - 0 fixed = 10 total (was 9)
{color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 16m
9s{color} | {color:green} root generated 0 new + 1317 unchanged - 10 fixed =
1317 total (was 1327) {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
3m 32s{color} | {color:orange} root: The patch generated 38 new + 594 unchanged
- 8 fixed = 632 total (was 602) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 5m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 25s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 8m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
57s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m
32s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
54s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
21s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 18m 59s{color}
| {color:red} hadoop-yarn-server-nodemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
59s{color} | {color:green} hadoop-yarn-server-web-proxy in the patch passed.
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 94m 39s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
43s{color} | {color:red}
[jira] [Commented] (YARN-8448) AM HTTPS Support
[ https://issues.apache.org/jira/browse/YARN-8448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16645928#comment-16645928 ] Robert Kanter commented on YARN-8448: - The 007 patch: - Rebased on latest trunk - Removed pom changes because they were taken care of by HADOOP-15832 - Renamed OFF, OPTIONAL, REQUIRED to NONE, LENIENT, and STRICT > AM HTTPS Support > > > Key: YARN-8448 > URL: https://issues.apache.org/jira/browse/YARN-8448 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Robert Kanter >Assignee: Robert Kanter >Priority: Major > Attachments: YARN-8448.001.patch, YARN-8448.002.patch, > YARN-8448.003.patch, YARN-8448.004.patch, YARN-8448.005.patch, > YARN-8448.006.patch, YARN-8448.007.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8448) AM HTTPS Support
[ https://issues.apache.org/jira/browse/YARN-8448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16644059#comment-16644059 ] Robert Kanter commented on YARN-8448: - I discussed this with [~haibochen] offline and we agreed that it's fine how it is now, but we should rename the values because REQUIRED is confusing. I've thought a bit about names, and how about: NONE, LENIENT, and STRICT. > AM HTTPS Support > > > Key: YARN-8448 > URL: https://issues.apache.org/jira/browse/YARN-8448 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Robert Kanter >Assignee: Robert Kanter >Priority: Major > Attachments: YARN-8448.001.patch, YARN-8448.002.patch, > YARN-8448.003.patch, YARN-8448.004.patch, YARN-8448.005.patch, > YARN-8448.006.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8448) AM HTTPS Support
[ https://issues.apache.org/jira/browse/YARN-8448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16643719#comment-16643719 ] Haibo Chen commented on YARN-8448: -- Thanks [~rkanter] for the elaboration. OFF is what YARN does as of today and OPTIONAL essentially allows applications to opt in secure/HTTPS connection between RM and AM web server. They are straightforward. IIUC, REQUIRED, without failing AM registration when an AM sends a http tracking URL, is OPTIONAL + warning/blocking users if the AM connection is HTTP. As running an application and accessing the running application are often times done together, it seems less intuitive to me that a user submits an application that runs fine, but could not access the application while it is running. If we're concerned about users with older apps that can't be updated to use HTTPS, IMO the proper solution is to use OPTIONAL as the policy. We can, when the policy is set of Optional, add an extra warning message (without blocking access) to the page returned to users. The way I think of the policy is it tells AMs what RM expects in terms of its requirement in the RM & AM web server connection (so more of an admin configuration): OFF means AMs can only choose HTTP connection; OPTIONAL means AMs can opt in HTTPS connection; REQUIRED means AMs must use HTTPS connection. From the users' perspective, if an application can run (without failing RM registration), they can always access the application. They may see a warning message if the connection is HTTP. > AM HTTPS Support > > > Key: YARN-8448 > URL: https://issues.apache.org/jira/browse/YARN-8448 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Robert Kanter >Assignee: Robert Kanter >Priority: Major > Attachments: YARN-8448.001.patch, YARN-8448.002.patch, > YARN-8448.003.patch, YARN-8448.004.patch, YARN-8448.005.patch, > YARN-8448.006.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8448) AM HTTPS Support
[
https://issues.apache.org/jira/browse/YARN-8448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16642647#comment-16642647
]
Robert Kanter commented on YARN-8448:
-
Thanks [~haibochen] for the review. That's a good idea about BouncyCastle;
that'll make it easier to iterate on this patch because we won't have to wait
6+ hours each time :). I've filed HADOOP-15832 and put up a patch with just
the pom changes there. Once that's in, I'll update this JIRA's patch.
On your second point, if the policy is REQUIRED, the RM won't proxy you to a
non-HTTPS AM. You'll instead get a warning page, similar to the
{{WebAppProxyServlet#warnUserPage}} code that warns the user in certain
situations when Kerberos is enabled. Take a look at the
{{WebAppProxyServlet#checkHttpsRequiredAndNotProvided}} method to see where
this is done. When set to OPTIONAL, this behavior doesn't trigger. If we fail
the AM, I'm concerned that it's going to make it harder for users with older
apps that can't be updated to use HTTPS. As it is now, with REQUIRED, you can
still run the AM if it's using HTTP, you just can't access it's web page (with
OPTIONAL, you'd still be able to access it's web page). Does that make sense?
> AM HTTPS Support
>
>
> Key: YARN-8448
> URL: https://issues.apache.org/jira/browse/YARN-8448
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Robert Kanter
>Assignee: Robert Kanter
>Priority: Major
> Attachments: YARN-8448.001.patch, YARN-8448.002.patch,
> YARN-8448.003.patch, YARN-8448.004.patch, YARN-8448.005.patch,
> YARN-8448.006.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8448) AM HTTPS Support
[ https://issues.apache.org/jira/browse/YARN-8448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16642603#comment-16642603 ] Haibo Chen commented on YARN-8448: -- Thanks [~rkanter] for the the patch. I took a high-level look. The overall approach looks good to me. I do have two comment/questions. 1) We can pull the bounty-castle upgrade into a separate patch 2) Does it make sense to add a check on the AM-RM registeration path? That is, if AM gives a HTTP tracking url when the policy is set to Required, maybe we should fail the AM? Otherwise, REQUIRED is just the same as optional. > AM HTTPS Support > > > Key: YARN-8448 > URL: https://issues.apache.org/jira/browse/YARN-8448 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Robert Kanter >Assignee: Robert Kanter >Priority: Major > Attachments: YARN-8448.001.patch, YARN-8448.002.patch, > YARN-8448.003.patch, YARN-8448.004.patch, YARN-8448.005.patch, > YARN-8448.006.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8448) AM HTTPS Support
[ https://issues.apache.org/jira/browse/YARN-8448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16638522#comment-16638522 ] Robert Kanter commented on YARN-8448: - Test failures seem unrelated - they all pass locally for me. > AM HTTPS Support > > > Key: YARN-8448 > URL: https://issues.apache.org/jira/browse/YARN-8448 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Robert Kanter >Assignee: Robert Kanter >Priority: Major > Attachments: YARN-8448.001.patch, YARN-8448.002.patch, > YARN-8448.003.patch, YARN-8448.004.patch, YARN-8448.005.patch, > YARN-8448.006.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8448) AM HTTPS Support
[
https://issues.apache.org/jira/browse/YARN-8448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16636430#comment-16636430
]
Hadoop QA commented on YARN-8448:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
23s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 11 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
14s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
36s{color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
26s{color} | {color:red} server-scm in trunk failed. {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
26s{color} | {color:red} ozone-manager in trunk failed. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
27m 50s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-project hadoop-client-modules/hadoop-client-check-invariants
hadoop-client-modules/hadoop-client-check-test-invariants
hadoop-client-modules/hadoop-client-minicluster
hadoop-client-modules/hadoop-client-runtime
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests
{color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
21s{color} | {color:red} server-scm in trunk failed. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
20s{color} | {color:red} ozone-manager in trunk failed. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
22s{color} | {color:red} server-scm in trunk failed. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
18s{color} | {color:red} ozone-manager in trunk failed. {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
19s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
22s{color} | {color:red} hadoop-kms in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
59s{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
24s{color} | {color:red} hadoop-hdfs-httpfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
21s{color} | {color:red} hadoop-hdfs-nfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
33s{color} | {color:red} hadoop-yarn-api in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
38s{color} | {color:red} hadoop-yarn-common in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
34s{color} | {color:red} hadoop-yarn-server-nodemanager in the patch failed.
{color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
16s{color} | {color:red} hadoop-yarn-server-web-proxy in the patch failed.
{color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
28s{color} | {color:red} hadoop-yarn-server-applicationhistoryservice in the
patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
26s{color} | {color:red} hadoop-yarn-server-resourcemanager in the patch
failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
25s{color} | {color:red} hadoop-yarn-server-tests in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
20s{color} | {color:red} hadoop-mapreduce-client-app in the patch failed.
{color} |
| {color:red}-1{color} | {color:red}
[jira] [Commented] (YARN-8448) AM HTTPS Support
[ https://issues.apache.org/jira/browse/YARN-8448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16636254#comment-16636254 ] Robert Kanter commented on YARN-8448: - I've kicked off another run after fixing trunk (it wasn't compiling before due to an incomplete pom change). > AM HTTPS Support > > > Key: YARN-8448 > URL: https://issues.apache.org/jira/browse/YARN-8448 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Robert Kanter >Assignee: Robert Kanter >Priority: Major > Attachments: YARN-8448.001.patch, YARN-8448.002.patch, > YARN-8448.003.patch, YARN-8448.004.patch, YARN-8448.005.patch, > YARN-8448.006.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8448) AM HTTPS Support
[
https://issues.apache.org/jira/browse/YARN-8448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16636229#comment-16636229
]
Hadoop QA commented on YARN-8448:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
25s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 11 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
29s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
10s{color} | {color:red} root in trunk failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
10s{color} | {color:red} root in trunk failed. {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
2m 15s{color} | {color:orange} The patch fails to run checkstyle in root
{color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
19s{color} | {color:red} server-scm in trunk failed. {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
17s{color} | {color:red} ozone-manager in trunk failed. {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
12s{color} | {color:red} hadoop-yarn-server-resourcemanager in trunk failed.
{color} |
| {color:red}-1{color} | {color:red} shadedclient {color} | {color:red} 11m
33s{color} | {color:red} branch has errors when building and testing our client
artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-project
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests
hadoop-client-modules/hadoop-client-runtime
hadoop-client-modules/hadoop-client-minicluster
hadoop-client-modules/hadoop-client-check-invariants
hadoop-client-modules/hadoop-client-check-test-invariants {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
13s{color} | {color:red} server-scm in trunk failed. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
12s{color} | {color:red} ozone-manager in trunk failed. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
14s{color} | {color:red} hadoop-yarn-server-resourcemanager in trunk failed.
{color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
12s{color} | {color:red} server-scm in trunk failed. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
10s{color} | {color:red} ozone-manager in trunk failed. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
11s{color} | {color:red} hadoop-yarn-server-resourcemanager in trunk failed.
{color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
21s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
20s{color} | {color:red} hadoop-yarn-server-nodemanager in the patch failed.
{color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
16s{color} | {color:red} hadoop-yarn-server-web-proxy in the patch failed.
{color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
17s{color} | {color:red} hadoop-client-runtime in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
19s{color} | {color:red} hadoop-client-minicluster in the patch failed. {color}
|
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
10s{color} | {color:red} server-scm in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
10s{color} | {color:red} ozone-manager in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
12s{color} | {color:red} hadoop-yarn-server-resourcemanager in the patch
failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
10s{color} | {color:red} root in the patch failed. {color} |
| {color:red}-1{color} | {color:red} cc {color} | {color:red} 0m 10s{color} |
{color:red} root in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac
[jira] [Commented] (YARN-8448) AM HTTPS Support
[
https://issues.apache.org/jira/browse/YARN-8448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16635808#comment-16635808
]
Robert Kanter commented on YARN-8448:
-
Thanks for the review [~snemeth]. I was on PTO but am back now, so I was able
to address your comments:
- In {{ProxyCAManager}}, {{LOG}} and {{rmContext}} are currently unused, but
they will be in the sibling JIRA, YARN-8449, to add support for RM HA. When
splitting up the patch, it was simpler to leave these in for now. There's a
{{TODO}} in {{ProxyCAManager#recover}}.
- I don't think there's a benefit to making the "to" date in
{{createCACertAndKeyPair}} a {{static final}} variable. It's only ever used
once. By not making it {{static}}, we can let it be garbage collected.
- It is intentional that {{ProxyCA#createCACertAndKeyPair}} passes the same
string for the issuer and subject. This is because the CA is issuing a
certificate for itself, so it is both the subject (the one who the certificate
is for) and the issuer (the one issuing the certificate).
The 006 patch:
- Rebased on latest trunk
- Addressed comments from [~snemeth]
-- Added {{}} to pom
-- Removed unused imports
-- Deleted unused methods {{KeyStoreTestUtil#setAllowAllSSL}}
-- Fixed exit codes in {{container-executor.c#create_script_paths}}. I had
blindly copy-pasted some existing code, which also had the wrong exit code
(I've fixed that too).
-- Made {{TestApplicationMasterLauncher#testSetupTokens}} {{private}}
-- Removed unnecessary type argument in
{{TestApplicationMasterLauncher.MyAMLauncher#createAndSetAMRMToken}}
-- {{createTrustManager#checkClientTrusted}} no longer declares throwing a
{{CertificateException}}
> AM HTTPS Support
>
>
> Key: YARN-8448
> URL: https://issues.apache.org/jira/browse/YARN-8448
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Robert Kanter
>Assignee: Robert Kanter
>Priority: Major
> Attachments: YARN-8448.001.patch, YARN-8448.002.patch,
> YARN-8448.003.patch, YARN-8448.004.patch, YARN-8448.005.patch,
> YARN-8448.006.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8448) AM HTTPS Support
[
https://issues.apache.org/jira/browse/YARN-8448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16582321#comment-16582321
]
Szilard Nemeth commented on YARN-8448:
--
Hey [~rkanter]!
Here are my review comments:
- hadoop project/pom.xml: Please use a variable for the bouncycastle version
instead of using 1.59 twice.
- There are unused imports in: ResourceManager, TestWebAppProxyServlet
- I don't really see any usage of the introduced method {{
org.apache.hadoop.security.ssl.KeyStoreTestUtil#setAllowAllSSL(javax.net.ssl.HttpsURLConnection)
}}
Actually there are two definitions of {{setAllowAllSSL}}, they both public and
I can't see any usage of those methods.
- {{KeyStoreTestUtil}}: Throws clause for {{CertificateException}} can be
removed since {{checkClientTrusted}} and {{checkServerTrusted}} never throws
this exception
- {{ProxyCAManager}}: LOG is unused, rmContext is unused. Did you intend to use
rmContext in recover? I would assume this as there is a TODO there.
- {{container-executor.c}}: The first block you added to
{{create_script_paths}} uses {{exit_code = OUT_OF_MEMORY;}} if the
keystore/truststore file destinations could not be created. Is this
intentional?
- {{container-executor.c}}: Still in {{create_script_paths}}, the code black
that try to open keystore/truststore files,
the exit codes seem to be bad here:
{{exit_code = INVALID_ARGUMENT_NUMBER;}}
Should be instead: {{COULD_NOT_CREATE_KEYSTORE_FILE or
COULD_NOT_CREATE_TRUSTSTORE_FILE}}
-
{{org.apache.hadoop.yarn.server.resourcemanager.TestApplicationMasterLauncher#testSetupTokens:
}} This method could be private.
Moreover, I would extract the code setting up {{proxyCA}} with the mocked
methods to a new method, for the sake of readability.
-
{{org.apache.hadoop.yarn.server.resourcemanager.TestApplicationMasterLauncher.MyAMLauncher#createAndSetAMRMToken:
}} Type argument AMRMTokenIdentifierfor Token can be removed
- {{org.apache.hadoop.yarn.server.webproxy.ProxyCA#createCACertAndKeyPair: }} I
think the {{to}} Date could be a private static final field of this class as it
is a fixed date.
- {{org.apache.hadoop.yarn.server.webproxy.ProxyCA#createCACertAndKeyPair }}:
When {{createCert}} is invoked, is it intentional that you used the same string
for the issuer and the subject?
- {{ProxyCA{{, in method {{createTrustManager}}: {{checkClientTrusted}} does
not throw a {{CertificateException}} so you can remove it from the signature
> AM HTTPS Support
>
>
> Key: YARN-8448
> URL: https://issues.apache.org/jira/browse/YARN-8448
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Robert Kanter
>Assignee: Robert Kanter
>Priority: Major
> Attachments: YARN-8448.001.patch, YARN-8448.002.patch,
> YARN-8448.003.patch, YARN-8448.004.patch, YARN-8448.005.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8448) AM HTTPS Support
[
https://issues.apache.org/jira/browse/YARN-8448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16571030#comment-16571030
]
genericqa commented on YARN-8448:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
24s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 11 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
21s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 27m
58s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 30m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 14m
16s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
25m 59s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-project
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests
hadoop-client-modules/hadoop-client-runtime
hadoop-client-modules/hadoop-client-check-invariants
hadoop-client-modules/hadoop-client-minicluster
hadoop-client-modules/hadoop-client-check-test-invariants {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
42s{color} | {color:red} hadoop-hdds/server-scm in trunk has 1 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 11m
6s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
19s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 29m
55s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} cc {color} | {color:red} 29m 55s{color} |
{color:red} root generated 1 new + 11 unchanged - 0 fixed = 12 total (was 11)
{color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 29m
55s{color} | {color:green} root generated 0 new + 1458 unchanged - 10 fixed =
1458 total (was 1468) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 14m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
23s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 39s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-project
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests
hadoop-client-modules/hadoop-client-runtime
hadoop-client-modules/hadoop-client-check-invariants
hadoop-client-modules/hadoop-client-minicluster
hadoop-client-modules/hadoop-client-check-test-invariants {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 17m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 11m
21s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
23s{color} | {color:green} hadoop-project in the patch passed.
[jira] [Commented] (YARN-8448) AM HTTPS Support
[
https://issues.apache.org/jira/browse/YARN-8448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16570703#comment-16570703
]
Robert Kanter commented on YARN-8448:
-
The 005 patch:
- Fixes the unit tests. It turns out we do still need the test scope
bouncycastle dependencies.
- Added some useful log messages to {{ProxyCA}}.
> AM HTTPS Support
>
>
> Key: YARN-8448
> URL: https://issues.apache.org/jira/browse/YARN-8448
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Robert Kanter
>Assignee: Robert Kanter
>Priority: Major
> Attachments: YARN-8448.001.patch, YARN-8448.002.patch,
> YARN-8448.003.patch, YARN-8448.004.patch, YARN-8448.005.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8448) AM HTTPS Support
[
https://issues.apache.org/jira/browse/YARN-8448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16569068#comment-16569068
]
genericqa commented on YARN-8448:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
22s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 11 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
19s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 24m
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 27m
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
26s{color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
44s{color} | {color:red} server-scm in trunk failed. {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
38s{color} | {color:red} ozone-manager in trunk failed. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
26m 2s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-project
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests
hadoop-client-modules/hadoop-client-runtime
hadoop-client-modules/hadoop-client-check-invariants
hadoop-client-modules/hadoop-client-minicluster
hadoop-client-modules/hadoop-client-check-test-invariants {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
31s{color} | {color:red} server-scm in trunk failed. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
31s{color} | {color:red} ozone-manager in trunk failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 11m
52s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
22s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
19s{color} | {color:red} server-scm in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
19s{color} | {color:red} ozone-manager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 27m
6s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} cc {color} | {color:red} 27m 6s{color} |
{color:red} root generated 1 new + 11 unchanged - 0 fixed = 12 total (was 11)
{color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 27m
6s{color} | {color:green} root generated 0 new + 1458 unchanged - 10 fixed =
1458 total (was 1468) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
42s{color} | {color:red} server-scm in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
37s{color} | {color:red} ozone-manager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
25s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 13s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-project
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests
hadoop-client-modules/hadoop-client-runtime
hadoop-client-modules/hadoop-client-check-invariants
[jira] [Commented] (YARN-8448) AM HTTPS Support
[
https://issues.apache.org/jira/browse/YARN-8448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16568819#comment-16568819
]
Robert Kanter commented on YARN-8448:
-
- Existing mvnsite failure and findbugs with ozone and hdds are not related
- The new cc warning is in {{test-container-executor.c}} and is due to printing
{{strerror}}, which is a function call, so the compiler is warning about it;
but this is consistent with a lot of other uses in the code.
- Test failures are all unrelated
- There was no link for the "shadedclient" failure, but I managed to find it
here:
https://builds.apache.org/job/PreCommit-YARN-Build/21488/artifact/out/patch-shadedclient.txt
-- Both hadoop-client-runtime and hadoop-client-minicluster were including the
bouncycastle artifacts transitively. It turns out that about 10 modules were
including bouncycastle even though we're only actually using it in 2 of them
(and 1 of those is solely due to my changes). I've removed the unnecessary
inclusions. I also removed it from being transitively pulled into the
hadoop-client-runtime and app because those don't actually need it.
-- I also had to exclude bouncycastle from being shaded. It's signed with a
special Oracle certificate so it can be a custom JCE security provider. If we
shade it, the signature won't match anymore, so we'd have to remove it; but
then we can't properly use it because the JVM will reject it due to the lack of
signature. See
https://stackoverflow.com/questions/13721579/jce-cannot-authenticate-the-provider-bc-in-java-swing-application
and
https://side-effects-bang.blogspot.com/2015/02/deploying-uberjars-that-use-bouncy.html
The 004 patch:
- Makes the pom changes described above
- Rebased on latest trunk
> AM HTTPS Support
>
>
> Key: YARN-8448
> URL: https://issues.apache.org/jira/browse/YARN-8448
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Robert Kanter
>Assignee: Robert Kanter
>Priority: Major
> Attachments: YARN-8448.001.patch, YARN-8448.002.patch,
> YARN-8448.003.patch, YARN-8448.004.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8448) AM HTTPS Support
[
https://issues.apache.org/jira/browse/YARN-8448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16567616#comment-16567616
]
genericqa commented on YARN-8448:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
24s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 11 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
20s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 27m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 29m
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
23s{color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
37s{color} | {color:red} ozone-manager in trunk failed. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
23m 32s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-project
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests
{color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
43s{color} | {color:red} hadoop-hdds/server-scm in trunk has 1 extant Findbugs
warnings. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
29s{color} | {color:red} ozone-manager in trunk failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 9m
5s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
22s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
19s{color} | {color:red} ozone-manager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 29m
49s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} cc {color} | {color:red} 29m 49s{color} |
{color:red} root generated 1 new + 11 unchanged - 0 fixed = 12 total (was 11)
{color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 29m
49s{color} | {color:green} root generated 0 new + 1458 unchanged - 10 fixed =
1458 total (was 1468) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
38s{color} | {color:red} ozone-manager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
20s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:red}-1{color} | {color:red} shadedclient {color} | {color:red} 12m
19s{color} | {color:red} patch has errors when building and testing our client
artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-project
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests
{color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
29s{color} | {color:red} ozone-manager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 8m
56s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
24s{color} | {color:green} hadoop-project in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m
26s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m
8s{color} |
[jira] [Commented] (YARN-8448) AM HTTPS Support
[ https://issues.apache.org/jira/browse/YARN-8448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16567078#comment-16567078 ] Robert Kanter commented on YARN-8448: - I'm not sure where Jenkins went, so I've just kicked one off manually > AM HTTPS Support > > > Key: YARN-8448 > URL: https://issues.apache.org/jira/browse/YARN-8448 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Robert Kanter >Assignee: Robert Kanter >Priority: Major > Attachments: YARN-8448.001.patch, YARN-8448.002.patch, > YARN-8448.003.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8448) AM HTTPS Support
[
https://issues.apache.org/jira/browse/YARN-8448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16562482#comment-16562482
]
Robert Kanter commented on YARN-8448:
-
There was a bunch of unrelated issues with HDFS, even before the patch applied.
Hopefully the next run will be cleaner.
The 003 patch:
- Replaces the deprecated {{X509v1CertificateGenerator}} and
{{X509v3CertificateGenerator}} with the newer {{X509v3CertificateBuilder}}, and
refactored it to share code now
-- This fixes all of the new deprecation warnings
- Added "BasicContraints" flags to indicate if a certificate is a CA
certificate or not
- Addressed whitespace and findbugs
- Fixed {{TestYarnConfigurationFields}} by adding
{{yarn.resourcemanager.application-https.policy}} to yarn-default.xml
> AM HTTPS Support
>
>
> Key: YARN-8448
> URL: https://issues.apache.org/jira/browse/YARN-8448
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Robert Kanter
>Assignee: Robert Kanter
>Priority: Major
> Attachments: YARN-8448.001.patch, YARN-8448.002.patch,
> YARN-8448.003.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8448) AM HTTPS Support
[
https://issues.apache.org/jira/browse/YARN-8448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560586#comment-16560586
]
genericqa commented on YARN-8448:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
18s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 11 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
19s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 24m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 28m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
19s{color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
40s{color} | {color:red} server-scm in trunk failed. {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
36s{color} | {color:red} ozone-manager in trunk failed. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
21m 53s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-project
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests
{color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
30s{color} | {color:red} server-scm in trunk failed. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
29s{color} | {color:red} ozone-manager in trunk failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 8m
17s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
21s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
16s{color} | {color:red} server-scm in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
15s{color} | {color:red} ozone-manager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 27m
3s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} cc {color} | {color:red} 27m 3s{color} |
{color:red} root generated 1 new + 11 unchanged - 0 fixed = 12 total (was 11)
{color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 27m 3s{color}
| {color:red} root generated 11 new + 1458 unchanged - 10 fixed = 1469 total
(was 1468) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
31s{color} | {color:red} server-scm in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
29s{color} | {color:red} ozone-manager in the patch failed. {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch 4 line(s) with tabs. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
18s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:red}-1{color} | {color:red} shadedclient {color} | {color:red} 10m
57s{color} | {color:red} patch has errors when building and testing our client
artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-project
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests
{color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
15s{color} | {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
28s{color} | {color:red}
[jira] [Commented] (YARN-8448) AM HTTPS Support
[
https://issues.apache.org/jira/browse/YARN-8448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560254#comment-16560254
]
Robert Kanter commented on YARN-8448:
-
The 002 patch is mostly just some refactoring to split {{ProxyCAManager}} into
{{ProxyCA}} and {{ProxyCAManager}}. This was necessary as I started looking at
YARN-8449 (RM HA support) because {{ProxyCA}} needs to be in the
{{hadoop-yarn-server-web-proxy}} module (to work with the proxy code) while
{{ProxyCAManager}} needs to be in the {{hadoop-yarn-server-resourcemanager}}
module (to work with the {{RMStateStore}}. Most of the code is the same, but
some things got moved around and reorganized. I also beefed up some tests in
{{TestProxyCA}} (which used to be named {{TestProxyCAManager}}.
> AM HTTPS Support
>
>
> Key: YARN-8448
> URL: https://issues.apache.org/jira/browse/YARN-8448
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Robert Kanter
>Assignee: Robert Kanter
>Priority: Major
> Attachments: YARN-8448.001.patch, YARN-8448.002.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8448) AM HTTPS Support
[
https://issues.apache.org/jira/browse/YARN-8448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16556364#comment-16556364
]
Robert Kanter commented on YARN-8448:
-
I've finished up a patch that implements everything described in YARN-6586,
other than the RM HA support (TODO in YARN-8449) and Documentation (just filed
YARN-8582 for this). I've put the bulk of the changes here
(YARN-8448.001.patch), and the MapReduce changes in MAPREDUCE-4669.
Some notes on the patch:
- Updated BouncyCastle library to a newer version and had to also change the
artifact from {{bcprov-jdk16}} to {{bcprov-jdk15on}}. I know that sounds
backwards, but jdk15on is actually newer and the one we should be using (see
http://bouncy-castle.1462172.n4.nabble.com/Bouncycaslte-bcprov-jdk15-vs-bcprov-jdk16-td4656252.html).
- The {{yarn.resourcemanager.application-https.policy}} property controls how
the RM should handle HTTPS when talking to AMs. It can be {{OFF}},
{{OPTIONAL}} (default), or {{REQUIRED}}. {{OFF}} makes it behave like today,
where it does nothing special. {{OPTIONAL}} makes it generate and provide the
keystore and truststore to the AM when it sees an HTTPS tracking URL, but HTTP
is also still allowed. And {{REQUIRED}} is like {{OPTIONAL}}, but it won't
follow HTTP tracking URLs.
- A lot of the code around the container executors is in providing/copying/etc
the keystore and truststore files. I've largely based this on the existing way
we handle the credentials (delegation tokens) file.
- When provided a keystore file, the AM will get env vars
{{KEYSTORE_FILE_LOCATION}} and {{KEYSTORE_PASSWORD}}; similarly,
{{TRUSTSTORE_FILE_LOCATION}} and {{TRUSTSTORE_PASSWORD}} for the truststore
file.
- Due to the (ugly) way we parse arguments in the LCE, I had to add an argument
that's either {{--http}} or {{--https}} to indicate if we'll be providing it
the keystore and truststore files. Otherwise, there isn't a good way to have
optional arguments.
- In order to keep things simple, I piggybacked passing the keystore and
truststore files and passwords via secrets in the Credentials, which is already
securely passed from the RM to the NM.
- {{ProxyCAManager}} is in charge of creating the certificates, keystores, and
truststores.
- When writing the unit tests, I found a number of tests that were about 80%
complete in what they were testing, which I completed in addition to adding
tests for my changes.
-- I also tried to simplify some things (e.g. {{TestDockerContainerRuntime}}
has ~30 tests that all duplicate the code for checking the arguments, and
because I changed the number of arguments, they all failed - instead of
updating them all, I created a helper method)
- I'm not sure what's up with {{test-container-executor}}, but unless my
environment was messed up, it doesn't work when run as {{root}}; maybe people
typically run it as a normal user? The test talks about running as {{root}} as
an option, and even has a few tests that only run when running as {{root}}. I
spent some time fixing this - it now runs in all 4 user configurations
described in the existing comments.
- I've tested in a real cluster with the DefaultContainerExecutor and
LinuxContainerExecutor using all combinations of
{{yarn.resourcemanager.application-https.policy}},
{{yarn.app.mapreduce.am.webapp.https.enabled}}, and
{{yarn.app.mapreduce.am.webapp.https.client.auth}} (see MAPREDUCE-4669), and
everything behaved correctly. I haven't tested out the
DockerContainerExecutor.
-- If you want to try this out yourself in a cluster, I'd recommend also
applying the MAPREDUCE-4669 patch so you have an AM that supports the changes.
You can then use {{openssl s_client -connect :}} to get
SSL details. You can also try {{curl}}.
> AM HTTPS Support
>
>
> Key: YARN-8448
> URL: https://issues.apache.org/jira/browse/YARN-8448
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Robert Kanter
>Assignee: Robert Kanter
>Priority: Major
> Attachments: YARN-8448.001.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
