[
https://issues.apache.org/jira/browse/YARN-8448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16556364#comment-16556364
]
Robert Kanter edited comment on YARN-8448 at 7/25/18 10:42 PM:
---------------------------------------------------------------
I've finished up a patch that implements everything described in YARN-6586,
other than the RM HA support (TODO in YARN-8449) and Documentation (just filed
YARN-8582 for this). I've put the bulk of the changes here
(YARN-8448.001.patch), and the MapReduce changes in MAPREDUCE-4669.
Some notes on the patch:
- Updated BouncyCastle library to a newer version and had to also change the
artifact from {{bcprov-jdk16}} to {{bcprov-jdk15on}}. I know that sounds
backwards, but jdk15on is actually newer and the one we should be using (see
http://bouncy-castle.1462172.n4.nabble.com/Bouncycaslte-bcprov-jdk15-vs-bcprov-jdk16-td4656252.html).
- The {{yarn.resourcemanager.application-https.policy}} property controls how
the RM should handle HTTPS when talking to AMs. It can be {{OFF}},
{{OPTIONAL}} (default), or {{REQUIRED}}. {{OFF}} makes it behave like today,
where it does nothing special. {{OPTIONAL}} makes it generate and provide the
keystore and truststore to the AM when it sees an HTTPS tracking URL, but HTTP
is also still allowed. And {{REQUIRED}} is like {{OPTIONAL}}, but it won't
follow HTTP tracking URLs.
- A lot of the code around the container executors is in providing/copying/etc
the keystore and truststore files. I've largely based this on the existing way
we handle the credentials (delegation tokens) file.
- When provided a keystore file, the AM will get env vars
{{KEYSTORE_FILE_LOCATION}} and {{KEYSTORE_PASSWORD}}; similarly,
{{TRUSTSTORE_FILE_LOCATION}} and {{TRUSTSTORE_PASSWORD}} for the truststore
file.
- Due to the (ugly) way we parse arguments in the LCE, I had to add an argument
that's either {{\-\-http}} or {{\-\-https}} to indicate if we'll be providing
it the keystore and truststore files. Otherwise, there isn't a good way to
have optional arguments.
- In order to keep things simple, I piggybacked passing the keystore and
truststore files and passwords via secrets in the Credentials, which is already
securely passed from the RM to the NM.
- {{ProxyCAManager}} is in charge of creating the certificates, keystores, and
truststores.
- When writing the unit tests, I found a number of tests that were about 80%
complete in what they were testing, which I completed in addition to adding
tests for my changes.
-- I also tried to simplify some things (e.g. {{TestDockerContainerRuntime}}
has ~30 tests that all duplicate the code for checking the arguments, and
because I changed the number of arguments, they all failed - instead of
updating them all, I created a helper method)
- I'm not sure what's up with {{test-container-executor}}, but unless my
environment was messed up, it doesn't work when run as {{root}}; maybe people
typically run it as a normal user? The test talks about running as {{root}} as
an option, and even has a few tests that only run when running as {{root}}. I
spent some time fixing this - it now runs in all 4 user configurations
described in the existing comments.
- I've tested in a real cluster with the DefaultContainerExecutor and
LinuxContainerExecutor using all combinations of
{{yarn.resourcemanager.application-https.policy}},
{{yarn.app.mapreduce.am.webapp.https.enabled}}, and
{{yarn.app.mapreduce.am.webapp.https.client.auth}} (see MAPREDUCE-4669), and
everything behaved correctly. I haven't tested out the
DockerContainerExecutor.
-- If you want to try this out yourself in a cluster, I'd recommend also
applying the MAPREDUCE-4669 patch so you have an AM that supports the changes.
You can then use {{openssl s_client -connect <am-host>:<am-web-port>}} to get
SSL details. You can also try {{curl}}.
was (Author: rkanter):
I've finished up a patch that implements everything described in YARN-6586,
other than the RM HA support (TODO in YARN-8449) and Documentation (just filed
YARN-8582 for this). I've put the bulk of the changes here
(YARN-8448.001.patch), and the MapReduce changes in MAPREDUCE-4669.
Some notes on the patch:
- Updated BouncyCastle library to a newer version and had to also change the
artifact from {{bcprov-jdk16}} to {{bcprov-jdk15on}}. I know that sounds
backwards, but jdk15on is actually newer and the one we should be using (see
http://bouncy-castle.1462172.n4.nabble.com/Bouncycaslte-bcprov-jdk15-vs-bcprov-jdk16-td4656252.html).
- The {{yarn.resourcemanager.application-https.policy}} property controls how
the RM should handle HTTPS when talking to AMs. It can be {{OFF}},
{{OPTIONAL}} (default), or {{REQUIRED}}. {{OFF}} makes it behave like today,
where it does nothing special. {{OPTIONAL}} makes it generate and provide the
keystore and truststore to the AM when it sees an HTTPS tracking URL, but HTTP
is also still allowed. And {{REQUIRED}} is like {{OPTIONAL}}, but it won't
follow HTTP tracking URLs.
- A lot of the code around the container executors is in providing/copying/etc
the keystore and truststore files. I've largely based this on the existing way
we handle the credentials (delegation tokens) file.
- When provided a keystore file, the AM will get env vars
{{KEYSTORE_FILE_LOCATION}} and {{KEYSTORE_PASSWORD}}; similarly,
{{TRUSTSTORE_FILE_LOCATION}} and {{TRUSTSTORE_PASSWORD}} for the truststore
file.
- Due to the (ugly) way we parse arguments in the LCE, I had to add an argument
that's either {{--http}} or {{--https}} to indicate if we'll be providing it
the keystore and truststore files. Otherwise, there isn't a good way to have
optional arguments.
- In order to keep things simple, I piggybacked passing the keystore and
truststore files and passwords via secrets in the Credentials, which is already
securely passed from the RM to the NM.
- {{ProxyCAManager}} is in charge of creating the certificates, keystores, and
truststores.
- When writing the unit tests, I found a number of tests that were about 80%
complete in what they were testing, which I completed in addition to adding
tests for my changes.
-- I also tried to simplify some things (e.g. {{TestDockerContainerRuntime}}
has ~30 tests that all duplicate the code for checking the arguments, and
because I changed the number of arguments, they all failed - instead of
updating them all, I created a helper method)
- I'm not sure what's up with {{test-container-executor}}, but unless my
environment was messed up, it doesn't work when run as {{root}}; maybe people
typically run it as a normal user? The test talks about running as {{root}} as
an option, and even has a few tests that only run when running as {{root}}. I
spent some time fixing this - it now runs in all 4 user configurations
described in the existing comments.
- I've tested in a real cluster with the DefaultContainerExecutor and
LinuxContainerExecutor using all combinations of
{{yarn.resourcemanager.application-https.policy}},
{{yarn.app.mapreduce.am.webapp.https.enabled}}, and
{{yarn.app.mapreduce.am.webapp.https.client.auth}} (see MAPREDUCE-4669), and
everything behaved correctly. I haven't tested out the
DockerContainerExecutor.
-- If you want to try this out yourself in a cluster, I'd recommend also
applying the MAPREDUCE-4669 patch so you have an AM that supports the changes.
You can then use {{openssl s_client -connect <am-host>:<am-web-port>}} to get
SSL details. You can also try {{curl}}.
> AM HTTPS Support
> ----------------
>
> Key: YARN-8448
> URL: https://issues.apache.org/jira/browse/YARN-8448
> Project: Hadoop YARN
> Issue Type: Sub-task
> Reporter: Robert Kanter
> Assignee: Robert Kanter
> Priority: Major
> Attachments: YARN-8448.001.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
