GitHub user StephanEwen opened a pull request:
https://github.com/apache/flink/pull/6326
Mutual authentication for internal communication
** This is based on #6324 - hence the first commit in this PR should be
discarded from review**
## What is the purpose of the change
Splits the SSL configuration into **internal communication** *(RPC, data
transport, blob server)* and **external/REST** communication. Also activates
mutual authentication for all internal communication.
This continues the security features of Flink as outlined in
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/Discuss-FLIP-26-SSL-Mutual-Authentication-td22188.html
Most of these changes are straightforward, the most important thing for
reviewers to in my opinion would be to check whether the configuration keys
make sense:
- One can configure SSL independently for internal and external/REST
communication
- This is due to feedback from users that internal communication needs to
be protected more and by Flink itself, while external communication is
frequently protected by REST proxies (often as side car processes to the
JobManager / Dispatcher)
- All keytore and password settings now exist additionally in the
`security.ssl.internal.*` and `security.ssl.rest.*` key namespace. The
`security.ssl.*` config keys still exist and used if the more specific key is
not set. This is meant both for backwards compatibility, and to make it easy to
use a uniform config across internal/external communication.
## Brief change log
- Introduces new config option families: `security.ssl.internal.*` and
`security.ssl.rest.*`
- Adds code to fall back to the `security.ssl.*` keys if no internal or
rest specific options are set
- Refactors all instantiation of `SSLEngine` and `SSL(Server)Socket` to
go through factories. That way, the different endpoint instantiations do not
need to apply configurations themselves.
- Activates mutual auth for akka/rpc via akka config, plus adds a test
- Activates mutual auth in the SSL Socket/Engine factories (netty / blob)
and adds a test
## Verifying this change
- There are additional unit tests checking that clients with untrusted
certificates cannot connect.
- Verifying end-to-end works by building the code, enabling internal SSL
in the flink-conf.yaml, starting a standalone cluster, checking the logs and
akka urls for SSL entries
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): **no**
- The public API, i.e., is any changed class annotated with
`@Public(Evolving)`: **no**
- The serializers: **no**
- The runtime per-record code paths (performance sensitive): **no**
- Anything that affects deployment or recovery: JobManager (and its
components), Checkpointing, Yarn/Mesos, ZooKeeper: **no**
- The S3 file system connector: **no**
## Documentation
- Does this pull request introduce a new feature? **yes**
- If yes, how is the feature documented? Docs coming in a separate PR
once we have agreement on the config keys
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/StephanEwen/incubator-flink client_auth
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/6326.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #6326
----
commit 37abf46f6030b6404707958e5a3a3fae0051dbea
Author: Stephan Ewen <sewen@...>
Date: 2018-07-13T07:31:18Z
[FLINK-9424] [security] Set default cipher suite to a more compatible
cipher suite.
The upgraded ciphers are not yet supported on all platforms and JDK
versions, making
the getting-started process rough. Instead, we document our recommendation
to set these
values in the configuration.
This reverts "[FLINK-9310] [security] Update standard cipher suites for
secure mode"
commit 0d33c8ab2be6502a56d8ea97a72fda5ec8b865c0
Author: Stephan Ewen <sewen@...>
Date: 2018-07-12T09:28:57Z
[FLINK-9313] [security] (part 1) Instantiate all SSLSocket and
SSLServerSocket through factories.
This removes hostname verification from SSL client sockets.
With client authentication, this is no longer needed and it is not
compatible with
various container environments.
commit 80cd8bec111bb91943bd691adf80275c79b57ca0
Author: Stephan Ewen <sewen@...>
Date: 2018-05-07T17:44:33Z
[FLINK-9313] [security] (part 3) Activate mutual authentication for RPC/akka
commit 97425b2962861922ac3d7e64fb57400de787966d
Author: Stephan Ewen <sewen@...>
Date: 2018-07-12T15:20:30Z
[FLINK-9313] [security] (part 2) Split SSL configuration into internal
(rpc, data transport, blob server) and external (REST)
This also uses SSLEngineFactory for all SSLEngine creations.
commit d64dd6c65deb013c13dd261a2bb7bb96f20cdcef
Author: Stephan Ewen <sewen@...>
Date: 2018-07-12T19:18:46Z
[FLINK-9314] [security] (part 4) Add mutual authentication for internal
Netty and Blob Server connections
----
---
