GitHub user vanzin opened a pull request:
https://github.com/apache/spark/pull/5377
[SPARK-6229] Add SASL encryption to network library.
There are two main parts of this change:
- Extending the bootstrap mechanism in the network library to add a
server-side
bootstrap (which works a little bit differently than the client-side
bootstrap), and
to allow the bootstraps to modify the underlying channel.
- Use SASL to encrypt data going through the RPC channel.
The second item requires some non-optimal code to be able to work around the
fact that the outbound path in netty is not thread-safe, and ordering is
very important
when encryption is in the picture.
A lot of the changes outside the network/common library are just to adjust
to the
changed API for initializing the RPC server.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/vanzin/spark SPARK-6229
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/5377.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #5377
----
commit fbe6ccb4bbf5cd8dd23b5bcbe5d013d7efb40f88
Author: Marcelo Vanzin <[email protected]>
Date: 2015-03-24T21:48:01Z
Add TransportServerBootstrap, make SASL code use it.
This commit changes the way a transport server is configured to use SASL. It
tried to be more in line with how you set up a client, by registering a
"bootstrap" that runs before the server is actually available. Due to how
servers internally work, the server bootstrap works differently from the
client
one, but a similar external interface is provided to clients of the library.
The SASL backend was reorganized to use this new mechanism. Later, the
bootstrap will be modified so that it's possible to also use SASL for
encryption, which requires inserting a new channel handler into the pipeline
- the main reason why this change was necessary in the first place.
commit 351a86f882ca082da963d645e5a66065fe7c21f2
Author: Marcelo Vanzin <[email protected]>
Date: 2015-03-25T00:39:26Z
Add SASL encryption to network library.
This change extends the "bootstrap" classes to expose the underlying netty
channel to the bootstrap implementations. This allows the SASL bootstraps
to register channel handlers that can then perform data encryption, when
that is enabled by the user.
Encryption is performed in a manner that is transparent to the other
existing
channel handlers, so perhaps it's not the most efficient implementation,
especially since it requires having 2 frame decoders in the pipeline. But
it's
the least intrusive way to do it.
commit 39539a7d4f889102ec9e12c61a7e2984011af594
Author: Marcelo Vanzin <[email protected]>
Date: 2015-03-26T20:18:04Z
Add config option to enable SASL encryption.
commit b923cae7b9e1e5616b5be0759ec810ee9d9fe5a4
Author: Marcelo Vanzin <[email protected]>
Date: 2015-03-27T18:50:31Z
Make SASL encryption handler thread-safe, handle FileRegion messages.
commit b00999a877c0a8562db7f7bcd2d8d79be6196924
Author: Marcelo Vanzin <[email protected]>
Date: 2015-04-03T20:25:10Z
Consolidate ByteArrayWritableChannel, fix SASL code to match master changes.
commit dad42fcff225fded1763ea2b7602b1fe6743b157
Author: Marcelo Vanzin <[email protected]>
Date: 2015-04-03T22:08:33Z
Make encryption thread-safe, less memory-intensive.
Two changes here:
First, wrap the code that does encryption in a FileRegion. Without resorting
to using a ThreadPerChannelEventLoopGroup, that's the only thread-safe way
to
do encryption in netty, apparently, since message ordering is extremely
important here. This requires some funny accounting in the SASL FileRegion
implementation, because we can't know before hand how much data we'll have
to write.
Second, instead of encrypting the whole original message, do it in smaller
chunks, to avoid using too much memory on the sender side. So it should be
possible to transfer a large file block without having to load the whole
block into memory at once.
commit e98bc5552591c5e33ae4e8b050128bd9b73f7440
Author: Marcelo Vanzin <[email protected]>
Date: 2015-04-06T22:50:56Z
Add option to only allow encrypted connections to the server.
commit 85843230d947f85939363388f5ceaa6fbc9e4896
Author: Marcelo Vanzin <[email protected]>
Date: 2015-04-06T23:00:04Z
Fix a comment.
commit cf2a605d02afd4f462270a0d808ff82c75ec04fd
Author: Marcelo Vanzin <[email protected]>
Date: 2015-04-06T23:02:40Z
Clean up some code.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
