GitHub user vanzin opened a pull request:
https://github.com/apache/spark/pull/9947
[SPARK-11140][CORE] Transfer files using network lib when using NettyRpcEnv
- 1.6.version.
This patch is the same code as in SPARK-11140 in master, but with some
added code to still use the HTTP file server by default in NettyRpcEnv. This is
mostly to avoid conflicts when backporting patches to 1.6.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/vanzin/spark SPARK-11140-branch-1.6
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/9947.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #9947
----
commit 1633793513535e5f38b17565cfc6d9c5849937ff
Author: Marcelo Vanzin <[email protected]>
Date: 2015-11-23T21:54:19Z
[SPARK-11140][CORE] Transfer files using network lib when using NettyRpcEnv.
This change abstracts the code that serves jars / files to executors so that
each RpcEnv can have its own implementation; the akka version uses the
existing
HTTP-based file serving mechanism, while the netty versions uses the new
stream support added to the network lib, which makes file transfers benefit
from the easier security configuration of the network library, and should
also
reduce overhead overall.
The change includes a small fix to TransportChannelHandler so that it
propagates
user events to downstream handlers.
Author: Marcelo Vanzin <[email protected]>
Closes #9530 from vanzin/SPARK-11140.
(cherry picked from commit c2467dadae8ce44010a912ee91c429310f8add65)
commit 08eb2020c6e7e4049dd70310dfff2d38c9c1df5f
Author: Marcelo Vanzin <[email protected]>
Date: 2015-11-24T21:58:18Z
Use HTTP-file server by default in Netty RPC.
Just fork the code in AkkaRpcEnv; the netty-based one can be enabled
via a conf option (which doesn't exist in master). Note that without
SPARK-11956 the netty file transfer is a bit broken.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
