GitHub user vanzin opened a pull request:
https://github.com/apache/spark/pull/9210
[SPARK-10997] [core] Add "client mode" to netty rpc env.
"Client mode" means the RPC env will not listen for incoming connections.
This allows certain processes in the Spark stack (such as Executors or
tha YARN client-mode AM) to act as pure clients when using the netty-based
RPC backend, reducing the number of sockets needed by the app and also the
number of open ports.
Client connections are also preferred when endpoints that actually have
a listening socket are involved; so, for example, if a Worker connects
to a Master and the Master needs to send a message to a Worker endpoint,
that client connection will be used, even though the Worker is also
listening for incoming connections.
With this change, the workaround for SPARK-10987 isn't necessary anymore,
and
is removed. The AM connects to the driver in "client mode", and that
connection
is used for all driver <-> AM communication, and so the AM is properly
notified
when the connection goes down.
This change also removes the workaround for SPARK-10987.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/vanzin/spark SPARK-10997
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/9210.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #9210
----
commit 48f3c55543878bf6ccbb4048f190eb5f2b3adc59
Author: Marcelo Vanzin <[email protected]>
Date: 2015-10-13T01:38:47Z
[SPARK-10997] [core] Add "client mode" to netty rpc env.
"Client mode" means the RPC env will not listen for incoming connections.
This allows certain processes in the Spark stack (such as Executors or
tha YARN client-mode AM) to act as pure clients when using the netty-based
RPC backend, reducing the number of sockets needed by the app and also the
number of open ports.
Client connections are also preferred when endpoints that actually have
a listening socket are involved; so, for example, if a Worker connects
to a Master and the Master needs to send a message to a Worker endpoint,
that client connection will be used, even though the Worker is also
listening for incoming connections.
With this change, the workaround for SPARK-10987 isn't necessary anymore,
and
is removed. The AM connects to the driver in "client mode", and that
connection
is used for all driver <-> AM communication, and so the AM is properly
notified
when the connection goes down.
This change also removes the workaround for SPARK-10987.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
