GitHub user ash211 opened a pull request:
https://github.com/apache/spark/pull/1107
[WIP] SPARK-2157 Ability to write tight firewall rules for Spark
https://issues.apache.org/jira/browse/SPARK-2157
This pull request adds the ability to specify every port opened by Spark
that I could find to configure. Specifically, it adds the following four
configuration options:
- `spark.fileserver.port`
- `spark.broadcast.port`
- `spark.replClassServer.port`
- `spark.blockManager.port`
Of those options, most will attempt to fall back to the n+1 incremental
port if the port is currently in use. This incremental fallback logic is done
for :
- `spark.broadcast.port`
- `spark.blockManager.port`
- `spark.replClassServer.port`
But not for `spark.fileserver.port`
Things still thinking about for this PR:
- I'm still observing instances of `CoarseGrainedExecutorBackend` listening
on ephemeral ports
- In earlier documentation `spark.driver.port` is described as being the
way to configure :7077 on Spark Master, and I'm not sure this is the case. If
so the same config item is used for multiple things, as it's also the port on
which the applications starts listening on Akka.
Many thanks to @epahomov whose work this is based off of.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/ash211/spark SPARK-2157
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/1107.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1107
----
commit 1d5b7f70900039d4a4fa0ec52129547d3dc35374
Author: Andrew Ash <[email protected]>
Date: 2014-06-17T05:09:59Z
Make port in HttpServer configurable
commit a8898f6a506f447e62177bff933f681a03f13150
Author: Andrew Ash <[email protected]>
Date: 2014-06-17T05:31:10Z
SPARK-1174 Add port configuration for HttpFileServer
Uses spark.fileserver.port
commit afba558655011c45ef4fe7fdb0f8104b2a212f72
Author: Andrew Ash <[email protected]>
Date: 2014-06-17T05:31:52Z
SPARK-1176 Add port configuration for HttpBroadcast
Uses spark.broadcast.port
commit 945a52661ec0386abb1d370364081177588d7615
Author: Andrew Ash <[email protected]>
Date: 2014-06-17T06:00:19Z
Add a configuration option for spark-shell's class server
spark.replClassServer.port
commit 5edb3956af6284f21899066c22e6b739b7dc0f54
Author: Andrew Ash <[email protected]>
Date: 2014-06-17T06:40:32Z
Make Spark's block manager port configurable
spark.blockManager.port
commit e743af886d915f8a1e1102d759b7622ac6664b13
Author: Andrew Ash <[email protected]>
Date: 2014-06-17T08:10:21Z
Fix ConnectionManager to retry with increment
Fails when running master+worker+executor+shell on the same machine. I
think
the issue is that both the shell and the executor attempt to start a
ConnectionManager, which causes port conflicts. Solution is to attempt and
increment on BindExceptions
commit 0013e2d053aafff453fcba1691cf68b79e06e4f7
Author: Andrew Ash <[email protected]>
Date: 2014-06-17T16:45:59Z
Add fallover increment logic for HttpServer
commit eac759e0577327efeaf70a42dc399fa22bf302b4
Author: Andrew Ash <[email protected]>
Date: 2014-06-17T17:08:49Z
Fix up HttpServer port increments
commit d102749d2be5930121d0ad7c9fbde43da239e780
Author: Andrew Ash <[email protected]>
Date: 2014-06-17T17:43:33Z
Document new port configuration options
- spark.fileserver.port
- spark.broadcast.port
- spark.replClassServer.port
- spark.blockManager.port
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
