GitHub user ash211 opened a pull request: https://github.com/apache/spark/pull/1107
[WIP] SPARK-2157 Ability to write tight firewall rules for Spark https://issues.apache.org/jira/browse/SPARK-2157 This pull request adds the ability to specify every port opened by Spark that I could find to configure. Specifically, it adds the following four configuration options: - `spark.fileserver.port` - `spark.broadcast.port` - `spark.replClassServer.port` - `spark.blockManager.port` Of those options, most will attempt to fall back to the n+1 incremental port if the port is currently in use. This incremental fallback logic is done for : - `spark.broadcast.port` - `spark.blockManager.port` - `spark.replClassServer.port` But not for `spark.fileserver.port` Things still thinking about for this PR: - I'm still observing instances of `CoarseGrainedExecutorBackend` listening on ephemeral ports - In earlier documentation `spark.driver.port` is described as being the way to configure :7077 on Spark Master, and I'm not sure this is the case. If so the same config item is used for multiple things, as it's also the port on which the applications starts listening on Akka. Many thanks to @epahomov whose work this is based off of. You can merge this pull request into a Git repository by running: $ git pull https://github.com/ash211/spark SPARK-2157 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/1107.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1107 ---- commit 1d5b7f70900039d4a4fa0ec52129547d3dc35374 Author: Andrew Ash <and...@andrewash.com> Date: 2014-06-17T05:09:59Z Make port in HttpServer configurable commit a8898f6a506f447e62177bff933f681a03f13150 Author: Andrew Ash <and...@andrewash.com> Date: 2014-06-17T05:31:10Z SPARK-1174 Add port configuration for HttpFileServer Uses spark.fileserver.port commit afba558655011c45ef4fe7fdb0f8104b2a212f72 Author: Andrew Ash <and...@andrewash.com> Date: 2014-06-17T05:31:52Z SPARK-1176 Add port configuration for HttpBroadcast Uses spark.broadcast.port commit 945a52661ec0386abb1d370364081177588d7615 Author: Andrew Ash <and...@andrewash.com> Date: 2014-06-17T06:00:19Z Add a configuration option for spark-shell's class server spark.replClassServer.port commit 5edb3956af6284f21899066c22e6b739b7dc0f54 Author: Andrew Ash <and...@andrewash.com> Date: 2014-06-17T06:40:32Z Make Spark's block manager port configurable spark.blockManager.port commit e743af886d915f8a1e1102d759b7622ac6664b13 Author: Andrew Ash <and...@andrewash.com> Date: 2014-06-17T08:10:21Z Fix ConnectionManager to retry with increment Fails when running master+worker+executor+shell on the same machine. I think the issue is that both the shell and the executor attempt to start a ConnectionManager, which causes port conflicts. Solution is to attempt and increment on BindExceptions commit 0013e2d053aafff453fcba1691cf68b79e06e4f7 Author: Andrew Ash <and...@andrewash.com> Date: 2014-06-17T16:45:59Z Add fallover increment logic for HttpServer commit eac759e0577327efeaf70a42dc399fa22bf302b4 Author: Andrew Ash <and...@andrewash.com> Date: 2014-06-17T17:08:49Z Fix up HttpServer port increments commit d102749d2be5930121d0ad7c9fbde43da239e780 Author: Andrew Ash <and...@andrewash.com> Date: 2014-06-17T17:43:33Z Document new port configuration options - spark.fileserver.port - spark.broadcast.port - spark.replClassServer.port - spark.blockManager.port ---- --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---