GitHub user andrewor14 opened a pull request:
https://github.com/apache/spark/pull/1777
[SPARK-2157] Enable tight firewall rules for Spark
The goal of this PR is to allow users of Spark to write tight firewall
rules for their clusters. This is currently not possible because Spark uses
random ports in many places, notably the communication between executors and
drivers. The changes in this PR are based on top of @ash211's changes in #1107.
The list covered here may or may not be the complete set of port needed for
Spark to operate perfectly. However, as of the latest commit there are no known
sources of random ports (except in tests). I have not documented a few of the
more obscure configs.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/andrewor14/spark configure-ports
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/1777.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1777
----
commit 1c0981a3f93c3bbb27425d76342c98c8c7d469cf
Author: Andrew Ash <[email protected]>
Date: 2014-06-17T05:09:59Z
Make port in HttpServer configurable
commit 49ee29b49e1275b48d18aef5182dba2937c11358
Author: Andrew Ash <[email protected]>
Date: 2014-06-17T05:31:10Z
SPARK-1174 Add port configuration for HttpFileServer
Uses spark.fileserver.port
commit f34115d59b83163d9542be09eb0c89a87ea89309
Author: Andrew Ash <[email protected]>
Date: 2014-06-17T05:31:52Z
SPARK-1176 Add port configuration for HttpBroadcast
Uses spark.broadcast.port
commit 17c79bbd66708d24c093be5f43e60c61f504d19d
Author: Andrew Ash <[email protected]>
Date: 2014-06-17T06:00:19Z
Add a configuration option for spark-shell's class server
spark.replClassServer.port
commit b80d2fd8e9b27a4d49561d31f100ffbb75393685
Author: Andrew Ash <[email protected]>
Date: 2014-06-17T06:40:32Z
Make Spark's block manager port configurable
spark.blockManager.port
commit c5a05684ace9332077dbf63848d08f39a8b91628
Author: Andrew Ash <[email protected]>
Date: 2014-06-17T08:10:21Z
Fix ConnectionManager to retry with increment
Fails when running master+worker+executor+shell on the same machine. I
think
the issue is that both the shell and the executor attempt to start a
ConnectionManager, which causes port conflicts. Solution is to attempt and
increment on BindExceptions
commit cad16dacb1b7dbac1122b38c2b02fe35f1303a59
Author: Andrew Ash <[email protected]>
Date: 2014-06-17T16:45:59Z
Add fallover increment logic for HttpServer
commit 066dc7ac936cfbf268e6ca7adfa1388f5c4049d6
Author: Andrew Ash <[email protected]>
Date: 2014-06-17T17:08:49Z
Fix up HttpServer port increments
commit 5d84e0e9285aec53aa9c57d64313c0e513e41d30
Author: Andrew Ash <[email protected]>
Date: 2014-06-17T17:43:33Z
Document new port configuration options
- spark.fileserver.port
- spark.broadcast.port
- spark.replClassServer.port
- spark.blockManager.port
commit 9e4ad9628f7ff0f96a3881a1a5aaedcb8be6b80d
Author: Andrew Ash <[email protected]>
Date: 2014-06-17T18:14:08Z
Reformat for style checker
commit 24a4c327c7441e6af6b82dbddacd71c57384dc04
Author: Andrew Ash <[email protected]>
Date: 2014-06-30T00:25:44Z
Remove type on val to match surrounding style
commit 0347aef2b686d1bcc1b8f5c230ba8ff99cbd0691
Author: Andrew Ash <[email protected]>
Date: 2014-06-30T05:26:48Z
Unify port fallback logic to a single place
commit 7c5bdc44df32fb550f375de3518b628fbb360d20
Author: Andrew Ash <[email protected]>
Date: 2014-06-30T05:34:47Z
Fix style issue
commit 038a579a26ffcfc1c5540f28176f236779eef12a
Author: Andrew Ash <[email protected]>
Date: 2014-06-30T07:02:17Z
Trust the server start function to report the port the service started on
commit ec676f4f74b7a8402047fb849b9dca7172cd32f5
Author: Andrew Or <[email protected]>
Date: 2014-08-04T21:46:50Z
Merge branch 'SPARK-2157' of github.com:ash211/spark into configure-ports
commit 73fbe892794a6f7e4a051401f356c89f4aa7f81f
Author: Andrew Or <[email protected]>
Date: 2014-08-04T22:39:01Z
Move start service logic to Utils
commit 6b550b0681ae8c0394685f6e929c4a14a48d10ec
Author: Andrew Or <[email protected]>
Date: 2014-08-04T23:56:17Z
Assorted fixes
commit ba322807d2e5ed1ce69dae449238a1df16a74ae9
Author: Andrew Or <[email protected]>
Date: 2014-08-05T00:00:31Z
Minor fixes
commit 1d7e40813e6ae98ee5cffb3e9e61807f3a01e941
Author: Andrew Or <[email protected]>
Date: 2014-08-05T00:40:27Z
Treat 0 ports specially + return correct ConnectionManager port
commit 470f38cf3c54941fbbcc358a358cc8a1fe2d6edd
Author: Andrew Or <[email protected]>
Date: 2014-08-05T00:43:24Z
Special case non-"Address already in use" exceptions
commit e111d080b4a7c0103c30b3a6e29c058d0ac4c3d0
Author: Andrew Or <[email protected]>
Date: 2014-08-05T01:46:11Z
Add names for UI services
commit 3f8e51bbb82669b43d7d52ece09ac957b35e994e
Author: Andrew Or <[email protected]>
Date: 2014-08-05T01:46:29Z
Correct erroneous docs...
commit 4d9e6f348cc408064173a91ecf9b509804eadf01
Author: Andrew Or <[email protected]>
Date: 2014-08-05T02:32:31Z
Fix super subtle bug
We were passing the Master/Worker UI names as base paths for Jetty
to use, which is obviously wrong in hindsight.
commit 8d836e6725120edaa36f4dd84ef9b4e88fc91f83
Author: Andrew Or <[email protected]>
Date: 2014-08-05T04:26:51Z
Also document SPARK_{MASTER/WORKER}_WEBUI_PORT
The existing documentation uses {master/worker}.ui.port, which doesn't
actually work because SparkConf doesn't pick up system properties that
are not prefixed by spark (SPARK-2857). We should at least document
a working alternative.
commit 6016e774e4034e0179f51042638bfd48b21b62e0
Author: Andrew Or <[email protected]>
Date: 2014-08-05T04:59:14Z
Add spark.executor.port
commit 986835897b42c4b7ab0d21878d4d825aa530f59c
Author: Andrew Or <[email protected]>
Date: 2014-08-05T05:32:09Z
Add a few miscellaneous ports
commit 151327a427d7513462739d050f7eac876edad741
Author: Andrew Or <[email protected]>
Date: 2014-08-05T05:32:24Z
Merge branch 'master' of github.com:apache/spark into configure-ports
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
