Github user mateiz commented on a diff in the pull request:
https://github.com/apache/spark/pull/880#discussion_r13159409
--- Diff: docs/configuration.md ---
@@ -774,104 +816,16 @@ The following variables can be set in `spark-env.sh`:
</tr>
</table>
-In addition to the above, there are also options for setting up the Spark
[standalone cluster
-scripts](spark-standalone.html#cluster-launch-scripts), such as number of
cores to use on each
-machine and maximum memory.
+In addition to the above, there are also options for setting up the Spark
+[standalone cluster
scripts](spark-standalone.html#cluster-launch-scripts), such as number of cores
+to use on each machine and maximum memory.
-Since `spark-env.sh` is a shell script, some of these can be set
programmatically -- for example,
-you might compute `SPARK_LOCAL_IP` by looking up the IP of a specific
network interface.
+Since `spark-env.sh` is a shell script, some of these can be set
programmatically -- for example, you might
+compute `SPARK_LOCAL_IP` by looking up the IP of a specific network
interface.
# Configuring Logging
Spark uses [log4j](http://logging.apache.org/log4j/) for logging. You can
configure it by adding a
`log4j.properties` file in the `conf` directory. One way to start is to
copy the existing
`log4j.properties.template` located there.
-
-# Configuring Ports for Network Security
-
-Spark makes heavy use of the network, and some environments have strict
requirements for using tight
-firewall settings. Below are the primary ports that Spark uses for its
communication and how to
-configure those ports.
-
-<table class="table">
- <tr>
- <th>From</th><th>To</th><th>Default
Port</th><th>Purpose</th><th>Configuration
- Setting</th><th>Notes</th>
- </tr>
- <!-- Web UIs -->
- <tr>
- <td>Browser</td>
- <td>Standalone Cluster Master</td>
- <td>8080</td>
- <td>Web UI</td>
- <td><code>master.ui.port</code></td>
- <td>Jetty-based</td>
- </tr>
- <tr>
- <td>Browser</td>
- <td>Worker</td>
- <td>8081</td>
- <td>Web UI</td>
- <td><code>worker.ui.port</code></td>
- <td>Jetty-based</td>
- </tr>
- <tr>
- <td>Browser</td>
- <td>Driver</td>
- <td>4040</td>
- <td>Web UI</td>
- <td><code>spark.ui.port</code></td>
- <td>Jetty-based</td>
- </tr>
- <tr>
- <td>Browser</td>
- <td>History Server</td>
- <td>18080</td>
- <td>Web UI</td>
- <td><code>spark.history.ui.port</code></td>
- <td>Jetty-based</td>
- </tr>
-
- <!-- Cluster interactions -->
- <tr>
- <td>Application</td>
- <td>Standalone Cluster Master</td>
- <td>7077</td>
- <td>Submit job to cluster</td>
- <td><code>spark.driver.port</code></td>
- <td>Akka-based. Set to "0" to choose a port randomly</td>
- </tr>
- <tr>
- <td>Worker</td>
- <td>Standalone Cluster Master</td>
- <td>7077</td>
- <td>Join cluster</td>
- <td><code>spark.driver.port</code></td>
- <td>Akka-based. Set to "0" to choose a port randomly</td>
- </tr>
- <tr>
- <td>Application</td>
- <td>Worker</td>
- <td>(random)</td>
- <td>Join cluster</td>
- <td><code>SPARK_WORKER_PORT</code> (standalone cluster)</td>
- <td>Akka-based</td>
- </tr>
-
- <!-- Other misc stuff -->
- <tr>
- <td>Driver and other Workers</td>
- <td>Worker</td>
- <td>(random)</td>
- <td>
- <ul>
- <li>File server for file and jars</li>
- <li>Http Broadcast</li>
- <li>Class file server (Spark Shell only)</li>
- </ul>
- </td>
- <td>None</td>
- <td>Jetty-based. Each of these services starts on a random port that
cannot be configured</td>
- </tr>
-
</table>
--- End diff --
You need to delete this `</table>` too.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
