Repository: spark Updated Branches: refs/heads/branch-1.0 898fc3480 -> 54c96c234
[Fix #204] Eliminate delay between binding and log checking **Bug**: In the existing history server, there is a `spark.history.updateInterval` seconds delay before application logs show up on the UI. **Cause**: This is because the following events happen in this order: (1) The background thread that checks for logs starts, but realizes the server has not yet bound and so waits for N seconds, (2) server binds, (3) N seconds later the background thread finds that the server has finally bound to a port, and so finally checks for application logs. **Fix**: This PR forces the log checking thread to start immediately after binding. It also documents two relevant environment variables that are currently missing. Author: Andrew Or <[email protected]> Closes #441 from andrewor14/history-server-fix and squashes the following commits: b2eb46e [Andrew Or] Document SPARK_PUBLIC_DNS and SPARK_HISTORY_OPTS for the history server e8d1fbc [Andrew Or] Eliminate delay between binding and checking for logs (cherry picked from commit 745e496c59cfece2fcd6120ecc366dcab07b293a) Signed-off-by: Patrick Wendell <[email protected]> Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/54c96c23 Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/54c96c23 Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/54c96c23 Branch: refs/heads/branch-1.0 Commit: 54c96c2343dff6e6defe4e2ad576ab8a82d45140 Parents: 898fc34 Author: Andrew Or <[email protected]> Authored: Tue Apr 22 14:27:49 2014 -0700 Committer: Patrick Wendell <[email protected]> Committed: Tue Apr 22 14:27:59 2014 -0700 ---------------------------------------------------------------------- .../spark/deploy/history/HistoryServer.scala | 5 +++++ docs/monitoring.md | 19 +++++++++++++++---- 2 files changed, 20 insertions(+), 4 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/spark/blob/54c96c23/core/src/main/scala/org/apache/spark/deploy/history/HistoryServer.scala ---------------------------------------------------------------------- diff --git a/core/src/main/scala/org/apache/spark/deploy/history/HistoryServer.scala b/core/src/main/scala/org/apache/spark/deploy/history/HistoryServer.scala index cf64700..b8f5623 100644 --- a/core/src/main/scala/org/apache/spark/deploy/history/HistoryServer.scala +++ b/core/src/main/scala/org/apache/spark/deploy/history/HistoryServer.scala @@ -98,6 +98,11 @@ class HistoryServer( def initialize() { attachPage(new HistoryPage(this)) attachHandler(createStaticHandler(STATIC_RESOURCE_DIR, "/static")) + } + + /** Bind to the HTTP server behind this web interface. */ + override def bind() { + super.bind() logCheckingThread.start() } http://git-wip-us.apache.org/repos/asf/spark/blob/54c96c23/docs/monitoring.md ---------------------------------------------------------------------- diff --git a/docs/monitoring.md b/docs/monitoring.md index 4c91c3a..144be3d 100644 --- a/docs/monitoring.md +++ b/docs/monitoring.md @@ -39,22 +39,33 @@ You can start a the history server by executing: The base logging directory must be supplied, and should contain sub-directories that each represents an application's event logs. This creates a web interface at -`http://<server-url>:18080` by default. The history server depends on the following variables: +`http://<server-url>:18080` by default. The history server can be configured as follows: <table class="table"> <tr><th style="width:21%">Environment Variable</th><th>Meaning</th></tr> <tr> <td><code>SPARK_DAEMON_MEMORY</code></td> - <td>Memory to allocate to the history server. (default: 512m).</td> + <td>Memory to allocate to the history server (default: 512m).</td> </tr> <tr> <td><code>SPARK_DAEMON_JAVA_OPTS</code></td> <td>JVM options for the history server (default: none).</td> </tr> + <tr> + <td><code>SPARK_PUBLIC_DNS</code></td> + <td> + The public address for the history server. If this is not set, links to application history + may use the internal address of the server, resulting in broken links (default: none). + </td> + </tr> + <tr> + <td><code>SPARK_HISTORY_OPTS</code></td> + <td> + <code>spark.history.*</code> configuration options for the history server (default: none). + </td> + </tr> </table> -Further, the history server can be configured as follows: - <table class="table"> <tr><th>Property Name</th><th>Default</th><th>Meaning</th></tr> <tr>
