GitHub user andrewor14 opened a pull request:
https://github.com/apache/spark/pull/441
[Fix #204] Eliminate delay between binding and log checking
In the existing history server, the user needs to wait
`spark.history.updateInterval` seconds before his/her logs show up on the UI.
This is because the following events happen in this order: (1) The
background thread that checks for logs starts, but realizes the server has not
yet bound and so waits for N seconds, (2) server binds, (3) N seconds later the
background thread finds that the server has finally bound to a port, and so
finally checks for application logs.
This PR forces the log checking thread to start immediately after binding.
It also documents two relevant environment variables that are currently missing.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/andrewor14/spark history-server-fix
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/441.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #441
----
commit e8d1fbcc4af9ff7a2d1c1352553f4d2d18a66ff6
Author: Andrew Or <[email protected]>
Date: 2014-04-18T03:53:43Z
Eliminate delay between binding and checking for logs
commit b2eb46e754d51a40f9d9ddfd4ab8598876f2bb3e
Author: Andrew Or <[email protected]>
Date: 2014-04-18T03:54:17Z
Document SPARK_PUBLIC_DNS and SPARK_HISTORY_OPTS for the history server
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
