GitHub user squito opened a pull request:
https://github.com/apache/spark/pull/11118
[WebUI][SPARK-7889] HistoryServer updates UI for incomplete apps
When the HistoryServer is showing an incomplete app, it needs to check if
there is a newer version of the app available. It does this by checking if a
version of the app has been loaded with a larger *filesize*. If so, it
detaches the current UI, attaches the new one, and redirects back to the same
URL to show the new UI.
https://issues.apache.org/jira/browse/SPARK-7889
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/squito/spark SPARK-7889-alternate
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/11118.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #11118
----
commit ae0b8b223698546f2e0bf89da52f3c8883586870
Author: Steve Loughran <[email protected]>
Date: 2015-06-22T12:54:25Z
SPARK-7889 cache with initial unit tests
commit 72945e22df00c4ca016aeb60a0697d33ec08dabe
Author: Steve Loughran <[email protected]>
Date: 2015-06-22T13:46:50Z
SPARK-7889 cache tests of refresh interval working.
commit 6764f499177d0ce0c422cb5206848ef0270bd0e3
Author: Steve Loughran <[email protected]>
Date: 2015-06-22T14:36:01Z
SPARK-7889 switch to explicit scan of SparkUI attempts to determine
completion state; use mock spark UI
commit bd47afb6f8b37426df1434a1bc2a43abdd81d65e
Author: Steve Loughran <[email protected]>
Date: 2015-06-22T14:45:00Z
SPARK-7889 use spark.history.cache.interval as config option; set default =
60; document
commit aaadf645d7ca82ae111cc3a61a0bf2d75cdb1d99
Author: Steve Loughran <[email protected]>
Date: 2015-11-23T14:39:51Z
SPARK-7889 resync with trunk & slightly improve code layout
commit fe6d9e0e74e7de1ba9db938b34343f412e883bec
Author: Steve Loughran <[email protected]>
Date: 2015-11-23T15:31:14Z
SPARK-7889 scalastyle
commit f69d391fc6bcd52fd9247d6a65b849dc805e0024
Author: Steve Loughran <[email protected]>
Date: 2015-11-23T15:43:21Z
SPARK-7889 scalastyle on ApplicationCacheSuite
commit 334fc91ebd6b7c3d934733adcef93bd5ff9a1325
Author: Steve Loughran <[email protected]>
Date: 2015-11-23T18:24:42Z
SPARK-7889 cleanup of comments and imports & the like
commit c19fee20cd74b866bc110b0b78bb53827f9ac1b6
Author: Steve Loughran <[email protected]>
Date: 2015-11-24T15:16:41Z
SPARK-7889 intermin checkpoint on changes; about to move cache map from
string to case class
commit ec5b65238639c7852d1ae0ff7dd25f4e70118079
Author: Steve Loughran <[email protected]>
Date: 2015-11-24T20:29:57Z
SPARK-7889 Cache designed to ask history server/provider for updates;
metrics used to track load & time âand for testing
commit feda232876ab5c466d0e009fe9318ed1fbbe5d49
Author: Steve Loughran <[email protected]>
Date: 2015-11-24T22:10:02Z
SPARK-7889 playing losing battles with a test
commit 04d8c64bbda500f0fe55fae262a3485da8118c5c
Author: Steve Loughran <[email protected]>
Date: 2015-11-24T22:58:49Z
SPARK-7889 test working, verified other tests in package work, review
comments & stylecheck
commit 9a7ca9f2860500f7833e649803dd637a0e3fb9c1
Author: Steve Loughran <[email protected]>
Date: 2015-11-24T23:11:59Z
SPARK-7889 - stylecheck on tests apparently skipped on mvn
commit a1024aa2e7a57028da34fa8b226438af58ddbbda
Author: Steve Loughran <[email protected]>
Date: 2015-11-25T15:05:17Z
SPARK-7889 style and javadoc only
commit 6e0e26dc9c5e960bf2b3369788d5d5ee180f41da
Author: Steve Loughran <[email protected]>
Date: 2015-11-25T18:11:05Z
SPARK-7889 pull out metrics into own class, make visible for testing
commit b3c70697175e1971e185b880b41b80183b6d66ea
Author: Steve Loughran <[email protected]>
Date: 2015-11-25T18:12:05Z
history server web UI update test from @squito
commit ea2afbbf01827215cfbdd713b22a8d8f52d42210
Author: Steve Loughran <[email protected]>
Date: 2015-11-25T18:12:48Z
SPARK-7889 more on the cache suite tests: there is no evidence that
eviction is taking place
commit d113c5b5f8b2dec57c3fa6a84969817f66a45cba
Author: Steve Loughran <[email protected]>
Date: 2015-11-25T19:34:09Z
SPARK-7889: tests to make sure app eviction is taking place and that this
triggers callbacs in the cache
commit a128d8c2141d394a2cee82d4eae542762a51a10d
Author: Steve Loughran <[email protected]>
Date: 2015-11-25T20:20:11Z
SPARK-7889 code style check
commit f1c7fe5ebca5cefebd2fadf70e52c85e5f865be5
Author: Steve Loughran <[email protected]>
Date: 2015-11-25T21:00:44Z
SPARK-7889: adding return type of tests embedded inside a test method to
keep scalastyle happy
commit 07f1af464b2784350045295f0803c32c38e504f1
Author: Steve Loughran <[email protected]>
Date: 2015-12-01T15:09:45Z
SPARK-7889 starting to add web filters
commit a33bdd76f1f14df89a98e04ea6f7eeb5d790deae
Author: Steve Loughran <[email protected]>
Date: 2015-12-01T18:18:37Z
SPARK-7889 ongoing test dev. Looks like the history provider isn't picking
up any changes
commit 9166ba65d89b445a5f6482e66bc3f51246d0845f
Author: Steve Loughran <[email protected]>
Date: 2015-12-01T18:52:20Z
SPARK-7889: looks like the test is failing because a new history isn't
being saved
commit 523390a0d8f974f75d784fb17091364ba6d186cd
Author: Steve Loughran <[email protected]>
Date: 2015-12-01T20:27:36Z
SPARK-7889 address scalastyle warnings
commit 9831ad46ea86c59e21d4fadaa94d52a668afffc5
Author: Steve Loughran <[email protected]>
Date: 2015-12-01T21:49:34Z
SPARK-7889 : we aren't getting an updated log file on the second
parallelize().count() call, so the FS history provider isn't seeing an update,
etc, etc.
commit 6fdaab1e6a6c47b69a1cc9320994f9c3964dd4ea
Author: Steve Loughran <[email protected]>
Date: 2015-12-02T15:54:53Z
SPARK-7889: filesize update time included in probe; update thread also
scans through modified files to verify this takes.
commit 163e218d812aa5448803db1e460c705a56df310f
Author: Steve Loughran <[email protected]>
Date: 2015-12-02T15:55:42Z
SPARK-7889: not all filesystems update modtime on a rename; the
EventLoggingListener attempts to do so afterwards, swallowing exceptions raised
commit bc3a2e35f0b32c3d908207cf8f3c700f1cfe06bf
Author: Steve Loughran <[email protected]>
Date: 2015-12-03T16:16:28Z
SPARK-7889 cache update works in tests -unreliably. Traces imply that its a
race condition between probe time and the scanner thread -if the initial load
is after the file update but before the scanner thread has looked @ the file,
the file isn't detected as updated. The provider has to return the actual file
timestamp of its choice for use in update checks, not the time that the initial
load took place
commit f81cfe13fc7ae2fb967a8ce0e556bb85ca38066b
Author: Steve Loughran <[email protected]>
Date: 2015-12-03T19:04:22Z
SPARK-7889 still looking at a race condition in the test. This adds more
time details, but I'm about to move the fshistory off time and into a generic
"attempt version" counter which will be compared on the probe. If an update has
happened, this will know
commit 78a463edf747e6df321b1e06f1960c5cc3456acf
Author: Steve Loughran <[email protected]>
Date: 2015-12-03T19:16:10Z
SPARK-7889 moved off time differences to a simple generational counter and
equality check
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
