GitHub user steveloughran opened a pull request:
https://github.com/apache/spark/pull/10545
[SPARK-1537] [YARN] Add history provider for YARN Application Timeline
Server
This is the successor to PR #5413; it incorporates SPARK-11315 (PR #8744),
which was split out for easier review.
It adds a history provider which uses the YARN timeline server for
histories, reading the events published in the application by way of the #8744
publisher. It's very efficient at getting attempt summary data, as that is
server
It also contains preparatory support for history server metrics
(SPARK-11373 / #9571) (i.e. it collect metrics, but does not publish them), and
the cache updating of incomplete work of SPARK-7889 /#6935, (the #8744
publisher includes an incrementing counter, which is used in the history server
to determine updates to histories.)
In comparison to the FS history provider, bootstrap time is fast as there
is no need to replay histories to extract that metadata. It does place load on
the timeline server, hence various options to configure the frequency of
probing for updates, including disabling background refreshes until users
actually reload pages. Because the YARN ATS service has different failure
modes from HDFS, there's some more startup checking of service availability,
with failure information collected and reported âas well as noted in metrics.
(More succinctly, the FS history provider assumes HDFS doesn't fail).
The new history server provider is added in `yarn/src/history`, along with
its various tests. The code is only included in compiles, tests and scalastyle
checks on Hadoop 2.6+, so does not cause any compatibility issues when Spark is
built against previous Hadoop versions.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/steveloughran/spark
stevel/feature/SPARK-1537-ATS
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/10545.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #10545
----
commit 5f9550bcef92835da8f8c7b72d0ae6299f11ea5b
Author: Steve Loughran <[email protected]>
Date: 2015-12-07T13:43:52Z
SPARK-1537 SPARK-11315 YarnHistoryService patch applied to latest master
(the one with the service API merged in)
commit 0cf226a7f58a099555e38fecee961f8e8b66ca52
Author: Steve Loughran <[email protected]>
Date: 2015-12-07T14:09:15Z
SPARK-1537 SPARK-11323 yarn history provider patch applied atop the new
SPARK-11315 publisher branch
commit 146d15230c8793b2edf8385dfa680f10635640ac
Author: Steve Loughran <[email protected]>
Date: 2015-12-07T14:10:07Z
SPARK-11315 scaladoc and import cleanup (minor)
commit 68c6bf4c35ad8df0feee604784fedabbcd765925
Author: Steve Loughran <[email protected]>
Date: 2015-12-07T15:29:28Z
SPARK-11315 event publisher to add a version counter to every entity
request pushed. This is for more reliable polling for changes during
integration with YARN-7889
commit 00b7ac7dd727de9808aa9d4339f1b13814f499a1
Author: Steve Loughran <[email protected]>
Date: 2015-12-07T15:43:18Z
SPARK-11323 yarn provider to use entity version counter to compare and
track attempt versions. This is for more reliable polling for changes during
integration with YARN-7889
commit 6f395fb9589520d1ce12f99dc66332426fa9b87e
Author: Steve Loughran <[email protected]>
Date: 2015-12-07T20:15:46Z
SPARK-11315 publish entity version counter as number; no need to stringify
commit 97dc9e8d0f1b80f77c743ec29fba3a221c2d191f
Author: Steve Loughran <[email protected]>
Date: 2015-12-07T20:16:48Z
SPARK-11323 tests to get the version counter marshalling working
commit 3cc77df5c9068a72edfef79de8fab911c08b835b
Author: Steve Loughran <[email protected]>
Date: 2015-12-08T16:37:53Z
SPARK-11323 fixing a regression which turned out to be a typo; lots of
improvement in test running in the process. Tests can register "failureActions"
for execution on a test failure; closures to dump the state of things & so have
better diagnostics
commit 9d92df2815a3c4896cbfcccc2972d17c50132e1a
Author: Steve Loughran <[email protected]>
Date: 2015-12-08T16:54:45Z
SPARK-11323 scalastyle fix
commit b3760310a5d52d996076f7fd79bed8e2da4c1a28
Author: Steve Loughran <[email protected]>
Date: 2015-12-08T17:29:42Z
SPARK-11323 chop min refresh interval from 10s to 1s, as testing likes
this. In production even 10s is probably too short, so it doesn't make things
much worse
commit 05499bd8a72b0e5b7fa07d85bf3cd7bd1915cf44
Author: Steve Loughran <[email protected]>
Date: 2015-12-10T11:59:35Z
SPARK-11315 ignore BlockUpdated and ExecutorMetricsUpdate events
commit 7ff89c2865c283d196bed1a8a0dd0885c6aeff18
Author: Steve Loughran <[email protected]>
Date: 2015-12-10T23:10:20Z
SPARK-11315 add info-level logging of post to ATS with entity ID and ATS
URL (as info level wasn't giving any details on whether/when entities were
published, or under what); downgrade event drop to info & not warning
commit 7cad9752009b8a7938af87ddbe6d6674737c5549
Author: Steve Loughran <[email protected]>
Date: 2015-12-11T21:49:04Z
SPARK-11315 style: methods whose arg list is multiline have newline before
first arg
commit 4c812cef839eb86c407bba2742159eec758f7b07
Author: Steve Loughran <[email protected]>
Date: 2015-12-11T21:49:28Z
SPARK-11323 style: methods whose arg list is multiline have newline before
first arg
commit 21e0c769928539a7d3734f142ae5c0a0cf586b60
Author: Steve Loughran <[email protected]>
Date: 2015-12-15T13:04:49Z
SPARK-11323 - some more style checking in the rest/ package
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
