GitHub user vanzin opened a pull request:
https://github.com/apache/spark/pull/5432
[SPARK-4705] Handle multiple app attempts event logs, history server.
This change modifies the event logging listener to write the logs for
different application
attempts to different files. The attempt ID is set by the scheduler
backend, so as long
as the backend returns that ID to SparkContext, things should work.
Currently, the
YARN backend does that.
The history server was also modified to model multiple attempts per
application. Each
attempt has its own UI and a separate row in the listing table, so that
users can look at
all the attempts separately. The UI "adapts" itself to avoid showing
attempt-specific info
when all the applications being shown have a single attempt.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/vanzin/spark SPARK-4705
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/5432.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #5432
----
commit 0eb7722da9e84cb8485e081b241d8260eceba971
Author: twinkle sachdeva <[email protected]>
Date: 2015-02-02T15:38:35Z
SPARK-4705: Doing cherry-pick of fix into master
commit 4c1fc262ea12ff6e4f16fd5ef66db12a7635716d
Author: twinkle sachdeva <[email protected]>
Date: 2015-02-25T04:45:43Z
SPARK-4705 Incorporating the review comments regarding formatting, will do
the rest of the changes after this
commit 6b2e521b2b0e0485e41489e1b3471c363d4b999b
Author: twinkle sachdeva <[email protected]>
Date: 2015-02-25T09:20:33Z
SPARK-4705 Incorporating the review comments regarding formatting, will do
the rest of the changes after this
commit 318525ad711d121e7edb4110c8ff433d47d0f59a
Author: twinkle.sachdeva <[email protected]>
Date: 2015-03-01T15:02:50Z
SPARK-4705: 1) moved from directory structure to single file, as per the
master branch. 2) Added the attempt id inside the
SparkListenerApplicationStart, to make the info available independent of
directory structure. 3) Changes in History Server to render the UI as per the
snaphot II
commit 5fd5c6f44a56d2826e134cd03edcf5711558ecc6
Author: Marcelo Vanzin <[email protected]>
Date: 2015-04-07T23:59:31Z
Fix my broken rebase.
commit 3245aa254bb2c9f2152db1d3be206c0d8e1f851d
Author: Marcelo Vanzin <[email protected]>
Date: 2015-04-08T18:24:41Z
Make app attempts part of the history server model.
This change explicitly models app attempts in the history server.
An app is now a collection of app attempts, instead of the logic
to match apps to app attempts being implicit in the rendering code.
This makes the rendering code a lot simpler, since it doesn't need
to do any fancy processing of the app list to figure out what to show.
commit 88b1de8d9ebfd3cef78161c9cbc4d3a06639a933
Author: Marcelo Vanzin <[email protected]>
Date: 2015-04-08T21:45:29Z
Add a test for apps with multiple attempts.
commit cbe8bba9b0ad62f51fa4a742ac2b2665f730ffd1
Author: Marcelo Vanzin <[email protected]>
Date: 2015-04-08T22:01:28Z
Attempt ID in listener event should be an option.
For backwards compatibility.
commit ce5ee5de4206d74491ecd13411ca2947a81ab4e1
Author: Marcelo Vanzin <[email protected]>
Date: 2015-04-08T23:57:21Z
Misc UI, test, style fixes.
commit c3e0a828a975053144479d023d3b639492119ac6
Author: Marcelo Vanzin <[email protected]>
Date: 2015-04-09T00:15:47Z
Move app name to app info, more UI fixes.
commit 657ec18b5affa0a3f77f51a3e3e05ddbbc21c85f
Author: Marcelo Vanzin <[email protected]>
Date: 2015-04-09T00:27:26Z
Fix yarn history URL, app links.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
