GitHub user JoshRosen opened a pull request:
https://github.com/apache/spark/pull/3197
Several progress API improvements / refactorings
This PR refactors / extends the status API introduced in #2696.
- Change StatusAPI from a mixin trait to a class. Before, the new status
API methods were directly accessible through SparkContext, whereas now they're
accessed through a `sc.statusAPI` field. As long as we were going to add these
methods directly to SparkContext, the mixin trait seemed like a good idea, but
this might be simpler to reason about and may avoid pitfalls that I've run into
while attempting to refactor other parts of SparkContext to use mixins (see
#3071, for example).
- Make `getJobIdsForGroup(null)` return ids for jobs that aren't associated
with any job group.
- Add `getActiveStageIds()` and `getActiveJobIds()` methods that return the
ids of whatever's currently active in this SparkContext. This should simplify
@davies's progress bar code.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/JoshRosen/spark progress-api-improvements
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/3197.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #3197
----
commit c47e294c1f065e377855d2030d54547d1407c968
Author: Josh Rosen <[email protected]>
Date: 2014-11-10T22:38:27Z
Remove StatusAPI mixin trait.
This makes binary compatibility easier to reason about and might avoid
some pitfalls that Iâve run into while attempting to refactor other
parts of SparkContext to use mixin traits (see #3071, for example).
Requiring users to access status API methods through `sc.statusAPI.*`
also avoids SparkContext bloat and buys us extra freedom for adding
parallel higher / lower-level APIs.
commit a227984df660c7e8eb108c35ac376dfc0acb58fd
Author: Josh Rosen <[email protected]>
Date: 2014-11-10T23:05:22Z
getJobIdsForGroup(null) should return jobs for default group
commit d5eab1f2378451ad86470d01147d9d63a5c89fc8
Author: Josh Rosen <[email protected]>
Date: 2014-11-11T00:50:52Z
Add getActive[Stage|Job]Ids() methods.
commit 2cc7353ba9af00c5dfe390b6b27d477187c1e090
Author: Josh Rosen <[email protected]>
Date: 2014-11-11T00:51:22Z
Add missing file.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
