GitHub user concretevitamin opened a pull request:
https://github.com/apache/spark/pull/221
Add getListenerBus() in SparkContext.
Motivation:
We had a job that collects and analyzes statistics collected by a custom
SparkListener. What we realized was that it was possible for our main job to
finish first, when the listener event poster thread is still processing events.
(In our particular case -- a small job consisting of linear algebra operations
-- an average of ~43ms is needed for the main job to wait.) A simple solution
to this seems to be gaining access to the listener bus, and thus do something
like `sc.getListenerBus().waitUntilEmpty(10000)`.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/concretevitamin/spark listener
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/221.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #221
----
commit a52de265966af39b67511597adb11e7a0b23b725
Author: Zongheng Yang <[email protected]>
Date: 2014-03-25T06:08:55Z
Add getListenerBus() in SparkContext.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
